tribuchet: building on eliza Machine state will be reset. To keep it, pass --keep-machine-state start all VLans (finished: start all VLans, in 0.00 seconds) Test will time out and terminate in 900.0 seconds run the VM test script start all VMs client: starting vm lb: starting vm worker1: starting vm worker2: starting vm lb # Disk image does not exist, creating the virtualisation disk image... lb: QEMU running (pid 49) worker1 # Disk image does not exist, creating the virtualisation disk image... worker1: QEMU running (pid 52) worker2: QEMU running (pid 51) client # Disk image does not exist, creating the virtualisation disk image... worker2 # Disk image does not exist, creating the virtualisation disk image... lb # Formatting '/build/vm-state-lb/tmp.g0NHmcM0IJ', fmt=raw size=1073741824 client # Formatting '/build/vm-state-client/tmp.cHu5Xh8VJ8', fmt=raw size=1073741824 worker2 # Formatting '/build/vm-state-worker2/tmp.Pa1gHSgwlP', fmt=raw size=1073741824 lb # mke2fs 1.47.4 (6-Mar-2025) client # mke2fs 1.47.4 (6-Mar-2025) worker2 # mke2fs 1.47.4 (6-Mar-2025) lb # Discarding device blocks: 0/262144 done client # Discarding device blocks: 0/262144 done worker2 # Discarding device blocks: 0/262144 done lb # Creating filesystem with 262144 4k blocks and 65536 inodes client # Creating filesystem with 262144 4k blocks and 65536 inodes worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes lb # Filesystem UUID: 88cbc442-c834-4d45-a9ef-88ee54739cc5 client # Filesystem UUID: 5bf538c6-46a3-4fdd-93a0-9651152ff582 worker2 # Filesystem UUID: ec749343-9b4b-467c-8ef1-28341be0bda4 lb # Superblock backups stored on blocks: client # Superblock backups stored on blocks: worker2 # Superblock backups stored on blocks: lb # 32768, 98304, 163840, 229376 client # 32768, 98304, 163840, 229376 worker2 # 32768, 98304, 163840, 229376 lb # client # worker2 # lb # Allocating group tables: 0/8 done client # Allocating group tables: 0/8 done worker2 # Allocating group tables: 0/8 done lb # Writing inode tables: 0/8 done client # Writing inode tables: 0/8 done worker2 # Writing inode tables: 0/8 done lb # Creating journal (8192 blocks): done client # Creating journal (8192 blocks): done worker2 # Creating journal (8192 blocks): done lb # Writing superblocks and filesystem accounting information: 0/8 done client # Writing superblocks and filesystem accounting information: 0/8 done worker2 # Writing superblocks and filesystem accounting information: 0/8 done lb # client # worker2 # lb # Virtualisation disk image created. client # Virtualisation disk image created. worker2 # Virtualisation disk image created. lb # Starting virtiofs daemons... client # Starting virtiofs daemons... worker2 # Starting virtiofs daemons... lb # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-25T15:23:45Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-25T15:23:45Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-25T15:23:45Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-25T15:23:45Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker2 # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-25T15:23:45Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker2 # [2026-09-25T15:23:45Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-25T15:23:45Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-25T15:23:45Z INFO virtiofsd] Waiting for vhost-user socket connection... worker2 # [2026-09-25T15:23:45Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-25T15:23:45Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-25T15:23:45Z INFO virtiofsd] Client connected, servicing requests worker2 # [2026-09-25T15:23:45Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-25T15:23:45Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-25T15:23:45Z INFO virtiofsd] Client connected, servicing requests worker2 # [2026-09-25T15:23:45Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-25T15:23:45Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-25T15:23:45Z INFO virtiofsd] Client connected, servicing requests worker2 # [2026-09-25T15:23:45Z INFO virtiofsd] Client connected, servicing requests client: QEMU running (pid 50) worker1 # Formatting '/build/vm-state-worker1/tmp.U00JDtIwEy', fmt=raw size=1073741824 worker1 # mke2fs 1.47.4 (6-Mar-2025) worker1 # Discarding device blocks: 0/262144 done worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes worker1 # Filesystem UUID: 8172407d-5c40-4235-8937-fc8748328c0f worker1 # Superblock backups stored on blocks: worker1 # 32768, 98304, 163840, 229376 worker1 # (finished: start all VMs, in 0.45 seconds) worker1 # Allocating group tables: 0/8 done client: waiting for unit network-addresses-eth1.service worker1 # Writing inode tables: 0/8 done client: waiting for the VM to finish booting worker1 # Creating journal (8192 blocks): done worker1 # Writing superblocks and filesystem accounting information: 0/8 done worker1 # worker1 # Virtualisation disk image created. worker1 # Starting virtiofs daemons... worker1 # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-25T15:23:45Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # [2026-09-25T15:23:45Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-25T15:23:45Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-25T15:23:45Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-25T15:23:45Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-25T15:23:45Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-25T15:23:45Z INFO virtiofsd] Client connected, servicing requests worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker1 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 lb # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 worker1 # [ 0.000000] KASLR enabled lb # [ 0.000000] KASLR enabled worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker1 # [ 0.000000] random: crng init done client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] lb # [ 0.000000] random: crng init done worker1 # [ 0.000000] Machine model: linux,dummy-virt lb # [ 0.000000] Machine model: linux,dummy-virt worker1 # [ 0.000000] efi: UEFI not found. lb # [ 0.000000] efi: UEFI not found. worker2 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 worker2 # [ 0.000000] KASLR enabled client # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT client # [ 0.000000] KASLR enabled worker2 # [ 0.000000] random: crng init done client # [ 0.000000] random: crng init done worker2 # [ 0.000000] Machine model: linux,dummy-virt worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] efi: UEFI not found. client # [ 0.000000] Machine model: linux,dummy-virt worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] client # [ 0.000000] efi: UEFI not found. lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker1 # [ 0.000000] Zone ranges: lb # [ 0.000000] Zone ranges: worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] DMA32 empty worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] DMA32 empty worker1 # [ 0.000000] Normal empty client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] Normal empty worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker1 # [ 0.000000] Device empty lb # [ 0.000000] Device empty worker2 # [ 0.000000] Zone ranges: client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker1 # [ 0.000000] Movable zone start for each node lb # [ 0.000000] Movable zone start for each node client # [ 0.000000] Zone ranges: worker1 # [ 0.000000] Early memory node ranges worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] Early memory node ranges worker2 # [ 0.000000] DMA32 empty client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] Normal empty client # [ 0.000000] DMA32 empty lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] Device empty client # [ 0.000000] Normal empty worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] Device empty worker2 # [ 0.000000] Movable zone start for each node lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] Early memory node ranges client # [ 0.000000] Movable zone start for each node worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 client # [ 0.000000] Early memory node ranges worker1 # [ 0.000000] psci: probing for conduit method from DT. worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] psci: probing for conduit method from DT. client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware. lb # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker1 # [ 0.000000] psci: Trusted OS migration not required lb # [ 0.000000] psci: Trusted OS migration not required client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker2 # [ 0.000000] psci: probing for conduit method from DT. worker1 # [ 0.000000] psci: SMC Calling Convention v1.1 lb # [ 0.000000] psci: SMC Calling Convention v1.1 client # [ 0.000000] psci: probing for conduit method from DT. worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware. client # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker2 # [ 0.000000] psci: Trusted OS migration not required worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 client # [ 0.000000] psci: Trusted OS migration not required lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker2 # [ 0.000000] psci: SMC Calling Convention v1.1 worker1 # [ 0.000000] Detected PIPT I-cache on CPU0 client # [ 0.000000] psci: SMC Calling Convention v1.1 lb # [ 0.000000] Detected PIPT I-cache on CPU0 worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface client # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker2 # [ 0.000000] Detected PIPT I-cache on CPU0 worker1 # [ 0.000000] CPU features: detected: Spectre-v4 lb # [ 0.000000] CPU features: detected: GICv3 CPU interface client # [ 0.000000] Detected PIPT I-cache on CPU0 worker1 # [ 0.000000] CPU features: detected: Spectre-BHB lb # [ 0.000000] CPU features: detected: Spectre-v4 worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) lb # [ 0.000000] CPU features: detected: Spectre-BHB worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 client # [ 0.000000] CPU features: detected: GICv3 CPU interface worker2 # [ 0.000000] CPU features: detected: Spectre-v4 client # [ 0.000000] CPU features: detected: Spectre-v4 worker1 # [ 0.000000] alternatives: applying boot alternatives worker2 # [ 0.000000] CPU features: detected: Spectre-BHB lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 client # [ 0.000000] CPU features: detected: Spectre-BHB lb # [ 0.000000] alternatives: applying boot alternatives worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker2 # [ 0.000000] alternatives: applying boot alternatives client # [ 0.000000] alternatives: applying boot alternatives worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/cw89fd0zkc3gaxrg6haynbv2xr4pxwgi-nixos-system-worker1-test/init regInfo=/nix/store/zym3rn263nhc2d17g7rlfr13br7bqpcz-closure-info/registration console=ttyAMA0,115200n8 console=tty0 lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/5hmvppg1fs8xkiccvl0m50n4dxjz92fr-nixos-system-lb-test/init regInfo=/nix/store/4s18yqhkjka9yjskv51fw7r9nw16dgfd-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/pv8jv0ma9kiark5g4gx6k5bl6dq0g11a-nixos-system-worker2-test/init regInfo=/nix/store/vl8jijfa43vpyq80f023biwj91a44kaj-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/zym3rn263nhc2d17g7rlfr13br7bqpcz-closure-info/registration", will be passed to user space. lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/4s18yqhkjka9yjskv51fw7r9nw16dgfd-closure-info/registration", will be passed to user space. client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/nwgxc9xy9a9l9j6z82df6cwp1rq7ly6z-nixos-system-client-test/init regInfo=/nix/store/nnlqyjr31s0109r7gpan7dzgxjg7rrvy-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/nnlqyjr31s0109r7gpan7dzgxjg7rrvy-closure-info/registration", will be passed to user space. worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/vl8jijfa43vpyq80f023biwj91a44kaj-closure-info/registration", will be passed to user space. lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker1 # [ 0.000000] software IO TLB: area num 1. lb # [ 0.000000] software IO TLB: area num 1. client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) client # [ 0.000000] software IO TLB: area num 1. worker2 # [ 0.000000] software IO TLB: area num 1. worker1 # [ 0.000000] Fallback order for Node 0: 0 lb # [ 0.000000] Fallback order for Node 0: 0 client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker1 # [ 0.000000] Policy zone: DMA lb # [ 0.000000] Policy zone: DMA client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) client # [ 0.000000] Fallback order for Node 0: 0 lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off worker2 # [ 0.000000] Fallback order for Node 0: 0 lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 client # [ 0.000000] Policy zone: DMA worker2 # [ 0.000000] Policy zone: DMA lb # [ 0.000000] allocated 3145728 bytes of page_ext worker1 # [ 0.000000] allocated 3145728 bytes of page_ext worker1 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages lb # [ 0.000000] ftrace: allocating 74950 entries in 294 pages client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 lb # [ 0.000000] rcu: Hierarchical RCU implementation. worker1 # [ 0.000000] rcu: Hierarchical RCU implementation. client # [ 0.000000] allocated 3145728 bytes of page_ext worker2 # [ 0.000000] allocated 3145728 bytes of page_ext worker1 # [ 0.000000] rcu: RCU event tracing is enabled. lb # [ 0.000000] rcu: RCU event tracing is enabled. client # [ 0.000000] ftrace: allocating 74950 entries in 294 pages worker2 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. client # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled. lb # [ 0.000000] Trampoline variant of Tasks RCU enabled. client # [ 0.000000] rcu: Hierarchical RCU implementation. worker2 # [ 0.000000] rcu: Hierarchical RCU implementation. lb # [ 0.000000] Rude variant of Tasks RCU enabled. worker1 # [ 0.000000] Rude variant of Tasks RCU enabled. client # [ 0.000000] rcu: RCU event tracing is enabled. worker2 # [ 0.000000] rcu: RCU event tracing is enabled. lb # [ 0.000000] Tracing variant of Tasks RCU enabled. worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled. client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. client # [ 0.000000] Trampoline variant of Tasks RCU enabled. lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled. client # [ 0.000000] Rude variant of Tasks RCU enabled. worker2 # [ 0.000000] Rude variant of Tasks RCU enabled. lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 client # [ 0.000000] Tracing variant of Tasks RCU enabled. worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled. worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] GICv3: 256 SPIs implemented worker1 # [ 0.000000] GICv3: 256 SPIs implemented lb # [ 0.000000] GICv3: 0 Extended SPIs implemented worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] Root IRQ handler: gic_handle_irq worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 client # [ 0.000000] GICv3: 256 SPIs implemented worker2 # [ 0.000000] GICv3: 256 SPIs implemented worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 client # [ 0.000000] GICv3: 0 Extended SPIs implemented worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 client # [ 0.000000] Root IRQ handler: gic_handle_irq worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1) lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1) lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns lb # [ 0.000028] arm-pv: using stolen time PV worker1 # [ 0.000028] arm-pv: using stolen time PV worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns lb # [ 0.000411] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker1 # [ 0.000405] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns lb # [ 0.000583] Console: colour dummy device 80x25 worker1 # [ 0.000592] Console: colour dummy device 80x25 client # [ 0.000030] arm-pv: using stolen time PV worker2 # [ 0.000028] arm-pv: using stolen time PV lb # [ 0.000590] printk: legacy console [tty0] enabled worker1 # [ 0.000599] printk: legacy console [tty0] enabled worker2 # [ 0.000407] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) client # [ 0.000411] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) lb # [ 0.000775] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker1 # [ 0.000788] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) client # [ 0.000588] Console: colour dummy device 80x25 worker2 # [ 0.000585] Console: colour dummy device 80x25 lb # [ 0.000781] pid_max: default: 32768 minimum: 301 worker1 # [ 0.000794] pid_max: default: 32768 minimum: 301 worker2 # [ 0.000593] printk: legacy console [tty0] enabled client # [ 0.000596] printk: legacy console [tty0] enabled lb # [ 0.000866] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker1 # [ 0.000864] LSM: initializing lsm=capability,landlock,yama,bpf,ima lb # [ 0.000996] landlock: Up and running. worker1 # [ 0.000989] landlock: Up and running. worker2 # [ 0.000785] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) lb # [ 0.000998] Yama: becoming mindful. client # [ 0.000789] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker1 # [ 0.000992] Yama: becoming mindful. lb # [ 0.001445] LSM support for eBPF active worker2 # [ 0.000791] pid_max: default: 32768 minimum: 301 client # [ 0.000795] pid_max: default: 32768 minimum: 301 worker1 # [ 0.001422] LSM support for eBPF active worker2 # [ 0.000877] LSM: initializing lsm=capability,landlock,yama,bpf,ima lb # [ 0.001588] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.000867] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker1 # [ 0.001612] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.001012] landlock: Up and running. client # [ 0.001012] landlock: Up and running. worker2 # [ 0.001014] Yama: becoming mindful. client # [ 0.001015] Yama: becoming mindful. lb # [ 0.001674] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker1 # [ 0.001652] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.001467] LSM support for eBPF active client # [ 0.001473] LSM support for eBPF active lb # [ 0.003524] rcu: Hierarchical SRCU implementation. worker1 # [ 0.003458] rcu: Hierarchical SRCU implementation. lb # [ 0.003529] rcu: Max phase no-delay instances is 1000. worker2 # [ 0.001604] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker1 # [ 0.003463] rcu: Max phase no-delay instances is 1000. client # [ 0.001613] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) lb # [ 0.004801] fsl-mc MSI: its@8080000 domain created worker1 # [ 0.004710] fsl-mc MSI: its@8080000 domain created worker2 # [ 0.001642] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.001652] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker1 # [ 0.004795] EFI services will not be available. worker2 # [ 0.003521] rcu: Hierarchical SRCU implementation. client # [ 0.003599] rcu: Hierarchical SRCU implementation. worker1 # [ 0.004866] smp: Bringing up secondary CPUs ... client # [ 0.003604] rcu: Max phase no-delay instances is 1000. worker2 # [ 0.003525] rcu: Max phase no-delay instances is 1000. worker1 # [ 0.004874] smp: Brought up 1 node, 1 CPU client # [ 0.004861] fsl-mc MSI: its@8080000 domain created worker2 # [ 0.004770] fsl-mc MSI: its@8080000 domain created worker1 # [ 0.004877] SMP: Total of 1 processors activated. client # [ 0.004950] EFI services will not be available. worker2 # [ 0.004857] EFI services will not be available. worker1 # [ 0.004882] CPU: All CPU(s) started at EL1 client # [ 0.005034] smp: Bringing up secondary CPUs ... worker2 # [ 0.004921] smp: Bringing up secondary CPUs ... worker1 # [ 0.004892] CPU features: detected: Branch Target Identification worker2 # [ 0.004929] smp: Brought up 1 node, 1 CPU client # [ 0.005042] smp: Brought up 1 node, 1 CPU worker1 # [ 0.004897] CPU features: detected: ARMv8.4 Translation Table Level client # [ 0.005045] SMP: Total of 1 processors activated. worker2 # [ 0.004932] SMP: Total of 1 processors activated. client # [ 0.005050] CPU: All CPU(s) started at EL1 worker2 # [ 0.004938] CPU: All CPU(s) started at EL1 worker1 # [ 0.004900] CPU features: detected: Instruction cache invalidation not required for I/D coherence client # [ 0.005060] CPU features: detected: Branch Target Identification worker2 # [ 0.004948] CPU features: detected: Branch Target Identification worker2 # [ 0.004952] CPU features: detected: ARMv8.4 Translation Table Level worker1 # [ 0.004904] CPU features: detected: Data cache clean to the PoU not required for I/D coherence client # [ 0.005065] CPU features: detected: ARMv8.4 Translation Table Level worker1 # [ 0.004907] CPU features: detected: Common not Private translations worker2 # [ 0.004955] CPU features: detected: Instruction cache invalidation not required for I/D coherence client # [ 0.005068] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker1 # [ 0.004910] CPU features: detected: CRC32 instructions lb # [ 0.004896] EFI services will not be available. worker2 # [ 0.004959] CPU features: detected: Data cache clean to the PoU not required for I/D coherence client # [ 0.005072] CPU features: detected: Data cache clean to the PoU not required for I/D coherence worker1 # [ 0.004913] CPU features: detected: Data cache clean to Point of Deep Persistence lb # [ 0.004982] smp: Bringing up secondary CPUs ... worker2 # [ 0.004962] CPU features: detected: Common not Private translations client # [ 0.005075] CPU features: detected: Common not Private translations lb # [ 0.004990] smp: Brought up 1 node, 1 CPU client # [ 0.005079] CPU features: detected: CRC32 instructions lb # [ 0.004993] SMP: Total of 1 processors activated. lb # [ 0.004998] CPU: All CPU(s) started at EL1 client # [ 0.005082] CPU features: detected: Data cache clean to Point of Deep Persistence lb # [ 0.005008] CPU features: detected: Branch Target Identification client # [ 0.005085] CPU features: detected: Data cache clean to Point of Persistence lb # [ 0.005012] CPU features: detected: ARMv8.4 Translation Table Level client # [ 0.005088] CPU features: detected: Data independent timing control (DIT) lb # [ 0.005015] CPU features: detected: Instruction cache invalidation not required for I/D coherence client # [ 0.005091] CPU features: detected: E0PD client # [ 0.005094] CPU features: detected: Enhanced Counter Virtualization lb # [ 0.005019] CPU features: detected: Data cache clean to the PoU not required for I/D coherence client # [ 0.005097] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) lb # [ 0.005023] CPU features: detected: Common not Private translations lb # [ 0.005026] CPU features: detected: CRC32 instructions client # [ 0.005101] CPU features: detected: Enhanced Virtualization Traps client # [ 0.005104] CPU features: detected: Fine Grained Traps lb # [ 0.005029] CPU features: detected: Data cache clean to Point of Deep Persistence client # [ 0.005107] CPU features: detected: Generic authentication (architected QARMA5 algorithm) lb # [ 0.005033] CPU features: detected: Data cache clean to Point of Persistence worker1 # [ 0.004916] CPU features: detected: Data cache clean to Point of Persistence worker2 # [ 0.004966] CPU features: detected: CRC32 instructions client # [ 0.005112] CPU features: detected: RCpc load-acquire (LDAPR) lb # [ 0.005036] CPU features: detected: Data independent timing control (DIT) worker1 # [ 0.004919] CPU features: detected: Data independent timing control (DIT) client # [ 0.005115] CPU features: detected: LSE atomic instructions lb # [ 0.005039] CPU features: detected: E0PD worker2 # [ 0.004968] CPU features: detected: Data cache clean to Point of Deep Persistence worker1 # [ 0.004922] CPU features: detected: E0PD client # [ 0.005118] CPU features: detected: Privileged Access Never lb # [ 0.005041] CPU features: detected: Enhanced Counter Virtualization worker2 # [ 0.004972] CPU features: detected: Data cache clean to Point of Persistence worker1 # [ 0.004925] CPU features: detected: Enhanced Counter Virtualization client # [ 0.005120] CPU features: detected: PMUv3 lb # [ 0.005044] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) worker2 # [ 0.004975] CPU features: detected: Data independent timing control (DIT) worker1 # [ 0.004928] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) client # [ 0.005123] CPU features: detected: RAS Extension Support worker2 # [ 0.004978] CPU features: detected: E0PD lb # [ 0.005048] CPU features: detected: Enhanced Virtualization Traps client # [ 0.005126] CPU features: detected: RASv1p1 Extension Support worker1 # [ 0.004932] CPU features: detected: Enhanced Virtualization Traps worker2 # [ 0.004981] CPU features: detected: Enhanced Counter Virtualization lb # [ 0.005051] CPU features: detected: Fine Grained Traps client # [ 0.005129] CPU features: detected: Random Number Generator worker1 # [ 0.004934] CPU features: detected: Fine Grained Traps client # [ 0.005131] CPU features: detected: Speculation barrier (SB) worker2 # [ 0.004984] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) lb # [ 0.005054] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker1 # [ 0.004938] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker2 # [ 0.004988] CPU features: detected: Enhanced Virtualization Traps client # [ 0.005134] CPU features: detected: Stage-2 Force Write-Back lb # [ 0.005059] CPU features: detected: RCpc load-acquire (LDAPR) worker1 # [ 0.004943] CPU features: detected: RCpc load-acquire (LDAPR) worker2 # [ 0.004991] CPU features: detected: Fine Grained Traps lb # [ 0.005062] CPU features: detected: LSE atomic instructions client # [ 0.005137] CPU features: detected: TLB range maintenance instructions worker1 # [ 0.004945] CPU features: detected: LSE atomic instructions lb # [ 0.005065] CPU features: detected: Privileged Access Never worker2 # [ 0.004994] CPU features: detected: Generic authentication (architected QARMA5 algorithm) client # [ 0.005141] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker1 # [ 0.004948] CPU features: detected: Privileged Access Never lb # [ 0.005067] CPU features: detected: PMUv3 worker1 # [ 0.004951] CPU features: detected: PMUv3 worker2 # [ 0.004999] CPU features: detected: RCpc load-acquire (LDAPR) client # [ 0.005180] alternatives: applying system-wide alternatives lb # [ 0.005071] CPU features: detected: RAS Extension Support worker1 # [ 0.004954] CPU features: detected: RAS Extension Support worker2 # [ 0.005002] CPU features: detected: LSE atomic instructions client # [ 0.008084] CPU features: detected: BBM Level 2 without TLB conflict abort lb # [ 0.005073] CPU features: detected: RASv1p1 Extension Support worker1 # [ 0.004957] CPU features: detected: RASv1p1 Extension Support worker2 # [ 0.005005] CPU features: detected: Privileged Access Never lb # [ 0.005076] CPU features: detected: Random Number Generator worker1 # [ 0.004959] CPU features: detected: Random Number Generator worker2 # [ 0.005007] CPU features: detected: PMUv3 lb # [ 0.005079] CPU features: detected: Speculation barrier (SB) worker2 # [ 0.005010] CPU features: detected: RAS Extension Support worker1 # [ 0.004962] CPU features: detected: Speculation barrier (SB) client # [ 0.008239] Memory: 1405644K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124660K reserved, 32768K cma-reserved) lb # [ 0.005081] CPU features: detected: Stage-2 Force Write-Back worker1 # [ 0.004964] CPU features: detected: Stage-2 Force Write-Back worker2 # [ 0.005013] CPU features: detected: RASv1p1 Extension Support client # [ 0.008585] devtmpfs: initialized worker2 # [ 0.005016] CPU features: detected: Random Number Generator lb # [ 0.005084] CPU features: detected: TLB range maintenance instructions worker1 # [ 0.004967] CPU features: detected: TLB range maintenance instructions client # [ 0.010427] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) worker2 # [ 0.005019] CPU features: detected: Speculation barrier (SB) lb # [ 0.005089] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker1 # [ 0.004972] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker2 # [ 0.005021] CPU features: detected: Stage-2 Force Write-Back lb # [ 0.005125] alternatives: applying system-wide alternatives client # [ 0.010451] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker1 # [ 0.005009] alternatives: applying system-wide alternatives worker2 # [ 0.005024] CPU features: detected: TLB range maintenance instructions client # [ 0.010651] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL lb # [ 0.007929] CPU features: detected: BBM Level 2 without TLB conflict abort worker1 # [ 0.007806] CPU features: detected: BBM Level 2 without TLB conflict abort client # [ 0.010655] 0 pages in range for non-PLT usage worker2 # [ 0.005029] CPU features: detected: Speculative Store Bypassing Safe (SSBS) client # [ 0.010656] 508272 pages in range for PLT usage worker2 # [ 0.005067] alternatives: applying system-wide alternatives lb # [ 0.008092] Memory: 1405660K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124636K reserved, 32768K cma-reserved) client # [ 0.010752] pinctrl core: initialized pinctrl subsystem worker1 # [ 0.007985] Memory: 1405620K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124664K reserved, 32768K cma-reserved) lb # [ 0.008434] devtmpfs: initialized client # [ 0.011518] DMI not present or invalid. worker2 # [ 0.007892] CPU features: detected: BBM Level 2 without TLB conflict abort worker1 # [ 0.008374] devtmpfs: initialized client # [ 0.014973] NET: Registered PF_NETLINK/PF_ROUTE protocol family lb # [ 0.010288] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) worker1 # [ 0.010198] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) client # [ 0.017348] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations worker2 # [ 0.008073] Memory: 1405668K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124632K reserved, 32768K cma-reserved) lb # [ 0.010311] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker1 # [ 0.010221] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker2 # [ 0.008477] devtmpfs: initialized client # [ 0.017513] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations lb # [ 0.010474] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL worker1 # [ 0.010397] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL worker2 # [ 0.010302] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) lb # [ 0.010478] 0 pages in range for non-PLT usage worker1 # [ 0.010401] 0 pages in range for non-PLT usage client # [ 0.017706] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations lb # [ 0.010479] 508272 pages in range for PLT usage worker1 # [ 0.010403] 508272 pages in range for PLT usage worker2 # [ 0.010324] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). client # [ 0.017726] audit: initializing netlink subsys (disabled) lb # [ 0.010591] pinctrl core: initialized pinctrl subsystem worker1 # [ 0.010506] pinctrl core: initialized pinctrl subsystem lb # [ 0.011368] DMI not present or invalid. worker2 # [ 0.010497] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL client # [ 0.018309] thermal_sys: Registered thermal governor 'fair_share' worker1 # [ 0.011238] DMI not present or invalid. worker2 # [ 0.010501] 0 pages in range for non-PLT usage lb # [ 0.014908] NET: Registered PF_NETLINK/PF_ROUTE protocol family client # [ 0.018311] thermal_sys: Registered thermal governor 'bang_bang' worker1 # [ 0.014804] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker2 # [ 0.010502] 508272 pages in range for PLT usage lb # [ 0.017211] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations client # [ 0.018314] thermal_sys: Registered thermal governor 'step_wise' worker1 # [ 0.017068] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations worker2 # [ 0.010608] pinctrl core: initialized pinctrl subsystem worker2 # [ 0.011411] DMI not present or invalid. client # [ 0.018317] thermal_sys: Registered thermal governor 'user_space' lb # [ 0.017378] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations worker1 # [ 0.017238] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations worker2 # [ 0.014872] NET: Registered PF_NETLINK/PF_ROUTE protocol family client # [ 0.018323] thermal_sys: Registered thermal governor 'power_allocator' lb # [ 0.017572] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker1 # [ 0.017430] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker2 # [ 0.017120] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations client # [ 0.018347] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 lb # [ 0.017592] audit: initializing netlink subsys (disabled) worker1 # [ 0.017449] audit: initializing netlink subsys (disabled) client # [ 0.018355] cpuidle: using governor ladder worker2 # [ 0.017290] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations lb # [ 0.018204] thermal_sys: Registered thermal governor 'fair_share' worker1 # [ 0.018073] thermal_sys: Registered thermal governor 'fair_share' client # [ 0.018361] cpuidle: using governor menu lb # [ 0.018206] thermal_sys: Registered thermal governor 'bang_bang' worker1 # [ 0.018075] thermal_sys: Registered thermal governor 'bang_bang' worker2 # [ 0.017483] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations client # [ 0.018553] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. lb # [ 0.018209] thermal_sys: Registered thermal governor 'step_wise' worker1 # [ 0.018079] thermal_sys: Registered thermal governor 'step_wise' worker2 # [ 0.017504] audit: initializing netlink subsys (disabled) client # [ 0.018567] ASID allocator initialised with 65536 entries lb # [ 0.018212] thermal_sys: Registered thermal governor 'user_space' client # [ 0.019783] Serial: AMBA PL011 UART driver worker1 # [ 0.018082] thermal_sys: Registered thermal governor 'user_space' worker2 # [ 0.018039] thermal_sys: Registered thermal governor 'fair_share' lb # [ 0.018217] thermal_sys: Registered thermal governor 'power_allocator' worker2 # [ 0.018041] thermal_sys: Registered thermal governor 'bang_bang' worker1 # [ 0.018087] thermal_sys: Registered thermal governor 'power_allocator' client # [ 0.024937] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 client # [ 0.025058] printk: console [ttyAMA0] enabled worker2 # [ 0.018044] thermal_sys: Registered thermal governor 'step_wise' lb # [ 0.018240] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 worker1 # [ 0.018109] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 lb # [ 0.018250] cpuidle: using governor ladder worker2 # [ 0.018047] thermal_sys: Registered thermal governor 'user_space' worker1 # [ 0.018118] cpuidle: using governor ladder lb # [ 0.018255] cpuidle: using governor menu worker1 # [ 0.018124] cpuidle: using governor menu worker2 # [ 0.018052] thermal_sys: Registered thermal governor 'power_allocator' lb # [ 0.018433] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. worker1 # [ 0.018309] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. worker2 # [ 0.018081] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 lb # [ 0.018447] ASID allocator initialised with 65536 entries worker1 # [ 0.018323] ASID allocator initialised with 65536 entries worker2 # [ 0.018092] cpuidle: using governor ladder lb # [ 0.019655] Serial: AMBA PL011 UART driver worker1 # [ 0.019510] Serial: AMBA PL011 UART driver worker2 # [ 0.018097] cpuidle: using governor menu lb # [ 0.024872] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 worker2 # [ 0.018282] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. worker1 # [ 0.024621] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 lb # [ 0.025029] printk: console [ttyAMA0] enabled worker1 # [ 0.024765] printk: console [ttyAMA0] enabled worker2 # [ 0.018298] ASID allocator initialised with 65536 entries worker2 # [ 0.019544] Serial: AMBA PL011 UART driver lb # [ 0.149291] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages worker1 # [ 0.149398] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages worker2 # [ 0.024722] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 lb # [ 0.149307] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker1 # [ 0.149414] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker2 # [ 0.024838] printk: console [ttyAMA0] enabled client # [ 0.147937] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages lb # [ 0.149312] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages worker1 # [ 0.149419] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages client # [ 0.147952] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page lb # [ 0.149317] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page worker2 # [ 0.149439] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages worker1 # [ 0.149424] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page client # [ 0.147957] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages lb # [ 0.149322] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages worker2 # [ 0.149453] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker1 # [ 0.149429] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages client # [ 0.147961] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page worker2 # [ 0.149458] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages lb # [ 0.149326] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page worker1 # [ 0.149433] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page client # [ 0.147966] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages worker2 # [ 0.149463] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page lb # [ 0.149330] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages worker1 # [ 0.149437] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages client # [ 0.147970] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page lb # [ 0.149335] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker2 # [ 0.149467] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages worker1 # [ 0.149442] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page client # [ 0.147974] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages worker2 # [ 0.149472] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page client # [ 0.147979] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker2 # [ 0.149476] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages lb # [ 0.156760] fbcon: Taking over console worker2 # [ 0.149480] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker1 # [ 0.156888] fbcon: Taking over console lb # [ 0.156773] ACPI: Interpreter disabled. worker1 # [ 0.156900] ACPI: Interpreter disabled. client # [ 0.155495] fbcon: Taking over console client # [ 0.155509] ACPI: Interpreter disabled. worker2 # [ 0.156974] fbcon: Taking over console worker2 # [ 0.156987] ACPI: Interpreter disabled. lb # [ 0.164725] iommu: Default domain type: Translated worker1 # [ 0.164823] iommu: Default domain type: Translated lb # [ 0.164734] iommu: DMA domain TLB invalidation policy: strict mode worker1 # [ 0.164833] iommu: DMA domain TLB invalidation policy: strict mode lb # [ 0.166449] SCSI subsystem initialized worker1 # [ 0.165278] SCSI subsystem initialized client # [ 0.164204] iommu: Default domain type: Translated worker2 # [ 0.164822] iommu: Default domain type: Translated lb # [ 0.167158] usbcore: registered new interface driver usbfs client # [ 0.164214] iommu: DMA domain TLB invalidation policy: strict mode worker2 # [ 0.164830] iommu: DMA domain TLB invalidation policy: strict mode worker1 # [ 0.167310] usbcore: registered new interface driver usbfs client # [ 0.164549] SCSI subsystem initialized lb # [ 0.167196] usbcore: registered new interface driver hub worker1 # [ 0.167341] usbcore: registered new interface driver hub worker2 # [ 0.166528] SCSI subsystem initialized lb # [ 0.167211] usbcore: registered new device driver usb worker1 # [ 0.167357] usbcore: registered new device driver usb client # [ 0.166588] usbcore: registered new interface driver usbfs lb # [ 0.167477] pps_core: LinuxPPS API ver. 1 registered worker1 # [ 0.167610] pps_core: LinuxPPS API ver. 1 registered worker2 # [ 0.167204] usbcore: registered new interface driver usbfs client # [ 0.166633] usbcore: registered new interface driver hub worker2 # [ 0.167246] usbcore: registered new interface driver hub lb # [ 0.167484] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti client # [ 0.166650] usbcore: registered new device driver usb worker1 # [ 0.167616] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti lb # [ 0.167494] PTP clock support registered worker2 # [ 0.167261] usbcore: registered new device driver usb worker1 # [ 0.167625] PTP clock support registered client # [ 0.166914] pps_core: LinuxPPS API ver. 1 registered lb # [ 0.167541] EDAC MC: Ver: 3.0.0 worker1 # [ 0.167673] EDAC MC: Ver: 3.0.0 worker2 # [ 0.167517] pps_core: LinuxPPS API ver. 1 registered lb # [ 0.172258] scmi_core: SCMI protocol bus registered client # [ 0.166920] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti worker1 # [ 0.172389] scmi_core: SCMI protocol bus registered worker2 # [ 0.167523] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti client # [ 0.166931] PTP clock support registered worker2 # [ 0.167533] PTP clock support registered client # [ 0.166978] EDAC MC: Ver: 3.0.0 lb # [ 0.173244] FPGA manager framework worker1 # [ 0.173378] FPGA manager framework worker2 # [ 0.167588] EDAC MC: Ver: 3.0.0 lb # [ 0.174182] vgaarb: loaded worker1 # [ 0.174287] vgaarb: loaded client # [ 0.171763] scmi_core: SCMI protocol bus registered worker2 # [ 0.172315] scmi_core: SCMI protocol bus registered lb # [ 0.174798] clocksource: Switched to clocksource arch_sys_counter worker1 # [ 0.174900] clocksource: Switched to clocksource arch_sys_counter client # [ 0.172760] FPGA manager framework worker2 # [ 0.173334] FPGA manager framework client # [ 0.173710] vgaarb: loaded worker2 # [ 0.174276] vgaarb: loaded client # [ 0.174325] clocksource: Switched to clocksource arch_sys_counter lb # [ 0.177067] VFS: Disk quotas dquot_6.6.0 worker1 # [ 0.176995] VFS: Disk quotas dquot_6.6.0 worker2 # [ 0.174905] clocksource: Switched to clocksource arch_sys_counter lb # [ 0.177101] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) worker1 # [ 0.177030] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) worker2 # [ 0.177090] VFS: Disk quotas dquot_6.6.0 worker2 # [ 0.177125] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) client # [ 0.177989] VFS: Disk quotas dquot_6.6.0 lb # [ 0.180800] netfs: FS-Cache loaded worker1 # [ 0.180728] netfs: FS-Cache loaded lb # [ 0.180918] pnp: PnP ACPI: disabled worker1 # [ 0.180841] pnp: PnP ACPI: disabled client # [ 0.178023] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) worker2 # [ 0.180714] netfs: FS-Cache loaded worker2 # [ 0.180827] pnp: PnP ACPI: disabled client # [ 0.181641] netfs: FS-Cache loaded client # [ 0.181764] pnp: PnP ACPI: disabled lb # [ 0.184838] NET: Registered PF_INET protocol family worker1 # [ 0.184632] NET: Registered PF_INET protocol family lb # [ 0.185112] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker1 # [ 0.184899] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) client # [ 0.185615] NET: Registered PF_INET protocol family worker2 # [ 0.184711] NET: Registered PF_INET protocol family client # [ 0.185885] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker2 # [ 0.184980] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) lb # [ 0.214007] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) worker1 # [ 0.213530] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) lb # [ 0.214045] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) worker1 # [ 0.213567] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) lb # [ 0.214070] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) client # [ 0.214548] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) worker1 # [ 0.213591] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) worker2 # [ 0.213974] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) lb # [ 0.214138] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) client # [ 0.214586] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) worker1 # [ 0.213661] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) worker2 # [ 0.214014] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) lb # [ 0.214280] TCP: Hash tables configured (established 16384 bind 16384) worker1 # [ 0.213804] TCP: Hash tables configured (established 16384 bind 16384) client # [ 0.214611] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) worker2 # [ 0.214040] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) lb # [ 0.214371] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) worker1 # [ 0.213875] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) client # [ 0.214682] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) worker2 # [ 0.214113] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) lb # [ 0.214407] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) worker1 # [ 0.213907] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.214825] TCP: Hash tables configured (established 16384 bind 16384) worker2 # [ 0.214256] TCP: Hash tables configured (established 16384 bind 16384) lb # [ 0.214443] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) worker1 # [ 0.213943] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.214901] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) worker2 # [ 0.214328] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) lb # [ 0.214531] NET: Registered PF_UNIX/PF_LOCAL protocol family worker1 # [ 0.214027] NET: Registered PF_UNIX/PF_LOCAL protocol family client # [ 0.214935] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.214557] NET: Registered PF_XDP protocol family worker2 # [ 0.214361] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) worker1 # [ 0.214052] NET: Registered PF_XDP protocol family lb # [ 0.214574] PCI: CLS 0 bytes, default 64 client # [ 0.214971] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) worker1 # [ 0.214069] PCI: CLS 0 bytes, default 64 worker2 # [ 0.214396] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.214789] Trying to unpack rootfs image as initramfs... client # [ 0.215069] NET: Registered PF_UNIX/PF_LOCAL protocol family worker1 # [ 0.214294] Trying to unpack rootfs image as initramfs... worker2 # [ 0.214497] NET: Registered PF_UNIX/PF_LOCAL protocol family client # [ 0.215138] NET: Registered PF_XDP protocol family worker2 # [ 0.214521] NET: Registered PF_XDP protocol family client # [ 0.215155] PCI: CLS 0 bytes, default 64 worker2 # [ 0.214537] PCI: CLS 0 bytes, default 64 client # [ 0.215378] Trying to unpack rootfs image as initramfs... worker2 # [ 0.214749] Trying to unpack rootfs image as initramfs... lb # [ 0.232588] kvm [1]: HYP mode not available worker1 # [ 0.232745] kvm [1]: HYP mode not available client # [ 0.232237] kvm [1]: HYP mode not available worker2 # [ 0.232758] kvm [1]: HYP mode not available worker1 # [ 0.316009] Initialise system trusted keyrings lb # [ 0.316336] Initialise system trusted keyrings worker1 # [ 0.316764] workingset: timestamp_bits=42 max_order=19 bucket_order=0 lb # [ 0.317081] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker1 # [ 0.318058] squashfs: version 4.0 (2009/01/31) Phillip Lougher lb # [ 0.318345] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker1 # [ 0.318855] 9p: Installing v9fs 9p2000 file system support worker2 # [ 0.316764] Initialise system trusted keyrings worker2 # [ 0.317520] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker2 # [ 0.318816] squashfs: version 4.0 (2009/01/31) Phillip Lougher lb # [ 0.319192] 9p: Installing v9fs 9p2000 file system support client # [ 0.322861] Initialise system trusted keyrings client # [ 0.323681] workingset: timestamp_bits=42 max_order=19 bucket_order=0 client # [ 0.325003] squashfs: version 4.0 (2009/01/31) Phillip Lougher client # [ 0.325805] 9p: Installing v9fs 9p2000 file system support worker2 # [ 0.326939] 9p: Installing v9fs 9p2000 file system support lb # [ 0.347307] Key type asymmetric registered worker1 # [ 0.347641] Key type asymmetric registered lb # [ 0.347328] Asymmetric key parser 'x509' registered worker1 # [ 0.347663] Asymmetric key parser 'x509' registered lb # [ 0.347390] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker1 # [ 0.347724] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker2 # [ 0.346963] Key type asymmetric registered lb # [ 0.349507] io scheduler mq-deadline registered worker2 # [ 0.346979] Asymmetric key parser 'x509' registered worker1 # [ 0.349896] io scheduler mq-deadline registered lb # [ 0.349518] io scheduler kyber registered worker1 # [ 0.349907] io scheduler kyber registered worker2 # [ 0.347033] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker2 # [ 0.349358] io scheduler mq-deadline registered worker2 # [ 0.349370] io scheduler kyber registered client # [ 0.354491] Key type asymmetric registered client # [ 0.354515] Asymmetric key parser 'x509' registered client # [ 0.354574] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) client # [ 0.356729] io scheduler mq-deadline registered client # [ 0.356741] io scheduler kyber registered lb # [ 0.358942] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker1 # [ 0.359036] pl061_gpio 9030000.pl061: PL061 GPIO chip registered lb # [ 0.360330] ledtrig-cpu: registered to indicate activity on CPUs lb # [ 0.360702] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: lb # [ 0.360720] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 lb # [ 0.360732] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker1 # [ 0.360425] ledtrig-cpu: registered to indicate activity on CPUs lb # [ 0.360741] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker1 # [ 0.360787] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: lb # [ 0.360762] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker1 # [ 0.360805] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 lb # [ 0.360793] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] client # [ 0.366436] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker1 # [ 0.360825] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 lb # [ 0.361224] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 lb # [ 0.361235] pci_bus 0000:00: root bus resource [bus 00-ff] worker1 # [ 0.360835] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 lb # [ 0.361242] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker1 # [ 0.360855] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits lb # [ 0.361247] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] client # [ 0.367843] ledtrig-cpu: registered to indicate activity on CPUs worker2 # [ 0.363067] pl061_gpio 9030000.pl061: PL061 GPIO chip registered lb # [ 0.361253] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.360878] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] client # [ 0.368201] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker2 # [ 0.363690] ledtrig-cpu: registered to indicate activity on CPUs worker1 # [ 0.360960] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 lb # [ 0.361306] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint client # [ 0.368218] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker2 # [ 0.364056] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker1 # [ 0.360970] pci_bus 0000:00: root bus resource [bus 00-ff] lb # [ 0.361814] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.360976] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker2 # [ 0.364073] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 client # [ 0.368230] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 lb # [ 0.362004] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.360981] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] lb # [ 0.362021] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.364086] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 client # [ 0.368239] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker1 # [ 0.360986] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] lb # [ 0.362051] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.368260] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker2 # [ 0.364094] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 lb # [ 0.362067] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.361041] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker2 # [ 0.364114] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits client # [ 0.368283] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] lb # [ 0.362530] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.361489] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.368389] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 lb # [ 0.362717] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.361763] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.364138] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] client # [ 0.368399] pci_bus 0000:00: root bus resource [bus 00-ff] lb # [ 0.362733] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.361781] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.364209] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 client # [ 0.368404] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] lb # [ 0.362763] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.361812] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.364218] pci_bus 0000:00: root bus resource [bus 00-ff] client # [ 0.368409] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.361829] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.364224] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] client # [ 0.368414] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker2 # [ 0.364230] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.362313] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.362503] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.364235] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] client # [ 0.368468] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker1 # [ 0.362519] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.368916] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.364339] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker1 # [ 0.362549] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.369110] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.364788] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.363037] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint client # [ 0.369127] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.364979] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.363228] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] client # [ 0.369158] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.364996] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.363244] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.369174] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.365026] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.363274] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.365042] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.369647] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.363741] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.369837] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.365508] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.363932] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] client # [ 0.369853] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.365694] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.363948] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.369883] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.365710] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.363978] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.365740] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.390128] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker1 # [ 0.363997] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.390321] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] worker2 # [ 0.366204] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker1 # [ 0.364473] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker2 # [ 0.366389] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] worker1 # [ 0.364667] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.387289] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker2 # [ 0.366405] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.364698] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.387500] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] worker2 # [ 0.366435] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.387517] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.392033] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.365168] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker2 # [ 0.366894] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint lb # [ 0.387547] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.365363] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.392063] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.367102] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.365393] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.388011] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.367118] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.392555] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint lb # [ 0.388201] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.367148] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.392747] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.365817] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint lb # [ 0.388217] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.367166] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.366006] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] client # [ 0.392763] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.388247] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.392793] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.367637] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker1 # [ 0.366285] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.388265] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.392812] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.367829] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.366479] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.388730] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint client # [ 0.393286] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker2 # [ 0.367859] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.366510] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.388925] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.393480] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.368327] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint lb # [ 0.388955] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.393511] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.368518] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.389419] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker2 # [ 0.368548] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.393981] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint lb # [ 0.389619] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.394176] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.368944] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint lb # [ 0.389650] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.394206] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.369129] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] lb # [ 0.390084] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker2 # [ 0.369410] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.390271] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] worker2 # [ 0.369602] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.390533] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.369632] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.390725] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.370103] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.390756] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.370293] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.391241] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.370323] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.391432] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.370800] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.391462] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.391921] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.392112] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.392143] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.392608] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint lb # [ 0.392863] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] lb # [ 0.392880] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.392910] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.393377] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.393565] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] client # [ 0.410683] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint lb # [ 0.393581] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.410886] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] worker1 # [ 0.415084] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.393659] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.411199] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.415294] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.394228] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned client # [ 0.411394] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.415326] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.394240] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned client # [ 0.411425] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.415839] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.394246] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned client # [ 0.411893] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.416034] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.412086] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.394290] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker1 # [ 0.416065] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.412116] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.419063] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.394337] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker1 # [ 0.416540] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker2 # [ 0.419106] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.412583] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker1 # [ 0.416796] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] lb # [ 0.394383] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned client # [ 0.412776] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.416813] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.419617] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint lb # [ 0.394430] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned client # [ 0.412807] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.419869] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] worker1 # [ 0.416844] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.419887] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.394477] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned client # [ 0.413283] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker1 # [ 0.417318] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker2 # [ 0.419917] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.413543] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] lb # [ 0.394524] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker1 # [ 0.417507] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] client # [ 0.413560] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.417574] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.420383] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.394570] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned client # [ 0.413590] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.420571] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.417607] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.394617] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned worker2 # [ 0.420587] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.418203] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned client # [ 0.414062] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.394665] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned client # [ 0.414250] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.420617] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.418215] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned client # [ 0.414266] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.421209] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned lb # [ 0.394734] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned worker1 # [ 0.418221] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned client # [ 0.414296] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.394780] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned worker2 # [ 0.421220] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker1 # [ 0.418266] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned client # [ 0.414894] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned worker2 # [ 0.421226] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker1 # [ 0.418314] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned client # [ 0.414906] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker2 # [ 0.421271] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker1 # [ 0.418363] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned client # [ 0.414911] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker2 # [ 0.421319] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker1 # [ 0.418411] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned client # [ 0.414958] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker2 # [ 0.421367] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned worker1 # [ 0.418460] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned client # [ 0.415006] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker2 # [ 0.421415] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned worker1 # [ 0.418508] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned client # [ 0.415054] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned worker2 # [ 0.421463] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned worker1 # [ 0.418557] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned client # [ 0.415103] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned worker2 # [ 0.421511] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker1 # [ 0.418604] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned client # [ 0.415151] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned worker2 # [ 0.421559] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker1 # [ 0.418652] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned client # [ 0.415205] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker2 # [ 0.421606] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned worker1 # [ 0.418724] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned lb # [ 0.442843] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned client # [ 0.415253] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker2 # [ 0.421653] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned worker1 # [ 0.418772] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned lb # [ 0.442885] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned client # [ 0.415300] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned worker1 # [ 0.418794] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker2 # [ 0.421721] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned lb # [ 0.442909] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned client # [ 0.415347] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned worker1 # [ 0.418816] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned worker2 # [ 0.421768] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned lb # [ 0.442932] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned client # [ 0.415429] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned worker1 # [ 0.418838] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned worker2 # [ 0.421790] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned lb # [ 0.442954] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned client # [ 0.415477] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned worker1 # [ 0.418860] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned worker2 # [ 0.421811] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned lb # [ 0.442977] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned client # [ 0.415499] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker1 # [ 0.418881] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned worker2 # [ 0.421833] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned lb # [ 0.443003] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned client # [ 0.415521] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned worker1 # [ 0.418928] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned worker2 # [ 0.421855] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned lb # [ 0.443025] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned client # [ 0.415543] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned worker1 # [ 0.418952] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned worker2 # [ 0.421876] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned lb # [ 0.443048] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned client # [ 0.415565] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned worker1 # [ 0.418975] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned worker2 # [ 0.421898] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned lb # [ 0.443071] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned worker1 # [ 0.418998] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned client # [ 0.415586] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned worker2 # [ 0.421924] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned lb # [ 0.443094] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned client # [ 0.415608] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned worker1 # [ 0.419021] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned worker2 # [ 0.421946] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned lb # [ 0.443116] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned client # [ 0.415634] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned worker1 # [ 0.419044] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned worker2 # [ 0.421968] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned lb # [ 0.443137] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned worker1 # [ 0.419066] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned client # [ 0.415656] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned lb # [ 0.443159] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned worker2 # [ 0.421990] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned worker1 # [ 0.419088] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned client # [ 0.415678] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned lb # [ 0.443181] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned worker2 # [ 0.422012] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned worker1 # [ 0.419109] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned lb # [ 0.443202] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned client # [ 0.415701] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned worker2 # [ 0.422034] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned worker1 # [ 0.419131] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned lb # [ 0.443224] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned worker2 # [ 0.422055] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned client # [ 0.415723] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned worker1 # [ 0.419153] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned lb # [ 0.443251] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] worker2 # [ 0.422076] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned client # [ 0.415745] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned worker1 # [ 0.419175] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned lb # [ 0.443261] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] worker2 # [ 0.422097] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned client # [ 0.415766] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned worker1 # [ 0.419201] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] lb # [ 0.443265] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] worker2 # [ 0.422118] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned client # [ 0.415788] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned worker1 # [ 0.419211] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] lb # [ 0.444092] pci 0000:00:07.0: enabling device (0000 -> 0002) worker2 # [ 0.422139] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned client # [ 0.415809] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned worker1 # [ 0.419215] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] worker2 # [ 0.422165] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] client # [ 0.415830] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned worker1 # [ 0.420082] pci 0000:00:07.0: enabling device (0000 -> 0002) worker2 # [ 0.422174] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] client # [ 0.415852] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned worker2 # [ 0.422179] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] client # [ 0.415878] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] worker2 # [ 0.423026] pci 0000:00:07.0: enabling device (0000 -> 0002) client # [ 0.415888] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] client # [ 0.415893] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] client # [ 0.416736] pci 0000:00:07.0: enabling device (0000 -> 0002) worker1 # [ 0.481703] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) worker2 # [ 0.481733] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) lb # [ 0.485174] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) client # [ 0.488980] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) worker2 # [ 0.487898] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker2 # [ 0.490162] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker1 # [ 0.492352] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker1 # [ 0.494572] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker1 # [ 0.498320] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) client # [ 0.495608] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) client # [ 0.497550] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) lb # [ 0.494987] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) lb # [ 0.497830] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) lb # [ 0.499695] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) worker2 # [ 0.500320] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) lb # [ 0.501630] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) worker2 # [ 0.502361] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) worker1 # [ 0.505489] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) client # [ 0.507663] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) client # [ 0.509589] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) lb # [ 0.511613] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) lb # [ 0.513442] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker2 # [ 0.512850] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) worker2 # [ 0.514799] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker1 # [ 0.515062] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) worker1 # [ 0.517201] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) client # [ 0.520197] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) worker1 # [ 0.518890] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) client # [ 0.521928] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker1 # [ 0.520644] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) worker1 # [ 0.522481] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) client # [ 0.525373] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker2 # [ 0.524292] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker2 # [ 0.526092] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) lb # [ 0.523327] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) lb # [ 0.525116] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) lb # [ 0.526925] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) worker2 # [ 0.529636] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) lb # [ 0.529077] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) client # [ 0.531810] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) client # [ 0.534234] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) worker1 # [ 0.536323] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) worker2 # [ 0.537164] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) client # [ 0.544170] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) lb # [ 0.550084] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled worker1 # [ 0.552066] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled lb # [ 0.552710] msm_serial: driver initialized worker1 # [ 0.554552] msm_serial: driver initialized worker2 # [ 0.552131] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled lb # [ 0.552858] SuperH (H)SCI(F) driver initialized worker1 # [ 0.554697] SuperH (H)SCI(F) driver initialized lb # [ 0.552915] STM32 USART driver initialized worker1 # [ 0.554748] STM32 USART driver initialized worker2 # [ 0.554550] msm_serial: driver initialized worker2 # [ 0.554682] SuperH (H)SCI(F) driver initialized worker2 # [ 0.554735] STM32 USART driver initialized client # [ 0.557146] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled client # [ 0.559678] msm_serial: driver initialized client # [ 0.559841] SuperH (H)SCI(F) driver initialized client # [ 0.559895] STM32 USART driver initialized lb # [ 0.582064] loop: module loaded lb # [ 0.582261] virtio_blk virtio2: 1/0/0 default/read/poll queues worker1 # [ 0.585695] loop: module loaded worker1 # [ 0.585867] virtio_blk virtio2: 1/0/0 default/read/poll queues worker1 # [ 0.586626] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker2 # [ 0.585710] loop: module loaded worker2 # [ 0.585873] virtio_blk virtio2: 1/0/0 default/read/poll queues worker2 # [ 0.586622] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) lb # [ 0.591106] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) client # [ 0.590915] loop: module loaded worker1 # [ 0.591415] megasas: 07.734.00.00-rc1 client # [ 0.591087] virtio_blk virtio2: 1/0/0 default/read/poll queues worker1 # [ 0.592059] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] client # [ 0.591814] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker2 # [ 0.591460] megasas: 07.734.00.00-rc1 worker2 # [ 0.592157] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] lb # [ 0.595366] megasas: 07.734.00.00-rc1 lb # [ 0.596092] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] lb # [ 0.599154] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.599177] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.600699] Using buffer write method lb # [ 0.600760] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] client # [ 0.602876] megasas: 07.734.00.00-rc1 lb # [ 0.602531] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.602940] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 client # [ 0.603648] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] worker2 # [ 0.604317] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.602551] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.602977] Intel/Sharp Extended Query Table at 0x0031 worker2 # [ 0.604348] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.605167] Using buffer write method client # [ 0.605586] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.606012] Using buffer write method client # [ 0.605610] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.605242] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] worker2 # [ 0.606071] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] worker1 # [ 0.607055] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.607075] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.608840] Using buffer write method worker1 # [ 0.608863] Concatenating MTD devices: worker1 # [ 0.608867] (0): "0.flash" lb # [ 0.615610] Using buffer write method worker1 # [ 0.608871] (1): "0.flash" lb # [ 0.615641] Concatenating MTD devices: worker1 # [ 0.608882] into device "0.flash" lb # [ 0.615645] (0): "0.flash" lb # [ 0.615649] (1): "0.flash" client # [ 0.610784] Using buffer write method lb # [ 0.615657] into device "0.flash" client # [ 0.610842] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] client # [ 0.612840] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 client # [ 0.612860] Intel/Sharp Extended Query Table at 0x0031 worker2 # [ 0.616911] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.616938] Intel/Sharp Extended Query Table at 0x0031 worker2 # [ 0.618539] Using buffer write method worker2 # [ 0.618563] Concatenating MTD devices: worker2 # [ 0.618567] (0): "0.flash" worker2 # [ 0.618571] (1): "0.flash" worker2 # [ 0.618579] into device "0.flash" client # [ 0.630872] Using buffer write method client # [ 0.630905] Concatenating MTD devices: client # [ 0.630909] (0): "0.flash" client # [ 0.630913] (1): "0.flash" client # [ 0.630921] into device "0.flash" worker1 # [ 0.858111] Freeing initrd memory: 26976K lb # [ 0.860773] Freeing initrd memory: 26948K worker2 # [ 0.861254] Freeing initrd memory: 26944K worker1 # [ 0.864022] tun: Universal TUN/TAP device driver, 1.6 client # [ 0.860240] Freeing initrd memory: 26972K lb # [ 0.866767] tun: Universal TUN/TAP device driver, 1.6 worker2 # [ 0.867262] tun: Universal TUN/TAP device driver, 1.6 lb # [ 0.870421] thunder_xcv, ver 1.0 lb # [ 0.870464] thunder_bgx, ver 1.0 lb # [ 0.870487] nicpf, ver 1.0 worker1 # [ 0.867880] thunder_xcv, ver 1.0 worker1 # [ 0.867921] thunder_bgx, ver 1.0 worker1 # [ 0.867943] nicpf, ver 1.0 client # [ 0.866269] tun: Universal TUN/TAP device driver, 1.6 worker1 # [ 0.868487] e1000: Intel(R) PRO/1000 Network Driver client # [ 0.869219] thunder_xcv, ver 1.0 client # [ 0.869257] thunder_bgx, ver 1.0 worker1 # [ 0.868495] e1000: Copyright (c) 1999-2006 Intel Corporation. client # [ 0.869279] nicpf, ver 1.0 worker1 # [ 0.868517] e1000e: Intel(R) PRO/1000 Network Driver client # [ 0.869816] e1000: Intel(R) PRO/1000 Network Driver worker1 # [ 0.868525] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. client # [ 0.869824] e1000: Copyright (c) 1999-2006 Intel Corporation. lb # [ 0.872294] e1000: Intel(R) PRO/1000 Network Driver worker1 # [ 0.868553] igb: Intel(R) Gigabit Ethernet Network Driver client # [ 0.869846] e1000e: Intel(R) PRO/1000 Network Driver lb # [ 0.872306] e1000: Copyright (c) 1999-2006 Intel Corporation. worker1 # [ 0.868559] igb: Copyright (c) 2007-2014 Intel Corporation. client # [ 0.869855] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. lb # [ 0.872330] e1000e: Intel(R) PRO/1000 Network Driver worker1 # [ 0.868580] igbvf: Intel(R) Gigabit Virtual Function Network Driver client # [ 0.869884] igb: Intel(R) Gigabit Ethernet Network Driver worker2 # [ 0.870992] thunder_xcv, ver 1.0 lb # [ 0.872340] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker1 # [ 0.868586] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker2 # [ 0.871027] thunder_bgx, ver 1.0 client # [ 0.869890] igb: Copyright (c) 2007-2014 Intel Corporation. worker2 # [ 0.871049] nicpf, ver 1.0 lb # [ 0.872369] igb: Intel(R) Gigabit Ethernet Network Driver worker1 # [ 0.868715] sky2: driver version 1.30 client # [ 0.869927] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker2 # [ 0.871631] e1000: Intel(R) PRO/1000 Network Driver lb # [ 0.872375] igb: Copyright (c) 2007-2014 Intel Corporation. worker1 # [ 0.870249] usbcore: registered new interface driver usb-storage worker2 # [ 0.871638] e1000: Copyright (c) 1999-2006 Intel Corporation. client # [ 0.869933] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. lb # [ 0.872398] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker1 # [ 0.870294] usbcore: registered new interface driver usbserial_generic client # [ 0.870065] sky2: driver version 1.30 worker2 # [ 0.871661] e1000e: Intel(R) PRO/1000 Network Driver lb # [ 0.872404] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker1 # [ 0.870307] usbserial: USB Serial support registered for generic worker2 # [ 0.871669] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. lb # [ 0.872538] sky2: driver version 1.30 worker1 # [ 0.870875] hv_vmbus: registering driver hyperv_keyboard worker2 # [ 0.871698] igb: Intel(R) Gigabit Ethernet Network Driver lb # [ 0.874240] usbcore: registered new interface driver usb-storage worker1 # [ 0.871545] ehci-pci 0000:00:07.0: EHCI Host Controller worker2 # [ 0.871704] igb: Copyright (c) 2007-2014 Intel Corporation. lb # [ 0.874367] usbcore: registered new interface driver usbserial_generic worker1 # [ 0.871567] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker2 # [ 0.871727] igbvf: Intel(R) Gigabit Virtual Function Network Driver lb # [ 0.874382] usbserial: USB Serial support registered for generic worker2 # [ 0.871732] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. lb # [ 0.875472] ehci-pci 0000:00:07.0: EHCI Host Controller client # [ 0.879811] usbcore: registered new interface driver usb-storage worker2 # [ 0.871863] sky2: driver version 1.30 worker1 # [ 0.871732] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 lb # [ 0.875495] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker1 # [ 0.883803] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker2 # [ 0.873396] usbcore: registered new interface driver usb-storage client # [ 0.879863] usbcore: registered new interface driver usbserial_generic worker1 # [ 0.884089] hub 1-0:1.0: USB hub found lb # [ 0.875735] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 client # [ 0.879877] usbserial: USB Serial support registered for generic worker2 # [ 0.873477] usbcore: registered new interface driver usbserial_generic worker1 # [ 0.884103] hub 1-0:1.0: 6 ports detected client # [ 0.880463] hv_vmbus: registering driver hyperv_keyboard lb # [ 0.886674] hv_vmbus: registering driver hyperv_keyboard worker2 # [ 0.873492] usbserial: USB Serial support registered for generic client # [ 0.881657] ehci-pci 0000:00:07.0: EHCI Host Controller worker2 # [ 0.874066] hv_vmbus: registering driver hyperv_keyboard lb # [ 0.887501] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker2 # [ 0.875394] ehci-pci 0000:00:07.0: EHCI Host Controller lb # [ 0.888530] hub 1-0:1.0: USB hub found client # [ 0.881680] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker1 # [ 0.887673] rtc-pl031 9010000.pl031: registered as rtc0 lb # [ 0.889032] hub 1-0:1.0: 6 ports detected client # [ 0.881841] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 worker2 # [ 0.875424] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker1 # [ 0.887701] rtc-pl031 9010000.pl031: setting system clock to 2026-09-25T15:23:47 UTC (1790349827) worker1 # [ 0.887994] i2c_dev: i2c /dev entries driver worker2 # [ 0.875633] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 lb # [ 0.890725] rtc-pl031 9010000.pl031: registered as rtc0 client # [ 0.887545] rtc-pl031 9010000.pl031: registered as rtc0 worker2 # [ 0.887643] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker2 # [ 0.887941] hub 1-0:1.0: USB hub found lb # [ 0.890753] rtc-pl031 9010000.pl031: setting system clock to 2026-09-25T15:23:47 UTC (1790349827) client # [ 0.887574] rtc-pl031 9010000.pl031: setting system clock to 2026-09-25T15:23:47 UTC (1790349827) worker2 # [ 0.887956] hub 1-0:1.0: 6 ports detected client # [ 0.887899] i2c_dev: i2c /dev entries driver lb # [ 0.892851] i2c_dev: i2c /dev entries driver client # [ 0.891006] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker2 # [ 0.890979] rtc-pl031 9010000.pl031: registered as rtc0 client # [ 0.891367] hub 1-0:1.0: USB hub found worker1 # [ 0.893329] sdhci: Secure Digital Host Controller Interface driver client # [ 0.891388] hub 1-0:1.0: 6 ports detected worker2 # [ 0.891005] rtc-pl031 9010000.pl031: setting system clock to 2026-09-25T15:23:47 UTC (1790349827) worker1 # [ 0.893341] sdhci: Copyright(c) Pierre Ossman worker2 # [ 0.891331] i2c_dev: i2c /dev entries driver worker1 # [ 0.893606] Synopsys Designware Multimedia Card Interface Driver worker1 # [ 0.893975] sdhci-pltfm: SDHCI platform and OF driver helper lb # [ 0.896432] sdhci: Secure Digital Host Controller Interface driver worker1 # [ 0.898302] hid: raw HID events driver (C) Jiri Kosina lb # [ 0.896445] sdhci: Copyright(c) Pierre Ossman worker1 # [ 0.898540] usbcore: registered new interface driver usbhid worker1 # [ 0.898546] usbhid: USB HID core driver lb # [ 0.896714] Synopsys Designware Multimedia Card Interface Driver client # [ 0.895860] sdhci: Secure Digital Host Controller Interface driver lb # [ 0.897084] sdhci-pltfm: SDHCI platform and OF driver helper worker2 # [ 0.896793] sdhci: Secure Digital Host Controller Interface driver client # [ 0.895871] sdhci: Copyright(c) Pierre Ossman lb # [ 0.898642] hid: raw HID events driver (C) Jiri Kosina worker2 # [ 0.896804] sdhci: Copyright(c) Pierre Ossman client # [ 0.896129] Synopsys Designware Multimedia Card Interface Driver worker2 # [ 0.897067] Synopsys Designware Multimedia Card Interface Driver lb # [ 0.902339] usbcore: registered new interface driver usbhid client # [ 0.896501] sdhci-pltfm: SDHCI platform and OF driver helper lb # [ 0.902349] usbhid: USB HID core driver worker2 # [ 0.897432] sdhci-pltfm: SDHCI platform and OF driver helper client # [ 0.897988] hid: raw HID events driver (C) Jiri Kosina client # [ 0.898221] usbcore: registered new interface driver usbhid client # [ 0.898228] usbhid: USB HID core driver worker2 # [ 0.901807] hid: raw HID events driver (C) Jiri Kosina worker2 # [ 0.902059] usbcore: registered new interface driver usbhid worker2 # [ 0.902066] usbhid: USB HID core driver worker2 # [ 0.923749] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available client # [ 0.923551] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker1 # [ 0.925907] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available lb # [ 0.925977] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker1 # [ 0.927408] drop_monitor: Initializing network drop monitor service client # [ 0.925070] drop_monitor: Initializing network drop monitor service worker2 # [ 0.925248] drop_monitor: Initializing network drop monitor service lb # [ 0.927502] drop_monitor: Initializing network drop monitor service lb # [ 0.927617] NET: Registered PF_INET6 protocol family client # [ 0.925187] NET: Registered PF_INET6 protocol family worker2 # [ 0.925366] NET: Registered PF_INET6 protocol family worker1 # [ 0.927520] NET: Registered PF_INET6 protocol family worker1 # [ 0.930504] Segment Routing with IPv6 client # [ 0.928227] Segment Routing with IPv6 worker2 # [ 0.928446] Segment Routing with IPv6 lb # [ 0.930652] Segment Routing with IPv6 worker1 # [ 0.930535] In-situ OAM (IOAM) with IPv6 client # [ 0.928247] In-situ OAM (IOAM) with IPv6 worker2 # [ 0.928465] In-situ OAM (IOAM) with IPv6 lb # [ 0.930671] In-situ OAM (IOAM) with IPv6 worker1 # [ 0.930562] NET: Registered PF_PACKET protocol family lb # [ 0.930697] NET: Registered PF_PACKET protocol family worker2 # [ 0.928493] NET: Registered PF_PACKET protocol family client # [ 0.928275] NET: Registered PF_PACKET protocol family client # [ 0.929922] 9pnet: Installing 9P2000 support worker2 # [ 0.930132] 9pnet: Installing 9P2000 support client # [ 0.929968] Key type dns_resolver registered worker2 # [ 0.930178] Key type dns_resolver registered lb # [ 0.932365] 9pnet: Installing 9P2000 support lb # [ 0.932410] Key type dns_resolver registered worker1 # [ 0.932223] 9pnet: Installing 9P2000 support worker1 # [ 0.932270] Key type dns_resolver registered worker1 # [ 0.938767] registered taskstats version 1 client # [ 0.936771] registered taskstats version 1 worker2 # [ 0.936930] registered taskstats version 1 client # [ 0.936916] Loading compiled-in X.509 certificates worker2 # [ 0.937069] Loading compiled-in X.509 certificates lb # [ 0.939199] registered taskstats version 1 worker1 # [ 0.939439] Loading compiled-in X.509 certificates lb # [ 0.939334] Loading compiled-in X.509 certificates worker1 # [ 0.947398] Demotion targets for Node 0: null worker2 # [ 0.945569] Demotion targets for Node 0: null client # [ 0.945437] Demotion targets for Node 0: null lb # [ 0.947921] Demotion targets for Node 0: null worker1 # [ 0.947492] Key type .fscrypt registered worker2 # [ 0.945663] Key type .fscrypt registered lb # [ 0.948020] Key type .fscrypt registered client # [ 0.945532] Key type .fscrypt registered worker1 # [ 0.947498] Key type fscrypt-provisioning registered worker2 # [ 0.945669] Key type fscrypt-provisioning registered lb # [ 0.948027] Key type fscrypt-provisioning registered client # [ 0.945539] Key type fscrypt-provisioning registered worker1 # [ 0.947585] ima: No TPM chip found, activating TPM-bypass! worker2 # [ 0.945758] ima: No TPM chip found, activating TPM-bypass! client # [ 0.945629] ima: No TPM chip found, activating TPM-bypass! lb # [ 0.948114] ima: No TPM chip found, activating TPM-bypass! worker1 # [ 0.947608] ima: Allocated hash algorithm: sha1 worker2 # [ 0.945778] ima: Allocated hash algorithm: sha1 client # [ 0.945648] ima: Allocated hash algorithm: sha1 lb # [ 0.948133] ima: Allocated hash algorithm: sha1 worker1 # [ 0.947628] ima: No architecture policies found worker2 # [ 0.945799] ima: No architecture policies found client # [ 0.945669] ima: No architecture policies found lb # [ 0.948154] ima: No architecture policies found worker1 # [ 0.951587] input: gpio-keys as /devices/platform/gpio-keys/input/input0 client # [ 0.949617] input: gpio-keys as /devices/platform/gpio-keys/input/input0 worker2 # [ 0.949847] input: gpio-keys as /devices/platform/gpio-keys/input/input0 lb # [ 0.952090] input: gpio-keys as /devices/platform/gpio-keys/input/input0 lb # [ 0.969197] clk: Disabling unused clocks worker1 # [ 0.968662] clk: Disabling unused clocks lb # [ 0.969217] PM: genpd: Disabling unused power domains worker1 # [ 0.968685] PM: genpd: Disabling unused power domains client # [ 0.967159] clk: Disabling unused clocks worker2 # [ 0.967524] clk: Disabling unused clocks client # [ 0.967235] PM: genpd: Disabling unused power domains worker2 # [ 0.967544] PM: genpd: Disabling unused power domains worker1 # [ 0.972909] Freeing unused kernel memory: 4736K lb # [ 0.973392] Freeing unused kernel memory: 4736K worker1 # [ 0.973100] Run /init as init process lb # [ 0.973565] Run /init as init process client # [ 0.971419] Freeing unused kernel memory: 4736K worker2 # [ 0.971702] Freeing unused kernel memory: 4736K client # [ 0.971630] Run /init as init process worker2 # [ 0.971899] Run /init as init process lb # [ 0.988948] systemd[1]: Successfully made /usr/ read-only. worker1 # [ 0.988984] systemd[1]: Successfully made /usr/ read-only. client # [ 0.988629] systemd[1]: Successfully made /usr/ read-only. worker2 # [ 0.989099] systemd[1]: Successfully made /usr/ read-only. lb # [ 1.146840] usb 1-1: new high-speed USB device number 2 using ehci-pci worker1 # [ 1.146941] usb 1-1: new high-speed USB device number 2 using ehci-pci client # [ 1.146362] usb 1-1: new high-speed USB device number 2 using ehci-pci worker2 # [ 1.146944] usb 1-1: new high-speed USB device number 2 using ehci-pci lb # [ 1.294449] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker1 # [ 1.294538] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker2 # [ 1.294569] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 client # [ 1.295963] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 lb # [ 1.323600] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker1 # [ 1.323626] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 1.330201] systemd[1]: Detected virtualization qemu. worker1 # [ 1.330121] systemd[1]: Detected virtualization qemu. client # [ 1.323321] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 1.331296] systemd[1]: Detected architecture arm64. worker1 # [ 1.331218] systemd[1]: Detected architecture arm64. worker2 # [ 1.323729] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 1.332317] systemd[1]: Running in initrd. worker1 # [ 1.332245] systemd[1]: Running in initrd. client # [ 1.329850] systemd[1]: Detected virtualization qemu. worker2 # [ 1.330287] systemd[1]: Detected virtualization qemu. client # [ 1.330990] systemd[1]: Detected architecture arm64. worker2 # [ 1.331426] systemd[1]: Detected architecture arm64. lb # [ 1.333660] systemd[1]: Initializing machine ID from random generator. worker1 # [ 1.333634] systemd[1]: Initializing machine ID from random generator. client # [ 1.332020] systemd[1]: Running in initrd. worker2 # [ 1.332454] systemd[1]: Running in initrd. lb # [ 1.335201] systemd[1]: Hostname set to . worker1 # [ 1.335145] systemd[1]: Hostname set to . client # [ 1.333389] systemd[1]: Initializing machine ID from random generator. worker2 # [ 1.333774] systemd[1]: Initializing machine ID from random generator. client # [ 1.334945] systemd[1]: Hostname set to . worker2 # [ 1.335333] systemd[1]: Hostname set to . lb # [ 1.375011] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 worker1 # [ 1.379090] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 worker2 # [ 1.379096] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 client # [ 1.386513] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 lb # [ 1.494841] usb 1-2: new high-speed USB device number 3 using ehci-pci worker1 # [ 1.498943] usb 1-2: new high-speed USB device number 3 using ehci-pci worker2 # [ 1.498946] usb 1-2: new high-speed USB device number 3 using ehci-pci client # [ 1.506367] usb 1-2: new high-speed USB device number 3 using ehci-pci lb # [ 1.585377] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker1 # [ 1.589479] systemd[1]: bpf-restrict-fs: LSM BPF program attached client # [ 1.589582] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 1.589875] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 1.649473] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 lb # [ 1.652297] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker1 # [ 1.653656] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker1 # [ 1.656512] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 client # [ 1.655856] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 client # [ 1.658405] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker2 # [ 1.662756] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.665567] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 lb # [ 1.670588] systemd[1]: Queued start job for default target Initrd Default Target. client # [ 1.671013] systemd[1]: Queued start job for default target Initrd Default Target. worker1 # [ 1.674376] systemd[1]: Queued start job for default target Initrd Default Target. worker2 # [ 1.677913] systemd[1]: Queued start job for default target Initrd Default Target. lb # [ 1.681161] systemd[1]: Created slice Slice /system/modprobe. lb # [ 1.682433] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 1.680723] systemd[1]: Created slice Slice /system/modprobe. lb # [ 1.683845] systemd[1]: Expecting device /dev/disk/by-label/nixos... client # [ 1.681938] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 1.684971] systemd[1]: Reached target Path Units. worker1 # [ 1.685227] systemd[1]: Created slice Slice /system/modprobe. lb # [ 1.685809] systemd[1]: Reached target Slice Units. client # [ 1.683459] systemd[1]: Expecting device /dev/disk/by-label/nixos... lb # [ 1.686642] systemd[1]: Reached target Swaps. client # [ 1.684566] systemd[1]: Reached target Path Units. worker1 # [ 1.686493] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 1.687463] systemd[1]: Reached target Timer Units. client # [ 1.685422] systemd[1]: Reached target Slice Units. worker1 # [ 1.687931] systemd[1]: Expecting device /dev/disk/by-label/nixos... client # [ 1.686282] systemd[1]: Reached target Swaps. lb # [ 1.688493] systemd[1]: Listening on D-Bus System Message Bus Socket. worker1 # [ 1.689065] systemd[1]: Reached target Path Units. client # [ 1.687141] systemd[1]: Reached target Timer Units. worker1 # [ 1.689913] systemd[1]: Reached target Slice Units. lb # [ 1.689895] systemd[1]: Listening on Journal Socket (/dev/log). worker1 # [ 1.690787] systemd[1]: Reached target Swaps. client # [ 1.688248] systemd[1]: Listening on D-Bus System Message Bus Socket. worker1 # [ 1.691662] systemd[1]: Reached target Timer Units. worker2 # [ 1.689548] systemd[1]: Created slice Slice /system/modprobe. client # [ 1.689514] systemd[1]: Listening on Journal Socket (/dev/log). lb # [ 1.691131] systemd[1]: Listening on Journal Sockets. worker1 # [ 1.692709] systemd[1]: Listening on D-Bus System Message Bus Socket. lb # [ 1.691283] systemd[1]: Listening on udev Control Socket. worker2 # [ 1.690797] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 1.694060] systemd[1]: Listening on Journal Socket (/dev/log). lb # [ 1.691412] systemd[1]: Listening on udev Kernel Socket. client # [ 1.690702] systemd[1]: Listening on Journal Sockets. lb # [ 1.691437] systemd[1]: Reached target Socket Units. client # [ 1.690838] systemd[1]: Listening on udev Control Socket. client # [ 1.690957] systemd[1]: Listening on udev Kernel Socket. lb # [ 1.696523] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 1.692362] systemd[1]: Expecting device /dev/disk/by-label/nixos... worker1 # [ 1.695294] systemd[1]: Listening on Journal Sockets. client # [ 1.690979] systemd[1]: Reached target Socket Units. worker2 # [ 1.692403] systemd[1]: Reached target Path Units. worker1 # [ 1.695435] systemd[1]: Listening on udev Control Socket. client # [ 1.695993] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 1.692424] systemd[1]: Reached target Slice Units. lb # [ 1.697763] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 1.695555] systemd[1]: Listening on udev Kernel Socket. worker2 # [ 1.692445] systemd[1]: Reached target Swaps. worker1 # [ 1.695579] systemd[1]: Reached target Socket Units. worker2 # [ 1.692465] systemd[1]: Reached target Timer Units. client # [ 1.697117] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 1.700668] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 1.692656] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 1.692808] systemd[1]: Listening on Journal Socket (/dev/log). worker1 # [ 1.701799] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 1.692956] systemd[1]: Listening on Journal Sockets. worker2 # [ 1.693089] systemd[1]: Listening on udev Control Socket. worker2 # [ 1.693198] systemd[1]: Listening on udev Kernel Socket. worker2 # [ 1.693220] systemd[1]: Reached target Socket Units. lb # [ 1.706920] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 1.705128] systemd[1]: Starting Create List of Static Device Nodes... client # [ 1.706449] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 1.706366] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 1.710781] systemd[1]: Mounting Kernel Configuration File System... lb # [ 1.715194] systemd[1]: Starting Journal Service... worker2 # [ 1.715069] systemd[1]: Mounting Kernel Configuration File System... client # [ 1.714747] systemd[1]: Starting Journal Service... worker1 # [ 1.719315] systemd[1]: Starting Journal Service... worker2 # [ 1.723189] systemd[1]: Starting Journal Service... worker1 # [ 1.735121] systemd[1]: Starting Load Kernel Modules... lb # [ 1.734429] systemd[1]: Starting Load Kernel Modules... worker1 # [ 1.735972] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 1.734535] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 1.733946] systemd[1]: Starting Load Kernel Modules... client # [ 1.734047] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 1.744658] systemd[1]: Starting Coldplug All udev Devices... client # [ 1.744236] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 1.746352] systemd[1]: Starting Load Kernel Modules... worker1 # [ 1.748010] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 1.746455] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 1.766057] systemd-journald[72]: Collecting audit messages is disabled. lb # [ 1.770923] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 1.771783] systemd[1]: Mounted Kernel Configuration File System. client # [ 1.770456] systemd[1]: Finished Create List of Static Device Nodes. worker1 # [ 1.771002] systemd[1]: Finished Create List of Static Device Nodes. client # [ 1.771362] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 1.771836] systemd[1]: Mounted Kernel Configuration File System. lb # [ 1.774696] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker2 # [ 1.775086] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 1.776925] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 1.774411] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker1 # [ 1.790141] systemd-journald[72]: Collecting audit messages is disabled. lb # [ 1.796290] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker2 # [ 1.791158] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 1.792135] systemd[1]: Mounted Kernel Configuration File System. client # [ 1.795747] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. lb # [ 1.799399] systemd-journald[72]: Collecting audit messages is disabled. worker1 # [ 1.799767] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. client # [ 1.798973] systemd-journald[72]: Collecting audit messages is disabled. worker2 # [ 1.799237] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 1.811406] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 1.813552] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev client # [ 1.810939] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 1.813347] systemd[1]: Starting Create Static Device Nodes in /dev... client # [ 1.817541] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker1 # [ 1.818957] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker1 # [ 1.824930] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 worker1 # [ 1.825182] [drm] features: -virgl +edid -resource_blob -host_visible worker1 # [ 1.825191] [drm] features: -context_init worker1 # [ 1.825937] [drm] number of scanouts: 1 worker1 # [ 1.825956] [drm] number of cap sets: 0 worker2 # [ 1.824742] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. lb # [ 1.831889] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 lb # [ 1.832145] [drm] features: -virgl +edid -resource_blob -host_visible lb # [ 1.832155] [drm] features: -context_init lb # [ 1.832849] [drm] number of scanouts: 1 lb # [ 1.832867] [drm] number of cap sets: 0 client # [ 1.831518] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 client # [ 1.831775] [drm] features: -virgl +edid -resource_blob -host_visible client # [ 1.831785] [drm] features: -context_init client # [ 1.832484] [drm] number of scanouts: 1 client # [ 1.832502] [drm] number of cap sets: 0 lb # [ 1.838987] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 1.839456] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker1 # [ 1.841876] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 1.839562] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 1.842002] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 1.846012] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker1 # [ 1.847294] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic worker1 # [ 1.847309] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 worker2 # [ 1.853198] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 worker2 # [ 1.853439] [drm] features: -virgl +edid -resource_blob -host_visible worker2 # [ 1.853449] [drm] features: -context_init worker2 # [ 1.854120] [drm] number of scanouts: 1 worker2 # [ 1.854138] [drm] number of cap sets: 0 client # [ 1.853662] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic lb # [ 1.855206] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic client # [ 1.853682] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 lb # [ 1.855223] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 client # [ 1.862896] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 1.863057] systemd[1]: Reached target Preparation for Local File Systems. client # [ 1.863081] systemd[1]: Reached target Local File Systems. lb # [ 1.871389] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 1.870689] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 1.871557] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 1.871584] systemd[1]: Reached target Local File Systems. lb # [ 1.879139] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker1 # [ 1.879151] Console: switching to colour frame buffer device 160x50 worker1 # [ 1.883044] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device worker1 # [ 1.885904] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 1.879251] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic worker1 # [ 1.886083] systemd[1]: Reached target Preparation for Local File Systems. worker2 # [ 1.879267] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 lb # [ 1.884089] Console: switching to colour frame buffer device 160x50 worker1 # [ 1.886117] systemd[1]: Reached target Local File Systems. lb # [ 1.888187] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device worker1 # [ 1.891674] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 1.886606] Console: switching to colour frame buffer device 160x50 client # [ 1.890870] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device worker2 # [ 1.899561] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 1.899746] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 1.903379] systemd[1]: Finished Load Kernel Modules. worker2 # [ 1.899771] systemd[1]: Reached target Local File Systems. lb # [ 1.905701] systemd[1]: Starting Apply Kernel Variables... worker1 # [ 1.908084] systemd[1]: Finished Load Kernel Modules. worker2 # [ 1.903702] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 1.907616] systemd[1]: Finished Load Kernel Modules. worker2 # [ 1.908355] Console: switching to colour frame buffer device 160x50 worker1 # [ 1.919757] systemd[1]: Starting Apply Kernel Variables... client # [ 1.918811] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 1.923767] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device worker2 # [ 1.940248] systemd[1]: Finished Load Kernel Modules. worker2 # [ 1.943070] systemd[1]: Starting Apply Kernel Variables... worker1 # [ 1.959499] systemd[1]: Finished Apply Kernel Variables. lb # [ 1.959480] systemd[1]: Finished Apply Kernel Variables. worker2 # [ 1.962471] systemd[1]: Started Journal Service. client # [ 1.962932] systemd[1]: Finished Apply Kernel Variables. lb # [ 1.971064] systemd[1]: Started Journal Service. lb # [ 1.968279] systemd-modules-load[73]: Inserted module 'dm_mod' worker2 # [ 1.966541] systemd-modules-load[74]: Inserted module 'dm_mod' client # [ 1.979889] systemd[1]: Started Journal Service. lb # [ 1.969626] systemd-modules-load[73]: Module 'virtio_balloon' is built in worker1 # [ 1.983121] systemd[1]: Started Journal Service. worker2 # [ 1.967822] systemd-modules-load[74]: Module 'virtio_balloon' is built in lb # [ 1.970717] systemd-modules-load[73]: Module 'virtio_console' is built in worker1 # [ 1.971178] systemd-modules-load[73]: Inserted module 'dm_mod' worker2 # [ 1.969226] systemd-modules-load[74]: Module 'virtio_console' is built in lb # [ 1.971834] systemd-modules-load[73]: Inserted module 'virtio_gpu' worker1 # [ 1.972297] systemd-modules-load[73]: Module 'virtio_balloon' is built in worker2 # [ 1.970350] systemd-modules-load[74]: Inserted module 'virtio_gpu' worker1 # [ 1.973436] systemd-modules-load[73]: Module 'virtio_console' is built in worker2 # [ 1.971358] systemd-modules-load[74]: Module 'virtio_rng' is built in worker1 # [ 1.974538] systemd-modules-load[73]: Inserted module 'virtio_gpu' worker1 # [ 1.975550] systemd-modules-load[73]: Module 'virtio_rng' is built in client # [ 1.976116] systemd-modules-load[73]: Inserted module 'dm_mod' client # [ 1.978195] systemd-modules-load[73]: Module 'virtio_balloon' is built in client # [ 1.979322] systemd-modules-load[73]: Module 'virtio_console' is built in lb # [ 1.983517] systemd-modules-load[73]: Module 'virtio_rng' is built in client # [ 1.985267] systemd-modules-load[73]: Inserted module 'virtio_gpu' client # [ 1.986291] systemd-modules-load[73]: Module 'virtio_rng' is built in client # [ 1.987585] systemd-udevd[78]: Using default interface naming scheme 'v261'. lb # [ 1.989708] systemd[1]: Starting Create System Files and Directories... worker2 # [ 1.989652] systemd[1]: Starting Create System Files and Directories... worker1 # [ 1.992413] systemd[1]: Starting Create System Files and Directories... worker1 # [ 1.993743] systemd-udevd[78]: Using default interface naming scheme 'v261'. lb # [ 1.996399] systemd-udevd[78]: Using default interface naming scheme 'v261'. client # [ 2.001486] systemd[1]: Starting Create System Files and Directories... lb # [ 2.004337] systemd[1]: Finished Create System Files and Directories. worker2 # [ 2.004878] systemd[1]: Finished Apply Kernel Variables. worker2 # [ 2.006423] systemd[1]: Finished Create System Files and Directories. client # [ 2.010369] systemd[1]: Finished Create System Files and Directories. worker1 # [ 2.013859] systemd[1]: Finished Create System Files and Directories. worker2 # [ 2.015658] systemd-udevd[78]: Using default interface naming scheme 'v261'. lb # [ 2.021061] systemd[1]: Started Rule-based Manager for Device Events and Files. client # [ 2.026787] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 2.030812] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 2.037455] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 2.068283] systemd[1]: Starting Virtual Console Setup... client # [ 2.084793] systemd[1]: Starting Virtual Console Setup... worker1 # [ 2.088104] systemd[1]: Starting Virtual Console Setup... worker2 # [ 2.089647] systemd[1]: Starting Virtual Console Setup... lb # [ 2.124505] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 2.127900] systemd[1]: Finished Virtual Console Setup. worker1 # [ 2.136488] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 2.136487] systemd-vconsole-setup[100]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 2.139964] systemd[1]: Finished Virtual Console Setup. client # [ 2.139969] systemd[1]: Finished Virtual Console Setup. worker2 # [ 2.145201] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 2.147871] systemd[1]: Finished Virtual Console Setup. worker1 # [ 2.753221] systemd[1]: Finished Coldplug All udev Devices. worker1 # [ 2.754203] systemd[1]: Reached target System Initialization. worker1 # [ 2.755111] systemd[1]: Reached target Basic System. client # [ 2.753181] systemd[1]: Finished Coldplug All udev Devices. client # [ 2.756136] systemd[1]: Reached target System Initialization. client # [ 2.757025] systemd[1]: Reached target Basic System. worker2 # [ 2.766741] systemd[1]: Finished Coldplug All udev Devices. worker2 # [ 2.767765] systemd[1]: Reached target System Initialization. worker2 # [ 2.768924] systemd[1]: Reached target Basic System. lb # [ 2.789482] systemd[1]: Finished Coldplug All udev Devices. lb # [ 2.790458] systemd[1]: Reached target System Initialization. lb # [ 2.791350] systemd[1]: Reached target Basic System. worker1 # [ 2.899999] (udev-worker)[104]: Network interface NamePolicy= disabled on kernel command line. client # [ 2.900166] (udev-worker)[90]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 2.918598] (udev-worker)[96]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 2.933842] (udev-worker)[99]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 2.940183] (udev-worker)[99]: Network interface NamePolicy= disabled on kernel command line. lb # [ 2.943903] (udev-worker)[96]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. lb # [ 2.956433] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line. client # [ 2.958749] (udev-worker)[93]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker2 # [ 2.960707] (udev-worker)[94]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 2.964258] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 2.965137] (udev-worker)[94]: Network interface NamePolicy= disabled on kernel command line. lb # [ 2.976510] (udev-worker)[96]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 3.001385] systemd[1]: Found device /dev/disk/by-label/nixos. worker1 # [ 3.007833] systemd[1]: Reached target Initrd Root Device. worker1 # [ 3.013588] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... lb # [ 3.027919] systemd[1]: Found device /dev/disk/by-label/nixos. lb # [ 3.029748] systemd[1]: Reached target Initrd Root Device. lb # [ 3.033352] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... client # [ 3.037034] systemd[1]: Found device /dev/disk/by-label/nixos. worker2 # [ 3.037145] systemd[1]: Found device /dev/disk/by-label/nixos. client # [ 3.040462] systemd[1]: Reached target Initrd Root Device. worker2 # [ 3.044096] systemd[1]: Reached target Initrd Root Device. client # [ 3.045318] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... worker2 # [ 3.048149] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... worker1 # [ 3.064202] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks worker1 # [ 3.072093] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker1 # [ 3.073918] systemd[1]: Mounting /sysroot... lb # [ 3.084823] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks client # [ 3.083039] systemd-fsck[109]: nixos: clean, 12/65536 files, 13019/262144 blocks worker2 # [ 3.087805] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks lb # [ 3.090558] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. client # [ 3.091998] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. lb # [ 3.096559] systemd[1]: Mounting /sysroot... worker2 # [ 3.095057] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. client # [ 3.096301] systemd[1]: Mounting /sysroot... worker2 # [ 3.100279] systemd[1]: Mounting /sysroot... worker1 # [ 3.120698] EXT4-fs (vda): mounted filesystem 8172407d-5c40-4235-8937-fc8748328c0f r/w with ordered data mode. Quota mode: none. worker1 # [ 3.112743] systemd[1]: Mounted /sysroot. worker1 # [ 3.114403] systemd[1]: Reached target Initrd Root File System. worker1 # [ 3.118273] systemd[1]: Starting Mountpoints Configured in the Real Root... lb # [ 3.146916] EXT4-fs (vda): mounted filesystem 88cbc442-c834-4d45-a9ef-88ee54739cc5 r/w with ordered data mode. Quota mode: none. client # [ 3.146795] EXT4-fs (vda): mounted filesystem 5bf538c6-46a3-4fdd-93a0-9651152ff582 r/w with ordered data mode. Quota mode: none. worker2 # [ 3.147162] EXT4-fs (vda): mounted filesystem ec749343-9b4b-467c-8ef1-28341be0bda4 r/w with ordered data mode. Quota mode: none. lb # [ 3.138227] systemd[1]: Mounted /sysroot. lb # [ 3.140516] systemd[1]: Reached target Initrd Root File System. worker2 # [ 3.138514] systemd[1]: Mounted /sysroot. client # [ 3.139013] systemd[1]: Mounted /sysroot. worker2 # [ 3.140386] systemd[1]: Reached target Initrd Root File System. client # [ 3.141769] systemd[1]: Reached target Initrd Root File System. lb # [ 3.144174] systemd[1]: Starting Mountpoints Configured in the Real Root... worker1 # [ 3.143868] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. worker2 # [ 3.143951] systemd[1]: Starting Mountpoints Configured in the Real Root... client # [ 3.146453] systemd[1]: Starting Mountpoints Configured in the Real Root... worker1 # [ 3.149859] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... worker1 # [ 3.155026] systemd[1]: Reloading... lb # [ 3.171267] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. client # [ 3.171711] systemd-sysroot-fstab-check[117]: /sysroot should be mounted in the initrd, will request daemon-reload. worker2 # [ 3.171915] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. client # [ 3.176109] systemd[1]: Reload requested from client PID 117 ('systemd-sysroot') (unit initrd-parse-etc.service)... lb # [ 3.178281] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... worker2 # [ 3.177545] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... client # [ 3.180678] systemd[1]: Reloading... lb # [ 3.182815] systemd[1]: Reloading... worker2 # [ 3.182253] systemd[1]: Reloading... worker1 # [ 3.358842] systemd[1]: Reloading finished in 205 ms. client # [ 3.384103] systemd[1]: Reloading finished in 204 ms. worker2 # [ 3.383985] systemd[1]: Reloading finished in 203 ms. lb # [ 3.389434] systemd[1]: Reloading finished in 210 ms. worker1 # [ 3.389959] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... worker1 # [ 3.392419] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... worker1 # [ 3.398046] systemd[1]: initrd-parse-etc.service: Deactivated successfully. worker1 # [ 3.400969] systemd[1]: Finished Mountpoints Configured in the Real Root. worker1 # [ 3.402586] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. worker2 # [ 3.419605] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... lb # [ 3.422657] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... client # [ 3.422370] systemd-sysroot-fstab-check[117]: Requesting initrd-fs.target/start/replace... lb # [ 3.426301] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... worker2 # [ 3.424156] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... client # [ 3.426477] systemd-sysroot-fstab-check[117]: Requesting swap.target/start/replace... worker2 # [ 3.430608] systemd[1]: initrd-parse-etc.service: Deactivated successfully. lb # [ 3.433148] systemd[1]: initrd-parse-etc.service: Deactivated successfully. client # [ 3.433035] systemd[1]: initrd-parse-etc.service: Deactivated successfully. lb # [ 3.436090] systemd[1]: Finished Mountpoints Configured in the Real Root. worker2 # [ 3.434687] systemd[1]: Finished Mountpoints Configured in the Real Root. client # [ 3.435621] systemd[1]: Finished Mountpoints Configured in the Real Root. lb # [ 3.437810] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. worker2 # [ 3.435710] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. client # [ 3.438305] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. lb # [ 3.735852] systemd[1]: Mounting /sysroot/nix/.ro-store... client # [ 3.736122] systemd[1]: Mounting /sysroot/nix/.ro-store... worker1 # [ 3.739684] systemd[1]: Mounting /sysroot/nix/.ro-store... client # [ 3.744195] systemd[1]: Mounting /sysroot/nix/.rw-store... lb # [ 3.746270] systemd[1]: Mounting /sysroot/nix/.rw-store... worker1 # [ 3.751468] systemd[1]: Mounting /sysroot/nix/.rw-store... lb # [ 3.758929] systemd[1]: Mounting /sysroot/run... client # [ 3.757246] systemd[1]: Mounting /sysroot/run... worker1 # [ 3.765274] systemd[1]: Mounting /sysroot/run... worker2 # [ 3.767559] systemd[1]: Mounting /sysroot/nix/.ro-store... client # [ 3.771527] systemd[1]: Mounting /sysroot/tmp/shared... lb # [ 3.777362] systemd[1]: Mounting /sysroot/tmp/shared... worker2 # [ 3.778426] systemd[1]: Mounting /sysroot/nix/.rw-store... client # [ 3.779613] systemd[1]: Mounting /sysroot/tmp/xchg... worker1 # [ 3.786395] systemd[1]: Mounting /sysroot/tmp/shared... worker2 # [ 3.791348] systemd[1]: Mounting /sysroot/run... lb # [ 3.798947] systemd[1]: Mounting /sysroot/tmp/xchg... worker2 # [ 3.804398] systemd[1]: Mounting /sysroot/tmp/shared... worker1 # [ 3.828084] systemd[1]: Mounting /sysroot/tmp/xchg... worker1 # [ 3.831011] systemd[1]: Mounted /sysroot/nix/.rw-store. worker2 # [ 3.834867] systemd[1]: Mounting /sysroot/tmp/xchg... client # [ 3.853041] fuse: init (API version 7.45) worker1 # [ 3.848963] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 3.861058] virtiofs virtio6: discovered new tag: nix-store client # [ 3.861881] virtiofs virtio6: virtio_fs_setup_dax: No cache capability worker1 # [ 3.867536] fuse: init (API version 7.45) lb # [ 3.869795] fuse: init (API version 7.45) worker1 # [ 3.874563] virtiofs virtio6: discovered new tag: nix-store lb # [ 3.863703] systemd[1]: Mounted /sysroot/nix/.rw-store. lb # [ 3.864914] systemd[1]: Mounted /sysroot/run. client # [ 3.875964] virtiofs virtio7: discovered new tag: shared client # [ 3.876727] virtiofs virtio7: virtio_fs_setup_dax: No cache capability client # [ 3.866843] systemd[1]: Mounted /sysroot/nix/.rw-store. worker1 # [ 3.884538] virtiofs virtio6: virtio_fs_setup_dax: No cache capability client # [ 3.883423] virtiofs virtio8: discovered new tag: xchg worker2 # [ 3.870986] systemd[1]: Mounted /sysroot/nix/.rw-store. client # [ 3.884157] virtiofs virtio8: virtio_fs_setup_dax: No cache capability worker1 # [ 3.873720] systemd[1]: Mounted /sysroot/run. lb # [ 3.886896] virtiofs virtio6: discovered new tag: nix-store lb # [ 3.887681] virtiofs virtio6: virtio_fs_setup_dax: No cache capability client # [ 3.874137] systemd[1]: Mounted /sysroot/run. worker2 # [ 3.890309] fuse: init (API version 7.45) lb # [ 3.884355] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 3.900473] virtiofs virtio7: discovered new tag: shared worker1 # [ 3.901230] virtiofs virtio7: virtio_fs_setup_dax: No cache capability lb # [ 3.904397] virtiofs virtio7: discovered new tag: shared lb # [ 3.905157] virtiofs virtio7: virtio_fs_setup_dax: No cache capability worker2 # [ 3.903130] virtiofs virtio6: discovered new tag: nix-store worker2 # [ 3.904010] virtiofs virtio6: virtio_fs_setup_dax: No cache capability client # [ 3.894227] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 3.910882] virtiofs virtio8: discovered new tag: xchg lb # [ 3.911596] virtiofs virtio8: discovered new tag: xchg lb # [ 3.912375] virtiofs virtio8: virtio_fs_setup_dax: No cache capability worker1 # [ 3.920943] virtiofs virtio8: virtio_fs_setup_dax: No cache capability client # [ 3.907815] systemd[1]: Mounted /sysroot/nix/.ro-store. worker2 # [ 3.920759] virtiofs virtio7: discovered new tag: shared lb # [ 3.910601] systemd[1]: Mounted /sysroot/nix/.ro-store. worker2 # [ 3.921548] virtiofs virtio7: virtio_fs_setup_dax: No cache capability client # [ 3.910806] systemd[1]: Mounted /sysroot/tmp/shared. worker2 # [ 3.910589] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 3.911833] systemd[1]: Mounted /sysroot/tmp/xchg. worker1 # [ 3.914099] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 3.928976] virtiofs virtio8: discovered new tag: xchg worker2 # [ 3.929719] virtiofs virtio8: virtio_fs_setup_dax: No cache capability lb # [ 3.922210] systemd[1]: Mounted /sysroot/tmp/shared. worker1 # [ 3.922411] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 3.924586] systemd[1]: Mounted /sysroot/tmp/xchg. worker1 # [ 3.926432] systemd[1]: Mounted /sysroot/nix/.ro-store. lb # [ 3.926416] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker1 # [ 3.927836] systemd[1]: Mounted /sysroot/tmp/shared. lb # [ 3.929889] systemd[1]: Finished rw-sysroot-nix-store.service. worker1 # [ 3.931608] systemd[1]: Mounted /sysroot/tmp/xchg. worker2 # [ 3.935854] systemd[1]: Mounted /sysroot/nix/.ro-store. client # [ 3.938037] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 3.939144] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 3.943138] systemd[1]: Mounted /sysroot/run. client # [ 3.944139] systemd[1]: Mounting /sysroot/nix/store... worker2 # [ 3.951148] systemd[1]: Mounted /sysroot/tmp/shared. worker2 # [ 3.954847] systemd[1]: Mounted /sysroot/tmp/xchg. worker2 # [ 3.962242] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 3.965416] systemd[1]: Finished rw-sysroot-nix-store.service. client # [ 3.981459] systemd[1]: Mounted /sysroot/nix/store. client # [ 3.983367] systemd[1]: Reached target Initrd File Systems. client # [ 3.985527] systemd[1]: Starting Find NixOS closure... client # [ 3.995366] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... client # [ 4.030636] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. client # [ 4.042553] systemd[1]: Finished Find NixOS closure. client # [ 4.044700] systemd[1]: Reached target Initrd Default Target. client # [ 4.048766] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... client # [ 4.079424] systemd[1]: Stopped target Initrd Default Target. client # [ 4.082900] systemd[1]: Stopped target Basic System. client # [ 4.084361] systemd[1]: Stopped target Initrd Root Device. client # [ 4.088941] systemd[1]: Stopped target Path Units. client # [ 4.090561] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. client # [ 4.092686] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. client # [ 4.096727] systemd[1]: Stopped target Slice Units. client # [ 4.100164] systemd[1]: Stopped target Socket Units. client # [ 4.101970] systemd[1]: Stopped target System Initialization. client # [ 4.104481] systemd[1]: Stopped target Swaps. client # [ 4.108143] systemd[1]: Stopped target Timer Units. client # [ 4.109871] systemd[1]: dbus.socket: Deactivated successfully. client # [ 4.113431] systemd[1]: Closed D-Bus System Message Bus Socket. client # [ 4.117442] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. client # [ 4.119412] systemd[1]: Stopped Find NixOS closure. client # [ 4.123229] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 4.130576] systemd[1]: systemd-sysctl.service: Deactivated successfully. client # [ 4.131678] systemd[1]: Stopped Apply Kernel Variables. client # [ 4.136717] systemd[1]: systemd-modules-load.service: Deactivated successfully. client # [ 4.137825] systemd[1]: Stopped Load Kernel Modules. client # [ 4.138552] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. client # [ 4.139656] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. client # [ 4.146433] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. client # [ 4.149065] systemd[1]: Stopped Create System Files and Directories. client # [ 4.150035] systemd[1]: Stopped target Local File Systems. client # [ 4.150838] systemd[1]: Stopped target Preparation for Local File Systems. client # [ 4.151805] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. client # [ 4.153048] systemd[1]: Stopped Coldplug All udev Devices. client # [ 4.153847] systemd[1]: Stopping Rule-based Manager for Device Events and Files... client # [ 4.154904] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 4.155945] systemd[1]: Stopped Virtual Console Setup. client # [ 4.160247] systemd[1]: initrd-cleanup.service: Deactivated successfully. client # [ 4.161346] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. client # [ 4.162281] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 4.163292] systemd[1]: Finished rw-sysroot-nix-store.service. client # [ 4.169391] systemd[1]: systemd-udevd.service: Deactivated successfully. client # [ 4.170467] systemd[1]: Stopped Rule-based Manager for Device Events and Files. client # [ 4.171507] systemd[1]: systemd-udevd.service: Consumed 1.098s CPU time over 2.302s wall clock time, 17.8M memory peak. client # [ 4.173115] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. client # [ 4.174139] systemd[1]: Closed udev Control Socket. client # [ 4.174871] systemd[1]: Starting Cleanup udev Database... client # [ 4.175658] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. client # [ 4.180228] systemd[1]: Stopped Create Static Device Nodes in /dev. client # [ 4.184336] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. client # [ 4.185729] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. client # [ 4.188147] systemd[1]: kmod-static-nodes.service: Deactivated successfully. client # [ 4.192140] systemd[1]: Stopped Create List of Static Device Nodes. client # [ 4.213500] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. client # [ 4.215853] systemd[1]: Finished Cleanup udev Database. client # [ 4.220708] systemd[1]: Reached target Switch Root. client # [ 4.221488] systemd[1]: Starting NixOS Activation... client # [ 4.301953] initrd-nixos-activation-start[168]: booting system configuration /nix/store/nwgxc9xy9a9l9j6z82df6cwp1rq7ly6z-nixos-system-client-test worker1 # [ 4.305679] (udev-worker)[100]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory worker1 # [ 4.310973] (udev-worker)[100]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory worker2 # [ 4.315200] (udev-worker)[99]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory worker2 # [ 4.321675] (udev-worker)[99]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory lb # [ 4.324412] (udev-worker)[99]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory lb # [ 4.330259] (udev-worker)[99]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory client # [ 4.333807] initrd-nixos-activation-start[168]: running activation script... worker1 # [ 4.340932] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 4.344882] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 4.348278] systemd[1]: Stopping Virtual Console Setup... worker1 # [ 4.349132] systemd[1]: Starting Virtual Console Setup... worker2 # [ 4.351356] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 4.352671] systemd[1]: Stopped Virtual Console Setup. lb # [ 4.356091] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 4.357212] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 4.356423] systemd[1]: Stopping Virtual Console Setup... worker2 # [ 4.357315] systemd[1]: Starting Virtual Console Setup... lb # [ 4.360150] systemd[1]: Stopping Virtual Console Setup... lb # [ 4.362342] systemd[1]: Starting Virtual Console Setup... worker1 # [ 4.364720] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 4.366194] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 4.369079] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 4.371648] systemd[1]: Starting Virtual Console Setup... worker2 # [ 4.370629] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 4.372957] systemd[1]: Starting Virtual Console Setup... lb # [ 4.378028] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 4.380182] systemd[1]: Stopped Virtual Console Setup. lb # [ 4.383707] systemd[1]: Starting Virtual Console Setup... worker1 # [ 4.389441] systemd-vconsole-setup[153]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 4.392772] systemd[1]: Finished Virtual Console Setup. worker2 # [ 4.394798] systemd-vconsole-setup[153]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 4.398218] systemd[1]: Finished Virtual Console Setup. lb # [ 4.401852] systemd-vconsole-setup[155]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 4.405248] systemd[1]: Finished Virtual Console Setup. client # [ 4.547159] initrd-nixos-activation-start[191]: setting up /etc... client # [ 4.657840] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. client # [ 4.660218] systemd[1]: Finished NixOS Activation. client # [ 4.662075] systemd[1]: Starting Switch Root... client # [ 4.679905] systemd[1]: Switching root. lb # [ 4.735154] systemd[1]: Mounting /sysroot/nix/store... worker1 # [ 4.737953] systemd[1]: Mounting /sysroot/nix/store... worker2 # [ 4.768562] systemd[1]: Mounting /sysroot/nix/store... lb # [ 4.776269] systemd[1]: Mounted /sysroot/nix/store. lb # [ 4.778432] systemd[1]: Reached target Initrd File Systems. worker1 # [ 4.780502] systemd[1]: Mounted /sysroot/nix/store. lb # [ 4.782188] systemd[1]: Starting Find NixOS closure... worker1 # [ 4.782763] systemd[1]: Reached target Initrd File Systems. worker1 # [ 4.786335] systemd[1]: Starting Find NixOS closure... lb # [ 4.789372] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker1 # [ 4.793670] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker2 # [ 4.811324] systemd[1]: Mounted /sysroot/nix/store. worker2 # [ 4.814584] systemd[1]: Reached target Initrd File Systems. worker2 # [ 4.817681] systemd[1]: Starting Find NixOS closure... worker2 # [ 4.828200] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... lb # [ 4.832660] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker1 # [ 4.835286] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. lb # [ 4.836159] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. worker1 # [ 4.840215] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. lb # [ 4.845421] systemd[1]: Finished Find NixOS closure. client # [ 4.859717] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). lb # [ 4.848167] systemd[1]: Reached target Initrd Default Target. worker1 # [ 4.848902] systemd[1]: Finished Find NixOS closure. lb # [ 4.850263] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker1 # [ 4.850568] systemd[1]: Reached target Initrd Default Target. worker1 # [ 4.853568] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker2 # [ 4.860859] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker2 # [ 4.864001] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. lb # [ 4.876364] systemd[1]: initrd-cleanup.service: Deactivated successfully. lb # [ 4.878003] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker2 # [ 4.877461] systemd[1]: Finished Find NixOS closure. worker1 # [ 4.879951] systemd[1]: Stopped target Initrd Default Target. lb # [ 4.881242] systemd[1]: Stopped target Initrd Default Target. worker1 # [ 4.881613] systemd[1]: Stopped target Basic System. lb # [ 4.882392] systemd[1]: Stopped target Basic System. worker2 # [ 4.880139] systemd[1]: Reached target Initrd Default Target. lb # [ 4.884186] systemd[1]: Stopped target Initrd Root Device. worker1 # [ 4.884143] systemd[1]: Stopped target Initrd Root Device. worker1 # [ 4.885409] systemd[1]: Stopped target Path Units. lb # [ 4.886904] systemd[1]: Stopped target Path Units. worker2 # [ 4.884287] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker1 # [ 4.888277] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker1 # [ 4.890069] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. lb # [ 4.887793] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker1 # [ 4.895695] systemd[1]: Stopped target Slice Units. lb # [ 4.896295] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker1 # [ 4.896702] systemd[1]: Stopped target Socket Units. lb # [ 4.897545] systemd[1]: Stopped target Slice Units. worker1 # [ 4.897557] systemd[1]: Stopped target System Initialization. lb # [ 4.898348] systemd[1]: Stopped target Socket Units. worker1 # [ 4.898511] systemd[1]: Stopped target Swaps. lb # [ 4.899236] systemd[1]: Stopped target System Initialization. worker1 # [ 4.899340] systemd[1]: Stopped target Timer Units. lb # [ 4.906982] systemd[1]: Stopped target Swaps. lb # [ 4.907764] systemd[1]: Stopped target Timer Units. worker2 # [ 4.906694] systemd[1]: Stopped target Initrd Default Target. worker2 # [ 4.908541] systemd[1]: Stopped target Basic System. lb # [ 4.911159] systemd[1]: dbus.socket: Deactivated successfully. worker1 # [ 4.912637] systemd[1]: dbus.socket: Deactivated successfully. worker1 # [ 4.913546] systemd[1]: Closed D-Bus System Message Bus Socket. worker1 # [ 4.914526] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker2 # [ 4.912559] systemd[1]: Stopped target Initrd Root Device. worker1 # [ 4.915700] systemd[1]: Stopped Find NixOS closure. worker2 # [ 4.913648] systemd[1]: Stopped target Path Units. worker2 # [ 4.914557] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. lb # [ 4.919171] systemd[1]: Closed D-Bus System Message Bus Socket. worker2 # [ 4.917170] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker2 # [ 4.920349] systemd[1]: Stopped target Slice Units. worker1 # [ 4.923843] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 4.924559] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. lb # [ 4.925762] systemd[1]: Stopped Find NixOS closure. worker2 # [ 4.924253] systemd[1]: Stopped target Socket Units. worker1 # [ 4.927231] systemd[1]: systemd-sysctl.service: Deactivated successfully. worker2 # [ 4.925239] systemd[1]: Stopped target System Initialization. worker2 # [ 4.926762] systemd[1]: Stopped target Swaps. lb # [ 4.930380] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 4.930703] systemd[1]: Stopped Apply Kernel Variables. lb # [ 4.931899] systemd[1]: systemd-sysctl.service: Deactivated successfully. worker1 # [ 4.933000] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker2 # [ 4.933800] systemd[1]: Stopped target Timer Units. lb # [ 4.936242] systemd[1]: Stopped Apply Kernel Variables. worker1 # [ 4.936191] systemd[1]: Stopped Load Kernel Modules. worker2 # [ 4.934735] systemd[1]: dbus.socket: Deactivated successfully. lb # [ 4.937043] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker1 # [ 4.936981] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker2 # [ 4.935693] systemd[1]: Closed D-Bus System Message Bus Socket. lb # [ 4.940228] systemd[1]: Stopped Load Kernel Modules. worker1 # [ 4.940168] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. lb # [ 4.941040] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker1 # [ 4.941379] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker2 # [ 4.939675] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. lb # [ 4.944208] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker1 # [ 4.944387] systemd[1]: Stopped Create System Files and Directories. lb # [ 4.945326] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker1 # [ 4.946415] systemd[1]: Stopped target Local File Systems. worker2 # [ 4.944388] systemd[1]: Stopped Find NixOS closure. worker1 # [ 4.947416] systemd[1]: Stopped target Preparation for Local File Systems. lb # [ 4.948188] systemd[1]: Stopped Create System Files and Directories. worker1 # [ 4.948540] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. worker1 # [ 4.949546] systemd[1]: Stopped Coldplug All udev Devices. worker1 # [ 4.950326] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker2 # [ 4.949025] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 4.951377] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 4.952260] systemd[1]: Stopped target Local File Systems. lb # [ 4.953075] systemd[1]: Stopped target Preparation for Local File Systems. lb # [ 4.954029] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. lb # [ 4.955028] systemd[1]: Stopped Coldplug All udev Devices. worker2 # [ 4.952471] systemd[1]: systemd-sysctl.service: Deactivated successfully. lb # [ 4.955801] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker1 # [ 4.956229] systemd[1]: Stopped Virtual Console Setup. lb # [ 4.960387] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 4.960441] systemd[1]: initrd-cleanup.service: Deactivated successfully. lb # [ 4.961426] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 4.961573] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. lb # [ 4.962158] systemd[1]: systemd-udevd.service: Deactivated successfully. worker1 # [ 4.962708] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 4.963104] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker2 # [ 4.961866] systemd[1]: Stopped Apply Kernel Variables. worker1 # [ 4.964161] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 4.963738] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker1 # [ 4.968122] systemd[1]: systemd-udevd.service: Deactivated successfully. lb # [ 4.968367] systemd[1]: systemd-udevd.service: Consumed 1.400s CPU time over 3.046s wall clock time, 22.4M memory peak. worker2 # [ 4.966913] systemd[1]: Stopped Load Kernel Modules. worker1 # [ 4.969102] systemd[1]: Stopped Rule-based Manager for Device Events and Files. lb # [ 4.969813] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. worker2 # [ 4.968182] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker1 # [ 4.970125] systemd[1]: systemd-udevd.service: Consumed 1.423s CPU time over 3.055s wall clock time, 22.2M memory peak. lb # [ 4.972098] systemd[1]: Closed udev Control Socket. worker2 # [ 4.970530] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. lb # [ 4.976404] systemd[1]: Starting Cleanup udev Database... worker1 # [ 4.976321] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. lb # [ 4.977224] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. worker1 # [ 4.977380] systemd[1]: Closed udev Control Socket. worker1 # [ 4.978104] systemd[1]: Starting Cleanup udev Database... lb # [ 4.978292] systemd[1]: Stopped Create Static Device Nodes in /dev. worker2 # [ 4.975725] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker1 # [ 4.978965] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. lb # [ 4.979238] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker2 # [ 4.977323] systemd[1]: Stopped Create System Files and Directories. worker2 # [ 4.978235] systemd[1]: Stopped target Local File Systems. worker2 # [ 4.979021] systemd[1]: Stopped target Preparation for Local File Systems. worker2 # [ 4.979973] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. worker2 # [ 4.981256] systemd[1]: Stopped Coldplug All udev Devices. lb # [ 4.984372] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. worker2 # [ 4.982059] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker1 # [ 4.984412] systemd[1]: Stopped Create Static Device Nodes in /dev. lb # [ 4.985411] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker2 # [ 4.983121] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 4.985339] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. lb # [ 4.986402] systemd[1]: Stopped Create List of Static Device Nodes. worker2 # [ 4.984295] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 4.986469] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. worker2 # [ 4.985109] systemd[1]: initrd-cleanup.service: Deactivated successfully. worker1 # [ 4.987532] systemd[1]: kmod-static-nodes.service: Deactivated successfully. lb # [ 4.988092] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 4.986075] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker2 # [ 4.987011] systemd[1]: systemd-udevd.service: Deactivated successfully. worker2 # [ 4.987938] systemd[1]: Stopped Rule-based Manager for Device Events and Files. lb # [ 4.992417] systemd[1]: Finished rw-sysroot-nix-store.service. worker1 # [ 4.992432] systemd[1]: Stopped Create List of Static Device Nodes. lb # [ 4.994876] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. lb # [ 4.996645] systemd[1]: Finished Cleanup udev Database. lb # [ 4.999124] systemd[1]: Reached target Switch Root. worker2 # [ 4.996200] systemd[1]: systemd-udevd.service: Consumed 1.412s CPU time over 3.076s wall clock time, 22.1M memory peak. worker2 # [ 4.997727] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 4.998994] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 5.003784] systemd[1]: Starting NixOS Activation... worker2 # [ 5.004134] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. worker2 # [ 5.005198] systemd[1]: Closed udev Control Socket. worker1 # [ 5.007537] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. worker2 # [ 5.005932] systemd[1]: Starting Cleanup udev Database... worker2 # [ 5.006739] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. worker2 # [ 5.008137] systemd[1]: Stopped Create Static Device Nodes in /dev. worker1 # [ 5.012428] systemd[1]: Finished Cleanup udev Database. worker1 # [ 5.013216] systemd[1]: Reached target Switch Root. worker1 # [ 5.013915] systemd[1]: Starting NixOS Activation... worker2 # [ 5.012398] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker2 # [ 5.013644] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. worker2 # [ 5.014663] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker2 # [ 5.020369] systemd[1]: Stopped Create List of Static Device Nodes. worker2 # [ 5.039009] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. worker2 # [ 5.040988] systemd[1]: Finished Cleanup udev Database. worker2 # [ 5.041766] systemd[1]: Reached target Switch Root. worker2 # [ 5.044240] systemd[1]: Starting NixOS Activation... lb # [ 5.072729] initrd-nixos-activation-start[178]: booting system configuration /nix/store/5hmvppg1fs8xkiccvl0m50n4dxjz92fr-nixos-system-lb-test worker1 # [ 5.079326] initrd-nixos-activation-start[177]: booting system configuration /nix/store/cw89fd0zkc3gaxrg6haynbv2xr4pxwgi-nixos-system-worker1-test lb # [ 5.101265] initrd-nixos-activation-start[178]: running activation script... worker1 # [ 5.107517] initrd-nixos-activation-start[177]: running activation script... worker2 # [ 5.107036] initrd-nixos-activation-start[177]: booting system configuration /nix/store/pv8jv0ma9kiark5g4gx6k5bl6dq0g11a-nixos-system-worker2-test worker2 # [ 5.135493] initrd-nixos-activation-start[177]: running activation script... lb # [ 5.285496] initrd-nixos-activation-start[201]: setting up /etc... worker1 # [ 5.301001] initrd-nixos-activation-start[200]: setting up /etc... worker2 # [ 5.322448] initrd-nixos-activation-start[200]: setting up /etc... client # [ 5.369028] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 5.376016] systemd[1]: Detected virtualization qemu. client # [ 5.377761] systemd[1]: Detected architecture arm64. client # [ 5.380148] systemd[1]: Detected first boot. client # [ 5.383812] systemd[1]: Initializing machine ID from random generator. lb # [ 5.389330] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. lb # [ 5.392086] systemd[1]: Finished NixOS Activation. lb # [ 5.393186] systemd[1]: Starting Switch Root... worker1 # [ 5.405647] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. worker1 # [ 5.408200] systemd[1]: Finished NixOS Activation. worker1 # [ 5.409634] systemd[1]: Starting Switch Root... lb # [ 5.414322] systemd[1]: Switching root. worker1 # [ 5.426803] systemd[1]: Switching root. worker2 # [ 5.428348] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. worker2 # [ 5.429517] systemd[1]: Finished NixOS Activation. worker2 # [ 5.432109] systemd[1]: Starting Switch Root... worker2 # [ 5.446602] systemd[1]: Switching root. lb # [ 5.595909] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). worker1 # [ 5.607870] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). worker2 # [ 5.627895] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). client # [ 5.653501] systemd[1]: bpf-restrict-fs: LSM BPF program attached client # [ 5.792785] systemd[1]: Applying preset policy. client # [ 5.981244] systemd[1]: Populated /etc with preset unit settings. lb # [ 6.070771] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 6.077528] systemd[1]: Detected virtualization qemu. lb # [ 6.079367] systemd[1]: Detected architecture arm64. lb # [ 6.081709] systemd[1]: Detected first boot. lb # [ 6.084868] systemd[1]: Initializing machine ID from random generator. worker1 # [ 6.086529] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker1 # [ 6.093681] systemd[1]: Detected virtualization qemu. worker1 # [ 6.095411] systemd[1]: Detected architecture arm64. worker1 # [ 6.097158] systemd[1]: Detected first boot. worker1 # [ 6.100939] systemd[1]: Initializing machine ID from random generator. worker2 # [ 6.098606] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 6.105655] systemd[1]: Detected virtualization qemu. worker2 # [ 6.107414] systemd[1]: Detected architecture arm64. worker2 # [ 6.109728] systemd[1]: Detected first boot. worker2 # [ 6.112947] systemd[1]: Initializing machine ID from random generator. client # [ 6.180741] systemd[1]: initrd-switch-root.service: Deactivated successfully. client # [ 6.182429] systemd[1]: Stopped initrd-switch-root.service. client # [ 6.185355] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. client # [ 6.188639] systemd[1]: Created slice Slice /system/getty. client # [ 6.190676] systemd[1]: Created slice User and Session Slice. client # [ 6.191924] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 6.193544] systemd[1]: Started Forward Password Requests to Wall Directory Watch. client # [ 6.195722] systemd[1]: Expecting device /dev/hvc0... client # [ 6.196715] systemd[1]: Expecting device /dev/ttyAMA0... client # [ 6.198319] systemd[1]: Reached target Local Encrypted Volumes. client # [ 6.199949] systemd[1]: Stopped target initrd-fs.target. client # [ 6.201436] systemd[1]: Stopped target initrd-root-fs.target. client # [ 6.203098] systemd[1]: Stopped target initrd-switch-root.target. client # [ 6.204279] systemd[1]: Reached target Virtual Machines and Containers. client # [ 6.205988] systemd[1]: Reached target Path Units. client # [ 6.207488] systemd[1]: Reached target Remote File Systems. client # [ 6.208549] systemd[1]: Reached target Slice Units. client # [ 6.209648] systemd[1]: Reached target Swaps. client # [ 6.211993] systemd[1]: Listening on Query the User Interactively for a Password. client # [ 6.214847] systemd[1]: Listening on Process Core Dump Socket. client # [ 6.216994] systemd[1]: Listening on Credential Encryption/Decryption. client # [ 6.219296] systemd[1]: Listening on Factory Reset Management. client # [ 6.220514] systemd[1]: Listening on Hostname Service Socket. client # [ 6.224007] systemd[1]: Starting Journal Log Access Socket... client # [ 6.226022] systemd[1]: Listening on Journal Audit Socket. client # [ 6.229175] systemd[1]: Listening on Console Output Muting Service Socket. client # [ 6.230749] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. client # [ 6.233403] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os client # [ 6.236102] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki client # [ 6.240882] systemd[1]: Listening on Disk Repartitioning Service Socket. client # [ 6.242219] systemd[1]: Listening on udev Control Socket. client # [ 6.243492] systemd[1]: Listening on udev Varlink Socket. client # [ 6.246586] systemd[1]: Mounting Huge Pages File System... client # [ 6.254695] systemd[1]: Mounting POSIX Message Queue File System... client # [ 6.262671] systemd[1]: Mounting Kernel Debug File System... client # [ 6.271123] systemd[1]: Mounting Kernel Trace File System... client # [ 6.283306] systemd[1]: Starting Create List of Static Device Nodes... client # [ 6.285452] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 6.297886] systemd[1]: Mounting Kernel Configuration File System... client # [ 6.301188] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm client # [ 6.311439] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore client # [ 6.313262] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse client # [ 6.336379] systemd[1]: Mounting FUSE Control File System... client # [ 6.337898] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 lb # [ 6.353886] systemd[1]: bpf-restrict-fs: LSM BPF program attached client # [ 6.367239] systemd[1]: Starting Journal Service... worker1 # [ 6.370022] systemd[1]: bpf-restrict-fs: LSM BPF program attached client # [ 6.374538] systemd[1]: Starting Load Kernel Modules... worker2 # [ 6.382453] systemd[1]: bpf-restrict-fs: LSM BPF program attached client # [ 6.404856] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... client # [ 6.421471] systemd[1]: Starting Remount Root and Kernel File Systems... client # [ 6.422988] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 6.437145] systemd-journald[262]: Collecting audit messages is enabled. client # [ 6.446054] systemd[1]: Starting Coldplug All udev Devices... client # [ 6.449624] systemd[1]: Listening on Journal Log Access Socket. client # [ 6.457195] systemd[1]: Started Journal Service. client # [ 6.442492] systemd[1]: Queued start job for default target Multi-User System. lb # [ 6.500995] systemd[1]: Applying preset policy. worker1 # [ 6.515372] systemd[1]: Applying preset policy. client # [ 6.515594] EXT4-fs (vda): re-mounted 5bf538c6-46a3-4fdd-93a0-9651152ff582. worker2 # [ 6.528876] systemd[1]: Applying preset policy. worker1 # [ 6.693670] systemd[1]: Populated /etc with preset unit settings. lb # [ 6.708763] systemd[1]: Populated /etc with preset unit settings. worker2 # [ 6.711848] systemd[1]: Populated /etc with preset unit settings. worker1 # [ 6.888659] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker1 # [ 6.890304] systemd[1]: Stopped initrd-switch-root.service. worker1 # [ 6.893239] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. worker1 # [ 6.897222] systemd[1]: Created slice Slice /system/getty. worker1 # [ 6.899297] systemd[1]: Created slice User and Session Slice. worker1 # [ 6.900502] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 6.902096] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker1 # [ 6.904257] systemd[1]: Expecting device /dev/hvc0... worker1 # [ 6.905865] systemd[1]: Expecting device /dev/ttyAMA0... worker1 # [ 6.907582] systemd[1]: Reached target Local Encrypted Volumes. worker1 # [ 6.909254] systemd[1]: Stopped target initrd-fs.target. worker2 # [ 6.908406] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker1 # [ 6.910844] systemd[1]: Stopped target initrd-root-fs.target. worker2 # [ 6.909681] systemd[1]: Stopped initrd-switch-root.service. worker1 # [ 6.912493] systemd[1]: Stopped target initrd-switch-root.target. worker1 # [ 6.914233] systemd[1]: Reached target Virtual Machines and Containers. worker1 # [ 6.916107] systemd[1]: Reached target Path Units. worker2 # [ 6.913390] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. lb # [ 6.916723] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker1 # [ 6.917644] systemd[1]: Reached target Remote File Systems. lb # [ 6.918362] systemd[1]: Stopped initrd-switch-root.service. worker2 # [ 6.917481] systemd[1]: Created slice Slice /system/getty. worker1 # [ 6.919288] systemd[1]: Reached target Slice Units. worker1 # [ 6.920826] systemd[1]: Reached target Swaps. worker2 # [ 6.919614] systemd[1]: Created slice User and Session Slice. lb # [ 6.922057] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. worker2 # [ 6.920857] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 6.923602] systemd[1]: Listening on Query the User Interactively for a Password. worker2 # [ 6.922475] systemd[1]: Started Forward Password Requests to Wall Directory Watch. lb # [ 6.926187] systemd[1]: Created slice Slice /system/getty. worker1 # [ 6.926316] systemd[1]: Listening on Process Core Dump Socket. worker2 # [ 6.924504] systemd[1]: Expecting device /dev/hvc0... lb # [ 6.928222] systemd[1]: Created slice User and Session Slice. worker1 # [ 6.928403] systemd[1]: Listening on Credential Encryption/Decryption. worker2 # [ 6.926084] systemd[1]: Expecting device /dev/ttyAMA0... lb # [ 6.929452] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker2 # [ 6.927870] systemd[1]: Reached target Local Encrypted Volumes. worker1 # [ 6.930698] systemd[1]: Listening on Factory Reset Management. lb # [ 6.931272] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker2 # [ 6.929560] systemd[1]: Stopped target initrd-fs.target. worker1 # [ 6.932378] systemd[1]: Listening on Hostname Service Socket. worker2 # [ 6.931232] systemd[1]: Stopped target initrd-root-fs.target. lb # [ 6.933376] systemd[1]: Expecting device /dev/hvc0... worker2 # [ 6.932334] systemd[1]: Stopped target initrd-switch-root.target. lb # [ 6.935117] systemd[1]: Expecting device /dev/ttyAMA0... worker1 # [ 6.935728] systemd[1]: Starting Journal Log Access Socket... worker2 # [ 6.933684] systemd[1]: Reached target Virtual Machines and Containers. lb # [ 6.936160] systemd[1]: Reached target Local Encrypted Volumes. lb # [ 6.937549] systemd[1]: Stopped target initrd-fs.target. worker1 # [ 6.937708] systemd[1]: Listening on Journal Audit Socket. worker2 # [ 6.935181] systemd[1]: Reached target Path Units. lb # [ 6.938902] systemd[1]: Stopped target initrd-root-fs.target. worker2 # [ 6.936822] systemd[1]: Reached target Remote File Systems. worker1 # [ 6.940256] systemd[1]: Listening on Console Output Muting Service Socket. lb # [ 6.940231] systemd[1]: Stopped target initrd-switch-root.target. worker2 # [ 6.938419] systemd[1]: Reached target Slice Units. lb # [ 6.941648] systemd[1]: Reached target Virtual Machines and Containers. worker1 # [ 6.941776] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. worker2 # [ 6.940028] systemd[1]: Reached target Swaps. lb # [ 6.943174] systemd[1]: Reached target Path Units. worker1 # [ 6.943172] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 6.942182] systemd[1]: Listening on Query the User Interactively for a Password. lb # [ 6.944922] systemd[1]: Reached target Remote File Systems. worker1 # [ 6.945686] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki lb # [ 6.946640] systemd[1]: Reached target Slice Units. worker2 # [ 6.945023] systemd[1]: Listening on Process Core Dump Socket. lb # [ 6.947656] systemd[1]: Reached target Swaps. worker2 # [ 6.947114] systemd[1]: Listening on Credential Encryption/Decryption. lb # [ 6.950059] systemd[1]: Listening on Query the User Interactively for a Password. worker1 # [ 6.950749] systemd[1]: Listening on Disk Repartitioning Service Socket. worker2 # [ 6.949380] systemd[1]: Listening on Factory Reset Management. worker1 # [ 6.952209] systemd[1]: Listening on udev Control Socket. worker2 # [ 6.950588] systemd[1]: Listening on Hostname Service Socket. lb # [ 6.952940] systemd[1]: Listening on Process Core Dump Socket. worker1 # [ 6.953364] systemd[1]: Listening on udev Varlink Socket. lb # [ 6.955132] systemd[1]: Listening on Credential Encryption/Decryption. worker1 # [ 6.956261] systemd[1]: Mounting Huge Pages File System... worker2 # [ 6.954168] systemd[1]: Starting Journal Log Access Socket... lb # [ 6.957358] systemd[1]: Listening on Factory Reset Management. worker2 # [ 6.956211] systemd[1]: Listening on Journal Audit Socket. lb # [ 6.958562] systemd[1]: Listening on Hostname Service Socket. worker2 # [ 6.959377] systemd[1]: Listening on Console Output Muting Service Socket. lb # [ 6.962349] systemd[1]: Starting Journal Log Access Socket... worker2 # [ 6.960962] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. worker1 # [ 6.963101] systemd[1]: Mounting POSIX Message Queue File System... lb # [ 6.964376] systemd[1]: Listening on Journal Audit Socket. worker2 # [ 6.962438] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 6.964239] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki lb # [ 6.967532] systemd[1]: Listening on Console Output Muting Service Socket. lb # [ 6.969033] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. lb # [ 6.970454] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 6.970878] systemd[1]: Mounting Kernel Debug File System... worker2 # [ 6.970378] systemd[1]: Listening on Disk Repartitioning Service Socket. lb # [ 6.972374] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker2 # [ 6.971848] systemd[1]: Listening on udev Control Socket. worker2 # [ 6.973063] systemd[1]: Listening on udev Varlink Socket. worker2 # [ 6.976041] systemd[1]: Mounting Huge Pages File System... lb # [ 6.978320] systemd[1]: Listening on Disk Repartitioning Service Socket. lb # [ 6.979734] systemd[1]: Listening on udev Control Socket. lb # [ 6.980903] systemd[1]: Listening on udev Varlink Socket. lb # [ 6.983771] systemd[1]: Mounting Huge Pages File System... worker1 # [ 6.983211] systemd[1]: Mounting Kernel Trace File System... worker2 # [ 6.983979] systemd[1]: Mounting POSIX Message Queue File System... worker2 # [ 6.987379] systemd[1]: Mounting Kernel Debug File System... lb # [ 6.991122] systemd[1]: Mounting POSIX Message Queue File System... worker1 # [ 6.992364] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 6.994566] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 6.995877] systemd[1]: Mounting Kernel Trace File System... lb # [ 6.999409] systemd[1]: Mounting Kernel Debug File System... lb # [ 7.011156] systemd[1]: Mounting Kernel Trace File System... worker2 # [ 7.009616] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 7.011933] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 7.011129] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 7.015467] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm lb # [ 7.023499] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 7.023162] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker1 # [ 7.023502] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse lb # [ 7.026758] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 7.028366] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 7.029674] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm worker2 # [ 7.035284] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker2 # [ 7.037521] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse lb # [ 7.043633] systemd[1]: Mounting Kernel Configuration File System... lb # [ 7.045370] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm worker1 # [ 7.044817] systemd[1]: Mounting FUSE Control File System... worker1 # [ 7.045176] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 lb # [ 7.056675] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore lb # [ 7.059128] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker2 # [ 7.059945] systemd[1]: Mounting FUSE Control File System... worker1 # [ 7.065988] systemd[1]: Starting Journal Service... worker2 # [ 7.060315] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker1 # [ 7.079574] systemd[1]: Starting Load Kernel Modules... lb # [ 7.084323] systemd[1]: Mounting FUSE Control File System... client # [ 6.446562] systemd[1]: systemd-journald.service: Deactivated successfully. client # [ 7.083692] systemd-journald[262]: Received client request to flush runtime journal. lb # [ 7.087888] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker2 # [ 7.087424] systemd[1]: Starting Journal Service... worker1 # [ 7.097552] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker2 # [ 7.101467] systemd[1]: Starting Load Kernel Modules... worker1 # [ 7.105692] systemd[1]: Starting Remount Root and Kernel File Systems... lb # [ 7.107058] systemd[1]: Starting Journal Service... worker1 # [ 7.107209] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 7.111533] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... lb # [ 7.117293] systemd[1]: Starting Load Kernel Modules... worker1 # [ 7.120561] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 7.122655] systemd[1]: Starting Remount Root and Kernel File Systems... worker1 # [ 7.124626] systemd[1]: Listening on Journal Log Access Socket. worker2 # [ 7.124347] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 7.128480] systemd[1]: Mounted Huge Pages File System. lb # [ 7.132606] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker1 # [ 7.131556] systemd[1]: Mounted POSIX Message Queue File System. worker1 # [ 7.132006] systemd[1]: Mounted Kernel Debug File System. worker1 # [ 7.132377] systemd[1]: Mounted Kernel Trace File System. worker1 # [ 7.132747] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 7.133105] systemd[1]: Mounted FUSE Control File System. worker2 # [ 7.136334] systemd[1]: Starting Coldplug All udev Devices... client # [ 7.124601] systemd-modules-load[263]: Module 'atkbd' is built in client # [ 7.125754] systemd-modules-load[263]: Module 'loop' is built in client # [ 7.126724] systemd[1]: Mounted Huge Pages File System. worker1 # [ 7.142881] systemd[1]: Finished Create List of Static Device Nodes. client # [ 7.127646] systemd[1]: Mounted POSIX Message Queue File System. worker2 # [ 7.144127] systemd[1]: Listening on Journal Log Access Socket. worker2 # [ 7.146092] systemd[1]: Mounted Huge Pages File System. worker2 # [ 7.148020] systemd[1]: Mounted POSIX Message Queue File System. worker1 # [ 7.150433] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker2 # [ 7.149950] systemd[1]: Mounted Kernel Debug File System. client # [ 7.136672] systemd[1]: Mounted Kernel Debug File System. client # [ 7.137822] systemd[1]: Mounted Kernel Trace File System. client # [ 7.138734] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 7.152626] systemd[1]: Mounted Kernel Trace File System. client # [ 7.139765] systemd[1]: Finished Load Kernel Modules. worker2 # [ 7.155305] systemd[1]: Mounted Kernel Configuration File System. worker2 # [ 7.158188] systemd[1]: Mounted FUSE Control File System. lb # [ 7.159277] systemd[1]: Starting Remount Root and Kernel File Systems... lb # [ 7.159634] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 7.148392] systemd[1]: Starting Apply Kernel Variables... client # [ 7.149342] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 7.165427] systemd[1]: Starting Coldplug All udev Devices... client # [ 7.150509] systemd[1]: Mounted Kernel Configuration File System. client # [ 7.151513] systemd[1]: Finished Remount Root and Kernel File Systems. lb # [ 7.170616] systemd[1]: Listening on Journal Log Access Socket. client # [ 7.160330] systemd[1]: Mounted FUSE Control File System. lb # [ 7.177439] systemd[1]: Mounted Huge Pages File System. client # [ 7.161339] systemd[1]: Listening on Disk Image Download Service Socket. client # [ 7.162419] systemd[1]: Starting Flush Journal to Persistent Storage... client # [ 7.163357] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 7.178087] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 7.184154] systemd[1]: Mounted POSIX Message Queue File System. worker2 # [ 7.182572] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 7.187178] systemd[1]: Mounted Kernel Debug File System. client # [ 7.172276] systemd[1]: Starting Load/Save OS Random Seed... lb # [ 7.189918] systemd[1]: Mounted Kernel Trace File System. client # [ 7.173127] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 7.174619] systemd-oomd[265]: No swap; memory pressure usage will be degraded client # [ 7.175953] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. client # [ 7.184127] systemd[1]: Finished Load/Save OS Random Seed. client # [ 7.185021] systemd[1]: Reached target First Boot Complete. client # [ 7.185829] systemd[1]: Finished Apply Kernel Variables. client # [ 7.186599] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 7.187615] systemd[1]: Starting Create Static Device Nodes in /dev... client # [ 7.192266] systemd[1]: Finished Flush Journal to Persistent Storage. lb # [ 7.210088] systemd-journald[271]: Collecting audit messages is enabled. lb # [ 7.218049] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 7.218628] systemd[1]: Mounted Kernel Configuration File System. lb # [ 7.229036] systemd[1]: Finished Load Kernel Modules. worker1 # [ 7.235234] EXT4-fs (vda): re-mounted 8172407d-5c40-4235-8937-fc8748328c0f. lb # [ 7.237384] systemd[1]: Starting Firewall... worker1 # [ 7.239755] systemd-journald[270]: Collecting audit messages is enabled. lb # [ 7.226288] systemd[1]: Queued start job for default target Multi-User System. worker1 # [ 7.241791] systemd[1]: Finished Load Kernel Modules. worker2 # [ 7.240056] systemd[1]: Finished Load Kernel Modules. lb # [ 7.247305] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 7.245487] systemd[1]: Starting Firewall... lb # [ 7.235451] systemd[1]: systemd-journald.service: Deactivated successfully. lb # [ 7.240999] systemd-modules-load[272]: Module 'atkbd' is built in lb # [ 7.242011] systemd-modules-load[272]: Module 'loop' is built in worker1 # [ 7.255483] systemd[1]: Starting Firewall... worker2 # [ 7.255732] systemd[1]: Starting Apply Kernel Variables... lb # [ 7.258985] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker2 # [ 7.263884] systemd-journald[270]: Collecting audit messages is enabled. lb # [ 7.267718] systemd[1]: Started Journal Service. worker1 # [ 7.267758] systemd[1]: Starting Apply Kernel Variables... worker1 # [ 7.270783] systemd[1]: Finished Remount Root and Kernel File Systems. worker1 # [ 7.259858] systemd[1]: Queued start job for default target Multi-User System. worker1 # [ 7.275113] systemd[1]: Started Journal Service. worker2 # [ 7.287015] EXT4-fs (vda): re-mounted ec749343-9b4b-467c-8ef1-28341be0bda4. worker1 # [ 7.276200] systemd[1]: systemd-journald.service: Deactivated successfully. worker2 # [ 7.291579] systemd[1]: Finished Remount Root and Kernel File Systems. worker2 # [ 7.292151] systemd[1]: Listening on Disk Image Download Service Socket. worker2 # [ 7.292433] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore client # [ 7.282385] systemd[1]: Finished Coldplug All udev Devices. lb # [ 7.285849] systemd-oomd[273]: No swap; memory pressure usage will be degraded worker1 # [ 7.289488] systemd-modules-load[271]: Module 'atkbd' is built in worker2 # [ 7.304417] systemd[1]: Starting Load/Save OS Random Seed... client # [ 7.291156] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 7.307798] EXT4-fs (vda): re-mounted 88cbc442-c834-4d45-a9ef-88ee54739cc5. worker2 # [ 7.306192] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 7.293726] systemd[1]: Reached target Preparation for Local File Systems. client # [ 7.297312] systemd[1]: Mounting /run/wrappers... worker1 # [ 7.300966] systemd-modules-load[271]: Module 'loop' is built in client # [ 7.300542] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker2 # [ 7.317591] systemd[1]: Started Journal Service. worker1 # [ 7.313160] systemd[1]: Listening on Disk Image Download Service Socket. lb # [ 7.317866] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker1 # [ 7.320352] systemd[1]: Starting Flush Journal to Persistent Storage... lb # [ 7.322038] systemd[1]: Finished Remount Root and Kernel File Systems. worker1 # [ 7.329960] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 7.328877] systemd[1]: Queued start job for default target Multi-User System. lb # [ 7.332371] systemd[1]: Listening on Disk Image Download Service Socket. worker2 # [ 7.330054] systemd[1]: systemd-journald.service: Deactivated successfully. worker2 # [ 7.336072] systemd-modules-load[271]: Module 'atkbd' is built in lb # [ 7.343677] systemd[1]: Starting Flush Journal to Persistent Storage... client # [ 7.342457] systemd[1]: Mounted /run/wrappers. client # [ 7.343221] systemd[1]: Reached target Local File Systems. worker1 # [ 7.346686] systemd[1]: Starting Load/Save OS Random Seed... lb # [ 7.348392] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 7.346931] systemd-modules-load[271]: Module 'loop' is built in client # [ 7.348079] systemd[1]: Listening on Boot Loader Control Service Socket. client # [ 7.352219] systemd[1]: Starting register-nix-paths.service... worker2 # [ 7.354293] systemd[1]: Starting Flush Journal to Persistent Storage... lb # [ 7.357980] systemd[1]: Starting Load/Save OS Random Seed... lb # [ 7.358895] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 7.357842] systemd[1]: Starting Create SUID/SGID Wrappers... worker1 # [ 7.360507] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 7.359992] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. client # [ 7.363991] systemd[1]: Starting Save Transient machine-id to Disk... lb # [ 7.366797] systemd[1]: Mounted FUSE Control File System. worker1 # [ 7.370820] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 7.372781] systemd[1]: Starting Create System Files and Directories... worker1 # [ 7.381969] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 7.396924] systemd-oomd[272]: No swap; memory pressure usage will be degraded worker1 # [ 7.419591] systemd-journald[270]: Received client request to flush runtime journal. worker2 # [ 7.417186] systemd[1]: Finished Load/Save OS Random Seed. lb # [ 7.424859] systemd[1]: Finished Apply Kernel Variables. worker2 # [ 7.429071] systemd[1]: Reached target First Boot Complete. worker2 # [ 7.434164] systemd[1]: Finished Apply Kernel Variables. worker2 # [ 7.435280] systemd-oomd[272]: No swap; memory pressure usage will be degraded lb # [ 7.452825] systemd-journald[271]: Received client request to flush runtime journal. worker2 # [ 7.456404] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 7.457520] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 7.458440] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker2 # [ 7.475784] systemd-journald[270]: Received client request to flush runtime journal. worker1 # [ 7.476593] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. client # [ 7.475587] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker1 # [ 7.477790] systemd[1]: Finished Load/Save OS Random Seed. worker1 # [ 7.478710] systemd[1]: Reached target First Boot Complete. worker1 # [ 7.479634] systemd[1]: Finished Apply Kernel Variables. client # [ 7.485002] systemd[1]: Finished Save Transient machine-id to Disk. lb # [ 7.492651] systemd[1]: Finished Load/Save OS Random Seed. lb # [ 7.493582] systemd[1]: Reached target First Boot Complete. lb # [ 7.501320] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 7.504056] systemd[1]: Starting Create Static Device Nodes in /dev... lb # [ 7.505041] systemd[1]: Finished Flush Journal to Persistent Storage. worker1 # [ 7.505984] systemd[1]: Finished Flush Journal to Persistent Storage. client # [ 7.510457] systemd-udevd[292]: Using default interface naming scheme 'v261'. worker2 # [ 7.550159] systemd[1]: Finished Flush Journal to Persistent Storage. client # [ 7.550712] systemd[1]: Finished Create System Files and Directories. worker1 # [ 7.556983] systemd[1]: Finished Create Static Device Nodes in /dev. worker1 # [ 7.560323] systemd[1]: Reached target Preparation for Local File Systems. client # [ 7.559365] systemd[1]: Starting Rebuild Journal Catalog... worker1 # [ 7.564346] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 7.563610] systemd[1]: Starting Record System Boot/Shutdown in UTMP... worker2 # [ 7.590331] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 7.592615] systemd[1]: Reached target Preparation for Local File Systems. worker2 # [ 7.596772] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 7.634332] systemd[1]: Finished Record System Boot/Shutdown in UTMP. lb # [ 7.636789] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 7.638041] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 7.648349] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 7.664439] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 7.673613] systemd-udevd[309]: Using default interface naming scheme 'v261'. client # [ 7.693897] systemd[1]: Finished Rebuild Journal Catalog. client # [ 7.696449] systemd[1]: Starting Update is Completed... worker2 # [ 7.709479] systemd-udevd[312]: Using default interface naming scheme 'v261'. client # [ 7.756312] systemd[1]: Finished Update is Completed. worker1 # [ 7.790953] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 7.791223] systemd-udevd[310]: Using default interface naming scheme 'v261'. worker2 # [ 7.873173] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 7.876740] systemd[1]: Mounting /run/wrappers... worker2 # [ 7.897886] systemd[1]: Mounting /run/wrappers... lb # [ 7.904647] systemd[1]: Mounting /run/wrappers... worker1 # [ 7.927191] systemd[1]: Mounted /run/wrappers. worker1 # [ 7.928618] systemd[1]: Reached target Local File Systems. worker1 # [ 7.934131] systemd[1]: Listening on Boot Loader Control Service Socket. worker1 # [ 7.937648] systemd[1]: Starting register-nix-paths.service... lb # [ 7.942875] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 7.948277] systemd[1]: Starting Create SUID/SGID Wrappers... worker2 # [ 7.950717] systemd[1]: Mounted /run/wrappers. worker1 # [ 7.955409] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker2 # [ 7.957900] systemd[1]: Reached target Local File Systems. worker2 # [ 7.958745] systemd[1]: Listening on Boot Loader Control Service Socket. worker1 # [ 7.964158] systemd[1]: Starting Save Transient machine-id to Disk... worker2 # [ 7.962130] systemd[1]: Starting register-nix-paths.service... lb # [ 7.965747] systemd[1]: Mounted /run/wrappers. lb # [ 7.966502] systemd[1]: Reached target Local File Systems. worker2 # [ 7.966321] systemd[1]: Starting Create SUID/SGID Wrappers... worker1 # [ 7.968706] systemd[1]: Starting Create System Files and Directories... lb # [ 7.976086] systemd[1]: Listening on Boot Loader Control Service Socket. lb # [ 7.980122] systemd[1]: Starting register-nix-paths.service... worker2 # [ 7.981603] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. lb # [ 7.985401] systemd[1]: Starting Create SUID/SGID Wrappers... lb # [ 7.986297] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. lb # [ 7.992647] systemd[1]: Starting Save Transient machine-id to Disk... worker2 # [ 7.991331] systemd[1]: Starting Save Transient machine-id to Disk... worker2 # [ 7.994320] systemd[1]: Starting Create System Files and Directories... lb # [ 8.012671] systemd[1]: Starting Create System Files and Directories... worker1 # [ 8.049846] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker1 # [ 8.057729] systemd[1]: Finished Save Transient machine-id to Disk. worker2 # [ 8.078678] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker2 # [ 8.089028] systemd[1]: Finished Save Transient machine-id to Disk. lb # [ 8.119741] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. lb # [ 8.130383] systemd[1]: Finished Save Transient machine-id to Disk. worker1 # [ 8.150445] systemd[1]: Finished Create System Files and Directories. worker1 # [ 8.161909] systemd[1]: Starting Rebuild Journal Catalog... worker1 # [ 8.162909] systemd[1]: Starting Record System Boot/Shutdown in UTMP... client # [ 8.193916] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker2 # [ 8.194286] systemd[1]: Finished Create System Files and Directories. client # [ 8.197846] systemd[1]: Finished Create SUID/SGID Wrappers. worker2 # [ 8.204733] systemd[1]: Starting Rebuild Journal Catalog... lb # [ 8.211727] systemd[1]: Finished Create System Files and Directories. worker2 # [ 8.213494] systemd[1]: Starting Record System Boot/Shutdown in UTMP... lb # [ 8.222272] systemd[1]: Starting Rebuild Journal Catalog... lb # [ 8.228414] systemd[1]: Starting Record System Boot/Shutdown in UTMP... worker1 # [ 8.283097] systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [ 8.338477] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 worker2 # [ 8.328380] systemd[1]: Finished Record System Boot/Shutdown in UTMP. lb # [ 8.337148] systemd[1]: Finished Record System Boot/Shutdown in UTMP. worker1 # [ 8.345087] systemd[1]: Finished Rebuild Journal Catalog. worker1 # [ 8.348869] systemd[1]: Starting Update is Completed... lb # [ 8.372792] systemd[1]: Finished Rebuild Journal Catalog. lb # [ 8.383355] systemd[1]: Starting Update is Completed... worker2 # [ 8.394725] systemd[1]: Finished Rebuild Journal Catalog. worker2 # [ 8.398556] systemd[1]: Starting Update is Completed... worker1 # [ 8.455653] systemd[1]: Finished Update is Completed. lb # [ 8.459699] systemd[1]: Finished Update is Completed. worker2 # [ 8.482160] systemd[1]: Finished Update is Completed. worker1 # [ 8.750245] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker1 # [ 8.756656] systemd[1]: Finished Create SUID/SGID Wrappers. worker2 # [ 8.776315] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker2 # [ 8.780712] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 8.813322] systemd[1]: Starting Virtual Console Setup... client # [ 8.827606] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 8.835471] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 8.839634] systemd[1]: Stopped Virtual Console Setup. client # [ 8.842808] systemd[1]: Starting Virtual Console Setup... worker2 # [ 8.846805] systemd[1]: Finished Firewall. client # [ 8.863299] systemd[1]: Finished register-nix-paths.service. worker1 # [ 8.934780] systemd[1]: Finished Firewall. lb # [ 8.986137] systemd[1]: Finished Coldplug All udev Devices. client # [ 8.988961] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. client # [ 9.021694] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. lb # [ 9.044203] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 9.074614] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse client # [ 9.082149] (udev-worker)[357]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 9.087063] (udev-worker)[357]: Network interface NamePolicy= disabled on kernel command line. client # [ 9.090184] (udev-worker)[354]: Network interface NamePolicy= disabled on kernel command line. lb # [ 9.100525] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. lb # [ 9.103098] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 9.227101] systemd[1]: Condition check resulted in Virtio network device being skipped. lb # [ 9.233459] systemd[1]: Finished register-nix-paths.service. client # [ 9.231358] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore lb # [ 9.237610] systemd[1]: Reached target System Initialization. lb # [ 9.238509] systemd[1]: Started Discard unused filesystem blocks once a week. lb # [ 9.239521] systemd[1]: Started niks3 garbage collection timer. client # [ 9.240246] systemd[1]: Starting Create SUID/SGID Wrappers... client # [ 9.241240] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. lb # [ 9.244163] systemd[1]: Started Daily Cleanup of Temporary Directories. client # [ 9.242498] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 lb # [ 9.246808] systemd[1]: Reached target Timer Units. lb # [ 9.247587] systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 9.249894] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore lb # [ 9.253948] systemd[1]: Listening on niks3 server socket. client # [ 9.251535] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 9.254801] systemd[1]: Listening on Nix Daemon Socket. lb # [ 9.255564] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. worker1 # [ 9.261301] systemd[1]: Finished Coldplug All udev Devices. client # [ 9.261716] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 9.264330] systemd[1]: Reached target Socket Units. lb # [ 9.265102] systemd[1]: Reached target Basic System. lb # [ 9.265837] systemd[1]: Starting Import lastlog data into lastlog2 database... lb # [ 9.266906] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 9.280381] systemd[1]: Starting Post-Boot Actions... lb # [ 9.281215] systemd[1]: Started Reset console on configuration changes. lb # [ 9.288721] systemd[1]: Starting resolvconf update... worker1 # [ 9.315001] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 9.331260] systemd[1]: Started rustfs.service. worker1 # [ 9.347198] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker2 # [ 9.357327] systemd[1]: Finished Coldplug All udev Devices. lb # [ 9.365561] systemd[1]: Starting rustfs-bucket.service... client # [ 9.382271] systemd-vconsole-setup[383]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 9.388930] systemd[1]: Finished Virtual Console Setup. worker2 # [ 9.413090] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 9.438189] systemd[1]: Finished register-nix-paths.service. worker1 # [ 9.449380] systemd[1]: Reached target System Initialization. worker2 # [ 9.447931] systemd[1]: Finished register-nix-paths.service. lb # [ 9.456244] systemd[1]: Finished Post-Boot Actions. worker2 # [ 9.454157] systemd[1]: Reached target System Initialization. worker1 # [ 9.463192] systemd[1]: Started Discard unused filesystem blocks once a week. lb # [ 9.463177] nsncd[459]: Sep 25 15:23:56.084 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 9.467692] systemd[1]: Started Discard unused filesystem blocks once a week. lb # [ 9.480395] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 9.495082] mousedev: PS/2 mouse device common for all mice worker2 # [ 9.481977] systemd[1]: Started Daily Cleanup of Temporary Directories. worker1 # [ 9.485629] systemd[1]: Started Daily Cleanup of Temporary Directories. worker2 # [ 9.490832] systemd[1]: Reached target Timer Units. worker2 # [ 9.492867] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 9.495748] systemd[1]: Listening on Nix Daemon Socket. lb # [ 9.500652] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker2 # [ 9.502881] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. worker1 # [ 9.506429] systemd[1]: Reached target Timer Units. lb # [ 9.510064] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker2 # [ 9.514040] systemd[1]: Reached target Socket Units. lb # [ 9.523419] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 9.523592] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 9.526980] systemd[1]: Reached target Basic System. lb # [ 9.531073] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 9.533267] systemd[1]: Listening on Nix Daemon Socket. worker2 # [ 9.535374] systemd[1]: Starting Import lastlog data into lastlog2 database... lb # [ 9.540185] systemd[1]: Started backdoor.service. worker1 # [ 9.546315] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. worker2 # [ 9.543999] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker2 # [ 9.554814] systemd[1]: Starting Post-Boot Actions... lb # [ 9.561459] systemd[1]: Starting D-Bus System Message Bus... worker2 # [ 9.561847] systemd[1]: Started Reset console on configuration changes. worker1 # [ 9.572602] systemd[1]: Reached target Socket Units. worker2 # [ 9.572569] systemd[1]: Starting resolvconf update... worker2 # [ 9.579151] systemd[1]: Starting D-Bus System Message Bus... worker2 # [ 9.587328] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker1 # [ 9.592830] systemd[1]: Reached target Basic System. worker2 # [ 9.599291] systemd[1]: Finished Post-Boot Actions. lb # [ 9.605589] systemd[1]: Starting User Login Management... worker1 # [ 9.609078] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 9.621919] systemd[1]: Started Name Service Cache Daemon (nsncd). worker1 # [ 9.630603] systemd[1]: Starting Import lastlog data into lastlog2 database... worker2 # [ 9.629551] nsncd[487]: Sep 25 15:23:56.244 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" lb # [ 9.634406] systemd[1]: Finished Import lastlog data into lastlog2 database. worker2 # [ 9.643830] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 9.650400] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker2 # [ 9.649502] systemd[1]: Reached target User and Group Name Lookups. worker2 # [ 9.656119] systemd[1]: Starting User Login Management... worker1 # [ 9.662733] systemd[1]: Starting Post-Boot Actions... worker2 # [ 9.669571] systemd[1]: Finished Import lastlog data into lastlog2 database. worker1 # [ 9.680582] nix-grpc-daemon[486]: ts=2026-09-25T15:23:56Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 9.682250] dbus-broker-launch[491]: Looking up NSS user entry for 'systemd-timesync'... worker2 # [ 9.692750] dbus-broker-launch[491]: NSS returned no entry for 'systemd-timesync' worker2 # [ 9.699928] dbus-broker-launch[491]: Invalid user-name in /nix/store/flsxa7d757b7zpdiv9wsbs42iawcisak-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" client # [ 9.714290] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker2 # [ 9.714592] systemd[1]: Started D-Bus System Message Bus. client # [ 9.718172] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 9.721079] systemd[1]: Reached target System Initialization. client # [ 9.723847] systemd[1]: Started Discard unused filesystem blocks once a week. client # [ 9.728178] systemd[1]: Started Daily Cleanup of Temporary Directories. client # [ 9.729260] systemd[1]: Reached target Timer Units. worker1 # [ 9.730209] nix-grpc-daemon[486]: ts=2026-09-25T15:23:56Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc client # [ 9.730010] systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 9.730935] systemd[1]: Listening on Nix Daemon Socket. client # [ 9.731714] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. client # [ 9.744608] systemd[1]: Reached target Socket Units. client # [ 9.745461] systemd[1]: Reached target Basic System. client # [ 9.746184] systemd[1]: Started backdoor.service. client # [ 9.746886] systemd[1]: Starting Import lastlog data into lastlog2 database... client # [ 9.752162] systemd[1]: Starting nftables firewall... worker2 # [ 9.755738] dbus-broker-launch[491]: Ready worker1 # [ 9.757190] systemd[1]: Started Reset console on configuration changes. lb # connecting to host... client # [ 9.776580] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 9.778783] nix-grpc-daemon[486]: ts=2026-09-25T15:23:56Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 client # [ 9.784114] systemd[1]: Starting Post-Boot Actions... worker1 # [ 9.809085] nix-grpc-daemon[486]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 9.808995] systemd-logind[499]: New seat seat0. worker2 # [ 9.815141] systemd[1]: Started User Login Management. worker1 # [ 9.816415] nix-grpc-daemon[486]: E0000 00:00:1790349836.341231 486 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. client # [ 9.822384] systemd[1]: Started Reset console on configuration changes. worker2 # [ 9.822497] systemd[1]: Starting linger-users.service... worker1 # [ 9.833189] nix-grpc-daemon[486]: E0000 00:00:1790349836.347194 486 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 9.833831] systemd[1]: Stopped target Host and Network Name Lookups. worker2 # [ 9.843271] systemd[1]: Stopping Host and Network Name Lookups... worker1 # [ 9.845174] nix-grpc-daemon[486]: ts=2026-09-25T15:23:56Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 9.848361] systemd[1]: Stopped target User and Group Name Lookups. worker1 # [ 9.853961] nix-grpc-daemon[486]: ts=2026-09-25T15:23:56Z level=info event=scheduler_lock_lost error="error: http://lb:5751/api/farm/lead: Could not connect to server" worker2 # [ 9.854053] systemd[1]: Stopping User and Group Name Lookups... worker2 # [ 9.856753] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker2 # [ 9.862990] systemd[1]: nscd.service: Deactivated successfully. client # [ 9.865111] systemd[1]: Starting resolvconf update... worker1 # [ 9.866797] nix-grpc-daemon[486]: E0000 00:00:1790349836.361310 540 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 9.866431] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker2 # [ 9.872735] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 9.880084] nix-grpc-daemon[486]: E0000 00:00:1790349836.390096 540 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. client # connecting to host... lb # [ 9.891956] systemd[1]: Stopped target Host and Network Name Lookups. lb # [ 9.893057] systemd[1]: Stopping Host and Network Name Lookups... worker1 # [ 9.891461] nix-grpc-daemon[486]: E0000 00:00:1790349836.390869 540 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 9.893914] systemd[1]: Stopped target User and Group Name Lookups. lb # [ 9.894818] systemd[1]: Stopping User and Group Name Lookups... worker1 # [ 9.907411] systemd[1]: Starting resolvconf update... worker1 # [ 9.912535] nsncd[488]: Sep 25 15:23:56.418 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" lb # [ 9.915469] dbus-broker-launch[498]: Looking up NSS user entry for 'systemd-timesync'... worker2 # [ 9.915094] systemd[1]: linger-users.service: Deactivated successfully. worker2 # [ 9.920931] systemd[1]: Finished linger-users.service. client # [ 9.923212] nsncd[454]: Sep 25 15:23:56.549 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 9.925533] nix-grpc-daemon[486]: E0000 00:00:1790349836.423128 540 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb # [ 9.926956] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... lb # [ 9.929318] systemd[1]: nscd.service: Deactivated successfully. lb # [ 9.930247] dbus-broker-launch[498]: NSS returned no entry for 'systemd-timesync' lb # [ 9.931290] dbus-broker-launch[498]: Invalid user-name in /nix/store/4yiprls1sll47jid3xzq0m9nmi1qz2ww-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker2 # [ 9.930604] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 9.933020] nsncd[554]: Sep 25 15:23:56.554 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 9.937197] systemd[1]: Started Name Service Cache Daemon (nsncd). client: Guest shell says: b'Spawning backdoor root shell...\n' worker2 # [ 9.941864] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 9.944248] nix-grpc-daemon[486]: ts=2026-09-25T15:23:56Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: FAILED_PRECONDITION: ipv4:192.168.1.2:50051: connect failed: addr: ipv4:192.168.1.2:50051 error: Network is unreachable" worker2 # [ 9.946002] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 9.960723] dbus-broker-launch[492]: Looking up NSS user entry for 'systemd-timesync'... client # [ 9.944159] systemd[1]: Finished Post-Boot Actions. worker2 # [ 9.969897] systemd[1]: Finished resolvconf update. worker1 # [ 9.968234] dbus-broker-launch[492]: NSS returned no entry for 'systemd-timesync' lb # [ 9.952668] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker1 # [ 9.971344] dbus-broker-launch[492]: Invalid user-name in /nix/store/flsxa7d757b7zpdiv9wsbs42iawcisak-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" client # [ 9.976143] systemd[1]: Reached target Host and Network Name Lookups. client: connected to guest root shell worker1 # [ 9.984152] systemd[1]: Starting D-Bus System Message Bus... client # [ 9.982677] systemd[1]: Reached target User and Group Name Lookups. client: (connecting took 10.41 seconds) worker1 # [ 9.988291] systemd[1]: Finished Post-Boot Actions. client # [ 9.996687] systemd[1]: Starting D-Bus System Message Bus... client: (finished: waiting for the VM to finish booting, in 10.41 seconds) worker1 # [ 9.992383] systemd[1]: Started Nix worker-protocol over gRPC. client # [ 10.009389] systemd[1]: Starting User Login Management... worker2 # [ 9.974034] systemd[1]: Reached target Preparation for Network. lb # [ 9.953789] systemd-logind[503]: New seat seat0. worker1 # [ 9.993824] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 10.013968] systemd[1]: Finished Import lastlog data into lastlog2 database. worker2 # [ 9.981665] systemd[1]: Starting DHCP Client... lb # [ 9.954471] systemd[1]: Started User Login Management. worker2 # [ 9.985358] systemd[1]: Starting Extra networking commands.... lb # [ 9.966386] systemd[1]: Starting linger-users.service... worker2 # [ 9.988597] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. lb # [ 9.969736] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 9.998671] systemd[1]: Reached target Host and Network Name Lookups. lb # [ 9.979055] (udev-worker)[333]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 10.003026] systemd[1]: Reached target User and Group Name Lookups. lb # [ 9.987983] (udev-worker)[337]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 10.008337] nix-grpc-daemon[486]: ts=2026-09-25T15:23:56Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: FAILED_PRECONDITION: ipv4:192.168.1.2:50051: connect failed: addr: ipv4:192.168.1.2:50051 error: Network is unreachable" lb # [ 9.997506] (udev-worker)[333]: Network interface NamePolicy= disabled on kernel command line. lb # [ 9.998608] systemd[1]: Started D-Bus System Message Bus. worker1 # [ 10.022460] systemd[1]: Starting User Login Management... worker1 # [ 10.025534] systemd[1]: Finished Import lastlog data into lastlog2 database. worker1 # [ 10.029271] systemd[1]: Started D-Bus System Message Bus. worker1 # [ 10.037643] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker1 # [ 10.041513] dbus-broker-launch[492]: Ready worker2 # [ 10.039189] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker2 # [ 10.045944] systemd[1]: Started backdoor.service. worker1 # [ 10.050769] systemd-logind[559]: New seat seat0. worker1 # [ 10.062335] systemd[1]: Started User Login Management. worker1 # [ 10.068425] systemd[1]: Starting linger-users.service... lb # [ 10.069674] dbus-broker-launch[498]: Ready worker1 # [ 10.069262] systemd[1]: Stopped target Host and Network Name Lookups. worker1 # [ 10.070177] systemd[1]: Stopping Host and Network Name Lookups... worker1 # [ 10.071048] systemd[1]: Stopped target User and Group Name Lookups. worker1 # [ 10.071942] systemd[1]: Stopping User and Group Name Lookups... worker1 # [ 10.090386] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker1 # [ 10.096732] systemd[1]: nscd.service: Deactivated successfully. worker1 # [ 10.104084] systemd[1]: Stopped Name Service Cache Daemon (nsncd). lb # [ 10.106558] systemd[1]: linger-users.service: Deactivated successfully. worker1 # [ 10.107023] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 10.111684] systemd[1]: Finished linger-users.service. worker1 # [ 10.115947] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker1 # [ 10.124865] systemd[1]: Started backdoor.service. lb # [ 10.142693] systemd[1]: Finished resolvconf update. worker1 # [ 10.145812] systemd[1]: linger-users.service: Deactivated successfully. worker2 # connecting to host... worker1 # [ 10.153544] systemd[1]: Finished linger-users.service. lb # [ 10.152752] nsncd[570]: Sep 25 15:23:56.774 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" lb # [ 10.165577] systemd[1]: Starting DHCP Client... lb # [ 10.172760] systemd[1]: Started Name Service Cache Daemon (nsncd). lb # [ 10.173670] systemd[1]: Reached target Host and Network Name Lookups. lb # [ 10.174550] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 10.174790] systemd[1]: Started Name Service Cache Daemon (nsncd). worker1 # [ 10.181179] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 10.185667] nsncd[607]: Sep 25 15:23:56.803 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 10.195358] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 10.211996] systemd[1]: Finished resolvconf update. worker1 # [ 10.217399] systemd[1]: Reached target Preparation for Network. worker1 # [ 10.228223] systemd[1]: Starting DHCP Client... worker1 # [ 10.230439] systemd[1]: Starting Extra networking commands.... worker1 # connecting to host... client # [ 10.243294] systemd-logind[484]: New seat seat0. worker2 # [ 10.243308] dhcpcd[590]: dhcpcd-10.3.2 starting client # [ 10.251610] dbus-broker-launch[483]: Looking up NSS user entry for 'systemd-timesync'... worker2 # [ 10.254072] dhcpcd[636]: dev: loaded udev client # [ 10.262547] systemd-logind[484]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) client # [ 10.263889] systemd-logind[484]: Watching system buttons on /dev/input/event0 (gpio-keys) worker2 # [ 10.293627] 8021q: 802.1Q VLAN Support v1.8 client # [ 10.281041] systemd[1]: Started User Login Management. client # [ 10.281831] dbus-broker-launch[483]: NSS returned no entry for 'systemd-timesync' client # [ 10.282812] dbus-broker-launch[483]: Invalid user-name in /nix/store/6wvgzhp3xj1v1xybf1ixpbqi5gif5fbv-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker2 # [ 10.297743] (udev-worker)[353]: Network interface NamePolicy= disabled on kernel command line. client # [ 10.303769] systemd[1]: Starting linger-users.service... lb # [ 10.306292] systemd[1]: Finished Firewall. worker2 # [ 10.306154] systemd[1]: Finished Extra networking commands.. worker2 # [ 10.307016] systemd[1]: Reached target Network. lb # [ 10.310576] systemd[1]: Reached target Preparation for Network. lb # [ 10.315409] systemd[1]: Starting Extra networking commands.... worker2 # [ 10.319238] systemd[1]: Starting Permit User Sessions... client # [ 10.327715] systemd[1]: Started D-Bus System Message Bus. worker2 # [ 10.329443] (udev-worker)[343]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker2 # [ 10.331723] (udev-worker)[343]: Network interface NamePolicy= disabled on kernel command line. client # [ 10.344956] systemd[1]: Stopped target Host and Network Name Lookups. client # [ 10.345953] systemd[1]: Stopping Host and Network Name Lookups... client # [ 10.346793] systemd[1]: Stopped target User and Group Name Lookups. client # [ 10.347642] systemd[1]: Stopping User and Group Name Lookups... client # [ 10.363401] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... client # [ 10.368283] systemd[1]: nscd.service: Deactivated successfully. client # [ 10.369229] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker2 # [ 10.390361] cfg80211: Loading compiled-in X.509 certificates for regulatory database client # [ 10.382003] dbus-broker-launch[483]: Ready client # [ 10.388077] systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 10.394255] systemd[1]: linger-users.service: Deactivated successfully. client # [ 10.399695] systemd[1]: Finished linger-users.service. worker1 # [ 10.401419] nix-grpc-daemon[486]: ts=2026-09-25T15:23:57Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: FAILED_PRECONDITION: ipv4:192.168.1.2:50051: connect failed: addr: ipv4:192.168.1.2:50051 error: Network is unreachable" worker1 # [ 10.413977] (udev-worker)[471]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 10.418006] systemd[1]: Finished Permit User Sessions. worker2 # [ 10.437760] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker2 # [ 10.438233] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' worker2 # [ 10.440811] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 worker2 # [ 10.441125] cfg80211: failed to load regulatory.db worker2 # [ 10.429255] systemd[1]: Started Getty on tty1. worker2 # [ 10.430011] systemd[1]: Reached target Login Prompts. lb # [ 10.440378] dhcpcd[624]: dhcpcd-10.3.2 starting worker1 # [ 10.453130] dhcpcd[645]: dhcpcd-10.3.2 starting lb # [ 10.453856] dhcpcd[635]: dev: loaded udev worker1 # [ 10.463436] dhcpcd[687]: dev: loaded udev client # [ 10.474909] systemd[1]: Finished resolvconf update. worker1 # [ 10.477998] (udev-worker)[336]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 10.509721] 8021q: 802.1Q VLAN Support v1.8 client # [ 10.491971] nsncd[549]: Sep 25 15:23:57.115 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 10.497888] (udev-worker)[336]: Network interface NamePolicy= disabled on kernel command line. client # [ 10.502914] systemd[1]: Starting DHCP Client... worker2 # [ 10.516585] 8021q: adding VLAN 0 to HW filter on device eth0 client # [ 10.503635] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 10.505493] dhcpcd[636]: eth0: waiting for carrier lb # [ 10.522420] 8021q: 802.1Q VLAN Support v1.8 worker2 # [ 10.507901] dhcpcd[636]: libudev: received NULL device worker2 # [ 10.510579] dhcpcd[636]: libudev: received NULL device client # [ 10.511094] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 10.511518] dhcpcd[636]: eth0: carrier acquired lb # [ 10.518382] systemd-logind[503]: Watching system buttons on /dev/input/event0 (gpio-keys) worker2 # [ 10.519893] dhcpcd[636]: DUID 00:01:00:01:32:49:4e:8d:52:54:00:12:34:56 client # [ 10.520871] systemd[1]: Reached target User and Group Name Lookups. worker2 # [ 10.522610] dhcpcd[636]: eth0: IAID 00:12:34:56 worker2 # [ 10.523295] dhcpcd[636]: eth0: adding address fe80::5054:ff:fe12:3456 lb # [ 10.543881] systemd[1]: Condition check resulted in Virtio network device being skipped. worker1 # [ 10.556181] systemd[1]: Finished Extra networking commands.. worker1 # [ 10.557055] systemd[1]: Reached target Network. lb # [ 10.557367] systemd[1]: Starting Address configuration of eth1... worker1 # [ 10.557745] systemd[1]: Starting Permit User Sessions... worker2 # [ 10.559172] dhcpcd[636]: eth0: IAID 00:12:34:56 worker2 # [ 10.576752] systemd[1]: Condition check resulted in Virtio network device being skipped. worker2 # [ 10.584851] systemd[1]: Starting Address configuration of eth1... worker1 # [ 10.611351] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker1 # [ 10.638574] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker1 # [ 10.639851] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' worker1 # [ 10.641718] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 worker1 # [ 10.642022] cfg80211: failed to load regulatory.db worker1 # [ 10.640360] systemd[1]: Finished Permit User Sessions. worker2 # [ 10.656193] 8021q: adding VLAN 0 to HW filter on device eth1 worker1 # [ 10.645678] systemd[1]: Started Getty on tty1. worker1 # [ 10.646400] systemd[1]: Reached target Login Prompts. worker2 # [ 10.654110] network-addresses-eth1-start[664]: adding address 192.168.1.4/24... done worker2 # [ 10.665155] network-addresses-eth1-start[664]: adding address 2001:db8:1::4/64... done client # [ 10.681563] dhcpcd[582]: dhcpcd-10.3.2 starting worker2 # [ 10.681644] systemd[1]: Finished Address configuration of eth1. worker2 # [ 10.683047] systemd-logind[499]: Watching system buttons on /dev/input/event0 (gpio-keys) client # [ 10.689028] dhcpcd[601]: dev: loaded udev worker1 # [ 10.714747] 8021q: adding VLAN 0 to HW filter on device eth0 worker1 # [ 10.702283] dhcpcd[687]: eth0: waiting for carrier worker1 # [ 10.703053] dhcpcd[687]: eth0: carrier acquired worker1 # [ 10.714249] dhcpcd[687]: DUID 00:01:00:01:32:49:4e:8d:52:54:00:12:34:56 worker1 # [ 10.715285] dhcpcd[687]: eth0: IAID 00:12:34:56 client # [ 10.732927] 8021q: 802.1Q VLAN Support v1.8 worker1 # [ 10.722660] dhcpcd[687]: eth0: adding address fe80::5054:ff:fe12:3456 worker2 # [ 10.735735] mousedev: PS/2 mouse device common for all mice worker1 # [ 10.723580] systemd[1]: Condition check resulted in Virtio network device being skipped. lb # [ 10.746056] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker1 # [ 10.733845] dhcpcd[687]: eth0: IAID 00:12:34:56 worker1 # [ 10.734910] nix-grpc-daemon[486]: ts=2026-09-25T15:23:57Z level=info event=scheduler_lock_lost error="error: http://lb:5751/api/farm/lead: Could not connect to server" worker1 # [ 10.740359] systemd[1]: Starting Address configuration of eth1... lb # [ 10.785821] 8021q: adding VLAN 0 to HW filter on device eth1 worker1 # [ 10.805843] 8021q: adding VLAN 0 to HW filter on device eth1 lb # [ 10.807260] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' lb # [ 10.807716] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' lb # [ 10.811319] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 lb # [ 10.811625] cfg80211: failed to load regulatory.db lb # [ 10.804363] network-addresses-eth1-start[651]: adding address 192.168.1.2/24... done worker1 # [ 10.804568] network-addresses-eth1-start[715]: adding address 192.168.1.3/24... done worker1 # [ 10.815367] network-addresses-eth1-start[715]: adding address 2001:db8:1::3/64... done lb # [ 10.816542] systemd[1]: Finished Extra networking commands.. lb # [ 10.821209] systemd[1]: Reached target Network. worker1 # [ 10.832909] systemd[1]: Finished Address configuration of eth1. client # [ 10.846745] cfg80211: Loading compiled-in X.509 certificates for regulatory database lb # [ 10.835346] systemd[1]: Starting PostgreSQL Server... worker2 # [ 10.832721] systemd-logind[499]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) lb # [ 10.851139] mousedev: PS/2 mouse device common for all mice lb # [ 10.841118] network-addresses-eth1-start[651]: adding address 2001:db8:1::2/64... done lb # [ 10.849997] systemd[1]: Starting Permit User Sessions... client # [ 10.881478] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' client # [ 10.881942] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' client # [ 10.885725] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 client # [ 10.886034] cfg80211: failed to load regulatory.db client # [ 10.878349] systemd[1]: Finished nftables firewall. worker1 # [ 10.882038] systemd-logind[559]: Watching system buttons on /dev/input/event0 (gpio-keys) client # [ 10.881373] systemd[1]: Reached target Preparation for Network. worker1 # [ 10.891157] dhcpcd[687]: eth0: soliciting a DHCP lease lb # [ 10.892094] systemd[1]: Finished Address configuration of eth1. client # [ 10.890840] systemd[1]: Starting Address configuration of eth1... worker1 # [ 10.896386] dhcpcd[687]: eth0: offered 10.0.2.15 from 10.0.2.2 client # [ 10.898755] systemd[1]: Starting Extra networking commands.... worker1 # [ 10.904230] dhcpcd[687]: eth0: probing address 10.0.2.15/24 lb # [ 10.959561] 8021q: adding VLAN 0 to HW filter on device eth0 lb # [ 10.947458] dhcpcd[635]: eth0: waiting for carrier lb # [ 10.958517] dhcpcd[635]: eth0: waiting for carrier lb # [ 10.961257] dhcpcd[635]: eth0: carrier acquired worker1 # [ 10.976817] mousedev: PS/2 mouse device common for all mice lb # [ 10.965268] systemd[1]: Finished Permit User Sessions. lb # [ 10.975654] systemd[1]: Started Getty on tty1. lb # [ 10.981378] systemd[1]: Reached target Login Prompts. lb # [ 10.986253] dhcpcd[635]: DUID 00:01:00:01:32:49:4e:8d:52:54:00:12:34:56 lb # [ 10.987341] dhcpcd[635]: eth0: IAID 00:12:34:56 lb # [ 10.987982] dhcpcd[635]: eth0: adding address fe80::5054:ff:fe12:3456 client # [ 11.000481] 8021q: adding VLAN 0 to HW filter on device eth0 client # [ 10.990260] dhcpcd[601]: eth0: waiting for carrier client # [ 10.993179] dhcpcd[601]: libudev: received NULL device client # [ 10.993968] dhcpcd[601]: libudev: received NULL device client # [ 10.994859] dhcpcd[601]: eth0: carrier acquired client # [ 11.010949] dhcpcd[601]: DUID 00:01:00:01:32:49:4e:8d:52:54:00:12:34:56 client # [ 11.011939] dhcpcd[601]: eth0: IAID 00:12:34:56 client # [ 11.017352] dhcpcd[601]: eth0: adding address fe80::5054:ff:fe12:3456 client # [ 11.035853] 8021q: adding VLAN 0 to HW filter on device eth1 client # [ 11.043980] network-addresses-eth1-start[615]: adding address 192.168.1.1/24... done client # [ 11.067951] network-addresses-eth1-start[615]: adding address 2001:db8:1::1/64... done worker1 # [ 11.075019] systemd-logind[559]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) client # [ 11.094303] systemd[1]: Finished Address configuration of eth1. client # [ 11.173892] systemd[1]: Finished Extra networking commands.. client # [ 11.177284] systemd[1]: Reached target Network. client # [ 11.181448] systemd[1]: Starting Permit User Sessions... worker1 # [ 11.200846] nix-grpc-daemon[486]: ts=2026-09-25T15:23:57Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused" lb # [ 11.226591] postgresql-pre-start[715]: The files belonging to this database system will be owned by user "postgres". lb # [ 11.230314] postgresql-pre-start[715]: This user must also own the server process. client # [ 11.230151] systemd[1]: Finished Permit User Sessions. client # [ 11.236408] systemd[1]: Started Getty on tty1. client # [ 11.237171] systemd[1]: Reached target Login Prompts. lb # [ 11.242591] postgresql-pre-start[715]: The database cluster will be initialized with locale "en_US.UTF-8". lb # [ 11.243970] postgresql-pre-start[715]: The default database encoding has accordingly been set to "UTF8". lb # [ 11.248152] postgresql-pre-start[715]: The default text search configuration will be set to "english". lb # [ 11.249397] postgresql-pre-start[715]: Data page checksums are enabled. lb # [ 11.250261] postgresql-pre-start[715]: fixing permissions on existing directory /var/lib/postgresql/18 ... ok lb # [ 11.251596] postgresql-pre-start[715]: creating subdirectories ... ok lb # [ 11.260164] postgresql-pre-start[715]: selecting dynamic shared memory implementation ... posix lb # [ 11.261617] systemd-logind[503]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) client # [ 11.298364] dhcpcd[601]: eth0: soliciting a DHCP lease client # [ 11.304542] dhcpcd[601]: eth0: offered 10.0.2.15 from 10.0.2.2 client # [ 11.312226] dhcpcd[601]: eth0: probing address 10.0.2.15/24 lb # [ 11.380240] postgresql-pre-start[715]: selecting default "max_connections" ... 100 client: (finished: waiting for unit network-addresses-eth1.service, in 11.79 seconds) client: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; } client: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; }, in 0.04 seconds) lb: waiting for unit network-addresses-eth1.service lb: waiting for the VM to finish booting lb: Guest shell says: b'Spawning backdoor root shell...\n' lb: connected to guest root shell lb: (connecting took 0.00 seconds) lb: (finished: waiting for the VM to finish booting, in 0.00 seconds) worker2 # [ 11.507617] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 lb # [ 11.510871] postgresql-pre-start[715]: selecting default "shared_buffers" ... 128MB lb: (finished: waiting for unit network-addresses-eth1.service, in 0.11 seconds) lb: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; } lb: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; }, in 0.05 seconds) worker1: waiting for unit network-addresses-eth1.service worker1: waiting for the VM to finish booting worker1: Guest shell says: b'Spawning backdoor root shell...\n' worker1: connected to guest root shell worker1: (connecting took 0.00 seconds) worker1: (finished: waiting for the VM to finish booting, in 0.00 seconds) worker1 # [ 11.682116] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 worker1: (finished: waiting for unit network-addresses-eth1.service, in 0.10 seconds) worker1: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; } worker1: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; }, in 0.04 seconds) worker2: waiting for unit network-addresses-eth1.service worker2: waiting for the VM to finish booting worker2: Guest shell says: b'Spawning backdoor root shell...\n' worker2: connected to guest root shell worker2: (connecting took 0.00 seconds) worker2: (finished: waiting for the VM to finish booting, in 0.00 seconds) worker2: (finished: waiting for unit network-addresses-eth1.service, in 0.08 seconds) worker2: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; } client # [ 11.809791] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 11.812730] systemd[1]: Stopped Virtual Console Setup. client # [ 11.813544] systemd[1]: Stopping Virtual Console Setup... worker2 # [ 11.813077] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. client # [ 11.818250] systemd[1]: Starting Virtual Console Setup... worker2 # [ 11.823593] systemd[1]: Starting Virtual Console Setup... client # [ 11.843322] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 11.846783] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 11.849643] systemd[1]: Stopped Virtual Console Setup. client # [ 11.851762] systemd[1]: Starting Virtual Console Setup... worker2 # [ 11.857238] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 11.858029] systemd[1]: Starting Virtual Console Setup... worker2: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; }, in 0.08 seconds) lb: waiting for unit niks3.service client # [ 11.894349] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. client # [ 11.906755] systemd-logind[484]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) worker2 # [ 11.946674] systemd-logind[499]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) client # [ 11.971795] systemd-vconsole-setup[698]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 11.975424] systemd[1]: Finished Virtual Console Setup. worker2 # [ 11.996459] systemd-vconsole-setup[701]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 12.000306] systemd[1]: Finished Virtual Console Setup. worker1 # [ 12.028598] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker1 # [ 12.035107] systemd[1]: Starting Virtual Console Setup... worker1 # [ 12.057363] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 12.062702] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 12.063491] systemd[1]: Starting Virtual Console Setup... lb # [ 12.066909] dhcpcd[635]: eth0: soliciting a DHCP lease lb # [ 12.072541] dhcpcd[635]: eth0: offered 10.0.2.15 from 10.0.2.2 lb # [ 12.080205] dhcpcd[635]: eth0: probing address 10.0.2.15/24 worker1 # [ 12.134441] systemd-logind[559]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) worker1 # [ 12.185102] systemd-vconsole-setup[756]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 12.186098] dhcpcd[636]: eth0: soliciting an IPv6 router worker1 # [ 12.189368] systemd[1]: Finished Virtual Console Setup. worker2 # [ 12.187356] dhcpcd[636]: eth0: Router Advertisement from fe80::2 lb # [ 12.204641] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 worker2 # [ 12.188898] dhcpcd[636]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker2 # [ 12.190350] dhcpcd[636]: eth0: adding route to fec0::/64 worker2 # [ 12.191495] dhcpcd[636]: eth0: adding default route via fe80::2 worker2 # [ 12.419089] dhcpcd[636]: eth0: soliciting a DHCP lease worker2 # [ 12.424569] dhcpcd[636]: eth0: offered 10.0.2.15 from 10.0.2.2 worker2 # [ 12.428247] dhcpcd[636]: eth0: probing address 10.0.2.15/24 lb # [ 12.542004] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. lb # [ 12.545216] systemd[1]: Starting Virtual Console Setup... lb # [ 12.593413] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 12.596175] systemd[1]: Stopped Virtual Console Setup. lb # [ 12.608135] systemd[1]: Starting Virtual Console Setup... lb # [ 12.666424] systemd-logind[503]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) worker1 # [ 12.801844] nix-grpc-daemon[486]: ts=2026-09-25T15:23:59Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused" lb # [ 12.819072] systemd-vconsole-setup[755]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 12.823101] systemd[1]: Finished Virtual Console Setup. worker1 # [ 12.875634] dhcpcd[687]: eth0: soliciting an IPv6 router worker1 # [ 12.877515] dhcpcd[687]: eth0: Router Advertisement from fe80::2 worker1 # [ 12.878880] dhcpcd[687]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker1 # [ 12.880347] dhcpcd[687]: eth0: adding route to fec0::/64 worker1 # [ 12.881532] dhcpcd[687]: eth0: adding default route via fe80::2 client # [ 13.139248] dhcpcd[601]: eth0: soliciting an IPv6 router client # [ 13.141196] dhcpcd[601]: eth0: Router Advertisement from fe80::2 client # [ 13.142530] dhcpcd[601]: eth0: adding address fec0::5054:ff:fe12:3456/64 client # [ 13.143996] dhcpcd[601]: eth0: adding route to fec0::/64 client # [ 13.145283] dhcpcd[601]: eth0: adding default route via fe80::2 lb # [ 13.158821] postgresql-pre-start[715]: selecting default time zone ... UTC lb # [ 13.161505] postgresql-pre-start[715]: creating configuration files ... ok lb # [ 13.378055] postgresql-pre-start[715]: running bootstrap script ... ok lb # [ 13.689876] dhcpcd[635]: eth0: soliciting an IPv6 router lb # [ 13.691182] dhcpcd[635]: eth0: Router Advertisement from fe80::2 lb # [ 13.692151] dhcpcd[635]: eth0: adding address fec0::5054:ff:fe12:3456/64 lb # [ 13.693062] dhcpcd[635]: eth0: adding route to fec0::/64 lb # [ 13.693793] dhcpcd[635]: eth0: adding default route via fe80::2 lb # [ 13.877959] postgresql-pre-start[715]: performing post-bootstrap initialization ... ok lb # [ 14.001813] postgresql-pre-start[715]: syncing data to disk ... ok lb # [ 14.003981] postgresql-pre-start[715]: initdb: warning: enabling "trust" authentication for local connections lb # [ 14.005450] postgresql-pre-start[715]: initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. lb # [ 14.007506] postgresql-pre-start[715]: Success. You can now start the database server using: lb # [ 14.008750] postgresql-pre-start[715]: pg_ctl -D /var/lib/postgresql/18 -l logfile start lb # [ 14.124106] postgres[774]: [774] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit lb # [ 14.127846] postgres[774]: [774] LOG: listening on IPv6 address "::1", port 5432 lb # [ 14.129387] postgres[774]: [774] LOG: listening on IPv4 address "127.0.0.1", port 5432 lb # [ 14.131282] postgres[774]: [774] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432" lb # [ 14.145259] postgres[783]: [783] LOG: database system was shut down at 2026-09-25 15:24:00 GMT lb # [ 14.153241] postgres[774]: [774] LOG: database system is ready to accept connections lb # [ 14.160155] systemd[1]: Started PostgreSQL Server. lb # [ 14.168113] systemd[1]: Starting PostgreSQL Setup Scripts... lb # [ 14.356830] postgresql-setup-start[804]: CREATE DATABASE lb # [ 14.391079] postgresql-setup-start[809]: CREATE ROLE lb # [ 14.405281] postgresql-setup-start[811]: ALTER DATABASE lb # [ 14.410529] systemd[1]: Finished PostgreSQL Setup Scripts. lb # [ 14.412694] systemd[1]: Reached target PostgreSQL. worker1 # [ 16.002314] nix-grpc-daemon[486]: ts=2026-09-25T15:24:02Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused" worker1 # [ 16.155035] dhcpcd[687]: eth0: leased 10.0.2.15 for 86400 seconds worker1 # [ 16.158965] dhcpcd[687]: eth0: adding route to 10.0.2.0/24 worker1 # [ 16.162071] dhcpcd[687]: eth0: adding default route via 10.0.2.2 worker1 # [ 16.260582] systemd[1]: Started DHCP Client. worker1 # [ 16.262575] systemd[1]: Reached target Multi-User System. worker1 # [ 16.263471] systemd[1]: Startup finished in 962ms (kernel) + 4.704s (initrd) + 10.595s (userspace) = 16.262s. client # [ 16.623663] dhcpcd[601]: eth0: leased 10.0.2.15 for 86400 seconds client # [ 16.623870] dhcpcd[601]: eth0: adding route to 10.0.2.0/24 client # [ 16.623942] dhcpcd[601]: eth0: adding default route via 10.0.2.2 client # [ 16.736136] systemd[1]: Started DHCP Client. client # [ 16.738250] systemd[1]: Reached target Multi-User System. client # [ 16.739476] systemd[1]: Startup finished in 961ms (kernel) + 3.966s (initrd) + 11.811s (userspace) = 16.739s. lb # [ 17.437475] dhcpcd[635]: eth0: leased 10.0.2.15 for 86400 seconds lb # [ 17.437638] dhcpcd[635]: eth0: adding route to 10.0.2.0/24 lb # [ 17.437701] dhcpcd[635]: eth0: adding default route via 10.0.2.2 lb # [ 17.555716] systemd[1]: Started DHCP Client. lb # [ 17.557706] systemd[1]: Reached target Network is Online. lb # [ 17.561816] systemd[1]: Started Envoy reverse proxy. lb # [ 17.570498] systemd[1]: Started mock-oidc.service. worker2 # [ 17.704734] dhcpcd[636]: eth0: leased 10.0.2.15 for 86400 seconds worker2 # [ 17.704953] dhcpcd[636]: eth0: adding route to 10.0.2.0/24 worker2 # [ 17.705023] dhcpcd[636]: eth0: adding default route via 10.0.2.2 lb # [ 17.716562] mock-oidc-server[903]: Mock OIDC Server running lb # [ 17.718447] mock-oidc-server[903]: OIDC Address: 192.168.1.2:8080 lb # [ 17.719558] mock-oidc-server[903]: Issue Address: 0.0.0.0:8081 lb # [ 17.721123] mock-oidc-server[903]: Issuer: http://192.168.1.2:8080/oidc lb # [ 17.722022] mock-oidc-server[903]: JWKS: http://192.168.1.2:8080/oidc/.well-known/jwks.json lb # [ 17.723189] mock-oidc-server[903]: Discovery: http://192.168.1.2:8080/oidc/.well-known/openid-configuration lb # [ 17.728089] mock-oidc-server[903]: Issue tokens: http://0.0.0.0:8081/issue?sub=... worker2 # [ 17.806360] systemd[1]: Started DHCP Client. worker2 # [ 17.808407] systemd[1]: Reached target Network is Online. worker2 # [ 17.811798] systemd[1]: Starting scheduler-token.service... worker2 # [ 17.860046] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 17.862225] systemd[1]: Finished scheduler-token.service. worker2 # [ 17.863115] systemd[1]: scheduler-token.service: Consumed 26ms CPU time over 49ms wall clock time, 3.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 17.873091] systemd[1]: Starting Nix worker-protocol over gRPC... lb # [ 17.901029] envoy[902]: [2026-09-25 15:24:04.524][902][info][main] [source/server/server.cc:467] initializing epoch 0 (base id=0, hot restart version=11.120) lb # [ 17.903814] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:469] statically linked extensions: lb # [ 17.905801] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr lb # [ 17.908829] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.dynamic_modules, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd lb # [ 17.918171] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router lb # [ 17.921548] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.formatter: envoy.formatter.cel, envoy.formatter.dynamic_modules, envoy.formatter.file_content, envoy.formatter.generic_secret, envoy.formatter.metadata, envoy.formatter.req_without_query lb # [ 17.925904] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.content_parsers: envoy.content_parsers.json lb # [ 17.927951] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.upstream_options: envoy.extensions.upstreams.http.reverse_tunnel.v3.ReverseTunnelUpstreamCodecOptions, envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options lb # [ 17.933549] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel, envoy.access_loggers.extension_filters.process_ratelimit lb # [ 17.940286] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.fault, envoy.filters.http.a2a, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.ai_protocol_manager, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.api_key_auth, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.bandwidth_share, envoy.filters.http.basic_auth, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cache_v2, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.credential_injector, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_server, envoy.filters.http.file_system_buffer, envoy.filters.http.filter_chain, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_reverse_transcoder, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.mcp, envoy.filters.http.mcp_json_rest_bridge, envoy.filters.http.mcp_router, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.proto_api_scrubber, envoy.filters.http.proto_message_extraction, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.sse_to_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.thrift_to_metadata, envoy.filters.http.transform, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router worker2 # [ 17.978791] nix-grpc-daemon[769]: ts=2026-09-25T15:24:04Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" lb # [ 17.985788] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo, envoy.network.dns_resolver.hickory worker2 # [ 17.986650] nix-grpc-daemon[769]: ts=2026-09-25T15:24:04Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc lb # [ 17.989086] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.matching.action: envoy.matching.action.transport_socket.name, envoy.matching.actions.format_string, filter-chain-name worker2 # [ 17.988886] nix-grpc-daemon[769]: ts=2026-09-25T15:24:04Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 lb # [ 17.992434] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.route_config_update_requester: envoy.route_config_update_requester.default lb # [ 17.994938] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter lb # [ 17.997074] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.access_loggers: envoy.access_loggers.dynamic_modules, envoy.access_loggers.file, envoy.access_loggers.fluentd, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stats, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.fluentd_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log worker2 # [ 17.999902] nix-grpc-daemon[769]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 18.003290] nix-grpc-daemon[769]: E0000 00:00:1790349844.623341 769 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 18.004736] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.resource_detectors: envoy.tracers.opentelemetry.resource_detectors.dynatrace, envoy.tracers.opentelemetry.resource_detectors.environment, envoy.tracers.opentelemetry.resource_detectors.static_config lb # [ 18.008490] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action lb # [ 18.010548] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] network.connection.client: default, envoy_internal worker2 # [ 18.007972] nix-grpc-daemon[769]: E0000 00:00:1790349844.631642 769 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 18.011478] nix-grpc-daemon[769]: ts=2026-09-25T15:24:04Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket lb # [ 18.014262] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.http.ext_proc.response_processors: envoy.extensions.http.ext_proc.save_processing_response lb # [ 18.016612] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.common.key_value: envoy.key_value.file_based lb # [ 18.018283] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.tls.certificate_mappers: envoy.tls.certificate_mappers.sni, envoy.tls.certificate_mappers.static_name lb # [ 18.021512] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso worker2 # [ 18.021611] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 18.025242] systemd[1]: Reached target Multi-User System. lb # [ 18.023734] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls lb # [ 18.030391] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.regex_engines: envoy.regex_engines.google_re2 worker2 # [ 18.027643] nix-grpc-daemon[769]: E0000 00:00:1790349844.647576 811 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 18.032324] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.built_in_formatters: envoy.built_in_formatters.cel, envoy.built_in_formatters.http.default, envoy.built_in_formatters.metadata, envoy.built_in_formatters.stream_info.default, envoy.built_in_formatters.xfcc_value lb # [ 18.035832] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator lb # [ 18.038523] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.health_checkers: envoy.health_checkers.dynamic_modules, envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift lb # [ 18.042019] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.matching.stat_tag_matching_data.input: stat_tag_value_input lb # [ 18.043946] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.dynamic_modules, envoy.filters.udp_listener.udp_proxy lb # [ 18.046807] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.tls.certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret lb # [ 18.049075] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.http.forward_client_cert_details: forward_client_cert lb # [ 18.050895] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.matching.generic_proxy_request_input.input: envoy.matching.generic_proxy.input.host, envoy.matching.generic_proxy.input.method, envoy.matching.generic_proxy.input.path, envoy.matching.generic_proxy.input.property, envoy.matching.generic_proxy.input.request, envoy.matching.generic_proxy.input.service lb # [ 18.055557] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector, envoy.upstream.local_address_selector.filter_state_override worker2 # [ 18.058147] systemd[1]: Startup finished in 961ms (kernel) + 4.723s (initrd) + 12.328s (userspace) = 18.013s. lb # [ 18.058911] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.clusters: envoy.cluster.dns, envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.composite, envoy.clusters.dynamic_forward_proxy, envoy.clusters.dynamic_modules, envoy.clusters.mcp_multicluster, envoy.clusters.redis, envoy.clusters.reverse_connection lb # [ 18.064415] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor worker2 # [ 18.062790] nix-grpc-daemon[769]: E0000 00:00:1790349844.667768 811 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb # [ 18.067473] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.serializers: dubbo.hessian2 lb # [ 18.069405] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.geoip_providers: envoy.geoip_providers.maxmind worker2 # [ 18.066202] nix-grpc-daemon[769]: E0000 00:00:1790349844.668148 811 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 18.071176] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.generic_proxy.codecs: envoy.generic_proxy.codecs.dubbo, envoy.generic_proxy.codecs.http1 worker2 # [ 18.069228] nix-grpc-daemon[769]: E0000 00:00:1790349844.685137 811 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 18.071625] nix-grpc-daemon[769]: ts=2026-09-25T15:24:04Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused" lb # [ 18.073861] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.matching.transport_socket.input: envoy.matching.inputs.endpoint_metadata, envoy.matching.inputs.locality_metadata, envoy.matching.inputs.transport_socket_filter_state lb # [ 18.077707] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.resource_monitors: envoy.resource_monitors.cgroup_memory, envoy.resource_monitors.cpu_utilization, envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource lb # [ 18.081082] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.path.match: envoy.path.match.uri_template.uri_template_matcher lb # [ 18.083197] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.ssl.server_context_factory: envoy.ssl.server_context_factory.default lb # [ 18.085584] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.quic.client_packet_writer: envoy.quic.packet_writer.default lb # [ 18.087449] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.grpc_credentials: envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata lb # [ 18.090209] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port lb # [ 18.093333] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory lb # [ 18.096988] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.retry_priorities: envoy.retry_priorities.previous_priorities lb # [ 18.099142] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.matching.input_matchers: envoy.matching.input_matchers.generic_request_matcher, envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.dynamic_modules, envoy.matching.matchers.ip, envoy.matching.matchers.metadata_matcher, envoy.matching.matchers.runtime_fraction lb # [ 18.104098] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme lb # [ 18.109239] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.quic.connection_id_generator: envoy.quic.connection_id_generator.quic_lb, envoy.quic.deterministic_connection_id_generator lb # [ 18.111749] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.srds_factory: envoy.srds_factory.default lb # [ 18.113824] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter lb # [ 18.115938] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.dynamic_modules, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.ext_proc, envoy.filters.network.generic_proxy, envoy.filters.network.geoip, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.reverse_tunnel, envoy.filters.network.reverse_tunnel_drain_aware_http_connection_manager, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_bandwidth_limit, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy lb # [ 18.130792] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.load_balancing_policies: envoy.load_balancing_policies.client_side_weighted_round_robin, envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.dynamic_modules, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.load_aware_locality, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.override_host, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset, envoy.load_balancing_policies.wrr_locality lb # [ 18.139879] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.ext_authz, envoy.filters.udp.session.http_capsule lb # [ 18.143070] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.upstreams: envoy.filters.connection_pools.tcp.generic lb # [ 18.148224] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple lb # [ 18.150803] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.quic.proof_source: envoy.quic.proof_source.filter_chain lb # [ 18.153944] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.filters.upstream_network: envoy.filters.upstream_network.reverse_tunnel_lifecycle lb # [ 18.156638] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.filters.listener: envoy.filters.listener.dynamic_modules, envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.set_filter_state, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector lb # [ 18.163121] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.protocols: dubbo lb # [ 18.164693] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.network_namespace, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san lb # [ 18.171786] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls lb # [ 18.176822] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher lb # [ 18.179411] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.matching.stat_matching_data.input: stat_full_name_match_input lb # [ 18.181378] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.envelope, envoy.http.stateful_session.header lb # [ 18.183975] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.filters.http.upstream: envoy.buffer, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.composite, envoy.filters.http.credential_injector, envoy.filters.http.ext_proc, envoy.filters.http.header_mutation, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.upstream_codec, envoy.filters.http.upstream_rbac, envoy.filters.http.wasm lb # [ 18.191206] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.http.ext_proc.processing_request_modifiers: envoy.extensions.http.ext_proc.mapped_attribute_builder lb # [ 18.193648] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest lb # [ 18.200397] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation lb # [ 18.202701] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.samplers: envoy.tracers.opentelemetry.samplers.always_on, envoy.tracers.opentelemetry.samplers.cel, envoy.tracers.opentelemetry.samplers.dynatrace, envoy.tracers.opentelemetry.samplers.parent_based, envoy.tracers.opentelemetry.samplers.trace_id_ratio_based lb # [ 18.207031] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy lb # [ 18.209771] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.rds_factory: envoy.rds_factory.default lb # [ 18.211405] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.dynamic_metadata, envoy.matching.inputs.dynamic_module_data_input, envoy.matching.inputs.filter_state, envoy.matching.inputs.local_reply, envoy.matching.inputs.network_namespace, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params lb # [ 18.222425] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.health_check.event_sinks: envoy.health_check.event_sink.file lb # [ 18.224381] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.string_matcher: envoy.string_matcher.lua lb # [ 18.226035] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] quic.http_server_connection: quic.http_server_connection.default lb # [ 18.228044] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts lb # [ 18.231159] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router lb # [ 18.232958] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable lb # [ 18.234996] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_mappers: envoy.tls.certificate_mappers.static_name, envoy.tls.upstream_certificate_mappers.filter_state_override lb # [ 18.237743] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.http.cache_v2: envoy.extensions.http.cache_v2.file_system_http_cache, envoy.extensions.http.cache_v2.simple lb # [ 18.240153] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche lb # [ 18.242043] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.route.early_data_policy: envoy.route.early_data_policy.default lb # [ 18.243912] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.http.injected_credentials: envoy.http.injected_credentials.generic, envoy.http.injected_credentials.oauth2 lb # [ 18.246393] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.thrift_proxy.transports: auto, framed, header, unframed lb # [ 18.248237] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.quic.server_preferred_address: quic.server_preferred_address.datasource, quic.server_preferred_address.fixed lb # [ 18.250612] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret lb # [ 18.252817] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.tracers: envoy.tracers.datadog, envoy.tracers.dynamic_modules, envoy.tracers.fluentd, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin lb # [ 18.256087] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.generic_proxy.filters: envoy.filters.generic.router lb # [ 18.257830] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.connection_handler: envoy.connection_handler.default lb # [ 18.259589] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.router.cluster_specifier_plugin: envoy.router.cluster_specifier_plugin.lua, envoy.router.cluster_specifier_plugin.matcher lb # [ 18.262185] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor lb # [ 18.264775] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case lb # [ 18.267055] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.resolvers: envoy.ip, envoy.resolvers.reverse_connection lb # [ 18.268937] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.quic.connection_debug_visitor: envoy.quic.connection_debug_visitor.basic, envoy.quic.connection_debug_visitor.quic_stats lb # [ 18.271424] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff lb # [ 18.273985] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8 worker2 # [ 18.269584] nix-grpc-daemon[769]: ts=2026-09-25T15:24:04Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused" lb # [ 18.275876] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation lb # [ 18.278096] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.bootstrap: envoy.bootstrap.dynamic_modules, envoy.bootstrap.internal_listener, envoy.bootstrap.reverse_tunnel.downstream_socket_interface, envoy.bootstrap.reverse_tunnel.upstream_socket_interface, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface, envoy.extensions.network.socket_interface.sockmap lb # [ 18.283119] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.dynamic_modules, envoy.tls.cert_validator.spiffe lb # [ 18.285787] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.http.header_validators: envoy.http.header_validators.envoy_default lb # [ 18.287778] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.request_id: envoy.request_id.uuid lb # [ 18.289423] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher lb # [ 18.291994] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.hashable_string, envoy.network.application_protocols, envoy.network.ip, envoy.network.network_namespace, envoy.network.transport_socket.http_11_proxy.info, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.ratelimit.hits_addend, envoy.router.debug_config, envoy.string, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.tcp_proxy.per_connection_idle_timeout_ms, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port lb # [ 18.300138] envoy[902]: [2026-09-25 15:24:04.527][902][info][main] [source/server/server.cc:471] envoy.rbac.principals: envoy.rbac.principals.mtls_authenticated lb # [ 18.301955] envoy[902]: [2026-09-25 15:24:04.541][902][info][main] [source/server/server.cc:531] HTTP header map info: lb # [ 18.303322] envoy[902]: [2026-09-25 15:24:04.541][902][info][main] [source/server/server.cc:534] request header map: 664 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-host,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-request-id lb # [ 18.319024] envoy[902]: [2026-09-25 15:24:04.541][902][info][main] [source/server/server.cc:534] request trailer map: 120 bytes: lb # [ 18.320569] envoy[902]: [2026-09-25 15:24:04.541][902][info][main] [source/server/server.cc:534] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id lb # [ 18.329447] envoy[902]: [2026-09-25 15:24:04.541][902][info][main] [source/server/server.cc:534] response trailer map: 144 bytes: grpc-message,grpc-status lb # [ 18.331191] envoy[902]: [2026-09-25 15:24:04.605][902][info][main] [source/server/server.cc:968] runtime: {} lb # [ 18.332461] envoy[902]: [2026-09-25 15:24:04.606][902][info][main] [source/server/server.cc:420] Raised file descriptor limits from 1024 to 524288. lb # [ 18.334098] envoy[902]: [2026-09-25 15:24:04.607][902][info][main] [source/server/server.cc:796] Starting admin HTTP server at 127.0.0.1:9901 lb # [ 18.335678] envoy[902]: [2026-09-25 15:24:04.607][902][info][admin] [source/server/admin/admin.cc:66] admin address: 127.0.0.1:9901 lb # [ 18.337278] envoy[902]: [2026-09-25 15:24:04.607][902][info][config] [source/server/configuration_impl.cc:181] loading tracing configuration lb # [ 18.338927] envoy[902]: [2026-09-25 15:24:04.607][902][info][config] [source/server/configuration_impl.cc:132] loading 0 static secret(s) lb # [ 18.340597] envoy[902]: [2026-09-25 15:24:04.607][902][info][config] [source/server/configuration_impl.cc:138] loading 2 cluster(s) lb # [ 18.342128] envoy[902]: [2026-09-25 15:24:04.732][902][info][config] [source/server/configuration_impl.cc:148] loading 1 listener(s) lb # [ 18.343645] envoy[902]: [2026-09-25 15:24:04.763][902][info][config] [source/server/configuration_impl.cc:164] loading stats configuration lb # [ 18.345311] envoy[902]: [2026-09-25 15:24:04.763][902][warning][main] [source/server/server.cc:1038] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.global_downstream_max_connections` resource monitor. lb # [ 18.348455] envoy[902]: [2026-09-25 15:24:04.763][902][info][main] [source/server/server.cc:1077] starting main dispatch loop lb # [ 18.349864] envoy[902]: [2026-09-25 15:24:04.777][902][info][runtime] [source/common/runtime/runtime_impl.cc:567] RTDS has finished initialization lb # [ 18.351492] envoy[902]: [2026-09-25 15:24:04.777][902][info][upstream] [source/common/upstream/cluster_manager_impl.cc:227] cm init: initializing secondary clusters worker2 # [ 18.669893] nix-grpc-daemon[769]: ts=2026-09-25T15:24:05Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused" worker2 # [ 19.470775] nix-grpc-daemon[769]: ts=2026-09-25T15:24:06Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2001:db8:1::2%5D:50051: Failed to connect to remote host: Connection refused" worker1 # [ 21.002886] nix-grpc-daemon[486]: ts=2026-09-25T15:24:07Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused" worker2 # [ 21.071234] nix-grpc-daemon[769]: ts=2026-09-25T15:24:07Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2001:db8:1::2%5D:50051: Failed to connect to remote host: Connection refused" lb # [ 21.320703] rustfs-bucket-start[945]: mb s3://farm lb # [ 21.332462] systemd[1]: Finished rustfs-bucket.service. lb # [ 21.344372] systemd[1]: Starting niks3 server... lb # [ 21.491547] postgres[960]: [960] ERROR: relation "goose_db_version" does not exist at character 36 lb # [ 21.496245] postgres[960]: [960] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC lb # [ 21.516819] niks3-server[955]: 2026/09/25 15:24:08 OK 20241026095416_initial_model.sql (14.44ms) lb # [ 21.519870] niks3-server[955]: 2026/09/25 15:24:08 OK 20251210153512_drop_unused_gin_index.sql (3.34ms) lb # [ 21.524851] niks3-server[955]: 2026/09/25 15:24:08 OK 20251218171726_add_pins.sql (4.92ms) lb # [ 21.529293] niks3-server[955]: 2026/09/25 15:24:08 OK 20260628120000_add_object_size_and_stats.sql (4.4ms) lb # [ 21.534085] niks3-server[955]: 2026/09/25 15:24:08 OK 20260905000000_add_claims.sql (4.75ms) lb # [ 21.537718] niks3-server[955]: 2026/09/25 15:24:08 OK 20260920000000_drop_claims.sql (3.6ms) lb # [ 21.540296] niks3-server[955]: 2026/09/25 15:24:08 OK 20260923120000_add_pushes.sql (2.47ms) lb # [ 21.542195] niks3-server[955]: 2026/09/25 15:24:08 goose: successfully migrated database to version: 20260923120000 lb # [ 21.545519] niks3-server[955]: 2026/09/25 15:24:08 OK 1_commit_pending_closure.sql (5.15ms) lb # [ 21.548092] niks3-server[955]: 2026/09/25 15:24:08 OK 2_object_stats_trigger.sql (2.5ms) lb # [ 21.550712] niks3-server[955]: 2026/09/25 15:24:08 OK 3_commit_push.sql (2.21ms) lb # [ 21.551803] niks3-server[955]: 2026/09/25 15:24:08 goose: up to current file version: 3 lb # [ 21.553629] niks3-server[955]: 2026/09/25 15:24:08 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key lb # [ 21.576393] niks3-server[955]: 2026/09/25 15:24:08 INFO Read proxy enabled — serving cache objects from S3 lb # [ 21.577774] niks3-server[955]: 2026/09/25 15:24:08 INFO Using socket-activated listener address=0.0.0.0:5751 lb # [ 21.585442] niks3-server[955]: 2026/09/25 15:24:08 INFO systemd watchdog enabled interval=15s lb # [ 21.586694] niks3-server[955]: 2026/09/25 15:24:08 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 21.587895] systemd[1]: Started niks3 server. lb # [ 21.589742] systemd[1]: Reached target Multi-User System. lb # [ 21.590673] systemd[1]: Startup finished in 963ms (kernel) + 4.692s (initrd) + 15.929s (userspace) = 21.585s. lb: (finished: waiting for unit niks3.service, in 9.88 seconds) lb: waiting for TCP port 5751 on localhost lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded! lb: (finished: waiting for TCP port 5751 on localhost, in 0.04 seconds) lb: waiting for unit envoy.service lb: (finished: waiting for unit envoy.service, in 0.03 seconds) lb: waiting for TCP port 50051 on localhost worker2 # [ 24.271577] nix-grpc-daemon[769]: ts=2026-09-25T15:24:10Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2001:db8:1::2%5D:50051: Failed to connect to remote host: Connection refused" worker1 # [ 26.003267] nix-grpc-daemon[486]: ts=2026-09-25T15:24:12Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused" worker2 # [ 29.271960] nix-grpc-daemon[769]: ts=2026-09-25T15:24:15Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2001:db8:1::2%5D:50051: Failed to connect to remote host: Connection refused" worker1 # [ 31.003653] nix-grpc-daemon[486]: ts=2026-09-25T15:24:17Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv4:192.168.1.2:50051: Failed to connect to remote host: Connection refused" lb # [ 31.584944] niks3-server[955]: 2026/09/25 15:24:18 INFO lead: acquired remote=192.168.1.4:50512 worker2 # [ 31.582880] nix-grpc-daemon[769]: ts=2026-09-25T15:24:18Z level=info event=scheduler_take_over lb # [ 33.155460] envoy[902]: [2026-09-25 15:24:19.778][902][warning][config] [source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:130] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment lb # [ 33.155615] envoy[902]: [2026-09-25 15:24:19.778][902][info][upstream] [source/common/upstream/cluster_manager_impl.cc:250] cm init: all clusters initialized lb # [ 33.155668] envoy[902]: [2026-09-25 15:24:19.778][902][info][main] [source/server/server.cc:1057] all clusters initialized. initializing init manager lb # [ 33.171575] envoy[902]: [2026-09-25 15:24:19.794][902][info][config] [source/common/listener_manager/listener_manager_impl.cc:1067] all dependencies initialized. starting workers lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded! lb # [ 34.210032] envoy[902]: conn peer=[::1]:59110 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=5 lb: (finished: waiting for TCP port 50051 on localhost, in 12.38 seconds) worker2 # [ 34.272754] nix-grpc-daemon[769]: ts=2026-09-25T15:24:20Z level=info event=scheduler_disconnected addr=lb:50051 error="failed to connect to all addresses; last error: UNKNOWN: ipv6:%5B2001:db8:1::2%5D:50051: Failed to connect to remote host: Connection refused" worker2 # [ 34.308666] systemd[1]: Starting scheduler-token.service... worker2 # [ 34.358750] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 34.363645] systemd[1]: Finished scheduler-token.service. worker2 # [ 34.364965] systemd[1]: scheduler-token.service: Consumed 34ms CPU time over 58ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. lb: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1 # [ 36.004450] nix-grpc-daemon[486]: ts=2026-09-25T15:24:22Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.006215] nix-grpc-daemon[769]: ts=2026-09-25T15:24:22Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:34328 worker2 # [ 36.009317] nix-grpc-daemon[769]: ts=2026-09-25T15:24:22Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2 # [ 39.273676] nix-grpc-daemon[769]: ts=2026-09-25T15:24:25Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.282521] nix-grpc-daemon[769]: ts=2026-09-25T15:24:25Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 39.285223] nix-grpc-daemon[769]: ts=2026-09-25T15:24:25Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:34328 worker2 # [ 39.288218] nix-grpc-daemon[769]: ts=2026-09-25T15:24:25Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1 # [ 39.740751] nix-grpc-daemon[486]: E0000 00:00:1790349866.362836 538 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 39.763202] nix-grpc-daemon[486]: E0000 00:00:1790349866.390280 538 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 39.792778] nix-grpc-daemon[486]: E0000 00:00:1790349866.394455 538 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 39.796387] nix-grpc-daemon[486]: E0000 00:00:1790349866.417327 538 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2 # [ 46.913348] nix-grpc-daemon[769]: ts=2026-09-25T15:24:33Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:56458 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: (finished: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux, in 13.52 seconds) subtest: one node schedules and both builders hold a WorkerSession on it lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 48.028789] nix-grpc-daemon[769]: E0000 00:00:1790349874.649567 809 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 48.053114] nix-grpc-daemon[769]: E0000 00:00:1790349874.676923 809 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 48.079603] nix-grpc-daemon[769]: E0000 00:00:1790349874.682073 809 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 48.083286] nix-grpc-daemon[769]: E0000 00:00:1790349874.702259 809 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open' worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) (finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 0.45 seconds) subtest: balancer auth: client cert, bearer token, nothing, unknown CN worker1 # [ 48.558536] nix-grpc-daemon[486]: ts=2026-09-25T15:24:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 48.566535] systemd[1]: Started Nix Daemon. worker1 # [ 48.698142] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 48.712210] nix-grpc-daemon[486]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 77 ms (attempt 1/5) worker1 # [ 48.792811] nix-grpc-daemon[486]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 18 ms (attempt 2/5) worker1 # [ 48.811855] nix-grpc-daemon[486]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 74 ms (attempt 3/5) worker1 # [ 48.890897] nix-grpc-daemon[486]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 690 ms (attempt 4/5) worker1 # [ 49.583012] nix-grpc-daemon[486]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 49.583166] nix-grpc-daemon[486]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 49.595136] nix-grpc-daemon[486]: ts=2026-09-25T15:24:36Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 49.602057] nix-grpc-daemon[486]: ts=2026-09-25T15:24:36Z level=debug event=path_info_miss path=nwgxc9xy9a9l9j6z82df6cwp1rq7ly6z-nixos-system-client-test error="error: path '/nix/store/nwgxc9xy9a9l9j6z82df6cwp1rq7ly6z-nixos-system-client-test' is required, but there is no substituter that can build it" worker2 # [ 49.614228] nix-grpc-daemon[769]: ts=2026-09-25T15:24:36Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 49.622447] systemd[1]: Started Nix Daemon. worker2 # [ 49.771226] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 49.786147] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 94 ms (attempt 1/5) worker2 # [ 49.883945] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 106 ms (attempt 2/5) worker2 # [ 49.990874] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 143 ms (attempt 3/5) worker2 # [ 50.135007] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 6 ms (attempt 4/5) worker2 # [ 50.142094] nix-grpc-daemon[769]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 50.145253] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 50.151165] nix-grpc-daemon[769]: ts=2026-09-25T15:24:36Z level=debug event=path_info_miss path=nwgxc9xy9a9l9j6z82df6cwp1rq7ly6z-nixos-system-client-test error="error: path '/nix/store/nwgxc9xy9a9l9j6z82df6cwp1rq7ly6z-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.05 seconds) client: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key client: (finished: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.03 seconds) worker2 # [ 50.406001] nix-grpc-daemon[769]: ts=2026-09-25T15:24:37Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:56488 worker2 # [ 50.415442] nix-grpc-daemon[769]: ts=2026-09-25T15:24:37Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:56488 worker2 # [ 50.422007] nix-grpc-daemon[769]: ts=2026-09-25T15:24:37Z level=debug event=path_info_miss path=nwgxc9xy9a9l9j6z82df6cwp1rq7ly6z-nixos-system-client-test error="error: path '/nix/store/nwgxc9xy9a9l9j6z82df6cwp1rq7ly6z-nixos-system-client-test' is required, but there is no substituter that can build it" worker2 # [ 50.430015] nix-grpc-daemon[769]: ts=2026-09-25T15:24:37Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:56488 worker2 # [ 50.434490] nix-grpc-daemon[769]: ts=2026-09-25T15:24:37Z level=debug event=path_info_miss path=nwgxc9xy9a9l9j6z82df6cwp1rq7ly6z-nixos-system-client-test error="error: path '/nix/store/nwgxc9xy9a9l9j6z82df6cwp1rq7ly6z-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: rm -r /var/lib/nix-grpc-store client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.02 seconds) worker1 # [ 50.619625] nix-grpc-daemon[486]: ts=2026-09-25T15:24:37Z level=info event=denied method=QueryMissing cn=- role=none worker2 # [ 50.768760] nix-grpc-daemon[769]: ts=2026-09-25T15:24:37Z level=info event=denied method=QueryMissing cn=stranger role=none (finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 2.42 seconds) subtest: DAG build is scheduled across workers and lands in the cache client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2 worker2 # [ 50.923766] nix-grpc-daemon[769]: ts=2026-09-25T15:24:37Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:56488 client # don't know how to build these paths: client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker1 # [ 50.934804] nix-grpc-daemon[486]: ts=2026-09-25T15:24:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53882 worker2 # [ 50.953493] nix-grpc-daemon[769]: ts=2026-09-25T15:24:37Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=0 worker1 # [ 50.957822] nix-grpc-daemon[486]: ts=2026-09-25T15:24:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker2 # [ 50.956649] nix-grpc-daemon[769]: ts=2026-09-25T15:24:37Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 50.966825] nix-grpc-daemon[486]: ts=2026-09-25T15:24:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53882 client # copying 1 paths to 'lb:50051'... worker1 # [ 50.975576] nix-grpc-daemon[486]: ts=2026-09-25T15:24:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'... worker1 # [ 50.988907] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 51.007801] nix-grpc-daemon[486]: ts=2026-09-25T15:24:37Z level=info event=niks3_push_started pid=850 worker1 # [ 51.179290] nix-grpc-daemon[850]: time=2026-09-25T15:24:37.806Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 51.281884] nix-daemon[838]: accepted connection from pid 862, user nix-grpc-daemon (trusted) lb # [ 51.299233] niks3-server[955]: 2026/09/25 15:24:37 INFO Received push request method=POST path=/api/pushes worker1 # [ 51.301826] nix-daemon[838]: reaped child process 865, status = succeeded worker1 # [ 51.310343] nix-grpc-daemon[850]: time=2026-09-25T15:24:37.936Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 51.311850] nix-grpc-daemon[850]: time=2026-09-25T15:24:37.936Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)" lb # [ 51.332097] niks3-server[955]: 2026/09/25 15:24:37 INFO Received sign narinfos request method=POST path=/api/pushes/1/sign lb # [ 51.353561] niks3-server[955]: 2026/09/25 15:24:37 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls lb # [ 51.371697] niks3-server[955]: 2026/09/25 15:24:37 INFO Signed narinfos id=1 count=1 worker1 # [ 51.374916] nix-grpc-daemon[850]: time=2026-09-25T15:24:38.000Z level=INFO msg="Uploading 1 narinfos" lb # [ 51.374493] niks3-server[955]: 2026/09/25 15:24:37 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst lb # [ 51.385560] niks3-server[955]: 2026/09/25 15:24:38 INFO Received complete push request method=POST path=/api/pushes/1/complete lb # [ 51.408554] niks3-server[955]: 2026/09/25 15:24:38 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo worker1 # [ 51.413938] nix-grpc-daemon[850]: time=2026-09-25T15:24:38.041Z level=INFO msg="Upload complete. (230ms)" worker1 # [ 51.419962] nix-grpc-daemon[486]: ts=2026-09-25T15:24:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=0 paths=1 nar_bytes_in=504 worker1 # [ 51.423287] nix-grpc-daemon[486]: ts=2026-09-25T15:24:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 51.435631] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 51.439678] nix-daemon[838]: reaped child process 848, status = succeeded client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051... client # farm-a> node-a: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker1 # [ 51.449322] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-t1 worker1 # [ 52.717988] nix-daemon[838]: reaped child process 869, status = succeeded worker1 # [ 52.777294] nix-daemon[838]: accepted connection from pid 885, user nix-grpc-daemon (trusted) worker1 # [ 52.794068] nix-daemon[838]: reaped child process 888, status = succeeded lb # [ 52.799672] niks3-server[955]: 2026/09/25 15:24:39 INFO Received push request method=POST path=/api/pushes worker1 # [ 52.808422] nix-grpc-daemon[850]: time=2026-09-25T15:24:39.434Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 52.821294] nix-grpc-daemon[850]: time=2026-09-25T15:24:39.434Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)" lb # [ 52.852535] niks3-server[955]: 2026/09/25 15:24:39 INFO Received sign narinfos request method=POST path=/api/pushes/2/sign worker1 # [ 52.861548] nix-grpc-daemon[850]: time=2026-09-25T15:24:39.488Z level=INFO msg="Uploading 1 narinfos" lb # [ 52.862830] niks3-server[955]: 2026/09/25 15:24:39 INFO Signed narinfos id=2 count=1 lb # [ 52.870548] niks3-server[955]: 2026/09/25 15:24:39 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst lb # [ 52.880558] niks3-server[955]: 2026/09/25 15:24:39 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv lb # [ 52.888766] niks3-server[955]: 2026/09/25 15:24:39 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls lb # [ 52.891192] niks3-server[955]: 2026/09/25 15:24:39 INFO Received complete push request method=POST path=/api/pushes/2/complete worker1 # [ 52.897510] nix-grpc-daemon[850]: time=2026-09-25T15:24:39.524Z level=INFO msg="Upload complete. (237ms)" lb # [ 52.900215] niks3-server[955]: 2026/09/25 15:24:39 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo worker1 # [ 52.904593] nix-grpc-daemon[486]: ts=2026-09-25T15:24:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 52.915958] nix-daemon[838]: reaped child process 867, status = succeeded worker2 # [ 52.913216] nix-grpc-daemon[769]: ts=2026-09-25T15:24:39Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=0 worker2 # [ 52.916616] nix-grpc-daemon[769]: ts=2026-09-25T15:24:39Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 52.924580] nix-grpc-daemon[486]: ts=2026-09-25T15:24:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 52.934459] nix-grpc-daemon[486]: ts=2026-09-25T15:24:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 52.943550] nix-grpc-daemon[850]: time=2026-09-25T15:24:39.570Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker1 # [ 52.949980] nix-grpc-daemon[486]: ts=2026-09-25T15:24:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'... worker1 # [ 52.963396] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 53.083726] nix-daemon[838]: accepted connection from pid 892, user nix-grpc-daemon (trusted) worker1 # [ 53.104150] nix-daemon[838]: reaped child process 895, status = succeeded lb # [ 53.107009] niks3-server[955]: 2026/09/25 15:24:39 INFO Received push request method=POST path=/api/pushes worker1 # [ 53.113358] nix-grpc-daemon[850]: time=2026-09-25T15:24:39.740Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 53.114970] nix-grpc-daemon[850]: time=2026-09-25T15:24:39.740Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)" lb # [ 53.126089] niks3-server[955]: 2026/09/25 15:24:39 INFO Received sign narinfos request method=POST path=/api/pushes/3/sign worker1 # [ 53.130866] nix-grpc-daemon[850]: time=2026-09-25T15:24:39.757Z level=INFO msg="Uploading 1 narinfos" lb # [ 53.134307] niks3-server[955]: 2026/09/25 15:24:39 INFO Signed narinfos id=3 count=1 lb # [ 53.139192] niks3-server[955]: 2026/09/25 15:24:39 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst lb # [ 53.150600] niks3-server[955]: 2026/09/25 15:24:39 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls lb # [ 53.157645] niks3-server[955]: 2026/09/25 15:24:39 INFO Received complete push request method=POST path=/api/pushes/3/complete lb # [ 53.163754] niks3-server[955]: 2026/09/25 15:24:39 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo worker1 # [ 53.166786] nix-grpc-daemon[850]: time=2026-09-25T15:24:39.793Z level=INFO msg="Upload complete. (193ms)" worker1 # [ 53.173452] nix-grpc-daemon[486]: ts=2026-09-25T15:24:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 53.180900] nix-grpc-daemon[486]: ts=2026-09-25T15:24:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 53.189999] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 53.194306] nix-daemon[838]: reaped child process 890, status = succeeded client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051... client # farm-b> node-a: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker1 # [ 53.204747] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-t1 worker1 # [ 54.260267] nix-daemon[838]: reaped child process 899, status = succeeded worker1 # [ 54.345094] nix-daemon[838]: accepted connection from pid 915, user nix-grpc-daemon (trusted) lb # [ 54.361744] niks3-server[955]: 2026/09/25 15:24:40 INFO Received push request method=POST path=/api/pushes worker1 # [ 54.363749] nix-daemon[838]: reaped child process 918, status = succeeded worker1 # [ 54.369510] nix-grpc-daemon[850]: time=2026-09-25T15:24:40.995Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 54.371027] nix-grpc-daemon[850]: time=2026-09-25T15:24:40.995Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)" lb # [ 54.384325] niks3-server[955]: 2026/09/25 15:24:41 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv lb # [ 54.393315] niks3-server[955]: 2026/09/25 15:24:41 INFO Received sign narinfos request method=POST path=/api/pushes/4/sign worker1 # [ 54.397589] nix-grpc-daemon[850]: time=2026-09-25T15:24:41.024Z level=INFO msg="Uploading 1 narinfos" lb # [ 54.400608] niks3-server[955]: 2026/09/25 15:24:41 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst lb # [ 54.403608] niks3-server[955]: 2026/09/25 15:24:41 INFO Signed narinfos id=4 count=1 lb # [ 54.417477] niks3-server[955]: 2026/09/25 15:24:41 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls lb # [ 54.419903] niks3-server[955]: 2026/09/25 15:24:41 INFO Received complete push request method=POST path=/api/pushes/4/complete worker1 # [ 54.427168] nix-grpc-daemon[850]: time=2026-09-25T15:24:41.054Z level=INFO msg="Upload complete. (215ms)" lb # [ 54.428481] niks3-server[955]: 2026/09/25 15:24:41 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo worker2 # [ 54.435352] nix-grpc-daemon[769]: ts=2026-09-25T15:24:41Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=0 worker1 # [ 54.436613] nix-grpc-daemon[486]: ts=2026-09-25T15:24:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=1 inputs=1 inputs_fetched=0 input_bytes=0 input_s=0 worker2 # [ 54.438945] nix-grpc-daemon[769]: ts=2026-09-25T15:24:41Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 54.446617] nix-grpc-daemon[486]: ts=2026-09-25T15:24:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 54.450722] nix-daemon[838]: reaped child process 897, status = succeeded worker1 # [ 54.455956] nix-grpc-daemon[486]: ts=2026-09-25T15:24:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 54.463645] nix-grpc-daemon[850]: time=2026-09-25T15:24:41.090Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... worker1 # [ 54.469924] nix-grpc-daemon[486]: ts=2026-09-25T15:24:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'... worker1 # [ 54.482299] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 54.597157] nix-daemon[838]: accepted connection from pid 922, user nix-grpc-daemon (trusted) worker1 # [ 54.612557] nix-daemon[838]: reaped child process 925, status = succeeded lb # [ 54.618065] niks3-server[955]: 2026/09/25 15:24:41 INFO Received push request method=POST path=/api/pushes worker1 # [ 54.622969] nix-grpc-daemon[850]: time=2026-09-25T15:24:41.249Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 54.624581] nix-grpc-daemon[850]: time=2026-09-25T15:24:41.249Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)" worker1 # [ 54.636269] nix-grpc-daemon[850]: time=2026-09-25T15:24:41.263Z level=INFO msg="Uploading 1 narinfos" lb # [ 54.636880] niks3-server[955]: 2026/09/25 15:24:41 INFO Received sign narinfos request method=POST path=/api/pushes/5/sign lb # [ 54.639297] niks3-server[955]: 2026/09/25 15:24:41 INFO Signed narinfos id=5 count=1 lb # [ 54.646932] niks3-server[955]: 2026/09/25 15:24:41 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst lb # [ 54.653083] niks3-server[955]: 2026/09/25 15:24:41 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls lb # [ 54.660459] niks3-server[955]: 2026/09/25 15:24:41 INFO Received complete push request method=POST path=/api/pushes/5/complete worker1 # [ 54.668962] nix-grpc-daemon[850]: time=2026-09-25T15:24:41.296Z level=INFO msg="Upload complete. (177ms)" lb # [ 54.669399] niks3-server[955]: 2026/09/25 15:24:41 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo worker1 # [ 54.674991] nix-grpc-daemon[486]: ts=2026-09-25T15:24:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 54.685543] nix-grpc-daemon[486]: ts=2026-09-25T15:24:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 54.690405] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 54.694234] nix-daemon[838]: reaped child process 920, status = succeeded client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051... worker1 # [ 54.703459] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) client # farm-top> node-a: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-t1 worker1 # [ 55.791822] nix-daemon[838]: reaped child process 929, status = succeeded worker1 # [ 55.849795] nix-daemon[838]: accepted connection from pid 945, user nix-grpc-daemon (trusted) worker1 # [ 55.870098] nix-daemon[838]: reaped child process 948, status = succeeded lb # [ 55.876356] niks3-server[955]: 2026/09/25 15:24:42 INFO Received push request method=POST path=/api/pushes worker1 # [ 55.884237] nix-grpc-daemon[850]: time=2026-09-25T15:24:42.509Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 55.885844] nix-grpc-daemon[850]: time=2026-09-25T15:24:42.509Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)" lb # [ 55.901486] niks3-server[955]: 2026/09/25 15:24:42 INFO Received sign narinfos request method=POST path=/api/pushes/6/sign worker1 # [ 55.906948] nix-grpc-daemon[850]: time=2026-09-25T15:24:42.533Z level=INFO msg="Uploading 1 narinfos" lb # [ 55.909890] niks3-server[955]: 2026/09/25 15:24:42 INFO Signed narinfos id=6 count=1 lb # [ 55.916572] niks3-server[955]: 2026/09/25 15:24:42 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls lb # [ 55.922421] niks3-server[955]: 2026/09/25 15:24:42 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker1 # [ 55.936570] nix-grpc-daemon[850]: time=2026-09-25T15:24:42.563Z level=INFO msg="Upload complete. (221ms)" lb # [ 55.931526] niks3-server[955]: 2026/09/25 15:24:42 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst lb # [ 55.940739] niks3-server[955]: 2026/09/25 15:24:42 INFO Received complete push request method=POST path=/api/pushes/6/complete lb # [ 55.944569] niks3-server[955]: 2026/09/25 15:24:42 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo worker1 # [ 55.944697] nix-grpc-daemon[486]: ts=2026-09-25T15:24:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=1 inputs=2 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 55.957693] nix-daemon[838]: reaped child process 927, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 5.20 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.15 seconds) client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.02 seconds) client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client # copying 3 paths... client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'... client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'... client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.22 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) (finished: subtest: DAG build is scheduled across workers and lands in the cache, in 5.84 seconds) subtest: repeat is answered Cached by the scheduler without building worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker1 # finding garbage collector roots... worker1 # deleting specified paths... worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker1 # deleting unused links... worker1 # note: hard linking is currently saving 0.0 KiB worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.18 seconds) worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker2 # finding garbage collector roots... worker2 # deleting specified paths... worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker2 # deleting unused links... worker2 # note: hard linking is currently saving 0.0 KiB worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.20 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2 client # don't know how to build these paths: client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker2 # [ 57.179409] nix-grpc-daemon[769]: ts=2026-09-25T15:24:43Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:56488 worker1 # [ 57.185892] nix-grpc-daemon[486]: ts=2026-09-25T15:24:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 57.193527] nix-grpc-daemon[850]: time=2026-09-25T15:24:43.819Z level=INFO msg="All 2 paths already cached" worker2 # [ 57.198730] nix-grpc-daemon[769]: ts=2026-09-25T15:24:43Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=1 client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 0.22 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.13 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) (finished: subtest: repeat is answered Cached by the scheduler without building, in 0.89 seconds) subtest: inputs already in the cache are not sent to the scheduler lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.03 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2 worker1 # [ 57.781456] nix-grpc-daemon[486]: ts=2026-09-25T15:24:44Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53882 client # don't know how to build these paths: client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker2 # [ 57.787144] nix-grpc-daemon[769]: ts=2026-09-25T15:24:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 57.807849] nix-grpc-daemon[769]: ts=2026-09-25T15:24:44Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=0 worker1 # [ 57.812391] nix-grpc-daemon[486]: ts=2026-09-25T15:24:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker2 # [ 57.811299] nix-grpc-daemon[769]: ts=2026-09-25T15:24:44Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 57.821744] nix-grpc-daemon[486]: ts=2026-09-25T15:24:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 57.830548] nix-grpc-daemon[850]: time=2026-09-25T15:24:44.457Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'... worker1 # [ 57.836848] nix-grpc-daemon[486]: ts=2026-09-25T15:24:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 57.845653] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 57.975475] nix-daemon[838]: accepted connection from pid 974, user nix-grpc-daemon (trusted) lb # [ 57.995397] niks3-server[955]: 2026/09/25 15:24:44 INFO Received push request method=POST path=/api/pushes worker1 # [ 57.998076] nix-daemon[838]: reaped child process 977, status = succeeded worker1 # [ 58.000993] nix-grpc-daemon[850]: time=2026-09-25T15:24:44.627Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 58.002528] nix-grpc-daemon[850]: time=2026-09-25T15:24:44.627Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)" lb # [ 58.014990] niks3-server[955]: 2026/09/25 15:24:44 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst worker1 # [ 58.022767] nix-grpc-daemon[850]: time=2026-09-25T15:24:44.649Z level=INFO msg="Uploading 1 narinfos" lb # [ 58.022947] niks3-server[955]: 2026/09/25 15:24:44 INFO Received sign narinfos request method=POST path=/api/pushes/7/sign lb # [ 58.030539] niks3-server[955]: 2026/09/25 15:24:44 INFO Signed narinfos id=7 count=1 lb # [ 58.033909] niks3-server[955]: 2026/09/25 15:24:44 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls lb # [ 58.044596] niks3-server[955]: 2026/09/25 15:24:44 INFO Received complete push request method=POST path=/api/pushes/7/complete worker1 # [ 58.050845] nix-grpc-daemon[850]: time=2026-09-25T15:24:44.677Z level=INFO msg="Upload complete. (194ms)" lb # [ 58.051542] niks3-server[955]: 2026/09/25 15:24:44 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo worker1 # [ 58.057737] nix-grpc-daemon[486]: ts=2026-09-25T15:24:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 58.064190] nix-grpc-daemon[486]: ts=2026-09-25T15:24:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 58.074423] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 58.079034] nix-daemon[838]: reaped child process 972, status = succeeded client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051... worker1 # [ 58.087442] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) client # farm-top> node-a: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv lb # [ 58.111258] envoy[902]: conn peer=[2001:db8:1::1]:60066 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=749 tx=422 ms=1618 lb # [ 58.114150] envoy[902]: conn peer=[2001:db8:1::1]:60082 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=362 ms=37 lb # [ 58.116562] envoy[902]: conn peer=[2001:db8:1::1]:60090 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=6 lb # [ 58.119399] envoy[902]: conn peer=[2001:db8:1::1]:60096 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=8 lb # [ 58.122160] envoy[902]: conn peer=[2001:db8:1::1]:52782 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9476 tx=4345 ms=5043 lb # [ 58.124678] envoy[902]: conn peer=[2001:db8:1::1]:52796 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=479 ms=34 client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-t1b worker1 # [ 58.713202] nix-daemon[838]: reaped child process 981, status = succeeded worker1 # [ 58.818529] nix-daemon[838]: accepted connection from pid 997, user nix-grpc-daemon (trusted) worker1 # [ 58.836219] nix-daemon[838]: reaped child process 1000, status = succeeded lb # [ 58.841971] niks3-server[955]: 2026/09/25 15:24:45 INFO Received push request method=POST path=/api/pushes worker1 # [ 58.848514] nix-grpc-daemon[850]: time=2026-09-25T15:24:45.474Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 58.850017] nix-grpc-daemon[850]: time=2026-09-25T15:24:45.474Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)" lb # [ 58.870594] niks3-server[955]: 2026/09/25 15:24:45 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst worker1 # [ 58.877552] nix-grpc-daemon[850]: time=2026-09-25T15:24:45.504Z level=INFO msg="Uploading 1 narinfos" lb # [ 58.879759] niks3-server[955]: 2026/09/25 15:24:45 INFO Received sign narinfos request method=POST path=/api/pushes/8/sign lb # [ 58.886435] niks3-server[955]: 2026/09/25 15:24:45 INFO Signed narinfos id=8 count=1 lb # [ 58.890640] niks3-server[955]: 2026/09/25 15:24:45 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv lb # [ 58.900140] niks3-server[955]: 2026/09/25 15:24:45 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls lb # [ 58.902768] niks3-server[955]: 2026/09/25 15:24:45 INFO Received complete push request method=POST path=/api/pushes/8/complete worker1 # [ 58.907525] nix-grpc-daemon[850]: time=2026-09-25T15:24:45.534Z level=INFO msg="Upload complete. (210ms)" lb # [ 58.908882] niks3-server[955]: 2026/09/25 15:24:45 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo worker1 # [ 58.916666] nix-grpc-daemon[486]: ts=2026-09-25T15:24:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=0 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=1 inputs=2 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 58.926991] nix-daemon[838]: reaped child process 979, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2, in 1.33 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.14 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.03 seconds) (finished: subtest: inputs already in the cache are not sent to the scheduler, in 1.68 seconds) subtest: outputs read back through any worker carry the cache signature client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2 worker1 # [ 59.357684] nix-grpc-daemon[486]: ts=2026-09-25T15:24:45Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53882 client # don't know how to build these paths: client # /nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker1 # [ 59.364831] nix-grpc-daemon[486]: ts=2026-09-25T15:24:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53882 worker2 # [ 59.378386] nix-grpc-daemon[769]: ts=2026-09-25T15:24:46Z level=debug event=want client=4 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv cached=0 worker2 # [ 59.382723] nix-grpc-daemon[769]: ts=2026-09-25T15:24:46Z level=debug event=assigned drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 59.386939] nix-grpc-daemon[769]: ts=2026-09-25T15:24:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 59.393229] nix-grpc-daemon[769]: ts=2026-09-25T15:24:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' to 'lb:50051'... worker2 # [ 59.404161] nix-grpc-daemon[769]: ts=2026-09-25T15:24:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 59.413239] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 59.431024] nix-grpc-daemon[769]: ts=2026-09-25T15:24:46Z level=info event=niks3_push_started pid=916 worker2 # [ 59.577668] nix-grpc-daemon[916]: time=2026-09-25T15:24:46.201Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 59.678999] nix-daemon[857]: accepted connection from pid 928, user nix-grpc-daemon (trusted) worker2 # [ 59.699197] nix-daemon[857]: reaped child process 931, status = succeeded lb # [ 59.705844] niks3-server[955]: 2026/09/25 15:24:46 INFO Received push request method=POST path=/api/pushes worker2 # [ 59.710165] nix-grpc-daemon[916]: time=2026-09-25T15:24:46.332Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 59.711723] nix-grpc-daemon[916]: time=2026-09-25T15:24:46.332Z level=INFO msg="Uploading 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv (512B)" lb # [ 59.731088] niks3-server[955]: 2026/09/25 15:24:46 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.ls worker2 # [ 59.737763] nix-grpc-daemon[916]: time=2026-09-25T15:24:46.360Z level=INFO msg="Uploading 1 narinfos" lb # [ 59.735422] niks3-server[955]: 2026/09/25 15:24:46 INFO Received sign narinfos request method=POST path=/api/pushes/9/sign lb # [ 59.742954] niks3-server[955]: 2026/09/25 15:24:46 INFO Signed narinfos id=9 count=1 lb # [ 59.746957] niks3-server[955]: 2026/09/25 15:24:46 INFO Registered completed upload object_key=nar/13mhmmy2qay13imbbdi287vhryz1hwnyg0w6agd9s0ph37wnsqll.nar.zst lb # [ 59.756498] niks3-server[955]: 2026/09/25 15:24:46 INFO Received complete push request method=POST path=/api/pushes/9/complete worker2 # [ 59.759521] nix-grpc-daemon[916]: time=2026-09-25T15:24:46.383Z level=INFO msg="Upload complete. (178ms)" lb # [ 59.763347] niks3-server[955]: 2026/09/25 15:24:46 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.narinfo worker2 # [ 59.766792] nix-grpc-daemon[769]: ts=2026-09-25T15:24:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 paths=1 nar_bytes_in=512 worker2 # [ 59.776168] nix-grpc-daemon[769]: ts=2026-09-25T15:24:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 59.781737] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 59.783821] nix-daemon[857]: reaped child process 914, status = succeeded client # building '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' on lb:50051... worker2 # [ 59.791794] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) client # farm-a> worker2: building 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-sig worker2 # [ 60.640657] nix-daemon[857]: reaped child process 935, status = succeeded worker2 # [ 60.714980] nix-daemon[857]: accepted connection from pid 951, user nix-grpc-daemon (trusted) worker2 # [ 60.733482] nix-daemon[857]: reaped child process 954, status = succeeded lb # [ 60.738894] niks3-server[955]: 2026/09/25 15:24:47 INFO Received push request method=POST path=/api/pushes worker2 # [ 60.742932] nix-grpc-daemon[916]: time=2026-09-25T15:24:47.365Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 60.746825] nix-grpc-daemon[916]: time=2026-09-25T15:24:47.365Z level=INFO msg="Uploading 2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a (128B)" lb # [ 60.760133] niks3-server[955]: 2026/09/25 15:24:47 INFO Registered completed upload object_key=log/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker2 # [ 60.766438] nix-grpc-daemon[916]: time=2026-09-25T15:24:47.390Z level=INFO msg="Uploading 1 narinfos" lb # [ 60.768960] niks3-server[955]: 2026/09/25 15:24:47 INFO Received sign narinfos request method=POST path=/api/pushes/10/sign lb # [ 60.771385] niks3-server[955]: 2026/09/25 15:24:47 INFO Signed narinfos id=10 count=1 lb # [ 60.778644] niks3-server[955]: 2026/09/25 15:24:47 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst lb # [ 60.788235] niks3-server[955]: 2026/09/25 15:24:47 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.ls lb # [ 60.790683] niks3-server[955]: 2026/09/25 15:24:47 INFO Received complete push request method=POST path=/api/pushes/10/complete worker2 # [ 60.793218] nix-grpc-daemon[916]: time=2026-09-25T15:24:47.417Z level=INFO msg="Upload complete. (192ms)" lb # [ 60.797423] niks3-server[955]: 2026/09/25 15:24:47 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.narinfo worker2 # [ 60.802186] nix-grpc-daemon[769]: ts=2026-09-25T15:24:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=1 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv assign_id=5 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker2 # [ 60.817002] nix-grpc-daemon[769]: ts=2026-09-25T15:24:47Z level=debug event=want client=4 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv cached=0 worker2 # [ 60.819675] nix-grpc-daemon[769]: ts=2026-09-25T15:24:47Z level=debug event=assigned drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 60.824378] nix-grpc-daemon[769]: ts=2026-09-25T15:24:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 60.828140] nix-daemon[857]: reaped child process 933, status = succeeded worker2 # [ 60.831359] nix-grpc-daemon[769]: ts=2026-09-25T15:24:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 60.839103] nix-grpc-daemon[916]: time=2026-09-25T15:24:47.462Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' to 'lb:50051'... worker2 # [ 60.845058] nix-grpc-daemon[769]: ts=2026-09-25T15:24:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 60.852090] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 60.967360] nix-daemon[857]: accepted connection from pid 960, user nix-grpc-daemon (trusted) worker2 # [ 60.985840] nix-daemon[857]: reaped child process 963, status = succeeded lb # [ 60.992302] niks3-server[955]: 2026/09/25 15:24:47 INFO Received push request method=POST path=/api/pushes worker2 # [ 60.994899] nix-grpc-daemon[916]: time=2026-09-25T15:24:47.618Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 60.996581] nix-grpc-daemon[916]: time=2026-09-25T15:24:47.618Z level=INFO msg="Uploading z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv (624B)" lb # [ 61.007827] niks3-server[955]: 2026/09/25 15:24:47 INFO Received sign narinfos request method=POST path=/api/pushes/11/sign worker2 # [ 61.009750] nix-grpc-daemon[916]: time=2026-09-25T15:24:47.633Z level=INFO msg="Uploading 1 narinfos" lb # [ 61.015152] niks3-server[955]: 2026/09/25 15:24:47 INFO Signed narinfos id=11 count=1 lb # [ 61.019913] niks3-server[955]: 2026/09/25 15:24:47 INFO Registered completed upload object_key=nar/0cnhkgsbmxv5hj7zw3fa59qfxikw6251mlhz2gz1v6pgn8c8shwp.nar.zst lb # [ 61.029318] niks3-server[955]: 2026/09/25 15:24:47 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.ls lb # [ 61.033688] niks3-server[955]: 2026/09/25 15:24:47 INFO Received complete push request method=POST path=/api/pushes/11/complete worker2 # [ 61.035092] nix-grpc-daemon[916]: time=2026-09-25T15:24:47.658Z level=INFO msg="Upload complete. (174ms)" lb # [ 61.037514] niks3-server[955]: 2026/09/25 15:24:47 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.narinfo worker2 # [ 61.040315] nix-grpc-daemon[769]: ts=2026-09-25T15:24:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 paths=1 nar_bytes_in=624 worker2 # [ 61.048488] nix-grpc-daemon[769]: ts=2026-09-25T15:24:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 61.056362] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 61.058502] nix-daemon[857]: reaped child process 958, status = succeeded client # building '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' on lb:50051... worker2 # [ 61.067113] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) client # farm-b> worker2: building z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-sig worker2 # [ 62.502249] nix-daemon[857]: reaped child process 967, status = succeeded worker2 # [ 62.611978] nix-daemon[857]: accepted connection from pid 983, user nix-grpc-daemon (trusted) worker2 # [ 62.627865] nix-daemon[857]: reaped child process 986, status = succeeded lb # [ 62.636133] niks3-server[955]: 2026/09/25 15:24:49 INFO Received push request method=POST path=/api/pushes worker2 # [ 62.639867] nix-grpc-daemon[916]: time=2026-09-25T15:24:49.262Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 62.641691] nix-grpc-daemon[916]: time=2026-09-25T15:24:49.262Z level=INFO msg="Uploading h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b (176B)" lb # [ 62.658399] niks3-server[955]: 2026/09/25 15:24:49 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.ls worker2 # [ 62.662946] nix-grpc-daemon[916]: time=2026-09-25T15:24:49.286Z level=INFO msg="Uploading 1 narinfos" lb # [ 62.664947] niks3-server[955]: 2026/09/25 15:24:49 INFO Registered completed upload object_key=log/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv lb # [ 62.672542] niks3-server[955]: 2026/09/25 15:24:49 INFO Received sign narinfos request method=POST path=/api/pushes/12/sign lb # [ 62.674857] niks3-server[955]: 2026/09/25 15:24:49 INFO Signed narinfos id=12 count=1 lb # [ 62.685119] niks3-server[955]: 2026/09/25 15:24:49 INFO Registered completed upload object_key=nar/0f75gpfkjj0g9xi2113yvvpdw0l16spc2f1xik1ifnzfbrvx4pfk.nar.zst worker2 # [ 62.688109] nix-grpc-daemon[916]: time=2026-09-25T15:24:49.311Z level=INFO msg="Upload complete. (211ms)" lb # [ 62.690787] niks3-server[955]: 2026/09/25 15:24:49 INFO Received complete push request method=POST path=/api/pushes/12/complete lb # [ 62.693995] niks3-server[955]: 2026/09/25 15:24:49 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.narinfo worker2 # [ 62.696079] nix-grpc-daemon[769]: ts=2026-09-25T15:24:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=1 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv assign_id=6 outputs=1 inputs=1 inputs_fetched=0 input_bytes=0 input_s=0 worker2 # [ 62.711764] nix-grpc-daemon[769]: ts=2026-09-25T15:24:49Z level=debug event=want client=4 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv cached=0 worker2 # [ 62.714870] nix-grpc-daemon[769]: ts=2026-09-25T15:24:49Z level=debug event=assigned drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 62.719486] nix-grpc-daemon[769]: ts=2026-09-25T15:24:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 62.723902] nix-daemon[857]: reaped child process 965, status = succeeded worker2 # [ 62.726812] nix-grpc-daemon[769]: ts=2026-09-25T15:24:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 62.732403] nix-grpc-daemon[916]: time=2026-09-25T15:24:49.356Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' to 'lb:50051'... worker2 # [ 62.738599] nix-grpc-daemon[769]: ts=2026-09-25T15:24:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 62.746936] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 62.860646] nix-daemon[857]: accepted connection from pid 992, user nix-grpc-daemon (trusted) lb # [ 62.881803] niks3-server[955]: 2026/09/25 15:24:49 INFO Received push request method=POST path=/api/pushes worker2 # [ 62.881620] nix-daemon[857]: reaped child process 995, status = succeeded worker2 # [ 62.884143] nix-grpc-daemon[916]: time=2026-09-25T15:24:49.507Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 62.886204] nix-grpc-daemon[916]: time=2026-09-25T15:24:49.509Z level=INFO msg="Uploading knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv (752B)" lb # [ 62.899696] niks3-server[955]: 2026/09/25 15:24:49 INFO Registered completed upload object_key=nar/0lpagyrngav67x80d1hq0njjlkmqdkvqawraz8dsp36ag50v6iwf.nar.zst worker2 # [ 62.903312] nix-grpc-daemon[916]: time=2026-09-25T15:24:49.526Z level=INFO msg="Uploading 1 narinfos" lb # [ 62.907343] niks3-server[955]: 2026/09/25 15:24:49 INFO Received sign narinfos request method=POST path=/api/pushes/13/sign lb # [ 62.913799] niks3-server[955]: 2026/09/25 15:24:49 INFO Signed narinfos id=13 count=1 lb # [ 62.921217] niks3-server[955]: 2026/09/25 15:24:49 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.ls lb # [ 62.923694] niks3-server[955]: 2026/09/25 15:24:49 INFO Received complete push request method=POST path=/api/pushes/13/complete worker2 # [ 62.928940] nix-grpc-daemon[916]: time=2026-09-25T15:24:49.552Z level=INFO msg="Upload complete. (173ms)" lb # [ 62.930573] niks3-server[955]: 2026/09/25 15:24:49 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.narinfo worker2 # [ 62.934736] nix-grpc-daemon[769]: ts=2026-09-25T15:24:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 paths=1 nar_bytes_in=752 worker2 # [ 62.944919] nix-grpc-daemon[769]: ts=2026-09-25T15:24:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 62.949213] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 62.954316] nix-daemon[857]: reaped child process 990, status = succeeded client # building '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' on lb:50051... client # farm-top> worker2: building knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker2 # [ 62.965936] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-sig worker2 # [ 64.003744] nix-daemon[857]: reaped child process 999, status = succeeded worker2 # [ 64.107103] nix-daemon[857]: accepted connection from pid 1015, user nix-grpc-daemon (trusted) worker2 # [ 64.124912] nix-daemon[857]: reaped child process 1018, status = succeeded lb # [ 64.130694] niks3-server[955]: 2026/09/25 15:24:50 INFO Received push request method=POST path=/api/pushes worker2 # [ 64.135623] nix-grpc-daemon[916]: time=2026-09-25T15:24:50.757Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 64.137471] nix-grpc-daemon[916]: time=2026-09-25T15:24:50.757Z level=INFO msg="Uploading 4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top (232B)" lb # [ 64.158207] niks3-server[955]: 2026/09/25 15:24:50 INFO Received sign narinfos request method=POST path=/api/pushes/14/sign worker2 # [ 64.161655] nix-grpc-daemon[916]: time=2026-09-25T15:24:50.785Z level=INFO msg="Uploading 1 narinfos" lb # [ 64.165700] niks3-server[955]: 2026/09/25 15:24:50 INFO Signed narinfos id=14 count=1 lb # [ 64.171028] niks3-server[955]: 2026/09/25 15:24:50 INFO Registered completed upload object_key=log/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv lb # [ 64.178430] niks3-server[955]: 2026/09/25 15:24:50 INFO Registered completed upload object_key=nar/07ar364rwqqniwi2nkdpqlgv7ws7702n5lqz703ackfmssrh40v8.nar.zst worker2 # [ 64.186167] nix-grpc-daemon[916]: time=2026-09-25T15:24:50.809Z level=INFO msg="Upload complete. (213ms)" lb # [ 64.188567] niks3-server[955]: 2026/09/25 15:24:50 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.ls lb # [ 64.191088] niks3-server[955]: 2026/09/25 15:24:50 INFO Received complete push request method=POST path=/api/pushes/14/complete worker2 # [ 64.192617] nix-grpc-daemon[769]: ts=2026-09-25T15:24:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=1 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv assign_id=7 outputs=1 inputs=2 inputs_fetched=0 input_bytes=0 input_s=0 lb # [ 64.199264] niks3-server[955]: 2026/09/25 15:24:50 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.narinfo worker2 # [ 64.206166] nix-daemon[857]: reaped child process 997, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2, in 5.04 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath, in 0.14 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker1 # [ 64.491462] nix-grpc-daemon[486]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:52702 client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked): client # /nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a client # /nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b client # /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker1 # [ 64.514096] nix-grpc-daemon[486]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:52702 worker1 # [ 64.518833] nix-grpc-daemon[486]: copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'http://lb:5751'... worker1 # [ 64.528768] nix-grpc-daemon[486]: copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'http://lb:5751'... worker1 # [ 64.536943] nix-grpc-daemon[486]: copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.19 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker2 # [ 64.695004] nix-grpc-daemon[769]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:36708 worker2 # [ 64.699580] nix-grpc-daemon[769]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:36708 client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.16 seconds) client: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top client # finding garbage collector roots... client # removing stale temporary roots file "/nix/var/nix/temproots/918" client # removing stale temporary roots file "/nix/var/nix/temproots/905" client # removing stale temporary roots file "/nix/var/nix/temproots/926" client # removing stale temporary roots file "/nix/var/nix/temproots/941" client # removing stale temporary roots file "/nix/var/nix/temproots/885" client # removing stale temporary roots file "/nix/var/nix/temproots/849" client # removing stale temporary roots file "/nix/var/nix/temproots/897" client # removing stale temporary roots file "/nix/var/nix/temproots/864" client # deleting specified paths... client # deleting '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' client # deleting unused links... client # note: hard linking is currently saving 0.0 KiB worker2 # [ 64.977711] nix-grpc-daemon[769]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 64.982946] nix-grpc-daemon[769]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 64.988657] nix-grpc-daemon[916]: time=2026-09-25T15:24:51.612Z level=INFO msg="All 1 paths already cached" worker2 # [ 64.993541] nix-grpc-daemon[769]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 64.998504] nix-grpc-daemon[769]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 65.003461] nix-grpc-daemon[769]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:56488 client # copying 3 paths... worker1 # [ 65.014649] nix-grpc-daemon[486]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:53882 client # copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'grpc://lb:50051'... worker2 # [ 65.028955] nix-grpc-daemon[769]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 65.033850] nix-grpc-daemon[769]: ts=2026-09-25T15:24:51Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 paths=1 nar_bytes_out=232 worker2 # [ 65.039270] nix-grpc-daemon[769]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 65.043327] nix-grpc-daemon[769]: ts=2026-09-25T15:24:51Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 paths=1 nar_bytes_out=176 worker1 # [ 65.050578] nix-grpc-daemon[486]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 65.054612] nix-grpc-daemon[486]: ts=2026-09-25T15:24:51Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=0 paths=1 nar_bytes_out=128 client # copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'grpc://lb:50051'... client # copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'grpc://lb:50051'... client: (finished: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.37 seconds) (finished: subtest: outputs read back through any worker carry the cache signature, in 5.90 seconds) subtest: two clients wanting the same drv build it once client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup client # [ 65.118062] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup. client # Running as unit: dedup1.service; invocation ID: 85196cd54fed48c881f208fb77325303 client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 0.04 seconds) client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup worker2 # [ 65.284337] nix-grpc-daemon[769]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:56488 client # [ 65.288986] nix[994]: don't know how to build these paths: client # [ 65.290672] nix[994]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker2 # [ 65.296783] nix-grpc-daemon[769]: ts=2026-09-25T15:24:51Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 worker2 # [ 65.299709] nix-grpc-daemon[769]: ts=2026-09-25T15:24:51Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 65.307464] nix-grpc-daemon[486]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 65.316118] nix-grpc-daemon[486]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53882 client # [ 65.323440] nix[994]: copying 1 paths to 'lb:50051'... worker1 # [ 65.324895] nix-grpc-daemon[486]: ts=2026-09-25T15:24:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 client # [ 65.324805] nix[994]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'... worker1 # [ 65.333791] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 65.466415] nix-daemon[838]: accepted connection from pid 1007, user nix-grpc-daemon (trusted) lb # [ 65.483255] niks3-server[955]: 2026/09/25 15:24:52 INFO Received push request method=POST path=/api/pushes worker1 # [ 65.486707] nix-daemon[838]: reaped child process 1010, status = succeeded worker1 # [ 65.488776] nix-grpc-daemon[850]: time=2026-09-25T15:24:52.115Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 65.490375] nix-grpc-daemon[850]: time=2026-09-25T15:24:52.115Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)" lb # [ 65.499001] niks3-server[955]: 2026/09/25 15:24:52 INFO Received sign narinfos request method=POST path=/api/pushes/15/sign worker1 # [ 65.504141] nix-grpc-daemon[850]: time=2026-09-25T15:24:52.131Z level=INFO msg="Uploading 1 narinfos" lb # [ 65.506784] niks3-server[955]: 2026/09/25 15:24:52 INFO Signed narinfos id=15 count=1 lb # [ 65.512131] niks3-server[955]: 2026/09/25 15:24:52 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls lb # [ 65.514613] niks3-server[955]: 2026/09/25 15:24:52 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst worker1 # [ 65.525716] nix-grpc-daemon[850]: time=2026-09-25T15:24:52.152Z level=INFO msg="Upload complete. (182ms)" lb # [ 65.526288] niks3-server[955]: 2026/09/25 15:24:52 INFO Received complete push request method=POST path=/api/pushes/15/complete lb # [ 65.531663] niks3-server[955]: 2026/09/25 15:24:52 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo worker1 # [ 65.532216] nix-grpc-daemon[486]: ts=2026-09-25T15:24:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=0 paths=1 nar_bytes_in=432 worker1 # [ 65.535194] nix-grpc-daemon[486]: ts=2026-09-25T15:24:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 65.546205] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 65.551729] nix-daemon[838]: reaped child process 1005, status = succeeded client # [ 65.555654] nix[994]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... worker1 # [ 65.558391] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) client # [ 66.183552] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup. client # Running as unit: dedup2.service; invocation ID: 29c15b6f4fc049178382ffaec7bcde0f client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 1.07 seconds) worker2 # [ 66.379451] nix-grpc-daemon[769]: ts=2026-09-25T15:24:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:56488 client # [ 66.384544] nix[1010]: don't know how to build these paths: client # [ 66.385449] nix[1010]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker2 # [ 66.393435] nix-grpc-daemon[769]: ts=2026-09-25T15:24:53Z level=debug event=want client=6 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 worker1 # [ 66.397667] nix-grpc-daemon[486]: ts=2026-09-25T15:24:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 client # [ 66.398716] nix[1010]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... client: must succeed: sleep 2 lb # [ 68.111864] envoy[902]: conn peer=[2001:db8:1::1]:52812 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4168 tx=1876 ms=1142 lb # [ 68.114697] envoy[902]: conn peer=[2001:db8:1::1]:52820 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9498 tx=4358 ms=4856 lb # [ 68.117293] envoy[902]: conn peer=[2001:db8:1::1]:39790 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1091 tx=2908 ms=96 lb # [ 68.119695] envoy[902]: conn peer=[2001:db8:1::1]:39810 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=329 ms=25 lb # [ 68.122199] envoy[902]: conn peer=[2001:db8:1::1]:39804 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=375 ms=36 lb # [ 68.124685] envoy[902]: conn peer=[2001:db8:1::1]:39796 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=384 tx=409 ms=45 client: (finished: must succeed: sleep 2, in 2.03 seconds) client: waiting for success: ! systemctl is-active dedup1 dedup2 worker1 # [ 69.623147] nix-daemon[838]: reaped child process 1014, status = succeeded worker1 # [ 69.687366] nix-daemon[838]: accepted connection from pid 1044, user nix-grpc-daemon (trusted) lb # [ 69.706898] niks3-server[955]: 2026/09/25 15:24:56 INFO Received push request method=POST path=/api/pushes worker1 # [ 69.708907] nix-daemon[838]: reaped child process 1047, status = succeeded worker1 # [ 69.715486] nix-grpc-daemon[850]: time=2026-09-25T15:24:56.339Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 69.717279] nix-grpc-daemon[850]: time=2026-09-25T15:24:56.339Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)" lb # [ 69.730005] niks3-server[955]: 2026/09/25 15:24:56 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls worker1 # [ 69.737959] nix-grpc-daemon[850]: time=2026-09-25T15:24:56.364Z level=INFO msg="Uploading 1 narinfos" lb # [ 69.737860] niks3-server[955]: 2026/09/25 15:24:56 INFO Received sign narinfos request method=POST path=/api/pushes/16/sign lb # [ 69.745122] niks3-server[955]: 2026/09/25 15:24:56 INFO Signed narinfos id=16 count=1 lb # [ 69.746922] niks3-server[955]: 2026/09/25 15:24:56 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv lb # [ 69.757620] niks3-server[955]: 2026/09/25 15:24:56 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst worker1 # [ 69.765221] nix-grpc-daemon[850]: time=2026-09-25T15:24:56.392Z level=INFO msg="Upload complete. (195ms)" lb # [ 69.767163] niks3-server[955]: 2026/09/25 15:24:56 INFO Received complete push request method=POST path=/api/pushes/16/complete lb # [ 69.771603] niks3-server[955]: 2026/09/25 15:24:56 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo worker1 # [ 69.772660] nix-grpc-daemon[486]: ts=2026-09-25T15:24:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 69.783296] nix-grpc-daemon[486]: ts=2026-09-25T15:24:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 69.788466] nix-daemon[838]: reaped child process 1012, status = succeeded client # [ 69.811899] systemd[1]: dedup1.service: Deactivated successfully. client # [ 69.815333] systemd[1]: dedup1.service: Consumed 150ms CPU time over 4.694s wall clock time, 41.5M memory peak, 7K incoming IP traffic, 8.3K outgoing IP traffic. client # [ 69.820889] systemd[1]: dedup2.service: Deactivated successfully. client # [ 69.823695] systemd[1]: dedup2.service: Consumed 183ms CPU time over 3.637s wall clock time, 35.2M memory peak, 5.9K incoming IP traffic, 5.7K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.07 seconds) client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; } client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.05 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.04 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: two clients wanting the same drv build it once, in 5.74 seconds) subtest: --timeout and --max-silent-time reach the builder worker2 # [ 70.994216] nix-grpc-daemon[769]: ts=2026-09-25T15:24:57Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 71.003984] nix-grpc-daemon[769]: ts=2026-09-25T15:24:57Z level=debug event=want client=7 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv cached=0 worker2 # [ 71.009058] nix-grpc-daemon[769]: ts=2026-09-25T15:24:57Z level=debug event=assigned drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 71.013657] nix-grpc-daemon[769]: ts=2026-09-25T15:24:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 71.017730] nix-grpc-daemon[769]: ts=2026-09-25T15:24:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 71.023557] nix-grpc-daemon[769]: ts=2026-09-25T15:24:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 71.033453] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 71.157616] nix-daemon[857]: accepted connection from pid 1049, user nix-grpc-daemon (trusted) worker2 # [ 71.178300] nix-daemon[857]: reaped child process 1052, status = succeeded lb # [ 71.181613] niks3-server[955]: 2026/09/25 15:24:57 INFO Received push request method=POST path=/api/pushes worker2 # [ 71.184533] nix-grpc-daemon[916]: time=2026-09-25T15:24:57.808Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 71.186081] nix-grpc-daemon[916]: time=2026-09-25T15:24:57.808Z level=INFO msg="Uploading sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv (424B)" lb # [ 71.203037] niks3-server[955]: 2026/09/25 15:24:57 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.ls worker2 # [ 71.206567] nix-grpc-daemon[916]: time=2026-09-25T15:24:57.830Z level=INFO msg="Uploading 1 narinfos" lb # [ 71.210541] niks3-server[955]: 2026/09/25 15:24:57 INFO Received sign narinfos request method=POST path=/api/pushes/17/sign lb # [ 71.214793] niks3-server[955]: 2026/09/25 15:24:57 INFO Signed narinfos id=17 count=1 lb # [ 71.220380] niks3-server[955]: 2026/09/25 15:24:57 INFO Received complete push request method=POST path=/api/pushes/17/complete worker2 # [ 71.226225] nix-grpc-daemon[916]: time=2026-09-25T15:24:57.850Z level=INFO msg="Upload complete. (183ms)" lb # [ 71.227985] niks3-server[955]: 2026/09/25 15:24:57 INFO Registered completed upload object_key=nar/1m12xmj6yzmb9y2j8ly0yvh5fyzy1nqk7ajhqwr4wn3b5h0bsch5.nar.zst lb # [ 71.234616] niks3-server[955]: 2026/09/25 15:24:57 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.narinfo worker2 # [ 71.231747] nix-grpc-daemon[769]: ts=2026-09-25T15:24:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 paths=1 nar_bytes_in=424 worker2 # [ 71.242825] nix-grpc-daemon[769]: ts=2026-09-25T15:24:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 71.248090] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 71.251511] nix-daemon[857]: reaped child process 1047, status = succeeded worker2 # [ 71.261801] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 77.075472] nix-grpc-daemon[769]: ts=2026-09-25T15:25:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=5 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv assign_id=9 outputs=0 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker2 # [ 77.100800] nix-daemon[857]: reaped child process 1054, status = succeeded worker2 # [ 77.102425] nix-daemon[857]: reaped child process 1056, status = succeeded worker1 # [ 77.276727] nix-grpc-daemon[486]: ts=2026-09-25T15:25:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53882 worker2 # [ 77.284538] nix-grpc-daemon[769]: ts=2026-09-25T15:25:03Z level=debug event=want client=8 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv cached=0 worker2 # [ 77.287545] nix-grpc-daemon[769]: ts=2026-09-25T15:25:03Z level=debug event=assigned drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 77.295346] nix-grpc-daemon[486]: ts=2026-09-25T15:25:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 77.304681] nix-grpc-daemon[486]: ts=2026-09-25T15:25:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 77.312898] nix-grpc-daemon[486]: ts=2026-09-25T15:25:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 77.321797] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 77.451546] nix-daemon[838]: accepted connection from pid 1060, user nix-grpc-daemon (trusted) worker1 # [ 77.471848] nix-daemon[838]: reaped child process 1063, status = succeeded lb # [ 77.470992] niks3-server[955]: 2026/09/25 15:25:04 INFO Received push request method=POST path=/api/pushes worker1 # [ 77.477036] nix-grpc-daemon[850]: time=2026-09-25T15:25:04.103Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 77.478609] nix-grpc-daemon[850]: time=2026-09-25T15:25:04.103Z level=INFO msg="Uploading pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv (424B)" lb # [ 77.488268] niks3-server[955]: 2026/09/25 15:25:04 INFO Received sign narinfos request method=POST path=/api/pushes/18/sign worker1 # [ 77.493724] nix-grpc-daemon[850]: time=2026-09-25T15:25:04.120Z level=INFO msg="Uploading 1 narinfos" lb # [ 77.496592] niks3-server[955]: 2026/09/25 15:25:04 INFO Signed narinfos id=18 count=1 lb # [ 77.501336] niks3-server[955]: 2026/09/25 15:25:04 INFO Registered completed upload object_key=nar/14f5yrnllfav3svmw4v4svw15r4zlf5yv1p972jy14mmq6gdw943.nar.zst lb # [ 77.508880] niks3-server[955]: 2026/09/25 15:25:04 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.ls lb # [ 77.511365] niks3-server[955]: 2026/09/25 15:25:04 INFO Received complete push request method=POST path=/api/pushes/18/complete worker1 # [ 77.518295] nix-grpc-daemon[850]: time=2026-09-25T15:25:04.145Z level=INFO msg="Upload complete. (186ms)" lb # [ 77.519461] niks3-server[955]: 2026/09/25 15:25:04 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.narinfo worker1 # [ 77.524235] nix-grpc-daemon[486]: ts=2026-09-25T15:25:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=0 paths=1 nar_bytes_in=424 worker1 # [ 77.534988] nix-grpc-daemon[486]: ts=2026-09-25T15:25:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 77.540344] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 77.544372] nix-daemon[838]: reaped child process 1058, status = succeeded worker1 # [ 77.555360] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) lb # [ 78.108948] envoy[902]: conn peer=[2001:db8:1::1]:39822 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2988 tx=1517 ms=4499 lb # [ 78.109086] envoy[902]: conn peer=[2001:db8:1::1]:39830 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1443 tx=1197 ms=3413 lb # [ 78.109138] envoy[902]: conn peer=[2001:db8:1::1]:56520 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2958 tx=934 ms=6098 worker1 # [ 83.767773] nix-grpc-daemon[486]: ts=2026-09-25T15:25:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=6 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv assign_id=10 outputs=0 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 83.788921] nix-daemon[838]: reaped child process 1065, status = succeeded worker1 # [ 83.790470] nix-daemon[838]: reaped child process 1067, status = succeeded worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: --timeout and --max-silent-time reach the builder, in 13.12 seconds) subtest: a failing build is counted with its own reason worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) worker1 # [ 84.192822] nix-grpc-daemon[486]: ts=2026-09-25T15:25:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53882 worker2 # [ 84.201889] nix-grpc-daemon[769]: ts=2026-09-25T15:25:10Z level=debug event=want client=9 drv=j4rjl229w5xzbr9614xcipiv7652fdka-fail-boom.drv cached=0 worker2 # [ 84.202132] nix-grpc-daemon[769]: ts=2026-09-25T15:25:10Z level=debug event=assigned drv=j4rjl229w5xzbr9614xcipiv7652fdka-fail-boom.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 84.209354] nix-grpc-daemon[769]: ts=2026-09-25T15:25:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 84.219414] nix-grpc-daemon[769]: ts=2026-09-25T15:25:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 84.227806] nix-grpc-daemon[769]: ts=2026-09-25T15:25:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 84.236593] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 84.366906] nix-daemon[857]: accepted connection from pid 1089, user nix-grpc-daemon (trusted) worker2 # [ 84.383826] nix-daemon[857]: reaped child process 1092, status = succeeded lb # [ 84.390595] niks3-server[955]: 2026/09/25 15:25:11 INFO Received push request method=POST path=/api/pushes worker2 # [ 84.395494] nix-grpc-daemon[916]: time=2026-09-25T15:25:11.018Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 84.397296] nix-grpc-daemon[916]: time=2026-09-25T15:25:11.018Z level=INFO msg="Uploading j4rjl229w5xzbr9614xcipiv7652fdka-fail-boom.drv (400B)" lb # [ 84.411259] niks3-server[955]: 2026/09/25 15:25:11 INFO Received sign narinfos request method=POST path=/api/pushes/19/sign worker2 # [ 84.413887] nix-grpc-daemon[916]: time=2026-09-25T15:25:11.037Z level=INFO msg="Uploading 1 narinfos" lb # [ 84.419388] niks3-server[955]: 2026/09/25 15:25:11 INFO Signed narinfos id=19 count=1 lb # [ 84.424159] niks3-server[955]: 2026/09/25 15:25:11 INFO Registered completed upload object_key=nar/0d79vqzwc4s1agk431w3csq2p4brahnzmanwj14fjf383ld797i6.nar.zst lb # [ 84.427109] niks3-server[955]: 2026/09/25 15:25:11 INFO Registered completed upload object_key=j4rjl229w5xzbr9614xcipiv7652fdka.ls lb # [ 84.438547] niks3-server[955]: 2026/09/25 15:25:11 INFO Received complete push request method=POST path=/api/pushes/19/complete worker2 # [ 84.440034] nix-grpc-daemon[916]: time=2026-09-25T15:25:11.063Z level=INFO msg="Upload complete. (194ms)" lb # [ 84.444329] niks3-server[955]: 2026/09/25 15:25:11 INFO Registered completed upload object_key=j4rjl229w5xzbr9614xcipiv7652fdka.narinfo worker2 # [ 84.447338] nix-grpc-daemon[769]: ts=2026-09-25T15:25:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 paths=1 nar_bytes_in=400 worker2 # [ 84.456259] nix-grpc-daemon[769]: ts=2026-09-25T15:25:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 84.463050] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 84.465702] nix-daemon[857]: reaped child process 1087, status = succeeded worker2 # [ 84.476449] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 84.884774] nix-grpc-daemon[769]: ts=2026-09-25T15:25:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 drv=j4rjl229w5xzbr9614xcipiv7652fdka-fail-boom.drv assign_id=11 outputs=0 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker2 # [ 84.907377] nix-daemon[857]: reaped child process 1094, status = succeeded worker2 # [ 84.909153] nix-daemon[857]: reaped child process 1096, status = succeeded worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) (finished: subtest: a failing build is counted with its own reason, in 1.24 seconds) subtest: the first client leaving does not fail the second client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share client # [ 85.225771] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share. client # Running as unit: share1.service; invocation ID: fccd3bc6337240718c93cd914f81a4d3 client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 0.05 seconds) client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share worker1 # [ 85.413483] nix-grpc-daemon[486]: ts=2026-09-25T15:25:12Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53882 client # [ 85.416508] nix[1096]: don't know how to build these paths: client # [ 85.417418] nix[1096]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker2 # [ 85.424190] nix-grpc-daemon[769]: ts=2026-09-25T15:25:12Z level=debug event=want client=10 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 worker1 # [ 85.427563] nix-grpc-daemon[486]: ts=2026-09-25T15:25:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker2 # [ 85.427201] nix-grpc-daemon[769]: ts=2026-09-25T15:25:12Z level=debug event=assigned drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 85.436360] nix-grpc-daemon[486]: ts=2026-09-25T15:25:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53882 client # [ 85.442684] nix[1096]: copying 1 paths to 'lb:50051'... worker1 # [ 85.444300] nix-grpc-daemon[486]: ts=2026-09-25T15:25:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 client # [ 85.443890] nix[1096]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' to 'lb:50051'... worker1 # [ 85.452874] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 85.579137] nix-daemon[838]: accepted connection from pid 1108, user nix-grpc-daemon (trusted) lb # [ 85.595401] niks3-server[955]: 2026/09/25 15:25:12 INFO Received push request method=POST path=/api/pushes worker1 # [ 85.597938] nix-daemon[838]: reaped child process 1111, status = succeeded worker1 # [ 85.600899] nix-grpc-daemon[850]: time=2026-09-25T15:25:12.227Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 85.602440] nix-grpc-daemon[850]: time=2026-09-25T15:25:12.227Z level=INFO msg="Uploading rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv (432B)" lb # [ 85.611815] niks3-server[955]: 2026/09/25 15:25:12 INFO Registered completed upload object_key=nar/0chdjpc9an2ckdblh08lnyrmzvw3h8qs76cn5ydwjwvgwkrm731l.nar.zst worker1 # [ 85.617799] nix-grpc-daemon[850]: time=2026-09-25T15:25:12.244Z level=INFO msg="Uploading 1 narinfos" lb # [ 85.618992] niks3-server[955]: 2026/09/25 15:25:12 INFO Received sign narinfos request method=POST path=/api/pushes/20/sign lb # [ 85.625335] niks3-server[955]: 2026/09/25 15:25:12 INFO Signed narinfos id=20 count=1 lb # [ 85.627527] niks3-server[955]: 2026/09/25 15:25:12 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.ls lb # [ 85.637814] niks3-server[955]: 2026/09/25 15:25:12 INFO Received complete push request method=POST path=/api/pushes/20/complete worker1 # [ 85.641302] nix-grpc-daemon[850]: time=2026-09-25T15:25:12.268Z level=INFO msg="Upload complete. (179ms)" lb # [ 85.642259] niks3-server[955]: 2026/09/25 15:25:12 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.narinfo worker1 # [ 85.647280] nix-grpc-daemon[486]: ts=2026-09-25T15:25:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=0 paths=1 nar_bytes_in=432 worker1 # [ 85.657007] nix-grpc-daemon[486]: ts=2026-09-25T15:25:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 85.661204] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 85.665258] nix-daemon[838]: reaped child process 1106, status = succeeded client # [ 85.670527] nix[1096]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... worker1 # [ 85.674960] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) client # [ 86.305877] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share. client # Running as unit: share2.service; invocation ID: 9d4416d26069475481dea6d2df06f50e client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 1.08 seconds) client # [ 86.485293] nix[1112]: don't know how to build these paths: worker2 # [ 86.483481] nix-grpc-daemon[769]: ts=2026-09-25T15:25:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:56488 client # [ 86.486470] nix[1112]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker2 # [ 86.490603] nix-grpc-daemon[769]: ts=2026-09-25T15:25:13Z level=debug event=want client=11 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 worker1 # [ 86.495279] nix-grpc-daemon[486]: ts=2026-09-25T15:25:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 client # [ 86.496423] nix[1112]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... client: must succeed: sleep 2; systemctl kill -s INT share1 lb # [ 88.112842] envoy[902]: conn peer=[2001:db8:1::1]:56524 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2972 tx=932 ms=6507 lb # [ 88.115491] envoy[902]: conn peer=[2001:db8:1::1]:41392 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2868 tx=1248 ms=710 client # [ 89.562268] systemd[1]: share1.service: Sent signal SIGINT to main process 1096 (nix) on client request. client # [ 89.567639] nix[1096]: shutting downrxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client # [ 89.569703] nix[1096]: error: interrupted by the user client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.05 seconds) client: waiting for success: ! systemctl is-active share1 client # [ 89.623445] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE client # [ 89.625680] systemd[1]: share1.service: Failed with result 'exit-code'. client # [ 89.629038] systemd[1]: share1.service: Consumed 157ms CPU time over 4.402s wall clock time, 41.4M memory peak, 6K incoming IP traffic, 8.1K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active share1, in 1.08 seconds) client: waiting for success: ! systemctl is-active share2 worker1 # [ 90.904706] nix-daemon[838]: reaped child process 1115, status = succeeded worker1 # [ 91.013744] nix-daemon[838]: accepted connection from pid 1149, user nix-grpc-daemon (trusted) worker1 # [ 91.031021] nix-daemon[838]: reaped child process 1152, status = succeeded lb # [ 91.036509] niks3-server[955]: 2026/09/25 15:25:17 INFO Received push request method=POST path=/api/pushes worker1 # [ 91.042837] nix-grpc-daemon[850]: time=2026-09-25T15:25:17.668Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 91.044430] nix-grpc-daemon[850]: time=2026-09-25T15:25:17.668Z level=INFO msg="Uploading fjww0fd4rwk1s35jav3cyal62l24n356-slow-share (120B)" lb # [ 91.062589] niks3-server[955]: 2026/09/25 15:25:17 INFO Received sign narinfos request method=POST path=/api/pushes/21/sign worker1 # [ 91.068273] nix-grpc-daemon[850]: time=2026-09-25T15:25:17.695Z level=INFO msg="Uploading 1 narinfos" lb # [ 91.071452] niks3-server[955]: 2026/09/25 15:25:17 INFO Signed narinfos id=21 count=1 lb # [ 91.076172] niks3-server[955]: 2026/09/25 15:25:17 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.ls lb # [ 91.078692] niks3-server[955]: 2026/09/25 15:25:17 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst worker1 # [ 91.089976] nix-grpc-daemon[850]: time=2026-09-25T15:25:17.717Z level=INFO msg="Upload complete. (205ms)" lb # [ 91.090772] niks3-server[955]: 2026/09/25 15:25:17 INFO Registered completed upload object_key=log/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv lb # [ 91.097065] niks3-server[955]: 2026/09/25 15:25:17 INFO Received complete push request method=POST path=/api/pushes/21/complete lb # [ 91.099938] niks3-server[955]: 2026/09/25 15:25:17 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.narinfo worker1 # [ 91.096557] nix-grpc-daemon[486]: ts=2026-09-25T15:25:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=5 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=12 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 91.109058] nix-grpc-daemon[486]: ts=2026-09-25T15:25:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=4 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=12 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 91.114131] nix-daemon[838]: reaped child process 1113, status = succeeded client # [ 91.123765] systemd[1]: share2.service: Deactivated successfully. client # [ 91.126169] systemd[1]: share2.service: Consumed 161ms CPU time over 4.820s wall clock time, 35.3M memory peak, 6K incoming IP traffic, 5.6K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active share2, in 1.07 seconds) client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; } client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.04 seconds) (finished: subtest: the first client leaving does not fail the second, in 6.75 seconds) subtest: build hook: nix-daemon with builders = grpc://lb client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log client # [ 92.062646] systemd[1]: Started Nix Daemon. client # [ 92.174572] nix-daemon[1164]: accepted connection from pid 1161, user root (trusted) worker2 # [ 92.316398] nix-grpc-daemon[769]: ts=2026-09-25T15:25:18Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 92.320186] nix-grpc-daemon[769]: ts=2026-09-25T15:25:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 92.332518] nix-grpc-daemon[769]: ts=2026-09-25T15:25:18Z level=debug event=want client=12 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=0 worker2 # [ 92.337844] nix-grpc-daemon[769]: ts=2026-09-25T15:25:18Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 92.342619] nix-grpc-daemon[769]: ts=2026-09-25T15:25:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 92.347162] nix-grpc-daemon[769]: ts=2026-09-25T15:25:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 92.353759] nix-grpc-daemon[769]: ts=2026-09-25T15:25:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 92.362423] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 92.488137] nix-daemon[857]: accepted connection from pid 1148, user nix-grpc-daemon (trusted) lb # [ 92.508153] niks3-server[955]: 2026/09/25 15:25:19 INFO Received push request method=POST path=/api/pushes worker2 # [ 92.508620] nix-daemon[857]: reaped child process 1151, status = succeeded worker2 # [ 92.512162] nix-grpc-daemon[916]: time=2026-09-25T15:25:19.135Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 92.513697] nix-grpc-daemon[916]: time=2026-09-25T15:25:19.135Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)" lb # [ 92.525159] niks3-server[955]: 2026/09/25 15:25:19 INFO Received sign narinfos request method=POST path=/api/pushes/22/sign worker2 # [ 92.528440] nix-grpc-daemon[916]: time=2026-09-25T15:25:19.152Z level=INFO msg="Uploading 1 narinfos" lb # [ 92.533670] niks3-server[955]: 2026/09/25 15:25:19 INFO Signed narinfos id=22 count=1 lb # [ 92.535483] niks3-server[955]: 2026/09/25 15:25:19 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst lb # [ 92.545949] niks3-server[955]: 2026/09/25 15:25:19 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls lb # [ 92.549770] niks3-server[955]: 2026/09/25 15:25:19 INFO Received complete push request method=POST path=/api/pushes/22/complete worker2 # [ 92.551729] nix-grpc-daemon[916]: time=2026-09-25T15:25:19.175Z level=INFO msg="Upload complete. (179ms)" lb # [ 92.557571] niks3-server[955]: 2026/09/25 15:25:19 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo worker2 # [ 92.559511] nix-grpc-daemon[769]: ts=2026-09-25T15:25:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 paths=1 nar_bytes_in=512 worker2 # [ 92.566606] nix-grpc-daemon[769]: ts=2026-09-25T15:25:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 92.571694] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 92.575385] nix-daemon[857]: reaped child process 1146, status = succeeded worker2 # [ 92.584251] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 94.232849] nix-daemon[857]: reaped child process 1155, status = succeeded worker2 # [ 94.338426] nix-daemon[857]: accepted connection from pid 1172, user nix-grpc-daemon (trusted) lb # [ 94.358579] niks3-server[955]: 2026/09/25 15:25:20 INFO Received push request method=POST path=/api/pushes worker2 # [ 94.358993] nix-daemon[857]: reaped child process 1175, status = succeeded worker2 # [ 94.364404] nix-grpc-daemon[916]: time=2026-09-25T15:25:20.986Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 94.365919] nix-grpc-daemon[916]: time=2026-09-25T15:25:20.986Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)" lb # [ 94.388345] niks3-server[955]: 2026/09/25 15:25:21 INFO Received sign narinfos request method=POST path=/api/pushes/23/sign worker2 # [ 94.391109] nix-grpc-daemon[916]: time=2026-09-25T15:25:21.014Z level=INFO msg="Uploading 1 narinfos" lb # [ 94.397536] niks3-server[955]: 2026/09/25 15:25:21 INFO Signed narinfos id=23 count=1 lb # [ 94.402245] niks3-server[955]: 2026/09/25 15:25:21 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv lb # [ 94.411062] niks3-server[955]: 2026/09/25 15:25:21 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst lb # [ 94.421593] niks3-server[955]: 2026/09/25 15:25:21 INFO Received complete push request method=POST path=/api/pushes/23/complete worker2 # [ 94.423419] nix-grpc-daemon[916]: time=2026-09-25T15:25:21.047Z level=INFO msg="Upload complete. (229ms)" lb # [ 94.425748] niks3-server[955]: 2026/09/25 15:25:21 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls lb # [ 94.428508] niks3-server[955]: 2026/09/25 15:25:21 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo worker2 # [ 94.430300] nix-grpc-daemon[769]: ts=2026-09-25T15:25:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=13 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker2 # [ 94.442847] nix-daemon[857]: reaped child process 1153, status = succeeded worker1 # [ 94.449204] nix-grpc-daemon[486]: ts=2026-09-25T15:25:21Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 94.456659] nix-grpc-daemon[486]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'... worker1 # [ 94.478622] nix-grpc-daemon[486]: ts=2026-09-25T15:25:21Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 94.482594] nix-grpc-daemon[486]: ts=2026-09-25T15:25:21Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=0 paths=1 nar_bytes_out=128 worker2 # [ 94.674783] nix-grpc-daemon[769]: ts=2026-09-25T15:25:21Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 94.679149] nix-grpc-daemon[769]: ts=2026-09-25T15:25:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 94.683164] nix-grpc-daemon[916]: time=2026-09-25T15:25:21.306Z level=INFO msg="All 1 paths already cached" worker2 # [ 94.690800] nix-grpc-daemon[769]: ts=2026-09-25T15:25:21Z level=debug event=want client=13 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=0 worker2 # [ 94.694547] nix-grpc-daemon[769]: ts=2026-09-25T15:25:21Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 94.699501] nix-grpc-daemon[769]: ts=2026-09-25T15:25:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 94.705600] nix-grpc-daemon[769]: ts=2026-09-25T15:25:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 94.713436] nix-grpc-daemon[916]: time=2026-09-25T15:25:21.337Z level=INFO msg="All 1 paths already cached" worker2 # [ 94.718598] nix-grpc-daemon[769]: ts=2026-09-25T15:25:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 94.726676] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 94.845037] nix-daemon[857]: accepted connection from pid 1181, user nix-grpc-daemon (trusted) lb # [ 94.865771] niks3-server[955]: 2026/09/25 15:25:21 INFO Received push request method=POST path=/api/pushes worker2 # [ 94.865464] nix-daemon[857]: reaped child process 1184, status = succeeded worker2 # [ 94.868843] nix-grpc-daemon[916]: time=2026-09-25T15:25:21.492Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 94.870353] nix-grpc-daemon[916]: time=2026-09-25T15:25:21.492Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)" lb # [ 94.883919] niks3-server[955]: 2026/09/25 15:25:21 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls worker2 # [ 94.888198] nix-grpc-daemon[916]: time=2026-09-25T15:25:21.511Z level=INFO msg="Uploading 1 narinfos" lb # [ 94.892395] niks3-server[955]: 2026/09/25 15:25:21 INFO Received sign narinfos request method=POST path=/api/pushes/24/sign lb # [ 94.897248] niks3-server[955]: 2026/09/25 15:25:21 INFO Signed narinfos id=24 count=1 lb # [ 94.899020] niks3-server[955]: 2026/09/25 15:25:21 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst lb # [ 94.911733] niks3-server[955]: 2026/09/25 15:25:21 INFO Received complete push request method=POST path=/api/pushes/24/complete lb # [ 94.916138] niks3-server[955]: 2026/09/25 15:25:21 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo worker2 # [ 94.917566] nix-grpc-daemon[916]: time=2026-09-25T15:25:21.541Z level=INFO msg="Upload complete. (181ms)" worker2 # [ 94.923671] nix-grpc-daemon[769]: ts=2026-09-25T15:25:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 paths=1 nar_bytes_in=624 worker2 # [ 94.933844] nix-grpc-daemon[769]: ts=2026-09-25T15:25:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 94.940754] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 94.943068] nix-daemon[857]: reaped child process 1179, status = succeeded worker2 # [ 94.952308] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 95.621695] nix-daemon[857]: reaped child process 1188, status = succeeded worker2 # [ 95.675169] nix-daemon[857]: accepted connection from pid 1204, user nix-grpc-daemon (trusted) worker2 # [ 95.691259] nix-daemon[857]: reaped child process 1207, status = succeeded lb # [ 95.699800] niks3-server[955]: 2026/09/25 15:25:22 INFO Received push request method=POST path=/api/pushes worker2 # [ 95.704380] nix-grpc-daemon[916]: time=2026-09-25T15:25:22.326Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 95.705913] nix-grpc-daemon[916]: time=2026-09-25T15:25:22.326Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)" lb # [ 95.718576] niks3-server[955]: 2026/09/25 15:25:22 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst lb # [ 95.730429] niks3-server[955]: 2026/09/25 15:25:22 INFO Received sign narinfos request method=POST path=/api/pushes/25/sign worker2 # [ 95.730908] nix-grpc-daemon[916]: time=2026-09-25T15:25:22.354Z level=INFO msg="Uploading 1 narinfos" lb # [ 95.736054] niks3-server[955]: 2026/09/25 15:25:22 INFO Signed narinfos id=25 count=1 lb # [ 95.740109] niks3-server[955]: 2026/09/25 15:25:22 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv lb # [ 95.747059] niks3-server[955]: 2026/09/25 15:25:22 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls lb # [ 95.755375] niks3-server[955]: 2026/09/25 15:25:22 INFO Received complete push request method=POST path=/api/pushes/25/complete worker2 # [ 95.757151] nix-grpc-daemon[916]: time=2026-09-25T15:25:22.380Z level=INFO msg="Upload complete. (194ms)" lb # [ 95.761701] niks3-server[955]: 2026/09/25 15:25:22 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo worker2 # [ 95.764348] nix-grpc-daemon[769]: ts=2026-09-25T15:25:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=14 outputs=1 inputs=1 inputs_fetched=0 input_bytes=0 input_s=0 worker2 # [ 95.776394] nix-daemon[857]: reaped child process 1186, status = succeeded worker1 # [ 95.778761] nix-grpc-daemon[486]: ts=2026-09-25T15:25:22Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 95.786283] nix-grpc-daemon[486]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'... worker1 # [ 95.807585] nix-grpc-daemon[486]: ts=2026-09-25T15:25:22Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 95.811527] nix-grpc-daemon[486]: ts=2026-09-25T15:25:22Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=0 paths=1 nar_bytes_out=176 worker2 # [ 96.008263] nix-grpc-daemon[769]: ts=2026-09-25T15:25:22Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 96.011996] nix-grpc-daemon[769]: ts=2026-09-25T15:25:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 96.016650] nix-grpc-daemon[916]: time=2026-09-25T15:25:22.639Z level=INFO msg="All 2 paths already cached" worker2 # [ 96.024725] nix-grpc-daemon[769]: ts=2026-09-25T15:25:22Z level=debug event=want client=14 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=0 worker2 # [ 96.028563] nix-grpc-daemon[769]: ts=2026-09-25T15:25:22Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 96.033325] nix-grpc-daemon[769]: ts=2026-09-25T15:25:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 96.039308] nix-grpc-daemon[769]: ts=2026-09-25T15:25:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 96.046844] nix-grpc-daemon[916]: time=2026-09-25T15:25:22.670Z level=INFO msg="All 2 paths already cached" worker2 # [ 96.052640] nix-grpc-daemon[769]: ts=2026-09-25T15:25:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 96.061420] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 96.184103] nix-daemon[857]: accepted connection from pid 1213, user nix-grpc-daemon (trusted) lb # [ 96.204491] niks3-server[955]: 2026/09/25 15:25:22 INFO Received push request method=POST path=/api/pushes worker2 # [ 96.204777] nix-daemon[857]: reaped child process 1216, status = succeeded worker2 # [ 96.207423] nix-grpc-daemon[916]: time=2026-09-25T15:25:22.830Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 96.210344] nix-grpc-daemon[916]: time=2026-09-25T15:25:22.830Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)" lb # [ 96.253689] niks3-server[955]: 2026/09/25 15:25:22 INFO Received sign narinfos request method=POST path=/api/pushes/26/sign worker2 # [ 96.254689] nix-grpc-daemon[916]: time=2026-09-25T15:25:22.878Z level=INFO msg="Uploading 1 narinfos" lb # [ 96.258955] niks3-server[955]: 2026/09/25 15:25:22 INFO Signed narinfos id=26 count=1 lb # [ 96.280897] niks3-server[955]: 2026/09/25 15:25:22 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls lb # [ 96.295044] niks3-server[955]: 2026/09/25 15:25:22 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst lb # [ 96.312454] niks3-server[955]: 2026/09/25 15:25:22 INFO Received complete push request method=POST path=/api/pushes/26/complete worker2 # [ 96.314611] nix-grpc-daemon[916]: time=2026-09-25T15:25:22.938Z level=INFO msg="Upload complete. (244ms)" lb # [ 96.319082] niks3-server[955]: 2026/09/25 15:25:22 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo worker2 # [ 96.323203] nix-grpc-daemon[769]: ts=2026-09-25T15:25:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 paths=1 nar_bytes_in=752 worker2 # [ 96.331134] nix-grpc-daemon[769]: ts=2026-09-25T15:25:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 96.349976] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 96.366540] nix-daemon[857]: reaped child process 1211, status = succeeded worker2 # [ 96.372788] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 97.213791] nix-daemon[857]: reaped child process 1220, status = succeeded worker2 # [ 97.303006] nix-daemon[857]: accepted connection from pid 1236, user nix-grpc-daemon (trusted) worker2 # [ 97.320665] nix-daemon[857]: reaped child process 1239, status = succeeded lb # [ 97.327304] niks3-server[955]: 2026/09/25 15:25:23 INFO Received push request method=POST path=/api/pushes worker2 # [ 97.331734] nix-grpc-daemon[916]: time=2026-09-25T15:25:23.954Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 97.333483] nix-grpc-daemon[916]: time=2026-09-25T15:25:23.954Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)" lb # [ 97.350956] niks3-server[955]: 2026/09/25 15:25:23 INFO Received sign narinfos request method=POST path=/api/pushes/27/sign worker2 # [ 97.352638] nix-grpc-daemon[916]: time=2026-09-25T15:25:23.976Z level=INFO msg="Uploading 1 narinfos" lb # [ 97.358313] niks3-server[955]: 2026/09/25 15:25:23 INFO Signed narinfos id=27 count=1 lb # [ 97.363828] niks3-server[955]: 2026/09/25 15:25:23 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst lb # [ 97.370968] niks3-server[955]: 2026/09/25 15:25:23 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls lb # [ 97.376207] niks3-server[955]: 2026/09/25 15:25:23 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker2 # [ 97.383540] nix-grpc-daemon[916]: time=2026-09-25T15:25:24.007Z level=INFO msg="Upload complete. (200ms)" lb # [ 97.385924] niks3-server[955]: 2026/09/25 15:25:24 INFO Received complete push request method=POST path=/api/pushes/27/complete lb # [ 97.392077] niks3-server[955]: 2026/09/25 15:25:24 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo worker2 # [ 97.392655] nix-grpc-daemon[769]: ts=2026-09-25T15:25:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=15 outputs=1 inputs=2 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 97.396936] nix-grpc-daemon[486]: ts=2026-09-25T15:25:24Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:53882 worker2 # [ 97.403019] nix-daemon[857]: reaped child process 1218, status = succeeded worker1 # [ 97.405113] nix-grpc-daemon[486]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'... worker2 # [ 97.426254] nix-grpc-daemon[769]: ts=2026-09-25T15:25:24Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 97.430248] nix-grpc-daemon[769]: ts=2026-09-25T15:25:24Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 paths=1 nar_bytes_out=232 client # [ 97.495509] nix-daemon[1164]: reaped child process 1166, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log, in 5.56 seconds) client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.02 seconds) client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.04 seconds) client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds) (finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 5.65 seconds) subtest: an input only one worker has still reaches the builder client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only client # this derivation will be built: client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only, in 0.29 seconds) client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.12 seconds) worker1: must succeed: nix-store --import < /tmp/shared/inp.closure lb # [ 98.116336] envoy[902]: conn peer=[2001:db8:1::1]:41408 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2986 tx=792 ms=4171 lb # [ 98.118997] envoy[902]: conn peer=[2001:db8:1::1]:41420 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1443 tx=1197 ms=4627 lb # [ 98.121625] envoy[902]: conn peer=[2001:db8:1::1]:51962 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=333 ms=20 lb # [ 98.124038] envoy[902]: conn peer=[2001:db8:1::1]:51956 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3397 tx=1924 ms=2180 lb # [ 98.126539] envoy[902]: conn peer=[2001:db8:1::1]:51964 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3845 tx=2136 ms=1145 lb # [ 98.129126] envoy[902]: conn peer=[2001:db8:1::1]:51966 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=375 ms=15 lb # [ 98.131512] envoy[902]: conn peer=[2001:db8:1::1]:51968 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4378 tx=2390 ms=1433 lb # [ 98.134091] envoy[902]: conn peer=[2001:db8:1::1]:51984 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=412 tx=406 ms=17 worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.15 seconds) worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.02 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2 client # [ 98.310617] nix-daemon[1164]: accepted connection from pid 1247, user root (trusted) client # this derivation will be built: client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker2 # [ 98.450975] nix-grpc-daemon[769]: ts=2026-09-25T15:25:25Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:56488 client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker2 # [ 98.456136] nix-grpc-daemon[769]: ts=2026-09-25T15:25:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 client # copying 1 paths... client # copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' to 'grpc://lb:50051'... worker2 # [ 98.463049] nix-grpc-daemon[769]: ts=2026-09-25T15:25:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 98.471989] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 98.603525] nix-daemon[857]: accepted connection from pid 1246, user nix-grpc-daemon (trusted) worker2 # [ 98.620453] nix-daemon[857]: reaped child process 1249, status = succeeded lb # [ 98.629077] niks3-server[955]: 2026/09/25 15:25:25 INFO Received push request method=POST path=/api/pushes worker2 # [ 98.632484] nix-grpc-daemon[916]: time=2026-09-25T15:25:25.255Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 98.634127] nix-grpc-daemon[916]: time=2026-09-25T15:25:25.255Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)" lb # [ 98.646473] niks3-server[955]: 2026/09/25 15:25:25 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst worker2 # [ 98.649919] nix-grpc-daemon[916]: time=2026-09-25T15:25:25.273Z level=INFO msg="Uploading 1 narinfos" lb # [ 98.654721] niks3-server[955]: 2026/09/25 15:25:25 INFO Received sign narinfos request method=POST path=/api/pushes/28/sign lb # [ 98.660408] niks3-server[955]: 2026/09/25 15:25:25 INFO Signed narinfos id=28 count=1 lb # [ 98.662167] niks3-server[955]: 2026/09/25 15:25:25 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls lb # [ 98.671587] niks3-server[955]: 2026/09/25 15:25:25 INFO Received complete push request method=POST path=/api/pushes/28/complete worker2 # [ 98.673154] nix-grpc-daemon[916]: time=2026-09-25T15:25:25.296Z level=INFO msg="Upload complete. (192ms)" lb # [ 98.676828] niks3-server[955]: 2026/09/25 15:25:25 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo worker2 # [ 98.680110] nix-grpc-daemon[769]: ts=2026-09-25T15:25:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 paths=1 nar_bytes_in=120 worker2 # [ 98.689067] nix-grpc-daemon[769]: ts=2026-09-25T15:25:25Z level=debug event=want client=15 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=0 worker1 # [ 98.694051] nix-grpc-daemon[486]: ts=2026-09-25T15:25:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker2 # [ 98.691956] nix-grpc-daemon[769]: ts=2026-09-25T15:25:25Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.3:50051 clients=1 worker2 # [ 98.697515] nix-daemon[857]: reaped child process 1244, status = succeeded worker1 # [ 98.703909] nix-grpc-daemon[486]: ts=2026-09-25T15:25:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53882 client # copying 2 paths to 'lb:50051'... worker1 # [ 98.714696] nix-grpc-daemon[486]: ts=2026-09-25T15:25:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'... client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'... worker1 # [ 98.724061] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 98.855378] nix-daemon[838]: accepted connection from pid 1164, user nix-grpc-daemon (trusted) lb # [ 98.872472] niks3-server[955]: 2026/09/25 15:25:25 INFO Received push request method=POST path=/api/pushes worker1 # [ 98.875554] nix-daemon[838]: reaped child process 1167, status = succeeded worker1 # [ 98.879744] nix-grpc-daemon[850]: time=2026-09-25T15:25:25.504Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker1 # [ 98.882129] nix-grpc-daemon[850]: time=2026-09-25T15:25:25.504Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)" worker1 # [ 98.884723] nix-grpc-daemon[850]: time=2026-09-25T15:25:25.504Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)" lb # [ 98.897422] niks3-server[955]: 2026/09/25 15:25:25 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst lb # [ 98.905895] niks3-server[955]: 2026/09/25 15:25:25 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst lb # [ 98.916452] niks3-server[955]: 2026/09/25 15:25:25 INFO Received sign narinfos request method=POST path=/api/pushes/29/sign worker1 # [ 98.919911] nix-grpc-daemon[850]: time=2026-09-25T15:25:25.546Z level=INFO msg="Uploading 2 narinfos" lb # [ 98.923851] niks3-server[955]: 2026/09/25 15:25:25 INFO Signed narinfos id=29 count=2 lb # [ 98.927371] niks3-server[955]: 2026/09/25 15:25:25 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls lb # [ 98.932868] niks3-server[955]: 2026/09/25 15:25:25 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls lb # [ 98.939463] niks3-server[955]: 2026/09/25 15:25:25 INFO Received complete push request method=POST path=/api/pushes/29/complete worker1 # [ 98.945439] nix-grpc-daemon[850]: time=2026-09-25T15:25:25.572Z level=INFO msg="Upload complete. (210ms)" lb # [ 98.947467] niks3-server[955]: 2026/09/25 15:25:25 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo lb # [ 98.952470] niks3-server[955]: 2026/09/25 15:25:25 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo worker1 # [ 98.952611] nix-grpc-daemon[486]: ts=2026-09-25T15:25:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=0 paths=2 nar_bytes_in=984 worker1 # [ 98.955955] nix-grpc-daemon[486]: ts=2026-09-25T15:25:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 98.967971] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 98.971944] nix-daemon[838]: reaped child process 1162, status = succeeded client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051... worker1 # [ 98.983750] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) client # uses-input-w1onlya> node-a: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 99.815007] nix-daemon[838]: reaped child process 1171, status = succeeded worker1 # [ 99.917759] nix-daemon[838]: accepted connection from pid 1187, user nix-grpc-daemon (trusted) worker1 # [ 99.934854] nix-daemon[838]: reaped child process 1190, status = succeeded lb # [ 99.941128] niks3-server[955]: 2026/09/25 15:25:26 INFO Received push request method=POST path=/api/pushes worker1 # [ 99.947735] nix-grpc-daemon[850]: time=2026-09-25T15:25:26.573Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 99.958427] niks3-server[955]: 2026/09/25 15:25:26 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls worker1 # [ 99.965492] nix-grpc-daemon[850]: time=2026-09-25T15:25:26.592Z level=INFO msg="Uploading 1 narinfos" lb # [ 99.966054] niks3-server[955]: 2026/09/25 15:25:26 INFO Received sign narinfos request method=POST path=/api/pushes/30/sign lb # [ 99.972680] niks3-server[955]: 2026/09/25 15:25:26 INFO Signed narinfos id=30 count=1 lb # [ 99.974423] niks3-server[955]: 2026/09/25 15:25:26 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv lb # [ 99.984242] niks3-server[955]: 2026/09/25 15:25:26 INFO Received complete push request method=POST path=/api/pushes/30/complete worker1 # [ 99.988113] nix-grpc-daemon[850]: time=2026-09-25T15:25:26.615Z level=INFO msg="Upload complete. (196ms)" lb # [ 99.989686] niks3-server[955]: 2026/09/25 15:25:26 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo client # copying 1 paths... worker1 # [ 99.995191] nix-grpc-daemon[486]: ts=2026-09-25T15:25:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=16 outputs=1 inputs=1 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 100.008465] nix-grpc-daemon[486]: ts=2026-09-25T15:25:26Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 100.011882] nix-daemon[838]: reaped child process 1169, status = succeeded client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'... worker2 # [ 100.024709] nix-grpc-daemon[769]: ts=2026-09-25T15:25:26Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 100.033051] nix-grpc-daemon[769]: copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'http://lb:5751'... worker2 # [ 100.043345] nix-grpc-daemon[769]: ts=2026-09-25T15:25:26Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 paths=1 nar_bytes_out=120 client # [ 100.107395] nix-daemon[1164]: reaped child process 1250, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2, in 1.94 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2 client # [ 100.241667] nix-daemon[1164]: accepted connection from pid 1269, user root (trusted) client # this derivation will be built: client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker2 # [ 100.378533] nix-grpc-daemon[769]: ts=2026-09-25T15:25:27Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:56488 client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker2 # [ 100.383414] nix-grpc-daemon[769]: ts=2026-09-25T15:25:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 100.389817] nix-grpc-daemon[916]: time=2026-09-25T15:25:27.013Z level=INFO msg="All 1 paths already cached" worker2 # [ 100.400538] nix-grpc-daemon[769]: ts=2026-09-25T15:25:27Z level=debug event=want client=16 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=0 worker2 # [ 100.404483] nix-grpc-daemon[769]: ts=2026-09-25T15:25:27Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 100.409223] nix-grpc-daemon[769]: ts=2026-09-25T15:25:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 100.415195] nix-grpc-daemon[769]: ts=2026-09-25T15:25:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56488 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'... worker2 # [ 100.426151] nix-grpc-daemon[769]: ts=2026-09-25T15:25:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 100.434867] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 100.449114] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 52 ms (attempt 1/5) worker2 # [ 100.506079] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 151 ms (attempt 2/5) worker2 # [ 100.662144] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 205 ms (attempt 3/5) worker2 # [ 100.868272] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 404 ms (attempt 4/5) worker2 # [ 101.274374] nix-grpc-daemon[769]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 101.277735] nix-grpc-daemon[769]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 101.283758] nix-grpc-daemon[769]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'... worker2 # [ 101.422835] nix-daemon[857]: accepted connection from pid 1263, user nix-grpc-daemon (trusted) worker2 # [ 101.445275] nix-daemon[857]: reaped child process 1266, status = succeeded lb # [ 101.448704] niks3-server[955]: 2026/09/25 15:25:28 INFO Received push request method=POST path=/api/pushes worker2 # [ 101.452561] nix-grpc-daemon[916]: time=2026-09-25T15:25:28.075Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 101.454968] nix-grpc-daemon[916]: time=2026-09-25T15:25:28.075Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)" lb # [ 101.469408] niks3-server[955]: 2026/09/25 15:25:28 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst worker2 # [ 101.473782] nix-grpc-daemon[916]: time=2026-09-25T15:25:28.097Z level=INFO msg="Uploading 1 narinfos" lb # [ 101.478525] niks3-server[955]: 2026/09/25 15:25:28 INFO Received sign narinfos request method=POST path=/api/pushes/31/sign lb # [ 101.483933] niks3-server[955]: 2026/09/25 15:25:28 INFO Signed narinfos id=31 count=1 lb # [ 101.487910] niks3-server[955]: 2026/09/25 15:25:28 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls worker2 # [ 101.494867] nix-grpc-daemon[916]: time=2026-09-25T15:25:28.118Z level=INFO msg="Upload complete. (200ms)" lb # [ 101.497183] niks3-server[955]: 2026/09/25 15:25:28 INFO Received complete push request method=POST path=/api/pushes/31/complete lb # [ 101.501078] niks3-server[955]: 2026/09/25 15:25:28 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo worker2 # [ 101.500186] nix-grpc-daemon[769]: ts=2026-09-25T15:25:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=1 paths=1 nar_bytes_in=568 worker2 # [ 101.508481] nix-grpc-daemon[769]: ts=2026-09-25T15:25:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 101.519725] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) worker2 # [ 101.522199] nix-daemon[857]: reaped child process 1256, status = succeeded client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051... worker2 # [ 101.531063] nix-daemon[857]: accepted connection from pid 769, user nix-grpc-daemon (trusted) client # uses-input-w1onlyb> worker2: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 102.988464] nix-daemon[857]: reaped child process 1270, status = succeeded worker2 # [ 103.078482] nix-daemon[857]: accepted connection from pid 1286, user nix-grpc-daemon (trusted) lb # [ 103.105365] niks3-server[955]: 2026/09/25 15:25:29 INFO Received push request method=POST path=/api/pushes worker2 # [ 103.108573] nix-daemon[857]: reaped child process 1289, status = succeeded worker2 # [ 103.114544] nix-grpc-daemon[916]: time=2026-09-25T15:25:29.734Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 103.129835] niks3-server[955]: 2026/09/25 15:25:29 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker2 # [ 103.135845] nix-grpc-daemon[916]: time=2026-09-25T15:25:29.759Z level=INFO msg="Uploading 1 narinfos" lb # [ 103.140423] niks3-server[955]: 2026/09/25 15:25:29 INFO Received sign narinfos request method=POST path=/api/pushes/32/sign lb # [ 103.146727] niks3-server[955]: 2026/09/25 15:25:29 INFO Signed narinfos id=32 count=1 lb # [ 103.153227] niks3-server[955]: 2026/09/25 15:25:29 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls lb # [ 103.161677] niks3-server[955]: 2026/09/25 15:25:29 INFO Received complete push request method=POST path=/api/pushes/32/complete worker2 # [ 103.163394] nix-grpc-daemon[916]: time=2026-09-25T15:25:29.787Z level=INFO msg="Upload complete. (219ms)" lb # [ 103.168757] niks3-server[955]: 2026/09/25 15:25:29 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo worker2 # [ 103.172708] nix-grpc-daemon[769]: ts=2026-09-25T15:25:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=17 outputs=1 inputs=1 inputs_fetched=0 input_bytes=0 input_s=0 client # copying 1 paths... worker2 # [ 103.186284] nix-grpc-daemon[769]: ts=2026-09-25T15:25:29Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 103.189396] nix-daemon[857]: reaped child process 1268, status = succeeded client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'... worker1 # [ 103.213240] nix-grpc-daemon[486]: ts=2026-09-25T15:25:29Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 103.217814] nix-grpc-daemon[486]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'... worker1 # [ 103.229049] nix-grpc-daemon[486]: ts=2026-09-25T15:25:29Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=0 paths=1 nar_bytes_out=120 client # [ 103.293679] nix-daemon[1164]: reaped child process 1272, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.19 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2 client # [ 103.427699] nix-daemon[1164]: accepted connection from pid 1291, user root (trusted) client # this derivation will be built: client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker1 # [ 103.575036] nix-grpc-daemon[486]: ts=2026-09-25T15:25:30Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:53882 client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker1 # [ 103.579675] nix-grpc-daemon[486]: ts=2026-09-25T15:25:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 103.584076] nix-grpc-daemon[850]: time=2026-09-25T15:25:30.210Z level=INFO msg="All 1 paths already cached" worker2 # [ 103.590671] nix-grpc-daemon[769]: ts=2026-09-25T15:25:30Z level=debug event=want client=17 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=0 worker1 # [ 103.595019] nix-grpc-daemon[486]: ts=2026-09-25T15:25:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker2 # [ 103.594457] nix-grpc-daemon[769]: ts=2026-09-25T15:25:30Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 103.604217] nix-grpc-daemon[486]: ts=2026-09-25T15:25:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 103.612450] nix-grpc-daemon[850]: time=2026-09-25T15:25:30.239Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'... worker1 # [ 103.618855] nix-grpc-daemon[486]: ts=2026-09-25T15:25:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 103.628211] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 103.755101] nix-daemon[838]: accepted connection from pid 1195, user nix-grpc-daemon (trusted) worker1 # [ 103.772460] nix-daemon[838]: reaped child process 1198, status = succeeded lb # [ 103.778293] niks3-server[955]: 2026/09/25 15:25:30 INFO Received push request method=POST path=/api/pushes worker1 # [ 103.783573] nix-grpc-daemon[850]: time=2026-09-25T15:25:30.410Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 103.785255] nix-grpc-daemon[850]: time=2026-09-25T15:25:30.410Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)" lb # [ 103.794553] niks3-server[955]: 2026/09/25 15:25:30 INFO Received sign narinfos request method=POST path=/api/pushes/33/sign worker1 # [ 103.797722] nix-grpc-daemon[850]: time=2026-09-25T15:25:30.424Z level=INFO msg="Uploading 1 narinfos" lb # [ 103.801788] niks3-server[955]: 2026/09/25 15:25:30 INFO Signed narinfos id=33 count=1 lb # [ 103.806107] niks3-server[955]: 2026/09/25 15:25:30 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls lb # [ 103.814411] niks3-server[955]: 2026/09/25 15:25:30 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst lb # [ 103.818806] niks3-server[955]: 2026/09/25 15:25:30 INFO Received complete push request method=POST path=/api/pushes/33/complete worker1 # [ 103.824147] nix-grpc-daemon[850]: time=2026-09-25T15:25:30.451Z level=INFO msg="Upload complete. (185ms)" lb # [ 103.824692] niks3-server[955]: 2026/09/25 15:25:30 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo worker1 # [ 103.830548] nix-grpc-daemon[486]: ts=2026-09-25T15:25:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=0 paths=1 nar_bytes_in=568 worker1 # [ 103.836211] nix-grpc-daemon[486]: ts=2026-09-25T15:25:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 worker1 # [ 103.845954] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) worker1 # [ 103.847980] nix-daemon[838]: reaped child process 1193, status = succeeded client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051... client # uses-input-w1onlyc> node-a: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker1 # [ 103.857874] nix-daemon[838]: accepted connection from pid 486, user nix-grpc-daemon (trusted) client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 105.090591] nix-daemon[838]: reaped child process 1202, status = succeeded worker1 # [ 105.204088] nix-daemon[838]: accepted connection from pid 1218, user nix-grpc-daemon (trusted) lb # [ 105.225192] niks3-server[955]: 2026/09/25 15:25:31 INFO Received push request method=POST path=/api/pushes worker1 # [ 105.226722] nix-daemon[838]: reaped child process 1221, status = succeeded worker1 # [ 105.231640] nix-grpc-daemon[850]: time=2026-09-25T15:25:31.857Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 105.244256] niks3-server[955]: 2026/09/25 15:25:31 INFO Received sign narinfos request method=POST path=/api/pushes/34/sign worker1 # [ 105.251037] nix-grpc-daemon[850]: time=2026-09-25T15:25:31.878Z level=INFO msg="Uploading 1 narinfos" lb # [ 105.252080] niks3-server[955]: 2026/09/25 15:25:31 INFO Signed narinfos id=34 count=1 lb # [ 105.256816] niks3-server[955]: 2026/09/25 15:25:31 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv lb # [ 105.265540] niks3-server[955]: 2026/09/25 15:25:31 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls worker1 # [ 105.269929] nix-grpc-daemon[850]: time=2026-09-25T15:25:31.897Z level=INFO msg="Upload complete. (205ms)" lb # [ 105.270471] niks3-server[955]: 2026/09/25 15:25:31 INFO Received complete push request method=POST path=/api/pushes/34/complete lb # [ 105.275075] niks3-server[955]: 2026/09/25 15:25:31 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo client # copying 1 paths... worker1 # [ 105.276546] nix-grpc-daemon[486]: ts=2026-09-25T15:25:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53882 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=18 outputs=1 inputs=1 inputs_fetched=0 input_bytes=0 input_s=0 worker2 # [ 105.282148] nix-grpc-daemon[769]: ts=2026-09-25T15:25:31Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:56488 worker1 # [ 105.289034] nix-daemon[838]: reaped child process 1200, status = succeeded worker2 # [ 105.290828] nix-grpc-daemon[769]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'... client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'... worker2 # [ 105.317658] nix-grpc-daemon[769]: ts=2026-09-25T15:25:31Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56488 worker2 # [ 105.321962] nix-grpc-daemon[769]: ts=2026-09-25T15:25:31Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56488 duration_s=0 paths=1 nar_bytes_out=120 client # [ 105.386451] nix-daemon[1164]: reaped child process 1294, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.09 seconds) (finished: subtest: an input only one worker has still reaches the builder, in 7.80 seconds) subtest: upload whose reference the worker lacks is completed from the cache client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache client # this derivation will be built: client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache, in 0.32 seconds) client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache client # this derivation will be built: client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache, in 0.28 seconds) client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.11 seconds) worker1: must succeed: nix-store --import < /tmp/shared/ref.closure worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.15 seconds) worker2 # [ 106.291043] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 107.054126] nix-grpc-daemon[769]: ts=2026-09-25T15:25:33Z level=info event=signal_exit lb # [ 107.588828] niks3-server[955]: 2026/09/25 15:25:34 INFO lead: released remote=192.168.1.4:50512 lb # [ 108.116315] envoy[902]: conn peer=[2001:db8:1::1]:51994 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4223 tx=2051 ms=1604 lb # [ 108.119029] envoy[902]: conn peer=[2001:db8:1::1]:51996 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=329 ms=30 lb # [ 108.121643] envoy[902]: conn peer=[2001:db8:1::1]:52002 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3752 tx=2122 ms=2860 lb # [ 108.124183] envoy[902]: conn peer=[2001:db8:1::1]:45392 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=329 ms=32 lb # [ 108.126572] envoy[902]: conn peer=[2001:db8:1::1]:45398 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3723 tx=2104 ms=1761 lb # [ 108.129107] envoy[902]: conn peer=[2001:db8:1::1]:45400 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=328 ms=16 worker2 # [ 108.585626] nix-grpc-daemon[769]: ts=2026-09-25T15:25:35Z level=info event=scheduler_restarting lb # [ 108.586509] envoy[902]: [2026-09-25 15:25:35.210][902][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream} worker2 # [ 108.590208] nix-grpc-daemon[769]: ts=2026-09-25T15:25:35Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 108.593016] nix-grpc-daemon[486]: ts=2026-09-25T15:25:35Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 108.597266] nix-grpc-daemon[769]: ts=2026-09-25T15:25:35Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:34328 worker2 # [ 108.600410] nix-grpc-daemon[769]: ts=2026-09-25T15:25:35Z level=info event=worker_gone addr=192.168.1.3:50051 worker2 # [ 108.602728] nix-grpc-daemon[769]: ts=2026-09-25T15:25:35Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:34328 worker2 # [ 108.605578] nix-grpc-daemon[769]: ts=2026-09-25T15:25:35Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 108.622380] nix-daemon[857]: reaped child process 859, status = succeeded worker1 # [ 108.793032] nix-grpc-daemon[486]: ts=2026-09-25T15:25:35Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 108.796109] nix-grpc-daemon[486]: ts=2026-09-25T15:25:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 108.806362] nix-grpc-daemon[769]: ts=2026-09-25T15:25:35Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker1 # [ 108.996496] nix-grpc-daemon[486]: ts=2026-09-25T15:25:35Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 108.999304] nix-grpc-daemon[486]: ts=2026-09-25T15:25:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 109.199745] nix-grpc-daemon[486]: ts=2026-09-25T15:25:35Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 109.202650] nix-grpc-daemon[486]: ts=2026-09-25T15:25:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 109.230566] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 109.234945] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 109.238129] systemd[1]: nix-grpc-daemon.service: Consumed 2.685s CPU time over 1min 31.364s wall clock time, 64.6M memory peak, 326.5K incoming IP traffic, 395.5K outgoing IP traffic. lb: waiting until exactly ['worker1'] are healthy members of aarch64-linux lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: (finished: waiting until exactly ['worker1'] are healthy members of aarch64-linux, in 0.04 seconds) client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2 worker1 # [ 109.403186] nix-grpc-daemon[486]: ts=2026-09-25T15:25:36Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 109.406173] nix-grpc-daemon[486]: ts=2026-09-25T15:25:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 109.462160] nix-grpc-daemon[486]: ts=2026-09-25T15:25:36Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53882 client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache worker1 # [ 109.466250] nix-grpc-daemon[486]: ts=2026-09-25T15:25:36Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:53882 client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.19 seconds) worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.02 seconds) worker2 # [ 109.541697] systemd[1]: Starting scheduler-token.service... worker2 # [ 109.603038] systemd[1]: scheduler-token.service: Deactivated successfully. worker1 # [ 109.607817] nix-grpc-daemon[486]: ts=2026-09-25T15:25:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 109.608304] systemd[1]: Finished scheduler-token.service. worker1 # [ 109.610235] nix-grpc-daemon[486]: ts=2026-09-25T15:25:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 109.609528] systemd[1]: scheduler-token.service: Consumed 35ms CPU time over 64ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 109.622609] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 109.727766] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:36Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 109.730304] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:36Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 109.734179] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:36Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 109.739568] nix-grpc-daemon[1316]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 109.742821] nix-grpc-daemon[1316]: E0000 00:00:1790349936.362686 1316 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 109.754299] niks3-server[955]: 2026/09/25 15:25:36 INFO lead: acquired remote=192.168.1.4:49670 worker2 # [ 109.764179] nix-grpc-daemon[1316]: E0000 00:00:1790349936.366447 1316 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 109.766597] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:36Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 109.791303] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:36Z level=info event=scheduler_take_over worker2 # [ 109.793182] nix-grpc-daemon[1316]: E0000 00:00:1790349936.375729 1357 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 109.798870] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 109.801955] nix-grpc-daemon[1316]: E0000 00:00:1790349936.397564 1357 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 109.804437] nix-grpc-daemon[1316]: E0000 00:00:1790349936.397767 1357 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 109.808792] nix-grpc-daemon[486]: ts=2026-09-25T15:25:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 109.806950] nix-grpc-daemon[1316]: E0000 00:00:1790349936.414297 1357 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 109.809619] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 109.810936] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 109.813543] nix-grpc-daemon[486]: ts=2026-09-25T15:25:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 109.831263] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 110.002817] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 110.005614] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 110.012899] nix-grpc-daemon[486]: ts=2026-09-25T15:25:36Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 110.015742] nix-grpc-daemon[486]: ts=2026-09-25T15:25:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 110.206464] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 110.209963] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 110.216298] nix-grpc-daemon[486]: ts=2026-09-25T15:25:36Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 110.219113] nix-grpc-daemon[486]: ts=2026-09-25T15:25:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 110.410757] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 110.419609] nix-grpc-daemon[486]: ts=2026-09-25T15:25:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 110.418695] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:37Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:38986 worker2 # [ 110.422483] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:37Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker2 # [ 110.429123] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:37Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 110.431371] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:37Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:38986 worker2 # [ 110.434687] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:37Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 110.761402] nix-grpc-daemon[486]: ts=2026-09-25T15:25:37Z level=info event=signal_exit worker1 # [ 112.593553] nix-grpc-daemon[486]: ts=2026-09-25T15:25:39Z level=info event=scheduler_restarting worker2 # [ 112.590604] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:39Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:38986 worker2 # [ 112.590814] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:39Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 112.608642] nix-grpc-daemon[486]: ts=2026-09-25T15:25:39Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker1 # [ 112.623878] nix-daemon[838]: reaped child process 840, status = succeeded worker2 # [ 113.068269] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:39Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:54530 worker1 # [ 113.204436] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 113.208443] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 113.209912] systemd[1]: nix-grpc-daemon.service: Consumed 2.633s CPU time over 1min 43.770s wall clock time, 65.2M memory peak, 177.4K incoming IP traffic, 345.8K outgoing IP traffic. lb: waiting until exactly ['worker2'] are healthy members of aarch64-linux lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: (finished: waiting until exactly ['worker2'] are healthy members of aarch64-linux, in 0.04 seconds) client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2 worker2 # [ 113.436197] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54542 worker2 # [ 113.445319] nix-daemon[857]: accepted connection from pid 1316, user nix-grpc-daemon (trusted) worker2 # [ 113.458999] nix-grpc-daemon[1316]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 98 ms (attempt 1/5) worker2 # [ 113.562094] nix-grpc-daemon[1316]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 143 ms (attempt 2/5) worker2 # [ 113.706759] nix-grpc-daemon[1316]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 354 ms (attempt 3/5) worker2 # [ 114.061207] nix-grpc-daemon[1316]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 363 ms (attempt 4/5) worker2 # [ 114.425682] nix-grpc-daemon[1316]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 114.427746] nix-grpc-daemon[1316]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org client # copying 2 paths... client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'... worker2 # [ 114.443330] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54542 client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'... worker2 # [ 114.454765] nix-daemon[857]: accepted connection from pid 1316, user nix-grpc-daemon (trusted) worker2 # [ 114.474528] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:41Z level=info event=niks3_push_started pid=1376 worker2 # [ 114.597747] nix-grpc-daemon[1376]: time=2026-09-25T15:25:41.221Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 114.702474] nix-daemon[857]: accepted connection from pid 1388, user nix-grpc-daemon (trusted) worker2 # [ 114.722731] nix-daemon[857]: reaped child process 1391, status = succeeded lb # [ 114.724139] niks3-server[955]: 2026/09/25 15:25:41 INFO Received push request method=POST path=/api/pushes worker2 # [ 114.728100] nix-grpc-daemon[1376]: time=2026-09-25T15:25:41.350Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker2 # [ 114.729637] nix-grpc-daemon[1376]: time=2026-09-25T15:25:41.350Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)" worker2 # [ 114.735131] nix-grpc-daemon[1376]: time=2026-09-25T15:25:41.351Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)" lb # [ 114.766338] niks3-server[955]: 2026/09/25 15:25:41 INFO Received sign narinfos request method=POST path=/api/pushes/35/sign lb # [ 114.776485] niks3-server[955]: 2026/09/25 15:25:41 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls lb # [ 114.785114] niks3-server[955]: 2026/09/25 15:25:41 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst lb # [ 114.793551] niks3-server[955]: 2026/09/25 15:25:41 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls lb # [ 114.800381] niks3-server[955]: 2026/09/25 15:25:41 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst worker2 # [ 114.798795] nix-grpc-daemon[1376]: time=2026-09-25T15:25:41.421Z level=INFO msg="Uploading 2 narinfos" lb # [ 114.803455] niks3-server[955]: 2026/09/25 15:25:41 INFO Signed narinfos id=35 count=2 lb # [ 114.821766] niks3-server[955]: 2026/09/25 15:25:41 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo lb # [ 114.830354] niks3-server[955]: 2026/09/25 15:25:41 INFO Received complete push request method=POST path=/api/pushes/35/complete worker2 # [ 114.832565] nix-grpc-daemon[1376]: time=2026-09-25T15:25:41.456Z level=INFO msg="Upload complete. (231ms)" lb # [ 114.835425] niks3-server[955]: 2026/09/25 15:25:41 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo worker2 # [ 114.839793] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54542 duration_s=0 paths=2 nar_bytes_in=312 worker2 # [ 114.849433] nix-daemon[857]: reaped child process 1374, status = succeeded client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.60 seconds) worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.02 seconds) worker1 # [ 114.926365] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 115.038160] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:41Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 115.040460] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:41Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 115.044258] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:41Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 115.050631] nix-grpc-daemon[1244]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 115.072321] nix-grpc-daemon[1244]: E0000 00:00:1790349941.675691 1244 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 115.074827] nix-grpc-daemon[1244]: E0000 00:00:1790349941.677638 1244 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 115.097820] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:41Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 115.099700] nix-grpc-daemon[1244]: E0000 00:00:1790349941.685399 1287 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 115.105271] nix-grpc-daemon[1244]: E0000 00:00:1790349941.708453 1287 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 115.109370] nix-grpc-daemon[1244]: E0000 00:00:1790349941.708601 1287 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 115.108650] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:41Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:38986 worker1 # [ 115.112693] nix-grpc-daemon[1244]: E0000 00:00:1790349941.723839 1287 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 115.113463] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:41Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 115.118502] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 115.119392] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:41Z level=info event=scheduler_connected addr=lb:50051 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: (finished: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux, in 0.06 seconds) (finished: subtest: upload whose reference the worker lacks is completed from the cache, in 9.75 seconds) subtest: a niks3 restart does not cost the leader its role lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb # [ 115.240774] systemd[1]: Stopping niks3 server... worker1 # [ 115.241923] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:41Z level=info event=scheduler_lock_lost status=200 lb # [ 115.244690] niks3-server[955]: 2026/09/25 15:25:41 INFO Shutdown signal received, draining in-flight requests timeout=10s worker2 # [ 115.242509] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:41Z level=info event=scheduler_lock_lost status=200 lb # [ 115.246389] niks3-server[955]: 2026/09/25 15:25:41 INFO lead: released remote=192.168.1.4:49670 lb # [ 115.268301] systemd[1]: niks3.service: Deactivated successfully. lb # [ 115.269234] systemd[1]: Stopped niks3 server. lb # [ 115.269903] systemd[1]: niks3.service: Consumed 439ms CPU time over 1min 33.922s wall clock time, 27.9M memory peak, 98K incoming IP traffic, 141.5K outgoing IP traffic. lb # [ 115.279310] systemd[1]: Starting niks3 server... lb # [ 115.345685] niks3-server[1153]: 2026/09/25 15:25:41 goose: no migrations to run. current version: 20260923120000 lb # [ 115.352124] niks3-server[1153]: 2026/09/25 15:25:41 OK 1_commit_pending_closure.sql (7.16ms) lb # [ 115.355482] niks3-server[1153]: 2026/09/25 15:25:41 OK 2_object_stats_trigger.sql (3.31ms) lb # [ 115.358525] niks3-server[1153]: 2026/09/25 15:25:41 OK 3_commit_push.sql (2.71ms) lb # [ 115.360540] niks3-server[1153]: 2026/09/25 15:25:41 goose: up to current file version: 3 lb # [ 115.362469] niks3-server[1153]: 2026/09/25 15:25:41 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key lb # [ 115.371979] niks3-server[1153]: 2026/09/25 15:25:41 INFO Read proxy enabled — serving cache objects from S3 lb # [ 115.373894] niks3-server[1153]: 2026/09/25 15:25:41 INFO Using socket-activated listener address=0.0.0.0:5751 lb # [ 115.375179] niks3-server[1153]: 2026/09/25 15:25:41 INFO systemd watchdog enabled interval=15s lb # [ 115.380493] niks3-server[1153]: 2026/09/25 15:25:41 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 115.382066] systemd[1]: Started niks3 server. lb: waiting for unit niks3.service lb: (finished: waiting for unit niks3.service, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.04 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2 worker1 # [ 115.733026] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:42Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:56188 worker1 # [ 115.741695] nix-daemon[838]: accepted connection from pid 1244, user nix-grpc-daemon (trusted) client # don't know how to build these paths: client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker2 # [ 115.752772] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54542 worker2 # [ 115.771171] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:42Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=0 worker2 # [ 115.776901] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:42Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 115.781526] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54542 worker2 # [ 115.785730] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54542 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'... worker2 # [ 115.792603] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54542 worker2 # [ 115.801449] nix-daemon[857]: accepted connection from pid 1316, user nix-grpc-daemon (trusted) worker2 # [ 115.927866] nix-daemon[857]: accepted connection from pid 1408, user nix-grpc-daemon (trusted) lb # [ 115.947399] niks3-server[1153]: 2026/09/25 15:25:42 INFO Received push request method=POST path=/api/pushes worker2 # [ 115.947469] nix-daemon[857]: reaped child process 1411, status = succeeded worker2 # [ 115.954291] nix-grpc-daemon[1376]: time=2026-09-25T15:25:42.577Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 115.955902] nix-grpc-daemon[1376]: time=2026-09-25T15:25:42.577Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)" lb # [ 115.970274] niks3-server[1153]: 2026/09/25 15:25:42 INFO Received sign narinfos request method=POST path=/api/pushes/36/sign lb # [ 115.989214] niks3-server[1153]: 2026/09/25 15:25:42 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls lb # [ 116.005606] niks3-server[1153]: 2026/09/25 15:25:42 INFO Signed narinfos id=36 count=1 worker2 # [ 116.006368] nix-grpc-daemon[1376]: time=2026-09-25T15:25:42.629Z level=INFO msg="Uploading 1 narinfos" lb # [ 116.020131] niks3-server[1153]: 2026/09/25 15:25:42 INFO Received complete push request method=POST path=/api/pushes/36/complete lb # [ 116.028262] niks3-server[1153]: 2026/09/25 15:25:42 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst lb # [ 116.031593] niks3-server[1153]: 2026/09/25 15:25:42 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.narinfo worker2 # [ 116.034411] nix-grpc-daemon[1376]: time=2026-09-25T15:25:42.658Z level=INFO msg="Upload complete. (223ms)" worker2 # [ 116.047580] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54542 duration_s=0 paths=1 nar_bytes_in=544 worker2 # [ 116.051173] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54542 worker2 # [ 116.058699] nix-daemon[857]: accepted connection from pid 1316, user nix-grpc-daemon (trusted) worker2 # [ 116.061005] nix-daemon[857]: reaped child process 1406, status = succeeded client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051... client # farm-a> worker2: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker2 # [ 116.071366] nix-daemon[857]: accepted connection from pid 1316, user nix-grpc-daemon (trusted) lb # [ 116.247825] niks3-server[1153]: 2026/09/25 15:25:42 INFO lead: acquired remote=192.168.1.4:41928 client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-after-niks3-restart worker2 # [ 117.352899] nix-daemon[857]: reaped child process 1415, status = succeeded worker2 # [ 117.408722] nix-daemon[857]: accepted connection from pid 1433, user nix-grpc-daemon (trusted) worker2 # [ 117.424103] nix-daemon[857]: reaped child process 1436, status = succeeded lb # [ 117.432500] niks3-server[1153]: 2026/09/25 15:25:44 INFO Received push request method=POST path=/api/pushes worker2 # [ 117.437728] nix-grpc-daemon[1376]: time=2026-09-25T15:25:44.059Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 117.439457] nix-grpc-daemon[1376]: time=2026-09-25T15:25:44.059Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)" lb # [ 117.454230] niks3-server[1153]: 2026/09/25 15:25:44 INFO Received sign narinfos request method=POST path=/api/pushes/37/sign worker2 # [ 117.458763] nix-grpc-daemon[1376]: time=2026-09-25T15:25:44.082Z level=INFO msg="Uploading 1 narinfos" lb # [ 117.464170] niks3-server[1153]: 2026/09/25 15:25:44 INFO Signed narinfos id=37 count=1 lb # [ 117.471738] niks3-server[1153]: 2026/09/25 15:25:44 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls lb # [ 117.484865] niks3-server[1153]: 2026/09/25 15:25:44 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv lb # [ 117.496731] niks3-server[1153]: 2026/09/25 15:25:44 INFO Received complete push request method=POST path=/api/pushes/37/complete lb # [ 117.504114] niks3-server[1153]: 2026/09/25 15:25:44 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst worker2 # [ 117.506447] nix-grpc-daemon[1376]: time=2026-09-25T15:25:44.130Z level=INFO msg="Upload complete. (223ms)" lb # [ 117.509414] niks3-server[1153]: 2026/09/25 15:25:44 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.narinfo worker2 # [ 117.520192] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54542 duration_s=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker2 # [ 117.531601] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:44Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=0 worker2 # [ 117.536161] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:44Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 117.540785] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54542 worker2 # [ 117.545210] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54542 worker2 # [ 117.549217] nix-daemon[857]: reaped child process 1413, status = succeeded worker2 # [ 117.551615] nix-grpc-daemon[1376]: time=2026-09-25T15:25:44.175Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'... worker2 # [ 117.557706] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54542 worker2 # [ 117.568190] nix-daemon[857]: accepted connection from pid 1316, user nix-grpc-daemon (trusted) worker2 # [ 117.686550] nix-daemon[857]: accepted connection from pid 1442, user nix-grpc-daemon (trusted) worker2 # [ 117.702861] nix-daemon[857]: reaped child process 1445, status = succeeded lb # [ 117.709060] niks3-server[1153]: 2026/09/25 15:25:44 INFO Received push request method=POST path=/api/pushes worker2 # [ 117.713522] nix-grpc-daemon[1376]: time=2026-09-25T15:25:44.337Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 117.715052] nix-grpc-daemon[1376]: time=2026-09-25T15:25:44.337Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)" lb # [ 117.728292] niks3-server[1153]: 2026/09/25 15:25:44 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls worker2 # [ 117.731284] nix-grpc-daemon[1376]: time=2026-09-25T15:25:44.354Z level=INFO msg="Uploading 1 narinfos" lb # [ 117.735905] niks3-server[1153]: 2026/09/25 15:25:44 INFO Received sign narinfos request method=POST path=/api/pushes/38/sign lb # [ 117.741303] niks3-server[1153]: 2026/09/25 15:25:44 INFO Signed narinfos id=38 count=1 lb # [ 117.743019] niks3-server[1153]: 2026/09/25 15:25:44 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst lb # [ 117.752673] niks3-server[1153]: 2026/09/25 15:25:44 INFO Received complete push request method=POST path=/api/pushes/38/complete lb # [ 117.762538] niks3-server[1153]: 2026/09/25 15:25:44 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo worker2 # [ 117.758116] nix-grpc-daemon[1376]: time=2026-09-25T15:25:44.381Z level=INFO msg="Upload complete. (183ms)" worker2 # [ 117.764367] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54542 duration_s=0 paths=1 nar_bytes_in=656 worker2 # [ 117.774573] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54542 worker2 # [ 117.782426] nix-daemon[857]: accepted connection from pid 1316, user nix-grpc-daemon (trusted) worker2 # [ 117.785369] nix-daemon[857]: reaped child process 1440, status = succeeded client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051... client # farm-b> worker2: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker2 # [ 117.795745] nix-daemon[857]: accepted connection from pid 1316, user nix-grpc-daemon (trusted) lb # [ 118.114318] envoy[902]: conn peer=[2001:db8:1::4]:39316 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=2958 tx=1121 ms=73408 lb # [ 118.117098] envoy[902]: conn peer=[2001:db8:1::1]:45414 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=519 ms=14 lb # [ 118.119524] envoy[902]: conn peer=192.168.1.3:35046 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=3624 tx=1380 ms=76669 lb # [ 118.122246] envoy[902]: conn peer=[2001:db8:1::1]:60750 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1005 tx=305 ms=1414 client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-after-niks3-restart worker2 # [ 119.044458] nix-daemon[857]: reaped child process 1449, status = succeeded worker2 # [ 119.139566] nix-daemon[857]: accepted connection from pid 1465, user nix-grpc-daemon (trusted) worker2 # [ 119.156426] nix-daemon[857]: reaped child process 1468, status = succeeded lb # [ 119.164971] niks3-server[1153]: 2026/09/25 15:25:45 INFO Received push request method=POST path=/api/pushes worker2 # [ 119.169097] nix-grpc-daemon[1376]: time=2026-09-25T15:25:45.791Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 119.170785] nix-grpc-daemon[1376]: time=2026-09-25T15:25:45.792Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)" lb # [ 119.187998] niks3-server[1153]: 2026/09/25 15:25:45 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst worker2 # [ 119.194063] nix-grpc-daemon[1376]: time=2026-09-25T15:25:45.817Z level=INFO msg="Uploading 1 narinfos" lb # [ 119.196135] niks3-server[1153]: 2026/09/25 15:25:45 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls lb # [ 119.202517] niks3-server[1153]: 2026/09/25 15:25:45 INFO Received sign narinfos request method=POST path=/api/pushes/39/sign lb # [ 119.208143] niks3-server[1153]: 2026/09/25 15:25:45 INFO Signed narinfos id=39 count=1 lb # [ 119.209953] niks3-server[1153]: 2026/09/25 15:25:45 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv lb # [ 119.221728] niks3-server[1153]: 2026/09/25 15:25:45 INFO Received complete push request method=POST path=/api/pushes/39/complete worker2 # [ 119.224506] nix-grpc-daemon[1376]: time=2026-09-25T15:25:45.848Z level=INFO msg="Upload complete. (220ms)" lb # [ 119.226738] niks3-server[1153]: 2026/09/25 15:25:45 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.narinfo worker2 # [ 119.235101] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54542 duration_s=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=1 inputs=1 inputs_fetched=0 input_bytes=0 input_s=0 worker2 # [ 119.248879] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:45Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=0 worker2 # [ 119.251725] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:45Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 119.256505] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54542 worker2 # [ 119.260753] nix-daemon[857]: reaped child process 1447, status = succeeded worker2 # [ 119.263666] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54542 worker2 # [ 119.269421] nix-grpc-daemon[1376]: time=2026-09-25T15:25:45.893Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'... worker2 # [ 119.275701] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54542 worker2 # [ 119.284720] nix-daemon[857]: accepted connection from pid 1316, user nix-grpc-daemon (trusted) worker2 # [ 119.397317] nix-daemon[857]: accepted connection from pid 1474, user nix-grpc-daemon (trusted) lb # [ 119.414998] niks3-server[1153]: 2026/09/25 15:25:46 INFO Received push request method=POST path=/api/pushes worker2 # [ 119.416694] nix-daemon[857]: reaped child process 1477, status = succeeded worker2 # [ 119.418491] nix-grpc-daemon[1376]: time=2026-09-25T15:25:46.041Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 119.420053] nix-grpc-daemon[1376]: time=2026-09-25T15:25:46.041Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)" lb # [ 119.432310] niks3-server[1153]: 2026/09/25 15:25:46 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst worker2 # [ 119.435854] nix-grpc-daemon[1376]: time=2026-09-25T15:25:46.059Z level=INFO msg="Uploading 1 narinfos" lb # [ 119.440194] niks3-server[1153]: 2026/09/25 15:25:46 INFO Received sign narinfos request method=POST path=/api/pushes/40/sign lb # [ 119.442638] niks3-server[1153]: 2026/09/25 15:25:46 INFO Signed narinfos id=40 count=1 lb # [ 119.452227] niks3-server[1153]: 2026/09/25 15:25:46 INFO Received complete push request method=POST path=/api/pushes/40/complete lb # [ 119.454605] niks3-server[1153]: 2026/09/25 15:25:46 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls lb # [ 119.466017] niks3-server[1153]: 2026/09/25 15:25:46 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo worker2 # [ 119.469015] nix-grpc-daemon[1376]: time=2026-09-25T15:25:46.092Z level=INFO msg="Upload complete. (176ms)" worker2 # [ 119.476141] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54542 duration_s=0 paths=1 nar_bytes_in=784 worker2 # [ 119.479543] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54542 worker2 # [ 119.490678] nix-daemon[857]: accepted connection from pid 1316, user nix-grpc-daemon (trusted) worker2 # [ 119.494729] nix-daemon[857]: reaped child process 1472, status = succeeded client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051... client # farm-top> worker2: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker2 # [ 119.505893] nix-daemon[857]: accepted connection from pid 1316, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-after-niks3-restart worker2 # [ 120.960798] nix-daemon[857]: reaped child process 1481, status = succeeded worker2 # [ 121.058921] nix-daemon[857]: accepted connection from pid 1497, user nix-grpc-daemon (trusted) worker2 # [ 121.078132] nix-daemon[857]: reaped child process 1500, status = succeeded lb # [ 121.078702] niks3-server[1153]: 2026/09/25 15:25:47 INFO Received push request method=POST path=/api/pushes worker2 # [ 121.084407] nix-grpc-daemon[1376]: time=2026-09-25T15:25:47.707Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 121.085942] nix-grpc-daemon[1376]: time=2026-09-25T15:25:47.707Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)" lb # [ 121.108064] niks3-server[1153]: 2026/09/25 15:25:47 INFO Received sign narinfos request method=POST path=/api/pushes/41/sign worker2 # [ 121.112749] nix-grpc-daemon[1376]: time=2026-09-25T15:25:47.736Z level=INFO msg="Uploading 1 narinfos" lb # [ 121.116447] niks3-server[1153]: 2026/09/25 15:25:47 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls lb # [ 121.122377] niks3-server[1153]: 2026/09/25 15:25:47 INFO Signed narinfos id=41 count=1 lb # [ 121.128644] niks3-server[1153]: 2026/09/25 15:25:47 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst lb # [ 121.135194] niks3-server[1153]: 2026/09/25 15:25:47 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv lb # [ 121.140370] niks3-server[1153]: 2026/09/25 15:25:47 INFO Received complete push request method=POST path=/api/pushes/41/complete worker2 # [ 121.141345] nix-grpc-daemon[1376]: time=2026-09-25T15:25:47.765Z level=INFO msg="Upload complete. (227ms)" lb # [ 121.144743] niks3-server[1153]: 2026/09/25 15:25:47 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo worker2 # [ 121.155651] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54542 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=1 inputs=2 inputs_fetched=0 input_bytes=0 input_s=0 worker2 # [ 121.161062] nix-daemon[857]: reaped child process 1479, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2, in 5.62 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath, in 0.14 seconds) (finished: subtest: a niks3 restart does not cost the leader its role, in 6.18 seconds) subtest: scheduler down: the other node takes the lock and keeps it lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2 # [ 121.505625] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 121.752944] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:48Z level=info event=signal_exit lb # [ 122.255065] niks3-server[1153]: 2026/09/25 15:25:48 INFO lead: released remote=192.168.1.4:41928 worker2 # [ 123.251665] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:49Z level=info event=scheduler_restarting worker1 # [ 123.255936] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:49Z level=info event=scheduler_restarting addr=lb:50051 lb # [ 123.254502] envoy[902]: [2026-09-25 15:25:49.876][902][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream} worker2 # [ 123.255862] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:49Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 123.261602] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:49Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:38986 worker2 # [ 123.265129] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:49Z level=info event=worker_gone addr=192.168.1.3:50051 worker2 # [ 123.267886] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:49Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:38986 worker2 # [ 123.270629] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:49Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 123.287766] nix-daemon[857]: reaped child process 1368, status = succeeded worker1 # [ 123.455983] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:50Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 123.459146] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 123.469038] nix-grpc-daemon[1316]: ts=2026-09-25T15:25:50Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 123.503943] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 123.508566] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 123.509969] systemd[1]: nix-grpc-daemon.service: Consumed 1.167s CPU time over 13.890s wall clock time, 39M memory peak, 253K incoming IP traffic, 109.4K outgoing IP traffic. worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over worker1 # [ 123.659841] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:50Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 123.662974] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 123.863638] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:50Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 123.866701] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 124.067351] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:50Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 124.070436] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 124.272182] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:50Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 124.274524] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 124.472561] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:51Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 124.475544] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 124.676083] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:51Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 124.678976] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 124.879611] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:51Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 124.882596] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 125.083090] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:51Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 125.086000] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 125.287427] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:51Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 125.289758] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 125.487761] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:52Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 125.490559] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 125.691058] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:52Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 125.693959] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 125.894463] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:52Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 125.897215] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 126.097855] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:52Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 126.100582] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 126.246604] niks3-server[1153]: 2026/09/25 15:25:52 INFO lead: acquired remote=192.168.1.3:57066 worker1 # [ 126.246960] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:52Z level=info event=scheduler_take_over worker1 # [ 126.301648] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:52Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 126.304070] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 126.502511] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:53Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 126.507408] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:53Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:44024 worker1 # [ 126.510801] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:53Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 3.17 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2 worker1 # [ 126.897416] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:56188 client # don't know how to build these paths: client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker1 # [ 126.903379] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56188 worker1 # [ 126.911383] nix-grpc-daemon[1244]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 5 ms (attempt 1/5) worker1 # [ 126.921154] nix-grpc-daemon[1244]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 190 ms (attempt 2/5) worker1 # [ 127.112318] nix-grpc-daemon[1244]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 77 ms (attempt 3/5) worker1 # [ 127.190586] nix-grpc-daemon[1244]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 481 ms (attempt 4/5) worker1 # [ 127.673845] nix-grpc-daemon[1244]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 127.677719] nix-grpc-daemon[1244]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 127.696085] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:54Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=0 worker1 # [ 127.699894] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:54Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 127.705350] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56188 worker1 # [ 127.711986] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56188 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'... worker1 # [ 127.718578] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56188 worker1 # [ 127.727647] nix-daemon[838]: accepted connection from pid 1244, user nix-grpc-daemon (trusted) worker1 # [ 127.743829] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:54Z level=info event=niks3_push_started pid=1335 worker1 # [ 127.870552] nix-grpc-daemon[1335]: time=2026-09-25T15:25:54.497Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 127.975934] nix-daemon[838]: accepted connection from pid 1347, user nix-grpc-daemon (trusted) lb # [ 127.994328] niks3-server[1153]: 2026/09/25 15:25:54 INFO Received push request method=POST path=/api/pushes worker1 # [ 127.996879] nix-daemon[838]: reaped child process 1350, status = succeeded worker1 # [ 128.000226] nix-grpc-daemon[1335]: time=2026-09-25T15:25:54.626Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 128.005904] nix-grpc-daemon[1335]: time=2026-09-25T15:25:54.627Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)" lb # [ 128.024981] niks3-server[1153]: 2026/09/25 15:25:54 INFO Received sign narinfos request method=POST path=/api/pushes/42/sign worker1 # [ 128.030333] nix-grpc-daemon[1335]: time=2026-09-25T15:25:54.656Z level=INFO msg="Uploading 1 narinfos" lb # [ 128.032088] niks3-server[1153]: 2026/09/25 15:25:54 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst lb # [ 128.035264] niks3-server[1153]: 2026/09/25 15:25:54 INFO Signed narinfos id=42 count=1 lb # [ 128.046675] niks3-server[1153]: 2026/09/25 15:25:54 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls worker1 # [ 128.052188] nix-grpc-daemon[1335]: time=2026-09-25T15:25:54.679Z level=INFO msg="Upload complete. (178ms)" lb # [ 128.052865] niks3-server[1153]: 2026/09/25 15:25:54 INFO Received complete push request method=POST path=/api/pushes/42/complete lb # [ 128.056045] niks3-server[1153]: 2026/09/25 15:25:54 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo worker1 # [ 128.059970] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56188 duration_s=0 paths=1 nar_bytes_in=520 worker1 # [ 128.062775] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56188 worker1 # [ 128.074208] nix-daemon[838]: accepted connection from pid 1244, user nix-grpc-daemon (trusted) worker1 # [ 128.075650] nix-daemon[838]: reaped child process 1333, status = succeeded client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051... worker1 # [ 128.083225] nix-daemon[838]: accepted connection from pid 1244, user nix-grpc-daemon (trusted) client # farm-a> node-a: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv lb # [ 128.113282] envoy[902]: conn peer=[2001:db8:1::1]:60756 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9762 tx=4424 ms=5432 lb # [ 128.115944] envoy[902]: conn peer=[2001:db8:1::4]:45604 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=3991 tx=586 ms=13663 client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-on-standby worker1 # [ 128.932663] nix-daemon[838]: reaped child process 1355, status = succeeded worker1 # [ 128.944483] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:55Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:44036 worker1 # [ 129.038313] nix-daemon[838]: accepted connection from pid 1371, user nix-grpc-daemon (trusted) lb # [ 129.056784] niks3-server[1153]: 2026/09/25 15:25:55 INFO Received push request method=POST path=/api/pushes worker1 # [ 129.059038] nix-daemon[838]: reaped child process 1374, status = succeeded worker1 # [ 129.064408] nix-grpc-daemon[1335]: time=2026-09-25T15:25:55.689Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 129.066009] nix-grpc-daemon[1335]: time=2026-09-25T15:25:55.689Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)" lb # [ 129.077403] niks3-server[1153]: 2026/09/25 15:25:55 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker1 # [ 129.089684] nix-grpc-daemon[1335]: time=2026-09-25T15:25:55.716Z level=INFO msg="Uploading 1 narinfos" lb # [ 129.089527] niks3-server[1153]: 2026/09/25 15:25:55 INFO Received sign narinfos request method=POST path=/api/pushes/43/sign lb # [ 129.095855] niks3-server[1153]: 2026/09/25 15:25:55 INFO Signed narinfos id=43 count=1 lb # [ 129.099784] niks3-server[1153]: 2026/09/25 15:25:55 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls lb # [ 129.108633] niks3-server[1153]: 2026/09/25 15:25:55 INFO Received complete push request method=POST path=/api/pushes/43/complete worker1 # [ 129.116045] nix-grpc-daemon[1335]: time=2026-09-25T15:25:55.743Z level=INFO msg="Upload complete. (224ms)" lb # [ 129.117249] niks3-server[1153]: 2026/09/25 15:25:55 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst worker1 # [ 129.125307] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56188 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 lb # [ 129.137976] niks3-server[1153]: 2026/09/25 15:25:55 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo worker1 # [ 129.140140] nix-daemon[838]: reaped child process 1353, status = succeeded worker1 # [ 129.144421] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:55Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=0 worker1 # [ 129.147888] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:55Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 129.153772] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56188 worker1 # [ 129.159945] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56188 worker1 # [ 129.168353] nix-grpc-daemon[1335]: time=2026-09-25T15:25:55.795Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'... worker1 # [ 129.174428] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56188 worker1 # [ 129.182192] nix-daemon[838]: accepted connection from pid 1244, user nix-grpc-daemon (trusted) worker1 # [ 129.299816] nix-daemon[838]: accepted connection from pid 1380, user nix-grpc-daemon (trusted) lb # [ 129.319905] niks3-server[1153]: 2026/09/25 15:25:55 INFO Received push request method=POST path=/api/pushes worker1 # [ 129.323079] nix-daemon[838]: reaped child process 1383, status = succeeded worker1 # [ 129.328847] nix-grpc-daemon[1335]: time=2026-09-25T15:25:55.955Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 129.330514] nix-grpc-daemon[1335]: time=2026-09-25T15:25:55.955Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)" lb # [ 129.342391] niks3-server[1153]: 2026/09/25 15:25:55 INFO Received sign narinfos request method=POST path=/api/pushes/44/sign worker1 # [ 129.346219] nix-grpc-daemon[1335]: time=2026-09-25T15:25:55.973Z level=INFO msg="Uploading 1 narinfos" lb # [ 129.348966] niks3-server[1153]: 2026/09/25 15:25:55 INFO Signed narinfos id=44 count=1 lb # [ 129.354849] niks3-server[1153]: 2026/09/25 15:25:55 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls lb # [ 129.362856] niks3-server[1153]: 2026/09/25 15:25:55 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst lb # [ 129.371876] niks3-server[1153]: 2026/09/25 15:25:55 INFO Received complete push request method=POST path=/api/pushes/44/complete worker1 # [ 129.376565] nix-grpc-daemon[1335]: time=2026-09-25T15:25:56.003Z level=INFO msg="Upload complete. (185ms)" lb # [ 129.381646] niks3-server[1153]: 2026/09/25 15:25:56 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo worker1 # [ 129.382675] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56188 duration_s=0 paths=1 nar_bytes_in=640 worker1 # [ 129.394628] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56188 worker1 # [ 129.399189] nix-daemon[838]: accepted connection from pid 1244, user nix-grpc-daemon (trusted) worker1 # [ 129.403236] nix-daemon[838]: reaped child process 1378, status = succeeded client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051... client # farm-b> node-a: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker1 # [ 129.415047] nix-daemon[838]: accepted connection from pid 1244, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-on-standby worker1 # [ 130.695110] nix-daemon[838]: reaped child process 1387, status = succeeded worker1 # [ 130.770534] nix-daemon[838]: accepted connection from pid 1403, user nix-grpc-daemon (trusted) worker1 # [ 130.786141] nix-daemon[838]: reaped child process 1406, status = succeeded lb # [ 130.792152] niks3-server[1153]: 2026/09/25 15:25:57 INFO Received push request method=POST path=/api/pushes worker1 # [ 130.798739] nix-grpc-daemon[1335]: time=2026-09-25T15:25:57.424Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 130.800294] nix-grpc-daemon[1335]: time=2026-09-25T15:25:57.424Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)" lb # [ 130.814433] niks3-server[1153]: 2026/09/25 15:25:57 INFO Received sign narinfos request method=POST path=/api/pushes/45/sign worker1 # [ 130.820804] nix-grpc-daemon[1335]: time=2026-09-25T15:25:57.447Z level=INFO msg="Uploading 1 narinfos" lb # [ 130.823678] niks3-server[1153]: 2026/09/25 15:25:57 INFO Signed narinfos id=45 count=1 lb # [ 130.829801] niks3-server[1153]: 2026/09/25 15:25:57 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv lb # [ 130.836306] niks3-server[1153]: 2026/09/25 15:25:57 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls lb # [ 130.845546] niks3-server[1153]: 2026/09/25 15:25:57 INFO Received complete push request method=POST path=/api/pushes/45/complete lb # [ 130.853125] niks3-server[1153]: 2026/09/25 15:25:57 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst lb # [ 130.855783] niks3-server[1153]: 2026/09/25 15:25:57 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo worker1 # [ 130.859331] nix-grpc-daemon[1335]: time=2026-09-25T15:25:57.486Z level=INFO msg="Upload complete. (236ms)" worker1 # [ 130.869537] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56188 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=1 inputs=1 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 130.882650] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:57Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=0 worker1 # [ 130.886631] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:57Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 130.890563] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56188 worker1 # [ 130.894226] nix-daemon[838]: reaped child process 1385, status = succeeded worker1 # [ 130.897062] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56188 worker1 # [ 130.902945] nix-grpc-daemon[1335]: time=2026-09-25T15:25:57.530Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'... worker1 # [ 130.909122] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56188 worker1 # [ 130.916777] nix-daemon[838]: accepted connection from pid 1244, user nix-grpc-daemon (trusted) worker1 # [ 131.032686] nix-daemon[838]: accepted connection from pid 1412, user nix-grpc-daemon (trusted) worker1 # [ 131.048160] nix-daemon[838]: reaped child process 1415, status = succeeded lb # [ 131.054385] niks3-server[1153]: 2026/09/25 15:25:57 INFO Received push request method=POST path=/api/pushes worker1 # [ 131.059683] nix-grpc-daemon[1335]: time=2026-09-25T15:25:57.686Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 131.061583] nix-grpc-daemon[1335]: time=2026-09-25T15:25:57.686Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)" lb # [ 131.072704] niks3-server[1153]: 2026/09/25 15:25:57 INFO Received sign narinfos request method=POST path=/api/pushes/46/sign worker1 # [ 131.077719] nix-grpc-daemon[1335]: time=2026-09-25T15:25:57.704Z level=INFO msg="Uploading 1 narinfos" lb # [ 131.081068] niks3-server[1153]: 2026/09/25 15:25:57 INFO Signed narinfos id=46 count=1 lb # [ 131.085066] niks3-server[1153]: 2026/09/25 15:25:57 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst lb # [ 131.094334] niks3-server[1153]: 2026/09/25 15:25:57 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls worker1 # [ 131.100335] nix-grpc-daemon[1335]: time=2026-09-25T15:25:57.727Z level=INFO msg="Upload complete. (174ms)" lb # [ 131.100953] niks3-server[1153]: 2026/09/25 15:25:57 INFO Received complete push request method=POST path=/api/pushes/46/complete lb # [ 131.106388] niks3-server[1153]: 2026/09/25 15:25:57 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo worker1 # [ 131.105417] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56188 duration_s=0 paths=1 nar_bytes_in=768 worker1 # [ 131.111070] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56188 worker1 # [ 131.121150] nix-daemon[838]: accepted connection from pid 1244, user nix-grpc-daemon (trusted) worker1 # [ 131.123194] nix-daemon[838]: reaped child process 1410, status = succeeded client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051... client # farm-top> node-a: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker1 # [ 131.132584] nix-daemon[838]: accepted connection from pid 1244, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-on-standby worker1 # [ 131.966896] nix-daemon[838]: reaped child process 1419, status = succeeded worker1 # [ 132.072873] nix-daemon[838]: accepted connection from pid 1435, user nix-grpc-daemon (trusted) worker1 # [ 132.091168] nix-daemon[838]: reaped child process 1438, status = succeeded lb # [ 132.096483] niks3-server[1153]: 2026/09/25 15:25:58 INFO Received push request method=POST path=/api/pushes worker1 # [ 132.103627] nix-grpc-daemon[1335]: time=2026-09-25T15:25:58.729Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 132.105351] nix-grpc-daemon[1335]: time=2026-09-25T15:25:58.729Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)" lb # [ 132.118119] niks3-server[1153]: 2026/09/25 15:25:58 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst worker1 # [ 132.126281] nix-grpc-daemon[1335]: time=2026-09-25T15:25:58.753Z level=INFO msg="Uploading 1 narinfos" lb # [ 132.126623] niks3-server[1153]: 2026/09/25 15:25:58 INFO Received sign narinfos request method=POST path=/api/pushes/47/sign lb # [ 132.133267] niks3-server[1153]: 2026/09/25 15:25:58 INFO Signed narinfos id=47 count=1 lb # [ 132.135031] niks3-server[1153]: 2026/09/25 15:25:58 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls lb # [ 132.147240] niks3-server[1153]: 2026/09/25 15:25:58 INFO Received complete push request method=POST path=/api/pushes/47/complete worker1 # [ 132.154442] nix-grpc-daemon[1335]: time=2026-09-25T15:25:58.781Z level=INFO msg="Upload complete. (215ms)" lb # [ 132.154515] niks3-server[1153]: 2026/09/25 15:25:58 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv lb # [ 132.157559] niks3-server[1153]: 2026/09/25 15:25:58 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo worker1 # [ 132.162553] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56188 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=1 inputs=2 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 132.173840] nix-daemon[838]: reaped child process 1417, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2, in 5.46 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath, in 0.14 seconds) worker2 # [ 132.374029] systemd[1]: Starting scheduler-token.service... worker2 # [ 132.434898] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 132.440414] systemd[1]: Finished scheduler-token.service. worker2 # [ 132.441757] systemd[1]: scheduler-token.service: Consumed 36ms CPU time over 63ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 132.452295] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 132.557391] nix-grpc-daemon[1521]: ts=2026-09-25T15:25:59Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 132.559874] nix-grpc-daemon[1521]: ts=2026-09-25T15:25:59Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 132.564064] nix-grpc-daemon[1521]: ts=2026-09-25T15:25:59Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 132.570843] nix-grpc-daemon[1521]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 132.592112] nix-grpc-daemon[1521]: E0000 00:00:1790349959.192546 1521 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 132.594691] nix-grpc-daemon[1521]: E0000 00:00:1790349959.194548 1521 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 132.619739] nix-grpc-daemon[1521]: ts=2026-09-25T15:25:59Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 132.621795] nix-grpc-daemon[1521]: E0000 00:00:1790349959.202394 1562 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 132.627499] nix-grpc-daemon[1521]: E0000 00:00:1790349959.226472 1562 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 132.632275] nix-grpc-daemon[1521]: E0000 00:00:1790349959.226625 1562 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 132.635273] nix-grpc-daemon[1521]: E0000 00:00:1790349959.242249 1562 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 132.637889] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 132.639363] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:59Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 132.638766] nix-grpc-daemon[1521]: ts=2026-09-25T15:25:59Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 132.642147] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:59Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:44024 worker1 # [ 132.646860] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:59Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: scheduler down: the other node takes the lock and keeps it, in 11.49 seconds) subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs client # [ 132.841362] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs. client # Running as unit: rs.service; invocation ID: 05b9f34f2cf54e438847431f4e45251c client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs, in 0.06 seconds) worker1 # [ 133.023755] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:59Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:56188 client # [ 133.026948] nix[1418]: don't know how to build these paths: client # [ 133.027971] nix[1418]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker1 # [ 133.033158] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:59Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=0 worker1 # [ 133.038116] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:59Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 133.042882] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56188 worker1 # [ 133.047077] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:56188 client # [ 133.051217] nix[1418]: copying 1 paths to 'lb:50051'... worker1 # [ 133.052890] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56188 client # [ 133.052590] nix[1418]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'... worker1 # [ 133.062104] nix-daemon[838]: accepted connection from pid 1244, user nix-grpc-daemon (trusted) worker1 # [ 133.181117] nix-daemon[838]: accepted connection from pid 1451, user nix-grpc-daemon (trusted) worker1 # [ 133.196624] nix-daemon[838]: reaped child process 1454, status = succeeded lb # [ 133.202649] niks3-server[1153]: 2026/09/25 15:25:59 INFO Received push request method=POST path=/api/pushes worker1 # [ 133.209201] nix-grpc-daemon[1335]: time=2026-09-25T15:25:59.835Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 133.210784] nix-grpc-daemon[1335]: time=2026-09-25T15:25:59.835Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)" lb # [ 133.222295] niks3-server[1153]: 2026/09/25 15:25:59 INFO Received sign narinfos request method=POST path=/api/pushes/48/sign worker1 # [ 133.226893] nix-grpc-daemon[1335]: time=2026-09-25T15:25:59.853Z level=INFO msg="Uploading 1 narinfos" lb # [ 133.228784] niks3-server[1153]: 2026/09/25 15:25:59 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls lb # [ 133.235421] niks3-server[1153]: 2026/09/25 15:25:59 INFO Signed narinfos id=48 count=1 lb # [ 133.243631] niks3-server[1153]: 2026/09/25 15:25:59 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst worker1 # [ 133.249220] nix-grpc-daemon[1335]: time=2026-09-25T15:25:59.876Z level=INFO msg="Upload complete. (177ms)" lb # [ 133.250226] niks3-server[1153]: 2026/09/25 15:25:59 INFO Received complete push request method=POST path=/api/pushes/48/complete lb # [ 133.254017] niks3-server[1153]: 2026/09/25 15:25:59 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo worker1 # [ 133.256622] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56188 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 133.265665] nix-grpc-daemon[1244]: ts=2026-09-25T15:25:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56188 worker1 # [ 133.273035] nix-daemon[838]: accepted connection from pid 1244, user nix-grpc-daemon (trusted) worker1 # [ 133.275601] nix-daemon[838]: reaped child process 1449, status = succeeded client # [ 133.281869] nix[1418]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051... worker1 # [ 133.284813] nix-daemon[838]: accepted connection from pid 1244, user nix-grpc-daemon (trusted) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true worker2 # [ 135.376476] systemd-journald[270]: Received client request to rotate journal, rotating. worker2 # [ 135.387786] systemd-journald[270]: Vacuuming done, freed 0B of archived journals from /var/log/journal/138482321de5455092b057d14e8cf27e. worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.05 seconds) worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service worker1 # [ 135.425605] systemd[1]: Stopping Nix worker-protocol over gRPC... worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds) client: must succeed: sleep 5.0 worker1 # [ 136.063251] nix-grpc-daemon[1244]: ts=2026-09-25T15:26:02Z level=info event=draining lb # [ 138.114404] envoy[902]: conn peer=[2001:db8:1::1]:54932 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9589 tx=4368 ms=5278 client: (finished: must succeed: sleep 5.0, in 5.03 seconds) worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=' worker1 # [ 140.761915] nix-daemon[838]: reaped child process 1458, status = succeeded worker1 # [ 140.839678] nix-daemon[838]: accepted connection from pid 1499, user nix-grpc-daemon (trusted) worker1 # [ 140.855906] nix-daemon[838]: reaped child process 1502, status = succeeded lb # [ 140.862546] niks3-server[1153]: 2026/09/25 15:26:07 INFO Received push request method=POST path=/api/pushes worker1 # [ 140.870417] nix-grpc-daemon[1335]: time=2026-09-25T15:26:07.495Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 140.871984] nix-grpc-daemon[1335]: time=2026-09-25T15:26:07.495Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)" lb # [ 140.886998] niks3-server[1153]: 2026/09/25 15:26:07 INFO Received sign narinfos request method=POST path=/api/pushes/49/sign worker1 # [ 140.893684] nix-grpc-daemon[1335]: time=2026-09-25T15:26:07.520Z level=INFO msg="Uploading 1 narinfos" lb # [ 140.895012] niks3-server[1153]: 2026/09/25 15:26:07 INFO Signed narinfos id=49 count=1 lb # [ 140.899845] niks3-server[1153]: 2026/09/25 15:26:07 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls lb # [ 140.905201] niks3-server[1153]: 2026/09/25 15:26:07 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv lb # [ 140.907966] niks3-server[1153]: 2026/09/25 15:26:07 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst lb # [ 140.919134] niks3-server[1153]: 2026/09/25 15:26:07 INFO Received complete push request method=POST path=/api/pushes/49/complete worker1 # [ 140.922384] nix-grpc-daemon[1335]: time=2026-09-25T15:26:07.549Z level=INFO msg="Upload complete. (217ms)" lb # [ 140.922539] niks3-server[1153]: 2026/09/25 15:26:07 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo worker1 # [ 140.930329] nix-grpc-daemon[1244]: ts=2026-09-25T15:26:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56188 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 140.942523] nix-daemon[838]: reaped child process 1456, status = succeeded client # [ 140.951399] systemd[1]: rs.service: Deactivated successfully. client # [ 140.954449] systemd[1]: rs.service: Consumed 166ms CPU time over 8.111s wall clock time, 41.4M memory peak, 7K incoming IP traffic, 8.3K outgoing IP traffic. worker1 # [ 141.064840] nix-grpc-daemon[1244]: ts=2026-09-25T15:26:07Z level=info event=signal_exit lb # [ 141.250635] niks3-server[1153]: 2026/09/25 15:26:07 INFO lead: released remote=192.168.1.3:57066 worker1 # [ 142.247179] nix-grpc-daemon[1244]: ts=2026-09-25T15:26:08Z level=info event=scheduler_restarting lb # [ 142.247457] envoy[902]: [2026-09-25 15:26:08.870][902][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream} worker1 # [ 142.254218] nix-grpc-daemon[1244]: ts=2026-09-25T15:26:08Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:44024 worker2 # [ 142.252461] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:08Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 142.257026] nix-grpc-daemon[1244]: ts=2026-09-25T15:26:08Z level=info event=worker_gone addr=192.168.1.4:50051 worker1 # [ 142.262142] nix-grpc-daemon[1244]: ts=2026-09-25T15:26:08Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 142.264910] nix-grpc-daemon[1244]: ts=2026-09-25T15:26:08Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:44024 worker1 # [ 142.267759] nix-grpc-daemon[1244]: ts=2026-09-25T15:26:08Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 142.284904] nix-daemon[838]: reaped child process 1296, status = succeeded worker2 # [ 142.452413] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:09Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 142.455388] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 142.470511] nix-grpc-daemon[1244]: ts=2026-09-25T15:26:09Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 142.578679] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:09Z level=info event=scheduler_take_over lb # [ 142.581261] niks3-server[1153]: 2026/09/25 15:26:09 INFO lead: acquired remote=192.168.1.4:60182 worker2 # [ 142.656072] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:09Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 142.658897] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 142.732453] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 142.736238] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 142.737701] systemd[1]: nix-grpc-daemon.service: Consumed 1.336s CPU time over 27.814s wall clock time, 39.8M memory peak, 329.1K incoming IP traffic, 137.7K outgoing IP traffic. worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.11 seconds) client: waiting for success: ! systemctl is-active rs worker1 # [ 142.753288] systemd[1]: Starting Nix worker-protocol over gRPC... client: (finished: waiting for success: ! systemctl is-active rs, in 0.03 seconds) client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; } client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.04 seconds) client: must fail: journalctl -u rs | grep -q 'reconnecting' client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 142.859872] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:09Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 142.863074] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 142.873616] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:09Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 142.875869] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:09Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 142.879948] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:09Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1 # [ 142.886463] nix-grpc-daemon[1510]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 142.887924] nix-grpc-daemon[1510]: E0000 00:00:1790349969.511531 1510 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 142.927893] nix-grpc-daemon[1510]: E0000 00:00:1790349969.513473 1510 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 142.933671] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:09Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 142.936959] nix-grpc-daemon[1510]: E0000 00:00:1790349969.523709 1552 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 142.939434] nix-grpc-daemon[1510]: E0000 00:00:1790349969.539403 1552 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 142.942238] nix-grpc-daemon[1510]: E0000 00:00:1790349969.539545 1552 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 142.944690] nix-grpc-daemon[1510]: E0000 00:00:1790349969.553961 1552 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 142.947047] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 142.947874] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 143.063903] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:09Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 143.066927] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 143.142860] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:09Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 143.145788] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 143.268777] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:09Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 143.271017] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 143.346124] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:09Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 143.347699] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:09Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:47948 worker2 # [ 143.350751] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:09Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker2 # [ 143.469183] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:10Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 143.476084] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:10Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 143.479260] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:10Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:47948 worker2 # [ 143.482373] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:10Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 11.16 seconds) subtest: scheduler restart mid-build: build finishes, no double build client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid client # [ 143.995874] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid. client # Running as unit: mid.service; invocation ID: b671e1ad54e84c90a7ac9b7eed3adfc8 client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid, in 0.05 seconds) worker2 # [ 144.179965] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:47956 worker2 # [ 144.188281] nix-daemon[857]: accepted connection from pid 1521, user nix-grpc-daemon (trusted) client # [ 144.198392] nix[1454]: don't know how to build these paths: client # [ 144.199484] nix[1454]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker2 # [ 144.206473] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:10Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=0 worker2 # [ 144.209741] nix-grpc-daemon[1521]: ts=2026-09-25T15:26:10Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 144.222420] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50094 worker1 # [ 144.230778] nix-daemon[838]: accepted connection from pid 1510, user nix-grpc-daemon (trusted) worker1 # [ 144.244799] nix-grpc-daemon[1510]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 26 ms (attempt 1/5) worker1 # [ 144.275571] nix-grpc-daemon[1510]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 72 ms (attempt 2/5) worker1 # [ 144.348603] nix-grpc-daemon[1510]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 364 ms (attempt 3/5) worker1 # [ 144.713750] nix-grpc-daemon[1510]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 279 ms (attempt 4/5) worker1 # [ 144.994631] nix-grpc-daemon[1510]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 144.997813] nix-grpc-daemon[1510]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 145.008561] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50094 client # [ 145.016127] nix[1454]: copying 1 paths to 'lb:50051'... worker1 # [ 145.017665] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50094 client # [ 145.016365] nix[1454]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'... worker1 # [ 145.026150] nix-daemon[838]: accepted connection from pid 1510, user nix-grpc-daemon (trusted) worker1 # [ 145.042441] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:11Z level=info event=niks3_push_started pid=1574 worker1 # [ 145.188600] nix-grpc-daemon[1574]: time=2026-09-25T15:26:11.813Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 145.303816] nix-daemon[838]: accepted connection from pid 1591, user nix-grpc-daemon (trusted) worker1 # [ 145.322371] nix-daemon[838]: reaped child process 1594, status = succeeded lb # [ 145.329325] niks3-server[1153]: 2026/09/25 15:26:11 INFO Received push request method=POST path=/api/pushes worker1 # [ 145.335187] nix-grpc-daemon[1574]: time=2026-09-25T15:26:11.962Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 145.340516] nix-grpc-daemon[1574]: time=2026-09-25T15:26:11.962Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)" lb # [ 145.357685] niks3-server[1153]: 2026/09/25 15:26:11 INFO Received sign narinfos request method=POST path=/api/pushes/50/sign lb # [ 145.367439] niks3-server[1153]: 2026/09/25 15:26:11 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls lb # [ 145.376932] niks3-server[1153]: 2026/09/25 15:26:11 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst lb # [ 145.387379] niks3-server[1153]: 2026/09/25 15:26:12 INFO Signed narinfos id=50 count=1 worker1 # [ 145.388070] nix-grpc-daemon[1574]: time=2026-09-25T15:26:12.014Z level=INFO msg="Uploading 1 narinfos" lb # [ 145.398576] niks3-server[1153]: 2026/09/25 15:26:12 INFO Received complete push request method=POST path=/api/pushes/50/complete worker1 # [ 145.402991] nix-grpc-daemon[1574]: time=2026-09-25T15:26:12.030Z level=INFO msg="Upload complete. (216ms)" lb # [ 145.405153] niks3-server[1153]: 2026/09/25 15:26:12 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo worker1 # [ 145.410172] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50094 duration_s=0 paths=1 nar_bytes_in=424 worker1 # [ 145.421844] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50094 worker1 # [ 145.427259] nix-daemon[838]: accepted connection from pid 1510, user nix-grpc-daemon (trusted) worker1 # [ 145.431444] nix-daemon[838]: reaped child process 1572, status = succeeded worker1 # [ 145.441126] nix-daemon[838]: accepted connection from pid 1510, user nix-grpc-daemon (trusted) client # [ 145.438965] nix[1454]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051... lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service worker2 # [ 147.671776] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1521 (nix-grpc-daemon) on client request. worker2 # [ 147.680342] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request. lb # [ 147.694901] niks3-server[1153]: 2026/09/25 15:26:14 INFO lead: released remote=192.168.1.4:60182 worker1 # [ 147.699687] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:14Z level=info event=scheduler_disconnected addr=lb:50051 error="Stream removed (RST_STREAM (Received RST_STREAM with error code 2))" worker2 # [ 147.698095] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL client # [ 147.698496] nix[1454]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting worker2 # [ 147.711699] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'. worker2 # [ 147.720864] systemd[1]: nix-grpc-daemon.service: Consumed 216ms CPU time over 15.250s wall clock time, 14.6M memory peak, 32.4K incoming IP traffic, 42K outgoing IP traffic. worker2 # [ 147.731242] systemd[1]: Starting scheduler-token.service... worker2 # [ 147.736516] nix-daemon[857]: reaped child process 1630, status = succeeded worker2 # [ 147.777859] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 147.780976] systemd[1]: Finished scheduler-token.service. worker2 # [ 147.782061] systemd[1]: scheduler-token.service: Consumed 26ms CPU time over 49ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 147.792177] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1. worker2 # [ 147.794066] systemd[1]: Starting scheduler-token.service... worker2 # [ 147.834833] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 147.837694] systemd[1]: Finished scheduler-token.service. worker2 # [ 147.838645] systemd[1]: scheduler-token.service: Consumed 25ms CPU time over 48ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 147.844339] systemd[1]: Starting Nix worker-protocol over gRPC... lb # [ 147.896714] niks3-server[1153]: 2026/09/25 15:26:14 INFO lead: acquired remote=192.168.1.3:44396 worker1 # [ 147.899041] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:14Z level=info event=scheduler_take_over worker1 # [ 147.901130] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:14Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 147.916065] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:14Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure" worker2 # [ 147.938919] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:14Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 147.941275] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:14Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 147.945127] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:14Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 147.950413] nix-grpc-daemon[1664]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 147.951982] nix-grpc-daemon[1664]: E0000 00:00:1790349974.573366 1664 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 147.955965] nix-grpc-daemon[1664]: E0000 00:00:1790349974.579591 1664 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.35 seconds) client: must succeed: sleep 5.0 worker2 # [ 147.998407] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:14Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 148.000803] nix-grpc-daemon[1664]: E0000 00:00:1790349974.588130 1704 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 148.006368] nix-grpc-daemon[1664]: E0000 00:00:1790349974.605800 1704 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 148.012870] nix-grpc-daemon[1664]: E0000 00:00:1790349974.605944 1704 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 148.016394] nix-grpc-daemon[1664]: E0000 00:00:1790349974.621356 1704 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 148.019482] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 148.020859] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:14Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 148.022214] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:14Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" lb # [ 148.113926] envoy[902]: conn peer=[2001:db8:1::1]:60340 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2962 tx=1497 ms=7916 worker1 # [ 148.116597] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:14Z level=info event=scheduler_connected addr=lb:50051 lb # [ 148.117418] envoy[902]: conn peer=[2001:db8:1::3]:37082 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=3527 tx=1140 ms=27360 worker1 # [ 148.120777] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:14Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" lb # [ 148.120936] envoy[902]: conn peer=[2001:db8:1::4]:39372 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=5710 tx=512 ms=15076 client # [ 148.201283] nix[1454]: scheduler at lb:50051: scheduler passive, another node serves, reconnecting worker2 # [ 148.221554] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:14Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 148.225049] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:14Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" worker1 # [ 148.321215] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:14Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 148.324135] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:14Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" worker2 # [ 148.425733] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 148.428574] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 148.523535] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 148.525493] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 148.629266] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 148.632232] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 148.726495] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 148.727880] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 148.832687] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 148.835440] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 148.926879] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 148.928720] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 149.036918] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 149.039052] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 149.129020] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 149.130897] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" client # [ 149.205627] nix[1454]: scheduler at lb:50051: no healthy upstream, reconnecting worker2 # [ 149.237333] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 149.240258] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 149.331241] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 149.333046] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 149.440846] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 149.452694] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:16Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 149.454062] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:16Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:50096 worker1 # [ 149.455755] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:16Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 149.533324] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 149.536140] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:16Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:50096 worker1 # [ 149.537794] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:16Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=1 worker1 # [ 151.586150] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:18Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:36568 client: (finished: must succeed: sleep 5.0, in 5.03 seconds) client: waiting for success: ! systemctl is-active mid worker1 # [ 153.285051] nix-daemon[838]: reaped child process 1598, status = succeeded worker1 # [ 153.405749] nix-daemon[838]: accepted connection from pid 1635, user nix-grpc-daemon (trusted) worker1 # [ 153.427687] nix-daemon[838]: reaped child process 1638, status = succeeded lb # [ 153.426452] niks3-server[1153]: 2026/09/25 15:26:20 INFO Received push request method=POST path=/api/pushes worker1 # [ 153.436511] nix-grpc-daemon[1574]: time=2026-09-25T15:26:20.062Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 153.438108] nix-grpc-daemon[1574]: time=2026-09-25T15:26:20.062Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)" lb # [ 153.470487] niks3-server[1153]: 2026/09/25 15:26:20 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst worker1 # [ 153.476812] nix-grpc-daemon[1574]: time=2026-09-25T15:26:20.103Z level=INFO msg="Uploading 1 narinfos" lb # [ 153.478919] niks3-server[1153]: 2026/09/25 15:26:20 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv lb # [ 153.486057] niks3-server[1153]: 2026/09/25 15:26:20 INFO Received sign narinfos request method=POST path=/api/pushes/51/sign lb # [ 153.493241] niks3-server[1153]: 2026/09/25 15:26:20 INFO Signed narinfos id=51 count=1 lb # [ 153.494849] niks3-server[1153]: 2026/09/25 15:26:20 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls lb # [ 153.502069] niks3-server[1153]: 2026/09/25 15:26:20 INFO Received complete push request method=POST path=/api/pushes/51/complete worker1 # [ 153.505439] nix-grpc-daemon[1574]: time=2026-09-25T15:26:20.132Z level=INFO msg="Upload complete. (249ms)" lb # [ 153.506398] niks3-server[1153]: 2026/09/25 15:26:20 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo worker1 # [ 153.516827] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50094 duration_s=8 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 153.528773] nix-daemon[838]: reaped child process 1596, status = succeeded client # [ 153.535200] systemd[1]: mid.service: Deactivated successfully. client # [ 153.537655] systemd[1]: mid.service: Consumed 174ms CPU time over 9.541s wall clock time, 41.5M memory peak, 8.1K incoming IP traffic, 9.2K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active mid, in 1.07 seconds) client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; } client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.04 seconds) (finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.37 seconds) subtest: interrupting the client stops the build on the worker client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr client # [ 154.361240] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr. client # Running as unit: intr.service; invocation ID: a0a0483c29da436f88430767f61c3365 client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr, in 0.05 seconds) client # [ 154.538234] nix[1490]: don't know how to build these paths: worker1 # [ 154.538373] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:21Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:50094 client # [ 154.539239] nix[1490]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker1 # [ 154.549504] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:21Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=0 worker1 # [ 154.553362] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:21Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 154.558169] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 154.567455] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker2 # [ 154.583567] nix-grpc-daemon[1664]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 30 ms (attempt 1/5) worker2 # [ 154.619279] nix-grpc-daemon[1664]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 51 ms (attempt 2/5) worker2 # [ 154.672919] nix-grpc-daemon[1664]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 119 ms (attempt 3/5) worker2 # [ 154.794410] nix-grpc-daemon[1664]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 651 ms (attempt 4/5) worker2 # [ 155.447751] nix-grpc-daemon[1664]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 155.451026] nix-grpc-daemon[1664]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 155.465427] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43716 client # [ 155.476142] nix[1490]: copying 1 paths to 'lb:50051'... client # [ 155.478888] nix[1490]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'... worker2 # [ 155.479508] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 155.489621] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker2 # [ 155.509282] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:22Z level=info event=niks3_push_started pid=1736 worker2 # [ 155.677244] nix-grpc-daemon[1736]: time=2026-09-25T15:26:22.298Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 155.802202] nix-daemon[857]: accepted connection from pid 1753, user nix-grpc-daemon (trusted) worker2 # [ 155.824426] nix-daemon[857]: reaped child process 1756, status = succeeded lb # [ 155.834685] niks3-server[1153]: 2026/09/25 15:26:22 INFO Received push request method=POST path=/api/pushes worker2 # [ 155.840365] nix-grpc-daemon[1736]: time=2026-09-25T15:26:22.464Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 155.845740] nix-grpc-daemon[1736]: time=2026-09-25T15:26:22.466Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)" lb # [ 155.868689] niks3-server[1153]: 2026/09/25 15:26:22 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls worker2 # [ 155.872095] nix-grpc-daemon[1736]: time=2026-09-25T15:26:22.495Z level=INFO msg="Uploading 1 narinfos" lb # [ 155.876370] niks3-server[1153]: 2026/09/25 15:26:22 INFO Received sign narinfos request method=POST path=/api/pushes/52/sign lb # [ 155.879395] niks3-server[1153]: 2026/09/25 15:26:22 INFO Signed narinfos id=52 count=1 lb # [ 155.890599] niks3-server[1153]: 2026/09/25 15:26:22 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst lb # [ 155.903237] niks3-server[1153]: 2026/09/25 15:26:22 INFO Received complete push request method=POST path=/api/pushes/52/complete lb # [ 155.911559] niks3-server[1153]: 2026/09/25 15:26:22 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo worker2 # [ 155.913754] nix-grpc-daemon[1736]: time=2026-09-25T15:26:22.537Z level=INFO msg="Upload complete. (238ms)" worker2 # [ 155.921788] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=0 paths=1 nar_bytes_in=424 worker2 # [ 155.934426] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 155.939949] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker2 # [ 155.944984] nix-daemon[857]: reaped child process 1734, status = succeeded client # [ 155.955585] nix[1490]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051... worker2 # [ 155.958034] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) client: must succeed: systemctl kill -s INT intr client # [ 157.952463] systemd[1]: intr.service: Sent signal SIGINT to main process 1490 (nix) on client request. worker2 # [ 157.956129] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:24Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv client # [ 157.957605] nix[1490]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client # [ 157.963597] nix[1490]: error: interrupted by the user client: (finished: must succeed: systemctl kill -s INT intr, in 0.04 seconds) client # [ 157.979686] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE client # [ 157.983089] systemd[1]: intr.service: Failed with result 'exit-code'. client # [ 157.984976] systemd[1]: intr.service: Consumed 164ms CPU time over 3.620s wall clock time, 41.4M memory peak, 6.5K incoming IP traffic, 8.3K outgoing IP traffic. worker2 # [ 157.985975] nix-daemon[857]: reaped child process 1758, status = succeeded worker2 # [ 157.988490] nix-daemon[1760]: unexpected Nix daemon error: error: interrupted by the user worker2 # [ 157.998018] nix-daemon[857]: reaped child process 1760, status = failed with exit code 1 (finished: subtest: interrupting the client stops the build on the worker, in 3.73 seconds) subtest: a deploy mid-build drains: the build finishes, the new generation takes over client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw client # [ 158.100480] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw. client # Running as unit: sw.service; invocation ID: 1d879ea566b34e6abd8aad404ebf9817 lb # [ 158.114220] envoy[902]: conn peer=[2001:db8:1::1]:39336 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3215 tx=1757 ms=9347 lb # [ 158.117010] envoy[902]: conn peer=[2001:db8:1::1]:41058 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3000 tx=850 ms=3435 client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw, in 0.06 seconds) worker2 # [ 158.284420] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:43716 client # [ 158.291227] nix[1511]: don't know how to build these paths: client # [ 158.293523] nix[1511]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker1 # [ 158.303496] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:24Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=0 worker1 # [ 158.303727] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:24Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 158.311142] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50094 worker1 # [ 158.321934] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:50094 client # [ 158.330216] nix[1511]: copying 1 paths to 'lb:50051'... worker1 # [ 158.331811] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50094 client # [ 158.331728] nix[1511]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'... worker1 # [ 158.341254] nix-daemon[838]: accepted connection from pid 1510, user nix-grpc-daemon (trusted) worker1 # [ 158.469717] nix-daemon[838]: accepted connection from pid 1670, user nix-grpc-daemon (trusted) worker1 # [ 158.490568] nix-daemon[838]: reaped child process 1673, status = succeeded lb # [ 158.489921] niks3-server[1153]: 2026/09/25 15:26:25 INFO Received push request method=POST path=/api/pushes worker1 # [ 158.494789] nix-grpc-daemon[1574]: time=2026-09-25T15:26:25.121Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 158.496540] nix-grpc-daemon[1574]: time=2026-09-25T15:26:25.121Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)" lb # [ 158.507934] niks3-server[1153]: 2026/09/25 15:26:25 INFO Received sign narinfos request method=POST path=/api/pushes/53/sign worker1 # [ 158.512042] nix-grpc-daemon[1574]: time=2026-09-25T15:26:25.138Z level=INFO msg="Uploading 1 narinfos" lb # [ 158.514869] niks3-server[1153]: 2026/09/25 15:26:25 INFO Signed narinfos id=53 count=1 lb # [ 158.520693] niks3-server[1153]: 2026/09/25 15:26:25 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls lb # [ 158.523189] niks3-server[1153]: 2026/09/25 15:26:25 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst lb # [ 158.534105] niks3-server[1153]: 2026/09/25 15:26:25 INFO Received complete push request method=POST path=/api/pushes/53/complete worker1 # [ 158.538055] nix-grpc-daemon[1574]: time=2026-09-25T15:26:25.165Z level=INFO msg="Upload complete. (186ms)" lb # [ 158.538780] niks3-server[1153]: 2026/09/25 15:26:25 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo worker1 # [ 158.544554] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50094 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 158.547487] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50094 worker1 # [ 158.558752] nix-daemon[838]: accepted connection from pid 1510, user nix-grpc-daemon (trusted) worker1 # [ 158.562581] nix-daemon[838]: reaped child process 1668, status = succeeded worker1 # [ 158.570853] nix-daemon[838]: accepted connection from pid 1510, user nix-grpc-daemon (trusted) client # [ 158.568655] nix[1511]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051... worker1: must succeed: systemctl reload nix-grpc-daemon.service worker1 # [ 160.519098] systemd[1]: Reloading Nix worker-protocol over gRPC... worker1 # [ 160.561227] systemd[1]: Reloaded Nix worker-protocol over gRPC. worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.08 seconds) worker1: must succeed: pgrep -f 'read -t [9]0 x' worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.04 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2 worker2 # [ 160.783214] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:43716 client # don't know how to build these paths: client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker2 # [ 160.790243] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43716 worker1 # [ 160.808675] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:27Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=0 worker1 # [ 160.810735] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:27Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 160.812973] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 160.822063] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43716 client # copying 1 paths to 'lb:50051'... worker2 # [ 160.831353] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'... worker2 # [ 160.844640] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker1 # [ 160.898048] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:27Z level=info event=draining worker2 # [ 160.972218] nix-daemon[857]: accepted connection from pid 1807, user nix-grpc-daemon (trusted) worker2 # [ 160.989166] nix-daemon[857]: reaped child process 1810, status = succeeded lb # [ 160.998422] niks3-server[1153]: 2026/09/25 15:26:27 INFO Received push request method=POST path=/api/pushes worker2 # [ 161.004685] nix-grpc-daemon[1736]: time=2026-09-25T15:26:27.627Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 161.006353] nix-grpc-daemon[1736]: time=2026-09-25T15:26:27.627Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)" lb # [ 161.023011] niks3-server[1153]: 2026/09/25 15:26:27 INFO Received sign narinfos request method=POST path=/api/pushes/54/sign worker2 # [ 161.026677] nix-grpc-daemon[1736]: time=2026-09-25T15:26:27.650Z level=INFO msg="Uploading 1 narinfos" lb # [ 161.031654] niks3-server[1153]: 2026/09/25 15:26:27 INFO Signed narinfos id=54 count=1 lb # [ 161.035981] niks3-server[1153]: 2026/09/25 15:26:27 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst lb # [ 161.043399] niks3-server[1153]: 2026/09/25 15:26:27 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls lb # [ 161.050627] niks3-server[1153]: 2026/09/25 15:26:27 INFO Received complete push request method=POST path=/api/pushes/54/complete lb # [ 161.062146] niks3-server[1153]: 2026/09/25 15:26:27 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo worker2 # [ 161.061308] nix-grpc-daemon[1736]: time=2026-09-25T15:26:27.683Z level=INFO msg="Upload complete. (205ms)" worker2 # [ 161.066797] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=0 paths=1 nar_bytes_in=528 worker2 # [ 161.077436] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 161.082731] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker2 # [ 161.086691] nix-daemon[857]: reaped child process 1805, status = succeeded client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051... client # farm-a> worker2: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker2 # [ 161.096345] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-during-drain worker2 # [ 162.552424] nix-daemon[857]: reaped child process 1814, status = succeeded worker2 # [ 162.649017] nix-daemon[857]: accepted connection from pid 1830, user nix-grpc-daemon (trusted) lb # [ 162.669728] niks3-server[1153]: 2026/09/25 15:26:29 INFO Received push request method=POST path=/api/pushes worker2 # [ 162.670229] nix-daemon[857]: reaped child process 1833, status = succeeded worker2 # [ 162.680428] nix-grpc-daemon[1736]: time=2026-09-25T15:26:29.302Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 162.682006] nix-grpc-daemon[1736]: time=2026-09-25T15:26:29.302Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)" lb # [ 162.696709] niks3-server[1153]: 2026/09/25 15:26:29 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv lb # [ 162.704202] niks3-server[1153]: 2026/09/25 15:26:29 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst worker2 # [ 162.708171] nix-grpc-daemon[1736]: time=2026-09-25T15:26:29.331Z level=INFO msg="Uploading 1 narinfos" lb # [ 162.711190] niks3-server[1153]: 2026/09/25 15:26:29 INFO Received sign narinfos request method=POST path=/api/pushes/55/sign lb # [ 162.717172] niks3-server[1153]: 2026/09/25 15:26:29 INFO Signed narinfos id=55 count=1 lb # [ 162.721014] niks3-server[1153]: 2026/09/25 15:26:29 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls lb # [ 162.728674] niks3-server[1153]: 2026/09/25 15:26:29 INFO Received complete push request method=POST path=/api/pushes/55/complete worker2 # [ 162.733066] nix-grpc-daemon[1736]: time=2026-09-25T15:26:29.356Z level=INFO msg="Upload complete. (226ms)" lb # [ 162.735871] niks3-server[1153]: 2026/09/25 15:26:29 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo worker1 # [ 162.745652] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:29Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=0 worker2 # [ 162.740674] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=3 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 162.747605] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:29Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 162.757025] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 162.761820] nix-daemon[857]: reaped child process 1812, status = succeeded worker2 # [ 162.769917] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 162.780458] nix-grpc-daemon[1736]: time=2026-09-25T15:26:29.404Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker2 # [ 162.787525] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'... worker2 # [ 162.800637] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker2 # [ 162.927920] nix-daemon[857]: accepted connection from pid 1837, user nix-grpc-daemon (trusted) worker2 # [ 162.951288] nix-daemon[857]: reaped child process 1840, status = succeeded lb # [ 162.951823] niks3-server[1153]: 2026/09/25 15:26:29 INFO Received push request method=POST path=/api/pushes worker2 # [ 162.957939] nix-grpc-daemon[1736]: time=2026-09-25T15:26:29.581Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 162.960813] nix-grpc-daemon[1736]: time=2026-09-25T15:26:29.581Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)" lb # [ 162.990401] niks3-server[1153]: 2026/09/25 15:26:29 INFO Received sign narinfos request method=POST path=/api/pushes/56/sign worker2 # [ 162.994316] nix-grpc-daemon[1736]: time=2026-09-25T15:26:29.617Z level=INFO msg="Uploading 1 narinfos" lb # [ 162.999387] niks3-server[1153]: 2026/09/25 15:26:29 INFO Signed narinfos id=56 count=1 lb # [ 163.007983] niks3-server[1153]: 2026/09/25 15:26:29 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst lb # [ 163.014665] niks3-server[1153]: 2026/09/25 15:26:29 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls lb # [ 163.021995] niks3-server[1153]: 2026/09/25 15:26:29 INFO Received complete push request method=POST path=/api/pushes/56/complete worker2 # [ 163.027343] nix-grpc-daemon[1736]: time=2026-09-25T15:26:29.651Z level=INFO msg="Upload complete. (216ms)" lb # [ 163.030498] niks3-server[1153]: 2026/09/25 15:26:29 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo worker2 # [ 163.036331] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=0 paths=1 nar_bytes_in=640 worker2 # [ 163.041846] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 163.055796] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker2 # [ 163.059961] nix-daemon[857]: reaped child process 1835, status = succeeded client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051... worker2 # [ 163.070824] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) client # farm-b> worker2: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-during-drain worker2 # [ 164.569298] nix-daemon[857]: reaped child process 1844, status = succeeded worker2 # [ 164.625881] nix-daemon[857]: accepted connection from pid 1860, user nix-grpc-daemon (trusted) worker2 # [ 164.650007] nix-daemon[857]: reaped child process 1863, status = succeeded lb # [ 164.657886] niks3-server[1153]: 2026/09/25 15:26:31 INFO Received push request method=POST path=/api/pushes worker2 # [ 164.662796] nix-grpc-daemon[1736]: time=2026-09-25T15:26:31.285Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 164.664878] nix-grpc-daemon[1736]: time=2026-09-25T15:26:31.285Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)" lb # [ 164.683117] niks3-server[1153]: 2026/09/25 15:26:31 INFO Received sign narinfos request method=POST path=/api/pushes/57/sign worker2 # [ 164.686645] nix-grpc-daemon[1736]: time=2026-09-25T15:26:31.310Z level=INFO msg="Uploading 1 narinfos" lb # [ 164.693872] niks3-server[1153]: 2026/09/25 15:26:31 INFO Signed narinfos id=57 count=1 lb # [ 164.695630] niks3-server[1153]: 2026/09/25 15:26:31 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls lb # [ 164.708750] niks3-server[1153]: 2026/09/25 15:26:31 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst lb # [ 164.721127] niks3-server[1153]: 2026/09/25 15:26:31 INFO Received complete push request method=POST path=/api/pushes/57/complete worker2 # [ 164.724831] nix-grpc-daemon[1736]: time=2026-09-25T15:26:31.348Z level=INFO msg="Upload complete. (236ms)" lb # [ 164.731722] niks3-server[1153]: 2026/09/25 15:26:31 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo lb # [ 164.736121] niks3-server[1153]: 2026/09/25 15:26:31 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker2 # [ 164.732733] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=4 outputs=1 inputs=1 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 164.742835] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:31Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=0 worker1 # [ 164.745276] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:31Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 164.746949] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 164.750142] nix-daemon[857]: reaped child process 1842, status = succeeded worker2 # [ 164.758494] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 164.767959] nix-grpc-daemon[1736]: time=2026-09-25T15:26:31.391Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... worker2 # [ 164.774860] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'... worker2 # [ 164.788369] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker2 # [ 164.914780] nix-daemon[857]: accepted connection from pid 1867, user nix-grpc-daemon (trusted) worker2 # [ 164.934401] nix-daemon[857]: reaped child process 1870, status = succeeded lb # [ 164.941543] niks3-server[1153]: 2026/09/25 15:26:31 INFO Received push request method=POST path=/api/pushes worker2 # [ 164.947609] nix-grpc-daemon[1736]: time=2026-09-25T15:26:31.571Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 164.949322] nix-grpc-daemon[1736]: time=2026-09-25T15:26:31.571Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)" lb # [ 164.968055] niks3-server[1153]: 2026/09/25 15:26:31 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst worker2 # [ 164.972123] nix-grpc-daemon[1736]: time=2026-09-25T15:26:31.595Z level=INFO msg="Uploading 1 narinfos" lb # [ 164.977029] niks3-server[1153]: 2026/09/25 15:26:31 INFO Received sign narinfos request method=POST path=/api/pushes/58/sign lb # [ 164.979684] niks3-server[1153]: 2026/09/25 15:26:31 INFO Signed narinfos id=58 count=1 lb # [ 164.992774] niks3-server[1153]: 2026/09/25 15:26:31 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls lb # [ 164.995514] niks3-server[1153]: 2026/09/25 15:26:31 INFO Received complete push request method=POST path=/api/pushes/58/complete worker2 # [ 164.999716] nix-grpc-daemon[1736]: time=2026-09-25T15:26:31.623Z level=INFO msg="Upload complete. (201ms)" lb # [ 165.004143] niks3-server[1153]: 2026/09/25 15:26:31 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo worker2 # [ 165.008557] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=0 paths=1 nar_bytes_in=768 worker2 # [ 165.011938] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 165.022883] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker2 # [ 165.028229] nix-daemon[857]: reaped child process 1865, status = succeeded client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051... client # farm-top> worker2: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker2 # [ 165.038559] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-during-drain worker2 # [ 166.465349] nix-daemon[857]: reaped child process 1874, status = succeeded worker2 # [ 166.589510] nix-daemon[857]: accepted connection from pid 1890, user nix-grpc-daemon (trusted) worker2 # [ 166.609828] nix-daemon[857]: reaped child process 1893, status = succeeded lb # [ 166.619669] niks3-server[1153]: 2026/09/25 15:26:33 INFO Received push request method=POST path=/api/pushes worker2 # [ 166.624447] nix-grpc-daemon[1736]: time=2026-09-25T15:26:33.247Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 166.626060] nix-grpc-daemon[1736]: time=2026-09-25T15:26:33.247Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)" lb # [ 166.649958] niks3-server[1153]: 2026/09/25 15:26:33 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls lb # [ 166.659765] niks3-server[1153]: 2026/09/25 15:26:33 INFO Received sign narinfos request method=POST path=/api/pushes/59/sign worker2 # [ 166.661867] nix-grpc-daemon[1736]: time=2026-09-25T15:26:33.285Z level=INFO msg="Uploading 1 narinfos" lb # [ 166.667371] niks3-server[1153]: 2026/09/25 15:26:33 INFO Signed narinfos id=59 count=1 lb # [ 166.674310] niks3-server[1153]: 2026/09/25 15:26:33 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv lb # [ 166.680427] niks3-server[1153]: 2026/09/25 15:26:33 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst lb # [ 166.688734] niks3-server[1153]: 2026/09/25 15:26:33 INFO Received complete push request method=POST path=/api/pushes/59/complete worker2 # [ 166.697950] nix-grpc-daemon[1736]: time=2026-09-25T15:26:33.321Z level=INFO msg="Upload complete. (250ms)" lb # [ 166.700650] niks3-server[1153]: 2026/09/25 15:26:33 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo worker2 # [ 166.704713] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=5 outputs=1 inputs=2 inputs_fetched=0 input_bytes=0 input_s=0 worker2 # [ 166.719352] nix-daemon[857]: reaped child process 1872, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2, in 6.12 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath, in 0.14 seconds) worker1: must succeed: pkill -f 'read -t [9]0 x' worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds) client: waiting for success: ! systemctl is-active sw worker1 # [ 167.047690] nix-daemon[838]: reaped child process 1677, status = succeeded worker1 # [ 167.108766] nix-daemon[838]: accepted connection from pid 1724, user nix-grpc-daemon (trusted) worker1 # [ 167.124358] nix-daemon[838]: reaped child process 1727, status = succeeded lb # [ 167.130035] niks3-server[1153]: 2026/09/25 15:26:33 INFO Received push request method=POST path=/api/pushes worker1 # [ 167.136251] nix-grpc-daemon[1574]: time=2026-09-25T15:26:33.762Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 167.137826] nix-grpc-daemon[1574]: time=2026-09-25T15:26:33.762Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)" lb # [ 167.154820] niks3-server[1153]: 2026/09/25 15:26:33 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls lb # [ 167.161826] niks3-server[1153]: 2026/09/25 15:26:33 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst worker1 # [ 167.169202] nix-grpc-daemon[1574]: time=2026-09-25T15:26:33.796Z level=INFO msg="Uploading 1 narinfos" lb # [ 167.169276] niks3-server[1153]: 2026/09/25 15:26:33 INFO Received sign narinfos request method=POST path=/api/pushes/60/sign lb # [ 167.171774] niks3-server[1153]: 2026/09/25 15:26:33 INFO Signed narinfos id=60 count=1 lb # [ 167.180563] niks3-server[1153]: 2026/09/25 15:26:33 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv lb # [ 167.190406] niks3-server[1153]: 2026/09/25 15:26:33 INFO Received complete push request method=POST path=/api/pushes/60/complete worker1 # [ 167.195452] nix-grpc-daemon[1574]: time=2026-09-25T15:26:33.822Z level=INFO msg="Upload complete. (208ms)" lb # [ 167.195889] niks3-server[1153]: 2026/09/25 15:26:33 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo worker1 # [ 167.204630] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50094 duration_s=8 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=2 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 167.215934] nix-daemon[838]: reaped child process 1675, status = succeeded client # [ 167.223040] systemd[1]: sw.service: Deactivated successfully. client # [ 167.226174] systemd[1]: sw.service: Consumed 174ms CPU time over 9.124s wall clock time, 41.5M memory peak, 7.1K incoming IP traffic, 8.3K outgoing IP traffic. worker1 # [ 167.900410] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:34Z level=info event=signal_exit client: (finished: waiting for success: ! systemctl is-active sw, in 1.07 seconds) client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; } client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.04 seconds) worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service lb # [ 168.114027] envoy[902]: conn peer=[2001:db8:1::1]:41078 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9634 tx=4379 ms=5937 lb # [ 168.116823] envoy[902]: conn peer=[2001:db8:1::1]:41070 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2961 tx=1498 ms=8926 lb # [ 168.904653] niks3-server[1153]: 2026/09/25 15:26:35 INFO lead: released remote=192.168.1.3:44396 worker1 # [ 169.900917] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:36Z level=info event=scheduler_restarting worker2 # [ 169.901250] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:36Z level=info event=scheduler_restarting addr=lb:50051 lb # [ 169.901671] envoy[902]: [2026-09-25 15:26:36.524][902][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream} worker1 # [ 169.904846] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:36Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 169.910326] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:36Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:50096 worker1 # [ 169.913773] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:36Z level=info event=worker_gone addr=192.168.1.4:50051 worker1 # [ 169.916098] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:36Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:50096 worker1 # [ 169.918645] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:36Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 169.933147] nix-daemon[838]: reaped child process 1566, status = succeeded worker2 # [ 170.101705] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 170.105187] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 170.116791] nix-grpc-daemon[1510]: ts=2026-09-25T15:26:36Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 170.305866] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:36Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 170.308792] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 170.509742] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 170.512833] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 170.713596] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 170.717004] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 170.913874] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 170.914979] systemd[1]: nix-grpc-daemon.service: Consumed 897ms CPU time over 28.161s wall clock time, 38.1M memory peak, 245.7K incoming IP traffic, 86.3K outgoing IP traffic. worker2 # [ 170.918320] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 170.921250] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 170.928158] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1. worker1 # [ 170.933240] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 171.054123] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:37Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 171.056263] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:37Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 171.060076] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:37Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 171.066616] nix-grpc-daemon[1742]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR lb # [ 171.076639] niks3-server[1153]: 2026/09/25 15:26:37 INFO lead: acquired remote=192.168.1.3:56838 worker1 # [ 171.088534] nix-grpc-daemon[1742]: E0000 00:00:1790349997.691562 1742 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 171.091010] nix-grpc-daemon[1742]: E0000 00:00:1790349997.693614 1742 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 3.08 seconds) lb: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 171.115305] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:37Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 171.117750] nix-grpc-daemon[1742]: E0000 00:00:1790349997.703125 1783 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 171.123398] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:37Z level=info event=scheduler_take_over worker2 # [ 171.122049] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 171.126772] nix-grpc-daemon[1742]: E0000 00:00:1790349997.725991 1783 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 171.125195] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 171.129317] nix-grpc-daemon[1742]: E0000 00:00:1790349997.726139 1783 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 171.131718] nix-grpc-daemon[1742]: E0000 00:00:1790349997.741574 1783 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 171.134231] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 171.135077] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 171.136404] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1 # [ 171.327267] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:37Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 171.325879] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:37Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 171.330148] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 171.328735] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 171.530600] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:38Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 171.529467] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:38Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 171.536099] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:38Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:58010 worker1 # [ 171.538714] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:38Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 171.545437] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:38Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 171.547529] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:38Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:58010 worker1 # [ 171.550636] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:38Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1 # [ 172.229739] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:38Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:58020 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: (finished: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux, in 2.12 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 15.24 seconds) subtest: queue gauges show a burst while every build blocks lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) client: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]' client # [ 173.445859] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto --impure --expr "map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ \"q0\" \"q1\" \"q2\" \"q3\" ]". client # Running as unit: burst.service; invocation ID: ea9f9e8e30ef478bbce119b94d9b89f3 client: (finished: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]', in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.03 seconds) worker2 # [ 173.624281] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:43716 client # [ 173.629689] nix[1567]: don't know how to build these paths: client # [ 173.630702] nix[1567]: /nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv client # [ 173.632399] nix[1567]: /nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv client # [ 173.633451] nix[1567]: /nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv client # [ 173.634447] nix[1567]: /nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker1 # [ 173.648271] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:40Z level=debug event=want client=1 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv cached=0 worker1 # [ 173.652415] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:40Z level=debug event=assigned drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv worker=192.168.1.3:50051 clients=1 worker2 # [ 173.651703] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker1 # [ 173.657321] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:40Z level=debug event=want client=1 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv cached=0 worker1 # [ 173.661985] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:40Z level=debug event=assigned drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 173.661934] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43716 worker1 # [ 173.670641] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:40Z level=debug event=want client=1 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv cached=0 client # [ 173.671252] nix[1567]: copying 1 paths to 'lb:50051'... worker2 # [ 173.670863] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 client # [ 173.672725] nix[1567]: copying path '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' to 'lb:50051'... worker1 # [ 173.679322] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:40Z level=debug event=want client=1 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv cached=0 worker2 # [ 173.680617] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker1 # [ 173.684451] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 173.690660] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 173.693933] nix-grpc-daemon[1742]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 47 ms (attempt 1/5) worker1 # [ 173.742129] nix-grpc-daemon[1742]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 198 ms (attempt 2/5) worker2 # [ 173.811609] nix-daemon[857]: accepted connection from pid 1897, user nix-grpc-daemon (trusted) worker2 # [ 173.833026] nix-daemon[857]: reaped child process 1900, status = succeeded lb # [ 173.833741] niks3-server[1153]: 2026/09/25 15:26:40 INFO Received push request method=POST path=/api/pushes worker2 # [ 173.837455] nix-grpc-daemon[1736]: time=2026-09-25T15:26:40.460Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 173.839092] nix-grpc-daemon[1736]: time=2026-09-25T15:26:40.460Z level=INFO msg="Uploading dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv (416B)" lb # [ 173.850377] niks3-server[1153]: 2026/09/25 15:26:40 INFO Received sign narinfos request method=POST path=/api/pushes/61/sign worker2 # [ 173.851276] nix-grpc-daemon[1736]: time=2026-09-25T15:26:40.474Z level=INFO msg="Uploading 1 narinfos" lb # [ 173.858227] niks3-server[1153]: 2026/09/25 15:26:40 INFO Signed narinfos id=61 count=1 lb # [ 173.859941] niks3-server[1153]: 2026/09/25 15:26:40 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.ls lb # [ 173.867622] niks3-server[1153]: 2026/09/25 15:26:40 INFO Registered completed upload object_key=nar/06mdwsnf16qmfi86y8r33asi4vcrp5z7kdxqki87psgfpgvvpdki.nar.zst lb # [ 173.876851] niks3-server[1153]: 2026/09/25 15:26:40 INFO Received complete push request method=POST path=/api/pushes/61/complete worker2 # [ 173.879457] nix-grpc-daemon[1736]: time=2026-09-25T15:26:40.503Z level=INFO msg="Upload complete. (188ms)" lb # [ 173.882966] niks3-server[1153]: 2026/09/25 15:26:40 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.narinfo worker2 # [ 173.887256] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=0 paths=1 nar_bytes_in=416 worker2 # [ 173.896563] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 173.902727] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker2 # [ 173.905159] nix-daemon[857]: reaped child process 1895, status = succeeded client # [ 173.913066] nix[1567]: building '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' on lb:50051... worker2 # [ 173.913467] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker1 # [ 173.941375] nix-grpc-daemon[1742]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 396 ms (attempt 3/5) worker1 # [ 174.377659] nix-grpc-daemon[1742]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 141 ms (attempt 4/5) worker1 # [ 174.519958] nix-grpc-daemon[1742]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 174.523863] nix-grpc-daemon[1742]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 174.534300] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58036 client # [ 174.542808] nix[1567]: copying 1 paths to 'lb:50051'... worker1 # [ 174.543614] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 client # [ 174.544316] nix[1567]: copying path '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' to 'lb:50051'... worker1 # [ 174.552177] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 174.571736] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:41Z level=info event=niks3_push_started pid=1813 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1 # [ 174.713670] nix-grpc-daemon[1813]: time=2026-09-25T15:26:41.338Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.14 seconds) worker1 # [ 174.842934] nix-daemon[838]: accepted connection from pid 1829, user nix-grpc-daemon (trusted) lb # [ 174.868876] niks3-server[1153]: 2026/09/25 15:26:41 INFO Received push request method=POST path=/api/pushes worker1 # [ 174.877627] nix-grpc-daemon[1813]: time=2026-09-25T15:26:41.503Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 174.879201] nix-grpc-daemon[1813]: time=2026-09-25T15:26:41.503Z level=INFO msg="Uploading d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv (416B)" worker1 # [ 174.888147] nix-daemon[838]: reaped child process 1835, status = succeeded lb # [ 174.897696] niks3-server[1153]: 2026/09/25 15:26:41 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.ls worker1 # [ 174.903339] nix-grpc-daemon[1813]: time=2026-09-25T15:26:41.529Z level=INFO msg="Uploading 1 narinfos" lb # [ 174.902603] niks3-server[1153]: 2026/09/25 15:26:41 INFO Received sign narinfos request method=POST path=/api/pushes/62/sign lb # [ 174.907625] niks3-server[1153]: 2026/09/25 15:26:41 INFO Signed narinfos id=62 count=1 lb # [ 174.913663] niks3-server[1153]: 2026/09/25 15:26:41 INFO Received complete push request method=POST path=/api/pushes/62/complete worker1 # [ 174.923207] nix-grpc-daemon[1813]: time=2026-09-25T15:26:41.550Z level=INFO msg="Upload complete. (210ms)" lb # [ 174.926829] niks3-server[1153]: 2026/09/25 15:26:41 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.narinfo worker1 # [ 174.928599] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 174.937313] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 174.941376] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 174.944648] nix-daemon[838]: reaped child process 1811, status = succeeded lb # [ 174.947411] niks3-server[1153]: 2026/09/25 15:26:41 INFO Registered completed upload object_key=nar/06ismykzsix0w29371dm1hbzzf3ljf9jn7dyh8skp5wqfkkw0vlk.nar.zst client # [ 174.950288] nix[1567]: building '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' on lb:50051... worker1 # [ 174.953035] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker2 # [ 176.175652] nix-daemon[857]: reaped child process 1904, status = succeeded worker2 # [ 176.230588] nix-daemon[857]: accepted connection from pid 1929, user nix-grpc-daemon (trusted) worker1 # [ 176.232715] nix-daemon[838]: reaped child process 1840, status = succeeded worker2 # [ 176.251352] nix-daemon[857]: reaped child process 1932, status = succeeded lb # [ 176.258433] niks3-server[1153]: 2026/09/25 15:26:42 INFO Received push request method=POST path=/api/pushes worker2 # [ 176.264378] nix-grpc-daemon[1736]: time=2026-09-25T15:26:42.885Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 176.265914] nix-grpc-daemon[1736]: time=2026-09-25T15:26:42.885Z level=INFO msg="Uploading y3qkgrgq2zzpgvi2rf5chq94i627wkdb-slow-q3 (120B)" lb # [ 176.288773] niks3-server[1153]: 2026/09/25 15:26:42 INFO Received sign narinfos request method=POST path=/api/pushes/63/sign worker2 # [ 176.291216] nix-grpc-daemon[1736]: time=2026-09-25T15:26:42.914Z level=INFO msg="Uploading 1 narinfos" worker1 # [ 176.294861] nix-daemon[838]: accepted connection from pid 1861, user nix-grpc-daemon (trusted) lb # [ 176.297162] niks3-server[1153]: 2026/09/25 15:26:42 INFO Signed narinfos id=63 count=1 lb # [ 176.303914] niks3-server[1153]: 2026/09/25 15:26:42 INFO Registered completed upload object_key=log/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv worker1 # [ 176.315473] nix-daemon[838]: reaped child process 1864, status = succeeded lb # [ 176.314956] niks3-server[1153]: 2026/09/25 15:26:42 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.ls lb # [ 176.323565] niks3-server[1153]: 2026/09/25 15:26:42 INFO Registered completed upload object_key=nar/1v1grzkvw3yj09x0dkdm2avihvqvnri156qzp768hk85zbw7lkn9.nar.zst worker1 # [ 176.335835] nix-grpc-daemon[1813]: time=2026-09-25T15:26:42.960Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 176.333718] nix-grpc-daemon[1736]: time=2026-09-25T15:26:42.957Z level=INFO msg="Upload complete. (210ms)" lb # [ 176.336778] niks3-server[1153]: 2026/09/25 15:26:42 INFO Received push request method=POST path=/api/pushes worker1 # [ 176.338213] nix-grpc-daemon[1813]: time=2026-09-25T15:26:42.960Z level=INFO msg="Uploading k6zivhvzgmlyxfmy64fzd759bxlhf9b5-slow-q0 (120B)" lb # [ 176.338879] niks3-server[1153]: 2026/09/25 15:26:42 INFO Received complete push request method=POST path=/api/pushes/63/complete worker1 # [ 176.346101] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:42Z level=debug event=assigned drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv worker=192.168.1.4:50051 clients=1 lb # [ 176.348823] niks3-server[1153]: 2026/09/25 15:26:42 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.narinfo worker2 # [ 176.340729] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=2 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv assign_id=2 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker2 # [ 176.355890] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 176.359033] nix-daemon[857]: reaped child process 1902, status = succeeded lb # [ 176.361694] niks3-server[1153]: 2026/09/25 15:26:42 INFO Registered completed upload object_key=log/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv lb # [ 176.368510] niks3-server[1153]: 2026/09/25 15:26:42 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.ls worker2 # [ 176.370165] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43716 worker1 # [ 176.374016] nix-grpc-daemon[1813]: time=2026-09-25T15:26:43.000Z level=INFO msg="Uploading 1 narinfos" lb # [ 176.374472] niks3-server[1153]: 2026/09/25 15:26:42 INFO Received sign narinfos request method=POST path=/api/pushes/64/sign lb # [ 176.380375] niks3-server[1153]: 2026/09/25 15:26:42 INFO Signed narinfos id=64 count=1 lb # [ 176.383937] niks3-server[1153]: 2026/09/25 15:26:43 INFO Registered completed upload object_key=nar/0xr809jj32p5n6mnc6i7lqsz48wrm5f9jp14m77k724041711iq7.nar.zst client # [ 176.381977] nix[1567]: copying 1 paths to 'lb:50051'... client # [ 176.385475] nix[1567]: copying path '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' to 'lb:50051'... worker2 # [ 176.385736] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 lb # [ 176.390350] niks3-server[1153]: 2026/09/25 15:26:43 INFO Received complete push request method=POST path=/api/pushes/64/complete worker2 # [ 176.393179] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) lb # [ 176.396860] niks3-server[1153]: 2026/09/25 15:26:43 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.narinfo worker1 # [ 176.396627] nix-grpc-daemon[1813]: time=2026-09-25T15:26:43.023Z level=INFO msg="Upload complete. (236ms)" worker1 # [ 176.404841] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=1 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv assign_id=1 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 176.420659] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:43Z level=debug event=assigned drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 176.423809] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 176.429330] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58036 client # [ 176.430190] nix[1567]: copying 1 paths to 'lb:50051'... client # [ 176.431636] nix[1567]: copying path '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' to 'lb:50051'... worker1 # [ 176.442670] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 176.447363] nix-daemon[838]: reaped child process 1838, status = succeeded worker1 # [ 176.449692] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker2 # [ 176.517473] nix-daemon[857]: accepted connection from pid 1936, user nix-grpc-daemon (trusted) worker2 # [ 176.537599] nix-daemon[857]: reaped child process 1939, status = succeeded lb # [ 176.542735] niks3-server[1153]: 2026/09/25 15:26:43 INFO Received push request method=POST path=/api/pushes worker2 # [ 176.545760] nix-grpc-daemon[1736]: time=2026-09-25T15:26:43.169Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 176.547611] nix-grpc-daemon[1736]: time=2026-09-25T15:26:43.169Z level=INFO msg="Uploading jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv (416B)" worker1 # [ 176.557753] nix-daemon[838]: accepted connection from pid 1868, user nix-grpc-daemon (trusted) lb # [ 176.557913] niks3-server[1153]: 2026/09/25 15:26:43 INFO Received sign narinfos request method=POST path=/api/pushes/65/sign worker2 # [ 176.558812] nix-grpc-daemon[1736]: time=2026-09-25T15:26:43.182Z level=INFO msg="Uploading 1 narinfos" lb # [ 176.565887] niks3-server[1153]: 2026/09/25 15:26:43 INFO Signed narinfos id=65 count=1 lb # [ 176.567616] niks3-server[1153]: 2026/09/25 15:26:43 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.ls worker1 # [ 176.574476] nix-daemon[838]: reaped child process 1871, status = succeeded lb # [ 176.578469] niks3-server[1153]: 2026/09/25 15:26:43 INFO Registered completed upload object_key=nar/1fyfjzlx8vymsfmdv5cv8ymgjkr18fwm737qvys29plwwiwmyvwr.nar.zst worker1 # [ 176.588242] nix-grpc-daemon[1813]: time=2026-09-25T15:26:43.214Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 176.589867] nix-grpc-daemon[1813]: time=2026-09-25T15:26:43.214Z level=INFO msg="Uploading pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv (416B)" lb # [ 176.589469] niks3-server[1153]: 2026/09/25 15:26:43 INFO Received complete push request method=POST path=/api/pushes/65/complete worker2 # [ 176.590318] nix-grpc-daemon[1736]: time=2026-09-25T15:26:43.214Z level=INFO msg="Upload complete. (187ms)" lb # [ 176.596381] niks3-server[1153]: 2026/09/25 15:26:43 INFO Received push request method=POST path=/api/pushes worker2 # [ 176.596256] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=0 paths=1 nar_bytes_in=416 lb # [ 176.603035] niks3-server[1153]: 2026/09/25 15:26:43 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.narinfo worker2 # [ 176.605425] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 lb # [ 176.610255] niks3-server[1153]: 2026/09/25 15:26:43 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.ls worker2 # [ 176.611245] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker1 # [ 176.616365] nix-grpc-daemon[1813]: time=2026-09-25T15:26:43.243Z level=INFO msg="Uploading 1 narinfos" worker2 # [ 176.615329] nix-daemon[857]: reaped child process 1934, status = succeeded lb # [ 176.616627] niks3-server[1153]: 2026/09/25 15:26:43 INFO Received sign narinfos request method=POST path=/api/pushes/66/sign lb # [ 176.622707] niks3-server[1153]: 2026/09/25 15:26:43 INFO Signed narinfos id=66 count=1 client # [ 176.624407] nix[1567]: building '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' on lb:50051... worker2 # [ 176.624593] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) lb # [ 176.628259] niks3-server[1153]: 2026/09/25 15:26:43 INFO Registered completed upload object_key=nar/1k21jrz411xc9gkca2pnr9mrjq1s3bg3zhsam3nnv4j066dgzgm7.nar.zst lb # [ 176.638122] niks3-server[1153]: 2026/09/25 15:26:43 INFO Received complete push request method=POST path=/api/pushes/66/complete worker1 # [ 176.643848] nix-grpc-daemon[1813]: time=2026-09-25T15:26:43.270Z level=INFO msg="Upload complete. (194ms)" lb # [ 176.644230] niks3-server[1153]: 2026/09/25 15:26:43 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.narinfo worker1 # [ 176.650513] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 176.662995] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 176.668434] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 176.672791] nix-daemon[838]: reaped child process 1866, status = succeeded client # [ 176.679524] nix[1567]: building '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' on lb:50051... worker1 # [ 176.684268] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) lb # [ 178.114534] envoy[902]: conn peer=[2001:db8:1::3]:37118 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2772 tx=928 ms=27177 worker2 # [ 178.503036] nix-daemon[857]: reaped child process 1943, status = succeeded worker1 # [ 178.570969] nix-daemon[838]: reaped child process 1875, status = succeeded worker2 # [ 178.568836] nix-daemon[857]: accepted connection from pid 1968, user nix-grpc-daemon (trusted) worker2 # [ 178.587903] nix-daemon[857]: reaped child process 1971, status = succeeded lb # [ 178.596363] niks3-server[1153]: 2026/09/25 15:26:45 INFO Received push request method=POST path=/api/pushes worker2 # [ 178.602132] nix-grpc-daemon[1736]: time=2026-09-25T15:26:45.225Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 178.607056] nix-grpc-daemon[1736]: time=2026-09-25T15:26:45.227Z level=INFO msg="Uploading gjyi8j90pk5flrsqqdhj6h132pcb7n33-slow-q2 (120B)" worker1 # [ 178.633745] nix-daemon[838]: accepted connection from pid 1900, user nix-grpc-daemon (trusted) worker2 # [ 178.632493] nix-grpc-daemon[1736]: time=2026-09-25T15:26:45.256Z level=INFO msg="Uploading 1 narinfos" lb # [ 178.634462] niks3-server[1153]: 2026/09/25 15:26:45 INFO Received sign narinfos request method=POST path=/api/pushes/67/sign lb # [ 178.642738] niks3-server[1153]: 2026/09/25 15:26:45 INFO Signed narinfos id=67 count=1 lb # [ 178.649404] niks3-server[1153]: 2026/09/25 15:26:45 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.ls worker1 # [ 178.652105] nix-daemon[838]: reaped child process 1903, status = succeeded lb # [ 178.651985] niks3-server[1153]: 2026/09/25 15:26:45 INFO Registered completed upload object_key=log/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv lb # [ 178.663758] niks3-server[1153]: 2026/09/25 15:26:45 INFO Registered completed upload object_key=nar/15imdx7zyy6mzjjv6qvdzywh0d9w1n6ikc0rq4rq95aw9x6swfzj.nar.zst worker1 # [ 178.671396] nix-grpc-daemon[1813]: time=2026-09-25T15:26:45.297Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" lb # [ 178.673072] niks3-server[1153]: 2026/09/25 15:26:45 INFO Received push request method=POST path=/api/pushes worker1 # [ 178.673299] nix-grpc-daemon[1813]: time=2026-09-25T15:26:45.297Z level=INFO msg="Uploading l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx-slow-q1 (120B)" lb # [ 178.675033] niks3-server[1153]: 2026/09/25 15:26:45 INFO Received complete push request method=POST path=/api/pushes/67/complete lb # [ 178.683588] niks3-server[1153]: 2026/09/25 15:26:45 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.narinfo worker2 # [ 178.677949] nix-grpc-daemon[1736]: time=2026-09-25T15:26:45.301Z level=INFO msg="Upload complete. (241ms)" worker2 # [ 178.684784] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=2 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv assign_id=3 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker2 # [ 178.697144] nix-daemon[857]: reaped child process 1941, status = succeeded worker1 # [ 178.699704] nix-grpc-daemon[1813]: time=2026-09-25T15:26:45.326Z level=INFO msg="Uploading 1 narinfos" lb # [ 178.699161] niks3-server[1153]: 2026/09/25 15:26:45 INFO Received sign narinfos request method=POST path=/api/pushes/68/sign lb # [ 178.705544] niks3-server[1153]: 2026/09/25 15:26:45 INFO Signed narinfos id=68 count=1 lb # [ 178.707001] niks3-server[1153]: 2026/09/25 15:26:45 INFO Registered completed upload object_key=log/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv lb # [ 178.717754] niks3-server[1153]: 2026/09/25 15:26:45 INFO Registered completed upload object_key=nar/0c1c367r3msa4fz6imwp70zxi4fxvrkb86v3hv921wsrf13yd8v3.nar.zst lb # [ 178.721232] niks3-server[1153]: 2026/09/25 15:26:45 INFO Received complete push request method=POST path=/api/pushes/68/complete worker1 # [ 178.723943] nix-grpc-daemon[1813]: time=2026-09-25T15:26:45.351Z level=INFO msg="Upload complete. (228ms)" lb # [ 178.725814] niks3-server[1153]: 2026/09/25 15:26:45 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.narinfo lb # [ 178.729430] niks3-server[1153]: 2026/09/25 15:26:45 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.ls worker1 # [ 178.732600] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=2 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv assign_id=4 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 178.744754] nix-daemon[838]: reaped child process 1873, status = succeeded client # [ 178.755957] systemd[1]: burst.service: Deactivated successfully. client # [ 178.759480] systemd[1]: burst.service: Consumed 189ms CPU time over 5.312s wall clock time, 51.7M memory peak, 13.6K incoming IP traffic, 18.4K outgoing IP traffic. client: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; } client: (finished: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: queue gauges show a burst while every build blocks, in 6.45 seconds) subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1 worker1 # [ 179.938454] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:58036 worker2 # [ 179.940134] nix-grpc-daemon[1664]: ts=2026-09-25T15:26:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43716 worker1 # [ 179.967159] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:46Z level=debug event=want client=2 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=0 worker1 # [ 179.971762] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:46Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 179.975876] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:46Z level=debug event=want client=2 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=0 worker1 # [ 179.980385] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:46Z level=debug event=want client=2 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=0 worker1 # [ 179.984585] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 179.988153] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 179.994003] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 180.003260] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 180.128357] nix-daemon[838]: accepted connection from pid 1919, user nix-grpc-daemon (trusted) worker1 # [ 180.145842] nix-daemon[838]: reaped child process 1922, status = succeeded lb # [ 180.152188] niks3-server[1153]: 2026/09/25 15:26:46 INFO Received push request method=POST path=/api/pushes worker1 # [ 180.157830] nix-grpc-daemon[1813]: time=2026-09-25T15:26:46.784Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 180.159384] nix-grpc-daemon[1813]: time=2026-09-25T15:26:46.784Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)" lb # [ 180.169511] niks3-server[1153]: 2026/09/25 15:26:46 INFO Received sign narinfos request method=POST path=/api/pushes/69/sign worker1 # [ 180.174613] nix-grpc-daemon[1813]: time=2026-09-25T15:26:46.801Z level=INFO msg="Uploading 1 narinfos" lb # [ 180.178529] niks3-server[1153]: 2026/09/25 15:26:46 INFO Signed narinfos id=69 count=1 lb # [ 180.181871] niks3-server[1153]: 2026/09/25 15:26:46 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst lb # [ 180.191466] niks3-server[1153]: 2026/09/25 15:26:46 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls lb # [ 180.195196] niks3-server[1153]: 2026/09/25 15:26:46 INFO Received complete push request method=POST path=/api/pushes/69/complete worker1 # [ 180.199798] nix-grpc-daemon[1813]: time=2026-09-25T15:26:46.826Z level=INFO msg="Upload complete. (186ms)" lb # [ 180.200289] niks3-server[1153]: 2026/09/25 15:26:46 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo worker1 # [ 180.206386] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 180.217850] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 180.222518] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 180.226432] nix-daemon[838]: reaped child process 1917, status = succeeded worker1 # [ 180.236480] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 181.678299] nix-daemon[838]: reaped child process 1926, status = succeeded worker1 # [ 181.786831] nix-daemon[838]: accepted connection from pid 1942, user nix-grpc-daemon (trusted) worker1 # [ 181.806434] nix-daemon[838]: reaped child process 1945, status = succeeded lb # [ 181.813489] niks3-server[1153]: 2026/09/25 15:26:48 INFO Received push request method=POST path=/api/pushes worker1 # [ 181.821462] nix-grpc-daemon[1813]: time=2026-09-25T15:26:48.446Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 181.822966] nix-grpc-daemon[1813]: time=2026-09-25T15:26:48.446Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)" lb # [ 181.839209] niks3-server[1153]: 2026/09/25 15:26:48 INFO Received sign narinfos request method=POST path=/api/pushes/70/sign worker1 # [ 181.843703] nix-grpc-daemon[1813]: time=2026-09-25T15:26:48.470Z level=INFO msg="Uploading 1 narinfos" lb # [ 181.847744] niks3-server[1153]: 2026/09/25 15:26:48 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv lb # [ 181.854147] niks3-server[1153]: 2026/09/25 15:26:48 INFO Signed narinfos id=70 count=1 lb # [ 181.855813] niks3-server[1153]: 2026/09/25 15:26:48 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst lb # [ 181.865875] niks3-server[1153]: 2026/09/25 15:26:48 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls worker1 # [ 181.869143] nix-grpc-daemon[1813]: time=2026-09-25T15:26:48.496Z level=INFO msg="Upload complete. (222ms)" lb # [ 181.870020] niks3-server[1153]: 2026/09/25 15:26:48 INFO Received complete push request method=POST path=/api/pushes/70/complete lb # [ 181.872778] niks3-server[1153]: 2026/09/25 15:26:48 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo worker1 # [ 181.876685] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=5 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 181.894270] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:48Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 181.897561] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 181.901826] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:48Z level=debug event=want client=2 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=0 worker1 # [ 181.905854] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 181.909494] nix-daemon[838]: reaped child process 1924, status = succeeded worker1 # [ 181.911946] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 181.919751] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 182.038496] nix-daemon[838]: accepted connection from pid 1951, user nix-grpc-daemon (trusted) worker1 # [ 182.058240] nix-daemon[838]: reaped child process 1954, status = succeeded lb # [ 182.061552] niks3-server[1153]: 2026/09/25 15:26:48 INFO Received push request method=POST path=/api/pushes worker1 # [ 182.066890] nix-grpc-daemon[1813]: time=2026-09-25T15:26:48.693Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 182.068636] nix-grpc-daemon[1813]: time=2026-09-25T15:26:48.693Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)" worker1 # [ 182.080377] nix-grpc-daemon[1813]: time=2026-09-25T15:26:48.707Z level=INFO msg="Uploading 1 narinfos" lb # [ 182.080185] niks3-server[1153]: 2026/09/25 15:26:48 INFO Received sign narinfos request method=POST path=/api/pushes/71/sign lb # [ 182.083009] niks3-server[1153]: 2026/09/25 15:26:48 INFO Signed narinfos id=71 count=1 lb # [ 182.093064] niks3-server[1153]: 2026/09/25 15:26:48 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls lb # [ 182.095691] niks3-server[1153]: 2026/09/25 15:26:48 INFO Received complete push request method=POST path=/api/pushes/71/complete worker1 # [ 182.101989] nix-grpc-daemon[1813]: time=2026-09-25T15:26:48.729Z level=INFO msg="Upload complete. (173ms)" lb # [ 182.103018] niks3-server[1153]: 2026/09/25 15:26:48 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo lb # [ 182.107319] niks3-server[1153]: 2026/09/25 15:26:48 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst worker1 # [ 182.108123] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 182.117376] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 182.122208] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 182.125932] nix-daemon[838]: reaped child process 1949, status = succeeded worker1 # [ 182.135332] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 183.180373] nix-daemon[838]: reaped child process 1958, status = succeeded worker1 # [ 183.281354] nix-daemon[838]: accepted connection from pid 1974, user nix-grpc-daemon (trusted) worker1 # [ 183.301882] nix-daemon[838]: reaped child process 1977, status = succeeded lb # [ 183.303180] niks3-server[1153]: 2026/09/25 15:26:49 INFO Received push request method=POST path=/api/pushes worker1 # [ 183.311113] nix-grpc-daemon[1813]: time=2026-09-25T15:26:49.935Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 183.312835] nix-grpc-daemon[1813]: time=2026-09-25T15:26:49.935Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)" lb # [ 183.327787] niks3-server[1153]: 2026/09/25 15:26:49 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls lb # [ 183.334429] niks3-server[1153]: 2026/09/25 15:26:49 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst worker1 # [ 183.342554] nix-grpc-daemon[1813]: time=2026-09-25T15:26:49.969Z level=INFO msg="Uploading 1 narinfos" lb # [ 183.342529] niks3-server[1153]: 2026/09/25 15:26:49 INFO Received sign narinfos request method=POST path=/api/pushes/72/sign lb # [ 183.346886] niks3-server[1153]: 2026/09/25 15:26:49 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv lb # [ 183.352839] niks3-server[1153]: 2026/09/25 15:26:49 INFO Signed narinfos id=72 count=1 lb # [ 183.360166] niks3-server[1153]: 2026/09/25 15:26:49 INFO Received complete push request method=POST path=/api/pushes/72/complete lb # [ 183.366698] niks3-server[1153]: 2026/09/25 15:26:49 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo worker1 # [ 183.366026] nix-grpc-daemon[1813]: time=2026-09-25T15:26:49.993Z level=INFO msg="Upload complete. (223ms)" worker1 # [ 183.376247] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=6 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 183.388230] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:49Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 183.395732] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 183.399628] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:50Z level=debug event=want client=2 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=0 worker1 # [ 183.403425] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 183.408216] nix-daemon[838]: reaped child process 1956, status = succeeded worker1 # [ 183.409728] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 183.422901] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 183.534470] nix-daemon[838]: accepted connection from pid 1983, user nix-grpc-daemon (trusted) lb # [ 183.550736] niks3-server[1153]: 2026/09/25 15:26:50 INFO Received push request method=POST path=/api/pushes worker1 # [ 183.554578] nix-daemon[838]: reaped child process 1986, status = succeeded worker1 # [ 183.556491] nix-grpc-daemon[1813]: time=2026-09-25T15:26:50.183Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 183.557986] nix-grpc-daemon[1813]: time=2026-09-25T15:26:50.183Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)" lb # [ 183.567975] niks3-server[1153]: 2026/09/25 15:26:50 INFO Received sign narinfos request method=POST path=/api/pushes/73/sign worker1 # [ 183.571943] nix-grpc-daemon[1813]: time=2026-09-25T15:26:50.198Z level=INFO msg="Uploading 1 narinfos" lb # [ 183.575581] niks3-server[1153]: 2026/09/25 15:26:50 INFO Signed narinfos id=73 count=1 lb # [ 183.580336] niks3-server[1153]: 2026/09/25 15:26:50 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst lb # [ 183.583795] niks3-server[1153]: 2026/09/25 15:26:50 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls lb # [ 183.592643] niks3-server[1153]: 2026/09/25 15:26:50 INFO Received complete push request method=POST path=/api/pushes/73/complete lb # [ 183.599784] niks3-server[1153]: 2026/09/25 15:26:50 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo worker1 # [ 183.603083] nix-grpc-daemon[1813]: time=2026-09-25T15:26:50.230Z level=INFO msg="Upload complete. (177ms)" worker1 # [ 183.608207] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 183.618685] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 183.623461] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 183.627631] nix-daemon[838]: reaped child process 1981, status = succeeded worker1 # [ 183.636099] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 184.884212] nix-daemon[838]: reaped child process 1990, status = succeeded worker1 # [ 184.981091] nix-daemon[838]: accepted connection from pid 2006, user nix-grpc-daemon (trusted) worker1 # [ 185.000181] nix-daemon[838]: reaped child process 2009, status = succeeded lb # [ 185.004358] niks3-server[1153]: 2026/09/25 15:26:51 INFO Received push request method=POST path=/api/pushes worker1 # [ 185.012403] nix-grpc-daemon[1813]: time=2026-09-25T15:26:51.636Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 185.013991] nix-grpc-daemon[1813]: time=2026-09-25T15:26:51.636Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)" lb # [ 185.027855] niks3-server[1153]: 2026/09/25 15:26:51 INFO Received sign narinfos request method=POST path=/api/pushes/74/sign worker1 # [ 185.031546] nix-grpc-daemon[1813]: time=2026-09-25T15:26:51.658Z level=INFO msg="Uploading 1 narinfos" lb # [ 185.035914] niks3-server[1153]: 2026/09/25 15:26:51 INFO Signed narinfos id=74 count=1 lb # [ 185.040862] niks3-server[1153]: 2026/09/25 15:26:51 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls lb # [ 185.046305] niks3-server[1153]: 2026/09/25 15:26:51 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst lb # [ 185.054818] niks3-server[1153]: 2026/09/25 15:26:51 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker1 # [ 185.058234] nix-grpc-daemon[1813]: time=2026-09-25T15:26:51.685Z level=INFO msg="Upload complete. (209ms)" lb # [ 185.057884] niks3-server[1153]: 2026/09/25 15:26:51 INFO Received complete push request method=POST path=/api/pushes/74/complete lb # [ 185.065154] niks3-server[1153]: 2026/09/25 15:26:51 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo worker1 # [ 185.069553] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=7 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 185.080331] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:51Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 185.087784] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 185.091320] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:51Z level=debug event=want client=2 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=0 worker1 # [ 185.095108] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 185.098869] nix-grpc-daemon[1813]: time=2026-09-25T15:26:51.725Z level=INFO msg="All 1 paths already cached" worker1 # [ 185.101742] nix-daemon[838]: reaped child process 1988, status = succeeded worker1 # [ 185.112243] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 185.114730] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 185.221504] nix-daemon[838]: accepted connection from pid 2015, user nix-grpc-daemon (trusted) lb # [ 185.237376] niks3-server[1153]: 2026/09/25 15:26:51 INFO Received push request method=POST path=/api/pushes worker1 # [ 185.240440] nix-daemon[838]: reaped child process 2018, status = succeeded worker1 # [ 185.242809] nix-grpc-daemon[1813]: time=2026-09-25T15:26:51.869Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 185.244350] nix-grpc-daemon[1813]: time=2026-09-25T15:26:51.869Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)" lb # [ 185.255604] niks3-server[1153]: 2026/09/25 15:26:51 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls worker1 # [ 185.260526] nix-grpc-daemon[1813]: time=2026-09-25T15:26:51.887Z level=INFO msg="Uploading 1 narinfos" lb # [ 185.263381] niks3-server[1153]: 2026/09/25 15:26:51 INFO Received sign narinfos request method=POST path=/api/pushes/75/sign lb # [ 185.268881] niks3-server[1153]: 2026/09/25 15:26:51 INFO Signed narinfos id=75 count=1 lb # [ 185.270603] niks3-server[1153]: 2026/09/25 15:26:51 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst lb # [ 185.280877] niks3-server[1153]: 2026/09/25 15:26:51 INFO Received complete push request method=POST path=/api/pushes/75/complete worker1 # [ 185.284944] nix-grpc-daemon[1813]: time=2026-09-25T15:26:51.912Z level=INFO msg="Upload complete. (166ms)" lb # [ 185.285548] niks3-server[1153]: 2026/09/25 15:26:51 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo worker1 # [ 185.291048] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 185.298484] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 185.303182] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 185.306941] nix-daemon[838]: reaped child process 2013, status = succeeded worker1 # [ 185.318034] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 186.934955] nix-daemon[838]: reaped child process 2022, status = succeeded worker1 # [ 187.064895] nix-daemon[838]: accepted connection from pid 2038, user nix-grpc-daemon (trusted) worker1 # [ 187.085743] nix-daemon[838]: reaped child process 2041, status = succeeded lb # [ 187.087309] niks3-server[1153]: 2026/09/25 15:26:53 INFO Received push request method=POST path=/api/pushes worker1 # [ 187.095190] nix-grpc-daemon[1813]: time=2026-09-25T15:26:53.719Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 187.096857] nix-grpc-daemon[1813]: time=2026-09-25T15:26:53.719Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)" lb # [ 187.115275] niks3-server[1153]: 2026/09/25 15:26:53 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls worker1 # [ 187.121669] nix-grpc-daemon[1813]: time=2026-09-25T15:26:53.748Z level=INFO msg="Uploading 1 narinfos" lb # [ 187.122862] niks3-server[1153]: 2026/09/25 15:26:53 INFO Received sign narinfos request method=POST path=/api/pushes/76/sign lb # [ 187.127467] niks3-server[1153]: 2026/09/25 15:26:53 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst lb # [ 187.135398] niks3-server[1153]: 2026/09/25 15:26:53 INFO Signed narinfos id=76 count=1 lb # [ 187.138689] niks3-server[1153]: 2026/09/25 15:26:53 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker1 # [ 187.140872] nix-grpc-daemon[1813]: time=2026-09-25T15:26:53.768Z level=INFO msg="Upload complete. (215ms)" lb # [ 187.141380] niks3-server[1153]: 2026/09/25 15:26:53 INFO Received complete push request method=POST path=/api/pushes/76/complete lb # [ 187.143476] niks3-server[1153]: 2026/09/25 15:26:53 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo worker1 # [ 187.150081] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=8 outputs=1 inputs=1 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 187.162201] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:53Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 187.166257] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 187.169646] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:53Z level=debug event=want client=2 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=0 worker1 # [ 187.173134] nix-daemon[838]: reaped child process 2020, status = succeeded worker1 # [ 187.175430] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 187.182331] nix-grpc-daemon[1813]: time=2026-09-25T15:26:53.809Z level=INFO msg="All 1 paths already cached" worker1 # [ 187.187883] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 187.195798] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 187.304216] nix-daemon[838]: accepted connection from pid 2047, user nix-grpc-daemon (trusted) worker1 # [ 187.324679] nix-daemon[838]: reaped child process 2050, status = succeeded lb # [ 187.327066] niks3-server[1153]: 2026/09/25 15:26:53 INFO Received push request method=POST path=/api/pushes worker1 # [ 187.332211] nix-grpc-daemon[1813]: time=2026-09-25T15:26:53.958Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 187.333765] nix-grpc-daemon[1813]: time=2026-09-25T15:26:53.958Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)" lb # [ 187.344408] niks3-server[1153]: 2026/09/25 15:26:53 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls worker1 # [ 187.349404] nix-grpc-daemon[1813]: time=2026-09-25T15:26:53.976Z level=INFO msg="Uploading 1 narinfos" lb # [ 187.352227] niks3-server[1153]: 2026/09/25 15:26:53 INFO Received sign narinfos request method=POST path=/api/pushes/77/sign lb # [ 187.354580] niks3-server[1153]: 2026/09/25 15:26:53 INFO Signed narinfos id=77 count=1 worker1 # [ 187.363739] nix-grpc-daemon[1813]: time=2026-09-25T15:26:53.990Z level=INFO msg="Upload complete. (160ms)" lb # [ 187.364084] niks3-server[1153]: 2026/09/25 15:26:53 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst worker1 # [ 187.368186] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 187.371902] niks3-server[1153]: 2026/09/25 15:26:53 INFO Received complete push request method=POST path=/api/pushes/77/complete lb # [ 187.374611] niks3-server[1153]: 2026/09/25 15:26:53 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo worker1 # [ 187.378235] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 187.381992] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 187.385435] nix-daemon[838]: reaped child process 2045, status = succeeded worker1 # [ 187.392880] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) lb # [ 188.114078] envoy[902]: conn peer=[2001:db8:1::1]:38828 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9641 tx=5163 ms=5118 worker1 # [ 188.431817] nix-daemon[838]: reaped child process 2054, status = succeeded worker1 # [ 188.535520] nix-daemon[838]: accepted connection from pid 2070, user nix-grpc-daemon (trusted) worker1 # [ 188.553949] nix-daemon[838]: reaped child process 2073, status = succeeded lb # [ 188.559196] niks3-server[1153]: 2026/09/25 15:26:55 INFO Received push request method=POST path=/api/pushes worker1 # [ 188.567186] nix-grpc-daemon[1813]: time=2026-09-25T15:26:55.191Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 188.568876] nix-grpc-daemon[1813]: time=2026-09-25T15:26:55.191Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)" lb # [ 188.584099] niks3-server[1153]: 2026/09/25 15:26:55 INFO Received sign narinfos request method=POST path=/api/pushes/78/sign worker1 # [ 188.587912] nix-grpc-daemon[1813]: time=2026-09-25T15:26:55.214Z level=INFO msg="Uploading 1 narinfos" lb # [ 188.592870] niks3-server[1153]: 2026/09/25 15:26:55 INFO Signed narinfos id=78 count=1 lb # [ 188.597023] niks3-server[1153]: 2026/09/25 15:26:55 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst lb # [ 188.607888] niks3-server[1153]: 2026/09/25 15:26:55 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls lb # [ 188.610718] niks3-server[1153]: 2026/09/25 15:26:55 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv lb # [ 188.613715] niks3-server[1153]: 2026/09/25 15:26:55 INFO Received complete push request method=POST path=/api/pushes/78/complete worker1 # [ 188.618086] nix-grpc-daemon[1813]: time=2026-09-25T15:26:55.245Z level=INFO msg="Upload complete. (216ms)" lb # [ 188.619563] niks3-server[1153]: 2026/09/25 15:26:55 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo worker1 # [ 188.629097] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=9 outputs=1 inputs=1 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 188.642347] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:55Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 188.646143] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 188.649898] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:55Z level=debug event=want client=2 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=0 worker1 # [ 188.654198] nix-daemon[838]: reaped child process 2052, status = succeeded worker1 # [ 188.656810] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 188.660723] nix-grpc-daemon[1813]: time=2026-09-25T15:26:55.286Z level=INFO msg="All 1 paths already cached" worker1 # [ 188.663604] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 188.675985] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 188.782284] nix-daemon[838]: accepted connection from pid 2079, user nix-grpc-daemon (trusted) worker1 # [ 188.799152] nix-daemon[838]: reaped child process 2082, status = succeeded lb # [ 188.806035] niks3-server[1153]: 2026/09/25 15:26:55 INFO Received push request method=POST path=/api/pushes worker1 # [ 188.810929] nix-grpc-daemon[1813]: time=2026-09-25T15:26:55.437Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 188.812535] nix-grpc-daemon[1813]: time=2026-09-25T15:26:55.437Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)" lb # [ 188.821016] niks3-server[1153]: 2026/09/25 15:26:55 INFO Received sign narinfos request method=POST path=/api/pushes/79/sign worker1 # [ 188.824105] nix-grpc-daemon[1813]: time=2026-09-25T15:26:55.451Z level=INFO msg="Uploading 1 narinfos" lb # [ 188.826798] niks3-server[1153]: 2026/09/25 15:26:55 INFO Signed narinfos id=79 count=1 lb # [ 188.832623] niks3-server[1153]: 2026/09/25 15:26:55 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst lb # [ 188.843277] niks3-server[1153]: 2026/09/25 15:26:55 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls worker1 # [ 188.848870] nix-grpc-daemon[1813]: time=2026-09-25T15:26:55.476Z level=INFO msg="Upload complete. (169ms)" lb # [ 188.849980] niks3-server[1153]: 2026/09/25 15:26:55 INFO Received complete push request method=POST path=/api/pushes/79/complete lb # [ 188.853318] niks3-server[1153]: 2026/09/25 15:26:55 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo worker1 # [ 188.856278] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 188.860088] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 188.868178] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 188.872098] nix-daemon[838]: reaped child process 2077, status = succeeded worker1 # [ 188.882145] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 190.316380] nix-daemon[838]: reaped child process 2086, status = succeeded worker1 # [ 190.409036] nix-daemon[838]: accepted connection from pid 2102, user nix-grpc-daemon (trusted) worker1 # [ 190.425733] nix-daemon[838]: reaped child process 2105, status = succeeded lb # [ 190.431700] niks3-server[1153]: 2026/09/25 15:26:57 INFO Received push request method=POST path=/api/pushes worker1 # [ 190.438606] nix-grpc-daemon[1813]: time=2026-09-25T15:26:57.064Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 190.440347] nix-grpc-daemon[1813]: time=2026-09-25T15:26:57.064Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)" lb # [ 190.455542] niks3-server[1153]: 2026/09/25 15:26:57 INFO Received sign narinfos request method=POST path=/api/pushes/80/sign worker1 # [ 190.460253] nix-grpc-daemon[1813]: time=2026-09-25T15:26:57.087Z level=INFO msg="Uploading 1 narinfos" lb # [ 190.463333] niks3-server[1153]: 2026/09/25 15:26:57 INFO Signed narinfos id=80 count=1 lb # [ 190.468708] niks3-server[1153]: 2026/09/25 15:26:57 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst lb # [ 190.471713] niks3-server[1153]: 2026/09/25 15:26:57 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv lb # [ 190.482525] niks3-server[1153]: 2026/09/25 15:26:57 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls lb # [ 190.485396] niks3-server[1153]: 2026/09/25 15:26:57 INFO Received complete push request method=POST path=/api/pushes/80/complete worker1 # [ 190.490963] nix-grpc-daemon[1813]: time=2026-09-25T15:26:57.118Z level=INFO msg="Upload complete. (201ms)" lb # [ 190.492369] niks3-server[1153]: 2026/09/25 15:26:57 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo worker1 # [ 190.501372] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=10 outputs=1 inputs=1 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 190.514553] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:57Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 190.519060] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 190.523299] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:57Z level=debug event=want client=2 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=0 worker1 # [ 190.527087] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 190.531041] nix-daemon[838]: reaped child process 2084, status = succeeded worker1 # [ 190.532668] nix-grpc-daemon[1813]: time=2026-09-25T15:26:57.157Z level=INFO msg="All 2 paths already cached" worker1 # [ 190.543891] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 190.548162] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 190.652863] nix-daemon[838]: accepted connection from pid 2111, user nix-grpc-daemon (trusted) worker1 # [ 190.668196] nix-daemon[838]: reaped child process 2114, status = succeeded lb # [ 190.673479] niks3-server[1153]: 2026/09/25 15:26:57 INFO Received push request method=POST path=/api/pushes worker1 # [ 190.678477] nix-grpc-daemon[1813]: time=2026-09-25T15:26:57.305Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 190.679980] nix-grpc-daemon[1813]: time=2026-09-25T15:26:57.305Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)" lb # [ 190.688977] niks3-server[1153]: 2026/09/25 15:26:57 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst worker1 # [ 190.694973] nix-grpc-daemon[1813]: time=2026-09-25T15:26:57.321Z level=INFO msg="Uploading 1 narinfos" lb # [ 190.696906] niks3-server[1153]: 2026/09/25 15:26:57 INFO Received sign narinfos request method=POST path=/api/pushes/81/sign lb # [ 190.699307] niks3-server[1153]: 2026/09/25 15:26:57 INFO Signed narinfos id=81 count=1 lb # [ 190.706742] niks3-server[1153]: 2026/09/25 15:26:57 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls lb # [ 190.715451] niks3-server[1153]: 2026/09/25 15:26:57 INFO Received complete push request method=POST path=/api/pushes/81/complete worker1 # [ 190.721119] nix-grpc-daemon[1813]: time=2026-09-25T15:26:57.348Z level=INFO msg="Upload complete. (170ms)" lb # [ 190.722466] niks3-server[1153]: 2026/09/25 15:26:57 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo worker1 # [ 190.728119] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 190.731091] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 190.740161] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 190.743877] nix-daemon[838]: reaped child process 2109, status = succeeded worker1 # [ 190.753671] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 192.208131] nix-daemon[838]: reaped child process 2118, status = succeeded worker1 # [ 192.304132] nix-daemon[838]: accepted connection from pid 2134, user nix-grpc-daemon (trusted) worker1 # [ 192.323501] nix-daemon[838]: reaped child process 2137, status = succeeded lb # [ 192.328706] niks3-server[1153]: 2026/09/25 15:26:58 INFO Received push request method=POST path=/api/pushes worker1 # [ 192.336397] nix-grpc-daemon[1813]: time=2026-09-25T15:26:58.961Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 192.338012] nix-grpc-daemon[1813]: time=2026-09-25T15:26:58.961Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)" lb # [ 192.351003] niks3-server[1153]: 2026/09/25 15:26:58 INFO Received sign narinfos request method=POST path=/api/pushes/82/sign worker1 # [ 192.356647] nix-grpc-daemon[1813]: time=2026-09-25T15:26:58.983Z level=INFO msg="Uploading 1 narinfos" lb # [ 192.358039] niks3-server[1153]: 2026/09/25 15:26:58 INFO Signed narinfos id=82 count=1 lb # [ 192.363924] niks3-server[1153]: 2026/09/25 15:26:58 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv lb # [ 192.370515] niks3-server[1153]: 2026/09/25 15:26:58 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls lb # [ 192.380962] niks3-server[1153]: 2026/09/25 15:26:58 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst lb # [ 192.383971] niks3-server[1153]: 2026/09/25 15:26:59 INFO Received complete push request method=POST path=/api/pushes/82/complete worker1 # [ 192.390262] nix-grpc-daemon[1813]: time=2026-09-25T15:26:59.017Z level=INFO msg="Upload complete. (229ms)" lb # [ 192.391261] niks3-server[1153]: 2026/09/25 15:26:59 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo worker1 # [ 192.398845] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=11 outputs=1 inputs=2 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 192.409442] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:59Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 192.420180] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 192.426460] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 192.430515] nix-daemon[838]: reaped child process 2116, status = succeeded worker1 # [ 192.433436] nix-grpc-daemon[1813]: time=2026-09-25T15:26:59.059Z level=INFO msg="All 2 paths already cached" worker1 # [ 192.436423] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 192.445117] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 192.557478] nix-daemon[838]: accepted connection from pid 2141, user nix-grpc-daemon (trusted) lb # [ 192.574085] niks3-server[1153]: 2026/09/25 15:26:59 INFO Received push request method=POST path=/api/pushes worker1 # [ 192.576241] nix-daemon[838]: reaped child process 2144, status = succeeded worker1 # [ 192.578656] nix-grpc-daemon[1813]: time=2026-09-25T15:26:59.205Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 192.580201] nix-grpc-daemon[1813]: time=2026-09-25T15:26:59.205Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)" worker1 # [ 192.596423] nix-grpc-daemon[1813]: time=2026-09-25T15:26:59.223Z level=INFO msg="Uploading 1 narinfos" lb # [ 192.590071] niks3-server[1153]: 2026/09/25 15:26:59 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst lb # [ 192.598216] niks3-server[1153]: 2026/09/25 15:26:59 INFO Received sign narinfos request method=POST path=/api/pushes/83/sign lb # [ 192.603646] niks3-server[1153]: 2026/09/25 15:26:59 INFO Signed narinfos id=83 count=1 lb # [ 192.611796] niks3-server[1153]: 2026/09/25 15:26:59 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls worker1 # [ 192.616825] nix-grpc-daemon[1813]: time=2026-09-25T15:26:59.243Z level=INFO msg="Upload complete. (164ms)" lb # [ 192.617670] niks3-server[1153]: 2026/09/25 15:26:59 INFO Received complete push request method=POST path=/api/pushes/83/complete lb # [ 192.623409] niks3-server[1153]: 2026/09/25 15:26:59 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo worker1 # [ 192.621733] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 192.630760] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 192.636765] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 192.638693] nix-daemon[838]: reaped child process 2139, status = succeeded worker1 # [ 192.647447] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 193.119020] nix-daemon[838]: reaped child process 2148, status = succeeded worker1 # [ 193.187646] nix-daemon[838]: accepted connection from pid 2164, user nix-grpc-daemon (trusted) worker1 # [ 193.202935] nix-daemon[838]: reaped child process 2167, status = succeeded lb # [ 193.208549] niks3-server[1153]: 2026/09/25 15:26:59 INFO Received push request method=POST path=/api/pushes worker1 # [ 193.214579] nix-grpc-daemon[1813]: time=2026-09-25T15:26:59.840Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 193.216153] nix-grpc-daemon[1813]: time=2026-09-25T15:26:59.840Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)" worker1 # [ 193.240159] nix-grpc-daemon[1813]: time=2026-09-25T15:26:59.867Z level=INFO msg="Uploading 1 narinfos" lb # [ 193.233139] niks3-server[1153]: 2026/09/25 15:26:59 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv lb # [ 193.242526] niks3-server[1153]: 2026/09/25 15:26:59 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls lb # [ 193.248421] niks3-server[1153]: 2026/09/25 15:26:59 INFO Received sign narinfos request method=POST path=/api/pushes/84/sign lb # [ 193.256209] niks3-server[1153]: 2026/09/25 15:26:59 INFO Signed narinfos id=84 count=1 lb # [ 193.258028] niks3-server[1153]: 2026/09/25 15:26:59 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst worker1 # [ 193.263180] nix-grpc-daemon[1813]: time=2026-09-25T15:26:59.890Z level=INFO msg="Upload complete. (209ms)" lb # [ 193.264993] niks3-server[1153]: 2026/09/25 15:26:59 INFO Received complete push request method=POST path=/api/pushes/84/complete lb # [ 193.268862] niks3-server[1153]: 2026/09/25 15:26:59 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo worker1 # [ 193.272542] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=0 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=12 outputs=1 inputs=2 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 193.284245] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:59Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 193.287389] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 193.291328] nix-daemon[838]: reaped child process 2146, status = succeeded worker1 # [ 193.296583] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 193.304764] nix-grpc-daemon[1813]: time=2026-09-25T15:26:59.931Z level=INFO msg="All 2 paths already cached" worker1 # [ 193.310705] nix-grpc-daemon[1742]: ts=2026-09-25T15:26:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 193.318580] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 193.435230] nix-daemon[838]: accepted connection from pid 2171, user nix-grpc-daemon (trusted) lb # [ 193.453749] niks3-server[1153]: 2026/09/25 15:27:00 INFO Received push request method=POST path=/api/pushes worker1 # [ 193.456189] nix-daemon[838]: reaped child process 2174, status = succeeded worker1 # [ 193.459257] nix-grpc-daemon[1813]: time=2026-09-25T15:27:00.085Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 193.460883] nix-grpc-daemon[1813]: time=2026-09-25T15:27:00.085Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)" worker1 # [ 193.473817] nix-grpc-daemon[1813]: time=2026-09-25T15:27:00.100Z level=INFO msg="Uploading 1 narinfos" lb # [ 193.469722] niks3-server[1153]: 2026/09/25 15:27:00 INFO Received sign narinfos request method=POST path=/api/pushes/85/sign lb # [ 193.477505] niks3-server[1153]: 2026/09/25 15:27:00 INFO Signed narinfos id=85 count=1 lb # [ 193.481417] niks3-server[1153]: 2026/09/25 15:27:00 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls lb # [ 193.491391] niks3-server[1153]: 2026/09/25 15:27:00 INFO Received complete push request method=POST path=/api/pushes/85/complete lb # [ 193.495796] niks3-server[1153]: 2026/09/25 15:27:00 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst worker1 # [ 193.499269] nix-grpc-daemon[1813]: time=2026-09-25T15:27:00.126Z level=INFO msg="Upload complete. (171ms)" lb # [ 193.500716] niks3-server[1153]: 2026/09/25 15:27:00 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo worker1 # [ 193.511220] nix-grpc-daemon[1742]: ts=2026-09-25T15:27:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 193.515282] nix-grpc-daemon[1742]: ts=2026-09-25T15:27:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 worker1 # [ 193.518066] nix-daemon[838]: reaped child process 2169, status = succeeded worker1 # [ 193.523531] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 193.531873] nix-daemon[838]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker1 # [ 194.158937] nix-daemon[838]: reaped child process 2178, status = succeeded worker1 # [ 194.277129] nix-daemon[838]: accepted connection from pid 2194, user nix-grpc-daemon (trusted) worker1 # [ 194.296112] nix-daemon[838]: reaped child process 2197, status = succeeded lb # [ 194.302036] niks3-server[1153]: 2026/09/25 15:27:00 INFO Received push request method=POST path=/api/pushes worker1 # [ 194.308257] nix-grpc-daemon[1813]: time=2026-09-25T15:27:00.934Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 194.314122] nix-grpc-daemon[1813]: time=2026-09-25T15:27:00.934Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)" lb # [ 194.331919] niks3-server[1153]: 2026/09/25 15:27:00 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv lb # [ 194.343831] niks3-server[1153]: 2026/09/25 15:27:00 INFO Received sign narinfos request method=POST path=/api/pushes/86/sign worker1 # [ 194.348762] nix-grpc-daemon[1813]: time=2026-09-25T15:27:00.975Z level=INFO msg="Uploading 1 narinfos" lb # [ 194.350769] niks3-server[1153]: 2026/09/25 15:27:00 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst lb # [ 194.357602] niks3-server[1153]: 2026/09/25 15:27:00 INFO Signed narinfos id=86 count=1 lb # [ 194.359337] niks3-server[1153]: 2026/09/25 15:27:00 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls lb # [ 194.368766] niks3-server[1153]: 2026/09/25 15:27:00 INFO Received complete push request method=POST path=/api/pushes/86/complete worker1 # [ 194.372298] nix-grpc-daemon[1813]: time=2026-09-25T15:27:00.999Z level=INFO msg="Upload complete. (231ms)" lb # [ 194.373343] niks3-server[1153]: 2026/09/25 15:27:00 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo worker1 # [ 194.380246] nix-grpc-daemon[1742]: ts=2026-09-25T15:27:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58036 duration_s=0 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=13 outputs=1 inputs=2 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 194.392297] nix-daemon[838]: reaped child process 2176, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 14.66 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]' client # [ 194.489980] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features "[\"gpu\"]". client # Running as unit: gpu.service; invocation ID: bdadfbd2dd764cdd9f66b89ce80c6b17 client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) worker1 # [ 194.662430] nix-grpc-daemon[1742]: ts=2026-09-25T15:27:01Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:58036 client # [ 194.665896] nix[1637]: don't know how to build these paths: client # [ 194.666864] nix[1637]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv worker2 # [ 194.667261] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43716 worker1 # [ 194.684917] nix-grpc-daemon[1742]: ts=2026-09-25T15:27:01Z level=debug event=want client=3 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=0 client # [ 194.683757] nix[1637]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip}, waiting up to 30s for a worker lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) client: waiting for success: systemctl show -p Result gpu | grep -q exit-code lb # [ 198.113656] envoy[902]: conn peer=[2001:db8:1::1]:38836 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27080 tx=12277 ms=14456 client # [ 224.684681] nix[1637]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip} client # [ 224.692065] nix[1637]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed client # [ 224.713562] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE client # [ 224.718050] systemd[1]: gpu.service: Failed with result 'exit-code'. client # [ 224.720136] systemd[1]: gpu.service: Consumed 166ms CPU time over 30.226s wall clock time, 33.4M memory peak, 5.7K incoming IP traffic, 5.2K outgoing IP traffic. client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.25 seconds) client: must succeed: journalctl -u gpu -o cat client: (finished: must succeed: journalctl -u gpu -o cat, in 0.02 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 45.34 seconds) subtest: low disk drains a worker and builds go to the other worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill worker1 # [ 226.093291] nix-grpc-daemon[1742]: ts=2026-09-25T15:27:32Z level=info event=unhealthy reason=min_free worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.17 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free' worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 0.03 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2 worker2 # [ 226.471326] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:33Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:43716 client # don't know how to build these paths: client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker2 # [ 226.477741] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43716 worker1 # [ 226.496092] nix-grpc-daemon[1742]: ts=2026-09-25T15:27:33Z level=debug event=want client=4 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=0 worker1 # [ 226.498956] nix-grpc-daemon[1742]: ts=2026-09-25T15:27:33Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 226.502299] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 226.511660] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43716 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'... worker2 # [ 226.520855] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 226.529561] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker2 # [ 226.659688] nix-daemon[857]: accepted connection from pid 1979, user nix-grpc-daemon (trusted) worker2 # [ 226.674907] nix-daemon[857]: reaped child process 1982, status = succeeded lb # [ 226.683931] niks3-server[1153]: 2026/09/25 15:27:33 INFO Received push request method=POST path=/api/pushes worker2 # [ 226.689109] nix-grpc-daemon[1736]: time=2026-09-25T15:27:33.312Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 226.690758] nix-grpc-daemon[1736]: time=2026-09-25T15:27:33.312Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)" lb # [ 226.704619] niks3-server[1153]: 2026/09/25 15:27:33 INFO Received sign narinfos request method=POST path=/api/pushes/87/sign worker2 # [ 226.707034] nix-grpc-daemon[1736]: time=2026-09-25T15:27:33.330Z level=INFO msg="Uploading 1 narinfos" lb # [ 226.712457] niks3-server[1153]: 2026/09/25 15:27:33 INFO Signed narinfos id=87 count=1 lb # [ 226.722127] niks3-server[1153]: 2026/09/25 15:27:33 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst lb # [ 226.734203] niks3-server[1153]: 2026/09/25 15:27:33 INFO Received complete push request method=POST path=/api/pushes/87/complete lb # [ 226.740257] niks3-server[1153]: 2026/09/25 15:27:33 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls worker2 # [ 226.741020] nix-grpc-daemon[1736]: time=2026-09-25T15:27:33.364Z level=INFO msg="Upload complete. (201ms)" lb # [ 226.747357] niks3-server[1153]: 2026/09/25 15:27:33 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo worker2 # [ 226.748110] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=0 paths=1 nar_bytes_in=512 worker2 # [ 226.757348] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 226.762490] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker2 # [ 226.766367] nix-daemon[857]: reaped child process 1977, status = succeeded client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051... client # farm-a> worker2: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker2 # [ 226.776448] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-drain worker2 # [ 227.401807] nix-daemon[857]: reaped child process 1986, status = succeeded worker2 # [ 227.522508] nix-daemon[857]: accepted connection from pid 2002, user nix-grpc-daemon (trusted) lb # [ 227.542842] niks3-server[1153]: 2026/09/25 15:27:34 INFO Received push request method=POST path=/api/pushes worker2 # [ 227.542647] nix-daemon[857]: reaped child process 2005, status = succeeded worker2 # [ 227.546703] nix-grpc-daemon[1736]: time=2026-09-25T15:27:34.169Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 227.548489] nix-grpc-daemon[1736]: time=2026-09-25T15:27:34.169Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)" worker2 # [ 227.567354] nix-grpc-daemon[1736]: time=2026-09-25T15:27:34.191Z level=INFO msg="Uploading 1 narinfos" lb # [ 227.566059] niks3-server[1153]: 2026/09/25 15:27:34 INFO Received sign narinfos request method=POST path=/api/pushes/88/sign lb # [ 227.573808] niks3-server[1153]: 2026/09/25 15:27:34 INFO Signed narinfos id=88 count=1 lb # [ 227.578964] niks3-server[1153]: 2026/09/25 15:27:34 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls lb # [ 227.585268] niks3-server[1153]: 2026/09/25 15:27:34 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst lb # [ 227.593708] niks3-server[1153]: 2026/09/25 15:27:34 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv lb # [ 227.601084] niks3-server[1153]: 2026/09/25 15:27:34 INFO Received complete push request method=POST path=/api/pushes/88/complete worker2 # [ 227.602394] nix-grpc-daemon[1736]: time=2026-09-25T15:27:34.226Z level=INFO msg="Upload complete. (216ms)" lb # [ 227.606307] niks3-server[1153]: 2026/09/25 15:27:34 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo worker1 # [ 227.616673] nix-grpc-daemon[1742]: ts=2026-09-25T15:27:34Z level=debug event=want client=4 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=0 worker1 # [ 227.619702] nix-grpc-daemon[1742]: ts=2026-09-25T15:27:34Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 227.615869] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=0 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=14 outputs=1 inputs=0 inputs_fetched=0 input_bytes=0 input_s=0 worker2 # [ 227.622192] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 227.626647] nix-daemon[857]: reaped child process 1984, status = succeeded worker2 # [ 227.631804] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 227.639911] nix-grpc-daemon[1736]: time=2026-09-25T15:27:34.263Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'... worker2 # [ 227.645952] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 227.653491] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker2 # [ 227.772956] nix-daemon[857]: accepted connection from pid 2009, user nix-grpc-daemon (trusted) worker2 # [ 227.789166] nix-daemon[857]: reaped child process 2012, status = succeeded lb # [ 227.797651] niks3-server[1153]: 2026/09/25 15:27:34 INFO Received push request method=POST path=/api/pushes worker2 # [ 227.801250] nix-grpc-daemon[1736]: time=2026-09-25T15:27:34.424Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 227.802819] nix-grpc-daemon[1736]: time=2026-09-25T15:27:34.424Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)" lb # [ 227.816345] niks3-server[1153]: 2026/09/25 15:27:34 INFO Received sign narinfos request method=POST path=/api/pushes/89/sign worker2 # [ 227.817126] nix-grpc-daemon[1736]: time=2026-09-25T15:27:34.440Z level=INFO msg="Uploading 1 narinfos" lb # [ 227.823399] niks3-server[1153]: 2026/09/25 15:27:34 INFO Signed narinfos id=89 count=1 lb # [ 227.827245] niks3-server[1153]: 2026/09/25 15:27:34 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls lb # [ 227.836188] niks3-server[1153]: 2026/09/25 15:27:34 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst lb # [ 227.839106] niks3-server[1153]: 2026/09/25 15:27:34 INFO Received complete push request method=POST path=/api/pushes/89/complete worker2 # [ 227.841370] nix-grpc-daemon[1736]: time=2026-09-25T15:27:34.465Z level=INFO msg="Upload complete. (178ms)" lb # [ 227.845649] niks3-server[1153]: 2026/09/25 15:27:34 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo worker2 # [ 227.848682] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=0 paths=1 nar_bytes_in=632 worker2 # [ 227.851890] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 227.863169] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker2 # [ 227.865023] nix-daemon[857]: reaped child process 2007, status = succeeded client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051... client # farm-b> worker2: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker2 # [ 227.875434] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) lb # [ 228.114689] envoy[902]: conn peer=[2001:db8:1::1]:59458 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=931 tx=636 ms=30042 client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-drain worker2 # [ 229.143661] nix-daemon[857]: reaped child process 2016, status = succeeded worker2 # [ 229.199199] nix-daemon[857]: accepted connection from pid 2032, user nix-grpc-daemon (trusted) worker2 # [ 229.219168] nix-daemon[857]: reaped child process 2035, status = succeeded lb # [ 229.220433] niks3-server[1153]: 2026/09/25 15:27:35 INFO Received push request method=POST path=/api/pushes worker2 # [ 229.227717] nix-grpc-daemon[1736]: time=2026-09-25T15:27:35.848Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 229.229574] nix-grpc-daemon[1736]: time=2026-09-25T15:27:35.848Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)" lb # [ 229.246519] niks3-server[1153]: 2026/09/25 15:27:35 INFO Received sign narinfos request method=POST path=/api/pushes/90/sign worker2 # [ 229.248195] nix-grpc-daemon[1736]: time=2026-09-25T15:27:35.872Z level=INFO msg="Uploading 1 narinfos" lb # [ 229.253343] niks3-server[1153]: 2026/09/25 15:27:35 INFO Signed narinfos id=90 count=1 lb # [ 229.259991] niks3-server[1153]: 2026/09/25 15:27:35 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv lb # [ 229.268511] niks3-server[1153]: 2026/09/25 15:27:35 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst lb # [ 229.271522] niks3-server[1153]: 2026/09/25 15:27:35 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls worker2 # [ 229.276067] nix-grpc-daemon[1736]: time=2026-09-25T15:27:35.899Z level=INFO msg="Upload complete. (192ms)" lb # [ 229.279415] niks3-server[1153]: 2026/09/25 15:27:35 INFO Received complete push request method=POST path=/api/pushes/90/complete lb # [ 229.283935] niks3-server[1153]: 2026/09/25 15:27:35 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo worker1 # [ 229.292365] nix-grpc-daemon[1742]: ts=2026-09-25T15:27:35Z level=debug event=want client=4 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=0 worker2 # [ 229.284551] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=15 outputs=1 inputs=1 inputs_fetched=0 input_bytes=0 input_s=0 worker1 # [ 229.295386] nix-grpc-daemon[1742]: ts=2026-09-25T15:27:35Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 229.295551] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 229.299762] nix-daemon[857]: reaped child process 2014, status = succeeded worker2 # [ 229.304773] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 229.312758] nix-grpc-daemon[1736]: time=2026-09-25T15:27:35.936Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'... worker2 # [ 229.318873] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 229.326962] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker2 # [ 229.439904] nix-daemon[857]: accepted connection from pid 2039, user nix-grpc-daemon (trusted) lb # [ 229.462091] niks3-server[1153]: 2026/09/25 15:27:36 INFO Received push request method=POST path=/api/pushes worker2 # [ 229.461852] nix-daemon[857]: reaped child process 2042, status = succeeded worker2 # [ 229.464951] nix-grpc-daemon[1736]: time=2026-09-25T15:27:36.088Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 229.466736] nix-grpc-daemon[1736]: time=2026-09-25T15:27:36.088Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)" lb # [ 229.480970] niks3-server[1153]: 2026/09/25 15:27:36 INFO Received sign narinfos request method=POST path=/api/pushes/91/sign worker2 # [ 229.482634] nix-grpc-daemon[1736]: time=2026-09-25T15:27:36.106Z level=INFO msg="Uploading 1 narinfos" lb # [ 229.487856] niks3-server[1153]: 2026/09/25 15:27:36 INFO Signed narinfos id=91 count=1 lb # [ 229.493300] niks3-server[1153]: 2026/09/25 15:27:36 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst lb # [ 229.500147] niks3-server[1153]: 2026/09/25 15:27:36 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls lb # [ 229.505609] niks3-server[1153]: 2026/09/25 15:27:36 INFO Received complete push request method=POST path=/api/pushes/91/complete worker2 # [ 229.509459] nix-grpc-daemon[1736]: time=2026-09-25T15:27:36.133Z level=INFO msg="Upload complete. (174ms)" lb # [ 229.512956] niks3-server[1153]: 2026/09/25 15:27:36 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo worker2 # [ 229.516204] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=0 paths=1 nar_bytes_in=760 worker2 # [ 229.524286] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 worker2 # [ 229.531051] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) worker2 # [ 229.532943] nix-daemon[857]: reaped child process 2037, status = succeeded client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051... client # farm-top> worker2: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker2 # [ 229.543334] nix-daemon[857]: accepted connection from pid 1664, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-drain worker2 # [ 230.625460] nix-daemon[857]: reaped child process 2046, status = succeeded worker2 # [ 230.693071] nix-daemon[857]: accepted connection from pid 2062, user nix-grpc-daemon (trusted) worker2 # [ 230.709963] nix-daemon[857]: reaped child process 2065, status = succeeded lb # [ 230.718544] niks3-server[1153]: 2026/09/25 15:27:37 INFO Received push request method=POST path=/api/pushes worker2 # [ 230.722834] nix-grpc-daemon[1736]: time=2026-09-25T15:27:37.345Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 230.724473] nix-grpc-daemon[1736]: time=2026-09-25T15:27:37.345Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)" lb # [ 230.743839] niks3-server[1153]: 2026/09/25 15:27:37 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst worker2 # [ 230.748138] nix-grpc-daemon[1736]: time=2026-09-25T15:27:37.371Z level=INFO msg="Uploading 1 narinfos" lb # [ 230.752310] niks3-server[1153]: 2026/09/25 15:27:37 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv lb # [ 230.755072] niks3-server[1153]: 2026/09/25 15:27:37 INFO Received sign narinfos request method=POST path=/api/pushes/92/sign lb # [ 230.766605] niks3-server[1153]: 2026/09/25 15:27:37 INFO Signed narinfos id=92 count=1 worker2 # [ 230.764249] nix-grpc-daemon[1736]: time=2026-09-25T15:27:37.388Z level=INFO msg="Upload complete. (213ms)" lb # [ 230.771966] niks3-server[1153]: 2026/09/25 15:27:37 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls lb # [ 230.776003] niks3-server[1153]: 2026/09/25 15:27:37 INFO Received complete push request method=POST path=/api/pushes/92/complete worker2 # [ 230.772622] nix-grpc-daemon[1664]: ts=2026-09-25T15:27:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43716 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=16 outputs=1 inputs=2 inputs_fetched=0 input_bytes=0 input_s=0 lb # [ 230.779086] niks3-server[1153]: 2026/09/25 15:27:37 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo worker2 # [ 230.783272] nix-daemon[857]: reaped child process 2044, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2, in 4.50 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath, in 0.14 seconds) worker1: must succeed: rm /nix/.rw-store/fill worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.12 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free' worker1 # [ 232.100498] nix-grpc-daemon[1742]: ts=2026-09-25T15:27:38Z level=info event=healthy reason=min_free worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.07 seconds) (finished: subtest: low disk drains a worker and builds go to the other, in 7.08 seconds) (finished: run the VM test script, in 233.04 seconds) test script finished in 233.18s cleanup kill QemuMachine (pid 50) client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) client # [2026-09-25T15:27:38Z INFO virtiofsd] Client disconnected, shutting down client # [2026-09-25T15:27:38Z INFO virtiofsd] Client disconnected, shutting down client # [2026-09-25T15:27:38Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 49) lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) lb # [2026-09-25T15:27:39Z INFO virtiofsd] Client disconnected, shutting down lb # [2026-09-25T15:27:39Z INFO virtiofsd] Client disconnected, shutting down lb # [2026-09-25T15:27:39Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 52) worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) worker1 # [2026-09-25T15:27:39Z INFO virtiofsd] Client disconnected, shutting down worker1 # [2026-09-25T15:27:39Z INFO virtiofsd] Client disconnected, shutting down worker1 # [2026-09-25T15:27:39Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 51) worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) worker2 # [2026-09-25T15:27:39Z INFO virtiofsd] Client disconnected, shutting down worker2 # [2026-09-25T15:27:39Z INFO virtiofsd] Client disconnected, shutting down worker2 # [2026-09-25T15:27:39Z INFO virtiofsd] Client disconnected, shutting down (finished: cleanup, in 1.09 seconds) additionally exposed symbols: client, lb, worker1, worker2, vlan1, start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh