vm-test-run-nix-grpc-farm
checks.aarch64-linux.farm
· build #490
· raw
1tribuchet: building on eliza2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12client # Disk image does not exist, creating the virtualisation disk image...13client # Formatting '/build/vm-state-client/tmp.ZuhhBwFpVt', fmt=raw size=107374182414client # mke2fs 1.47.4 (6-Mar-2025)15lb # Disk image does not exist, creating the virtualisation disk image...16worker2 # Disk image does not exist, creating the virtualisation disk image...17lb: QEMU running (pid 49)18client: QEMU running (pid 50)19worker1: QEMU running (pid 52)20client # Discarding device blocks: 0/262144 done21worker1 # Disk image does not exist, creating the virtualisation disk image...22worker2: QEMU running (pid 51)23client # Creating filesystem with 262144 4k blocks and 65536 inodes24worker1 # Formatting '/build/vm-state-worker1/tmp.0U3O84ZunV', fmt=raw size=107374182425lb # Formatting '/build/vm-state-lb/tmp.uM7Oduvt1P', fmt=raw size=107374182426worker2 # Formatting '/build/vm-state-worker2/tmp.oSagt32KKb', fmt=raw size=107374182427client # Filesystem UUID: d4dd8969-73c0-43cf-ab04-047d2883e63d28(finished: start all VMs, in 0.53 seconds)29worker2 # mke2fs 1.47.4 (6-Mar-2025)30lb # mke2fs 1.47.4 (6-Mar-2025)31worker1 # mke2fs 1.47.4 (6-Mar-2025)32client # Superblock backups stored on blocks:33lb # Discarding device blocks: 0/262144 done34worker1 # Discarding device blocks: 0/262144 done35client: waiting for unit network-addresses-eth1.service36worker2 # Discarding device blocks: 0/262144 done37client # 32768, 98304, 163840, 22937638lb # Creating filesystem with 262144 4k blocks and 65536 inodes39worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes40client # 41lb # Filesystem UUID: 263eff33-f374-436f-b58d-fd57b4da33c142worker2 # Filesystem UUID: 47eff782-01e5-4cab-8c65-8fec002074a143client # Allocating group tables: 0/8 done44lb # Superblock backups stored on blocks:45worker2 # Superblock backups stored on blocks:46client # Writing inode tables: 0/8 done47lb # 32768, 98304, 163840, 22937648worker2 # 32768, 98304, 163840, 22937649client # Creating journal (8192 blocks): done50lb # 51worker2 # 52client # Writing superblocks and filesystem accounting information: 0/8 done53lb # Allocating group tables: 0/8 done54worker2 # Allocating group tables: 0/8 done55client # 56lb # Writing inode tables: 0/8 done57worker2 # Writing inode tables: 0/8 done58client # Virtualisation disk image created.59lb # Creating journal (8192 blocks): done60worker2 # Creating journal (8192 blocks): done61client: waiting for the VM to finish booting62worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes63lb # Writing superblocks and filesystem accounting information: 0/8 done64worker1 # Filesystem UUID: bf9ef386-e4ac-4152-8954-4575120e86c065lb # 66worker1 # Superblock backups stored on blocks:67lb # Virtualisation disk image created.68worker1 # 32768, 98304, 163840, 22937669worker2 # Writing superblocks and filesystem accounting information: 0/8 done70worker1 # 71worker2 # 72worker1 # Allocating group tables: 0/8 done73worker2 # Virtualisation disk image created.74worker1 # Writing inode tables: 0/8 done75worker1 # Creating journal (8192 blocks): done76worker1 # Writing superblocks and filesystem accounting information: 0/8 done77worker1 # 78worker1 # Virtualisation disk image created.79client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]80client # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 202681client # [ 0.000000] KASLR enabled82client # [ 0.000000] random: crng init done83client # [ 0.000000] Machine model: linux,dummy-virt84client # [ 0.000000] efi: UEFI not found.85client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT86client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]87client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]88client # [ 0.000000] Zone ranges:89client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]90client # [ 0.000000] DMA32 empty91client # [ 0.000000] Normal empty92client # [ 0.000000] Device empty93client # [ 0.000000] Movable zone start for each node94client # [ 0.000000] Early memory node ranges95client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]96client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]97client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c40000098client # [ 0.000000] psci: probing for conduit method from DT.99client # [ 0.000000] psci: PSCIv1.3 detected in firmware.100client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs101client # [ 0.000000] psci: Trusted OS migration not required102client # [ 0.000000] psci: SMC Calling Convention v1.1103client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)104client # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296105client # [ 0.000000] Detected PIPT I-cache on CPU0106client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)107client # [ 0.000000] CPU features: detected: GICv3 CPU interface108client # [ 0.000000] CPU features: detected: Spectre-v4109client # [ 0.000000] CPU features: detected: Spectre-BHB110client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38111client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23112client # [ 0.000000] alternatives: applying boot alternatives113client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/5252da3xcqz5fxzqz01xd82cxl855ak1-nixos-system-client-test/init regInfo=/nix/store/a9vnspps9sb5b5czzzz1l864fbjlmq8d-closure-info/registration console=ttyAMA0,115200n8 console=tty0114client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/a9vnspps9sb5b5czzzz1l864fbjlmq8d-closure-info/registration", will be passed to user space.115worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]116client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes117lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]118client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)119worker2 # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026120worker2 # [ 0.000000] KASLR enabled121lb # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026122client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)123worker2 # [ 0.000000] random: crng init done124lb # [ 0.000000] KASLR enabled125lb # [ 0.000000] random: crng init done126worker2 # [ 0.000000] Machine model: linux,dummy-virt127client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB128lb # [ 0.000000] Machine model: linux,dummy-virt129worker2 # [ 0.000000] efi: UEFI not found.130client # [ 0.000000] software IO TLB: area num 1.131lb # [ 0.000000] efi: UEFI not found.132worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT133client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB134lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT135worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]136client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)137lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]138client # [ 0.000000] Fallback order for Node 0: 0139worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]140lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]141worker2 # [ 0.000000] Zone ranges:142lb # [ 0.000000] Zone ranges:143worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]144client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216145worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]146client # [ 0.000000] Policy zone: DMA147lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]148worker2 # [ 0.000000] DMA32 empty149lb # [ 0.000000] DMA32 empty150worker2 # [ 0.000000] Normal empty151client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off152lb # [ 0.000000] Normal empty153worker1 # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026154worker2 # [ 0.000000] Device empty155worker1 # [ 0.000000] KASLR enabled156lb # [ 0.000000] Device empty157worker2 # [ 0.000000] Movable zone start for each node158client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1159worker1 # [ 0.000000] random: crng init done160lb # [ 0.000000] Movable zone start for each node161worker2 # [ 0.000000] Early memory node ranges162worker1 # [ 0.000000] Machine model: linux,dummy-virt163lb # [ 0.000000] Early memory node ranges164client # [ 0.000000] allocated 3145728 bytes of page_ext165worker1 # [ 0.000000] efi: UEFI not found.166worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]167client # [ 0.000000] ftrace: allocating 74885 entries in 294 pages168lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]169client # [ 0.000000] ftrace: allocated 294 pages with 4 groups170worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT171worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]172lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]173client # [ 0.000000] rcu: Hierarchical RCU implementation.174worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000175worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]176lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000177client # [ 0.000000] rcu: RCU event tracing is enabled.178worker2 # [ 0.000000] psci: probing for conduit method from DT.179lb # [ 0.000000] psci: probing for conduit method from DT.180worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]181worker1 # [ 0.000000] Zone ranges:182client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.183worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware.184lb # [ 0.000000] psci: PSCIv1.3 detected in firmware.185worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs186worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]187client # [ 0.000000] Trampoline variant of Tasks RCU enabled.188lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs189worker1 # [ 0.000000] DMA32 empty190worker2 # [ 0.000000] psci: Trusted OS migration not required191client # [ 0.000000] Rude variant of Tasks RCU enabled.192lb # [ 0.000000] psci: Trusted OS migration not required193worker1 # [ 0.000000] Normal empty194worker2 # [ 0.000000] psci: SMC Calling Convention v1.1195client # [ 0.000000] Tracing variant of Tasks RCU enabled.196worker1 # [ 0.000000] Device empty197lb # [ 0.000000] psci: SMC Calling Convention v1.1198worker1 # [ 0.000000] Movable zone start for each node199client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.200worker1 # [ 0.000000] Early memory node ranges201worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)202lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)203worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]204worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296205client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1206lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296207worker2 # [ 0.000000] Detected PIPT I-cache on CPU0208lb # [ 0.000000] Detected PIPT I-cache on CPU0209worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]210client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.211worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000212worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)213lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)214worker1 # [ 0.000000] psci: probing for conduit method from DT.215worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface216client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.217lb # [ 0.000000] CPU features: detected: GICv3 CPU interface218worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware.219worker2 # [ 0.000000] CPU features: detected: Spectre-v4220lb # [ 0.000000] CPU features: detected: Spectre-v4221worker2 # [ 0.000000] CPU features: detected: Spectre-BHB222client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.223worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs224lb # [ 0.000000] CPU features: detected: Spectre-BHB225worker1 # [ 0.000000] psci: Trusted OS migration not required226worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38227client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0228lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38229worker1 # [ 0.000000] psci: SMC Calling Convention v1.1230client # [ 0.000000] GICv3: 256 SPIs implemented231worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23232client # [ 0.000000] GICv3: 0 Extended SPIs implemented233lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23234worker2 # [ 0.000000] alternatives: applying boot alternatives235worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)236client # [ 0.000000] Root IRQ handler: gic_handle_irq237lb # [ 0.000000] alternatives: applying boot alternatives238client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI239worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296240worker1 # [ 0.000000] Detected PIPT I-cache on CPU0241client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0242client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000243worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)244client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]245worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface246worker1 # [ 0.000000] CPU features: detected: Spectre-v4247worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/fhjd99g778w0n7n58v9pngxdn87svwlf-nixos-system-worker2-test/init regInfo=/nix/store/8clhlxs2384kagmjxsysmqfk4aqyyqmr-closure-info/registration console=ttyAMA0,115200n8 console=tty0248client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)249lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/a9n40kyqx09p2mx9kz6ck1ddzmmfmfqv-nixos-system-lb-test/init regInfo=/nix/store/kxfc7nc3yvm950aydaljrdz4dxr491hv-closure-info/registration console=ttyAMA0,115200n8 console=tty0250worker1 # [ 0.000000] CPU features: detected: Spectre-BHB251worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38252client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)253worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/8clhlxs2384kagmjxsysmqfk4aqyyqmr-closure-info/registration", will be passed to user space.254worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23255lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/kxfc7nc3yvm950aydaljrdz4dxr491hv-closure-info/registration", will be passed to user space.256client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000257worker1 # [ 0.000000] alternatives: applying boot alternatives258client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000259worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes260lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes261client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.262worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)263lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)264client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns265worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)266lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)267client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).268worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB269lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB270worker2 # [ 0.000000] software IO TLB: area num 1.271worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/h490qvdfgh8rdlrafpm8wgj56x94xnlb-nixos-system-worker1-test/init regInfo=/nix/store/pznpw35xiyr2l3gwkj88qbrfsnnwgdih-closure-info/registration console=ttyAMA0,115200n8 console=tty0272lb # [ 0.000000] software IO TLB: area num 1.273client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns274worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB275lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB276client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns277worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)278worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/pznpw35xiyr2l3gwkj88qbrfsnnwgdih-closure-info/registration", will be passed to user space.279lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)280client # [ 0.000033] arm-pv: using stolen time PV281worker2 # [ 0.000000] Fallback order for Node 0: 0282lb # [ 0.000000] Fallback order for Node 0: 0283worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes284worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216285lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216286worker2 # [ 0.000000] Policy zone: DMA287client # [ 0.000653] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)288worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)289lb # [ 0.000000] Policy zone: DMA290client # [ 0.000834] Console: colour dummy device 80x25291worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off292client # [ 0.000841] printk: legacy console [tty0] enabled293lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off294worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)295worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1296lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1297worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB298worker2 # [ 0.000000] allocated 3145728 bytes of page_ext299client # [ 0.001058] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)300lb # [ 0.000000] allocated 3145728 bytes of page_ext301worker1 # [ 0.000000] software IO TLB: area num 1.302worker2 # [ 0.000000] ftrace: allocating 74885 entries in 294 pages303client # [ 0.001065] pid_max: default: 32768 minimum: 301304lb # [ 0.000000] ftrace: allocating 74885 entries in 294 pages305worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB306worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups307lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups308client # [ 0.001141] LSM: initializing lsm=capability,landlock,yama,bpf,ima309worker2 # [ 0.000000] rcu: Hierarchical RCU implementation.310worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)311client # [ 0.001315] landlock: Up and running.312lb # [ 0.000000] rcu: Hierarchical RCU implementation.313worker2 # [ 0.000000] rcu: RCU event tracing is enabled.314client # [ 0.001317] Yama: becoming mindful.315worker1 # [ 0.000000] Fallback order for Node 0: 0316lb # [ 0.000000] rcu: RCU event tracing is enabled.317client # [ 0.001955] LSM support for eBPF active318worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.319worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216320lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.321worker1 # [ 0.000000] Policy zone: DMA322worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled.323client # [ 0.002118] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)324lb # [ 0.000000] Trampoline variant of Tasks RCU enabled.325worker2 # [ 0.000000] Rude variant of Tasks RCU enabled.326worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off327lb # [ 0.000000] Rude variant of Tasks RCU enabled.328client # [ 0.002153] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)329worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled.330lb # [ 0.000000] Tracing variant of Tasks RCU enabled.331worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1332client # [ 0.003282] cacheinfo: Unable to detect cache hierarchy for CPU 0333worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.334worker1 # [ 0.000000] allocated 3145728 bytes of page_ext335lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.336client # [ 0.004093] rcu: Hierarchical SRCU implementation.337worker1 # [ 0.000000] ftrace: allocating 74885 entries in 294 pages338worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1339client # [ 0.004098] rcu: Max phase no-delay instances is 1000.340lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1341worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups342client # [ 0.005332] fsl-mc MSI: its@8080000 domain created343worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.344worker1 # [ 0.000000] rcu: Hierarchical RCU implementation.345client # [ 0.005423] EFI services will not be available.346lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.347client # [ 0.005533] smp: Bringing up secondary CPUs ...348worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.349client # [ 0.005542] smp: Brought up 1 node, 1 CPU350worker1 # [ 0.000000] rcu: RCU event tracing is enabled.351lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.352client # [ 0.005545] SMP: Total of 1 processors activated.353worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.354worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.355client # [ 0.005551] CPU: All CPU(s) started at EL1356lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.357worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled.358worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0359lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0360client # [ 0.005563] CPU features: detected: Branch Target Identification361worker1 # [ 0.000000] Rude variant of Tasks RCU enabled.362worker2 # [ 0.000000] GICv3: 256 SPIs implemented363lb # [ 0.000000] GICv3: 256 SPIs implemented364worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled.365worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented366client # [ 0.005568] CPU features: detected: ARMv8.4 Translation Table Level367lb # [ 0.000000] GICv3: 0 Extended SPIs implemented368worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq369lb # [ 0.000000] Root IRQ handler: gic_handle_irq370worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.371client # [ 0.005571] CPU features: detected: Instruction cache invalidation not required for I/D coherence372worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI373lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI374worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1375worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0376lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0377client # [ 0.005575] CPU features: detected: Data cache clean to the PoU not required for I/D coherence378worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000379lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000380worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.381client # [ 0.005578] CPU features: detected: Common not Private translations382worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]383lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]384client # [ 0.005581] CPU features: detected: CRC32 instructions385worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.386worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)387lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1)388client # [ 0.005584] CPU features: detected: Data cache clean to Point of Deep Persistence389worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.390client # [ 0.005588] CPU features: detected: Data cache clean to Point of Persistence391worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)392lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1)393worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0394client # [ 0.005591] CPU features: detected: Data independent timing control (DIT)395worker1 # [ 0.000000] GICv3: 256 SPIs implemented396worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000397lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000398client # [ 0.005594] CPU features: detected: E0PD399worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented400worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000401lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000402worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq403client # [ 0.005597] CPU features: detected: Enhanced Counter Virtualization404worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI405worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.406lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.407client # [ 0.005600] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)408worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0409worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns410client # [ 0.005603] CPU features: detected: Enhanced Virtualization Traps411lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns412worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000413worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).414client # [ 0.005607] CPU features: detected: Fine Grained Traps415worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]416lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).417client # [ 0.005611] CPU features: detected: Generic authentication (architected QARMA5 algorithm)418worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)419worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns420lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns421client # [ 0.005616] CPU features: detected: RCpc load-acquire (LDAPR)422worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns423client # [ 0.005619] CPU features: detected: LSE atomic instructions424worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)425lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns426worker2 # [ 0.000032] arm-pv: using stolen time PV427lb # [ 0.000031] arm-pv: using stolen time PV428client # [ 0.005622] CPU features: detected: Privileged Access Never429worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000430client # [ 0.005625] CPU features: detected: PMUv3431worker2 # [ 0.000416] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)432worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000433lb # [ 0.000434] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)434client # [ 0.005627] CPU features: detected: RAS Extension Support435worker2 # [ 0.000582] Console: colour dummy device 80x25436lb # [ 0.000607] Console: colour dummy device 80x25437worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.438client # [ 0.005630] CPU features: detected: RASv1p1 Extension Support439worker2 # [ 0.000589] printk: legacy console [tty0] enabled440lb # [ 0.000615] printk: legacy console [tty0] enabled441client # [ 0.005633] CPU features: detected: Random Number Generator442worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns443worker2 # [ 0.000775] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)444client # [ 0.005635] CPU features: detected: Speculation barrier (SB)445lb # [ 0.000810] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)446worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).447worker2 # [ 0.000782] pid_max: default: 32768 minimum: 301448client # [ 0.005639] CPU features: detected: Stage-2 Force Write-Back449lb # [ 0.000816] pid_max: default: 32768 minimum: 301450worker2 # [ 0.000850] LSM: initializing lsm=capability,landlock,yama,bpf,ima451client # [ 0.005641] CPU features: detected: TLB range maintenance instructions452lb # [ 0.000893] LSM: initializing lsm=capability,landlock,yama,bpf,ima453worker2 # [ 0.000990] landlock: Up and running.454worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns455lb # [ 0.001014] landlock: Up and running.456worker2 # [ 0.000992] Yama: becoming mindful.457client # [ 0.005646] CPU features: detected: Speculative Store Bypassing Safe (SSBS)458lb # [ 0.001017] Yama: becoming mindful.459worker2 # [ 0.001445] LSM support for eBPF active460worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns461lb # [ 0.001489] LSM support for eBPF active462client # [ 0.005687] alternatives: applying system-wide alternatives463worker1 # [ 0.000030] arm-pv: using stolen time PV464worker2 # [ 0.001578] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)465lb # [ 0.001631] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)466client # [ 0.008750] CPU features: detected: BBM Level 2 without TLB conflict abort467worker2 # [ 0.001610] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)468worker1 # [ 0.000412] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)469lb # [ 0.001664] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)470worker1 # [ 0.000593] Console: colour dummy device 80x25471worker2 # [ 0.002794] cacheinfo: Unable to detect cache hierarchy for CPU 0472lb # [ 0.002830] cacheinfo: Unable to detect cache hierarchy for CPU 0473client # [ 0.008968] Memory: 1406528K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123784K reserved, 32768K cma-reserved)474worker1 # [ 0.000602] printk: legacy console [tty0] enabled475worker2 # [ 0.003489] rcu: Hierarchical SRCU implementation.476client # [ 0.009371] devtmpfs: initialized477lb # [ 0.003546] rcu: Hierarchical SRCU implementation.478worker2 # [ 0.003493] rcu: Max phase no-delay instances is 1000.479lb # [ 0.003550] rcu: Max phase no-delay instances is 1000.480client # [ 0.011226] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)481worker1 # [ 0.000794] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)482worker2 # [ 0.004662] fsl-mc MSI: its@8080000 domain created483lb # [ 0.004720] fsl-mc MSI: its@8080000 domain created484worker1 # [ 0.000801] pid_max: default: 32768 minimum: 301485worker2 # [ 0.004748] EFI services will not be available.486lb # [ 0.004819] EFI services will not be available.487client # [ 0.011250] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).488worker2 # [ 0.004832] smp: Bringing up secondary CPUs ...489worker1 # [ 0.000876] LSM: initializing lsm=capability,landlock,yama,bpf,ima490lb # [ 0.004888] smp: Bringing up secondary CPUs ...491worker2 # [ 0.004839] smp: Brought up 1 node, 1 CPU492worker1 # [ 0.001050] landlock: Up and running.493client # [ 0.011464] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL494lb # [ 0.004896] smp: Brought up 1 node, 1 CPU495worker1 # [ 0.001053] Yama: becoming mindful.496worker2 # [ 0.004842] SMP: Total of 1 processors activated.497client # [ 0.011469] 0 pages in range for non-PLT usage498lb # [ 0.004899] SMP: Total of 1 processors activated.499worker1 # [ 0.001502] LSM support for eBPF active500worker2 # [ 0.004849] CPU: All CPU(s) started at EL1501client # [ 0.011470] 508288 pages in range for PLT usage502lb # [ 0.004904] CPU: All CPU(s) started at EL1503worker2 # [ 0.004860] CPU features: detected: Branch Target Identification504client # [ 0.011567] pinctrl core: initialized pinctrl subsystem505worker1 # [ 0.001639] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)506lb # [ 0.004915] CPU features: detected: Branch Target Identification507client # [ 0.012311] DMI not present or invalid.508worker2 # [ 0.004865] CPU features: detected: ARMv8.4 Translation Table Level509lb # [ 0.004920] CPU features: detected: ARMv8.4 Translation Table Level510worker1 # [ 0.001671] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)511client # [ 0.016182] NET: Registered PF_NETLINK/PF_ROUTE protocol family512worker1 # [ 0.002710] cacheinfo: Unable to detect cache hierarchy for CPU 0513worker2 # [ 0.004868] CPU features: detected: Instruction cache invalidation not required for I/D coherence514lb # [ 0.004923] CPU features: detected: Instruction cache invalidation not required for I/D coherence515client # [ 0.018481] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations516worker1 # [ 0.003431] rcu: Hierarchical SRCU implementation.517worker2 # [ 0.004872] CPU features: detected: Data cache clean to the PoU not required for I/D coherence518worker1 # [ 0.003434] rcu: Max phase no-delay instances is 1000.519client # [ 0.018660] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations520lb # [ 0.004926] CPU features: detected: Data cache clean to the PoU not required for I/D coherence521worker1 # [ 0.004641] fsl-mc MSI: its@8080000 domain created522worker2 # [ 0.004876] CPU features: detected: Common not Private translations523lb # [ 0.004930] CPU features: detected: Common not Private translations524client # [ 0.018865] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations525worker1 # [ 0.004731] EFI services will not be available.526worker2 # [ 0.004879] CPU features: detected: CRC32 instructions527lb # [ 0.004933] CPU features: detected: CRC32 instructions528worker1 # [ 0.004810] smp: Bringing up secondary CPUs ...529client # [ 0.018887] audit: initializing netlink subsys (disabled)530worker1 # [ 0.004818] smp: Brought up 1 node, 1 CPU531worker2 # [ 0.004881] CPU features: detected: Data cache clean to Point of Deep Persistence532lb # [ 0.004936] CPU features: detected: Data cache clean to Point of Deep Persistence533client # [ 0.019498] thermal_sys: Registered thermal governor 'fair_share'534worker1 # [ 0.004821] SMP: Total of 1 processors activated.535worker2 # [ 0.004885] CPU features: detected: Data cache clean to Point of Persistence536lb # [ 0.004940] CPU features: detected: Data cache clean to Point of Persistence537client # [ 0.019500] thermal_sys: Registered thermal governor 'bang_bang'538worker1 # [ 0.004827] CPU: All CPU(s) started at EL1539worker2 # [ 0.004888] CPU features: detected: Data independent timing control (DIT)540client # [ 0.019504] thermal_sys: Registered thermal governor 'step_wise'541worker1 # [ 0.004838] CPU features: detected: Branch Target Identification542lb # [ 0.004943] CPU features: detected: Data independent timing control (DIT)543worker2 # [ 0.004891] CPU features: detected: E0PD544lb # [ 0.004946] CPU features: detected: E0PD545client # [ 0.019510] thermal_sys: Registered thermal governor 'user_space'546worker1 # [ 0.004843] CPU features: detected: ARMv8.4 Translation Table Level547lb # [ 0.004949] CPU features: detected: Enhanced Counter Virtualization548worker2 # [ 0.004893] CPU features: detected: Enhanced Counter Virtualization549client # [ 0.019512] thermal_sys: Registered thermal governor 'power_allocator'550worker1 # [ 0.004847] CPU features: detected: Instruction cache invalidation not required for I/D coherence551worker2 # [ 0.004896] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)552lb # [ 0.004952] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)553client # [ 0.019539] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1554worker1 # [ 0.004850] CPU features: detected: Data cache clean to the PoU not required for I/D coherence555worker2 # [ 0.004900] CPU features: detected: Enhanced Virtualization Traps556lb # [ 0.004955] CPU features: detected: Enhanced Virtualization Traps557client # [ 0.019548] cpuidle: using governor ladder558client # [ 0.019554] cpuidle: using governor menu559lb # [ 0.004958] CPU features: detected: Fine Grained Traps560worker2 # [ 0.004903] CPU features: detected: Fine Grained Traps561worker1 # [ 0.004854] CPU features: detected: Common not Private translations562worker1 # [ 0.004857] CPU features: detected: CRC32 instructions563client # [ 0.019764] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.564worker2 # [ 0.004906] CPU features: detected: Generic authentication (architected QARMA5 algorithm)565lb # [ 0.004963] CPU features: detected: Generic authentication (architected QARMA5 algorithm)566client # [ 0.019781] ASID allocator initialised with 65536 entries567worker1 # [ 0.004860] CPU features: detected: Data cache clean to Point of Deep Persistence568worker2 # [ 0.004911] CPU features: detected: RCpc load-acquire (LDAPR)569lb # [ 0.004968] CPU features: detected: RCpc load-acquire (LDAPR)570client # [ 0.021076] Serial: AMBA PL011 UART driver571worker2 # [ 0.004914] CPU features: detected: LSE atomic instructions572lb # [ 0.004971] CPU features: detected: LSE atomic instructions573worker1 # [ 0.004864] CPU features: detected: Data cache clean to Point of Persistence574client # [ 0.026389] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1575worker2 # [ 0.004917] CPU features: detected: Privileged Access Never576lb # [ 0.004974] CPU features: detected: Privileged Access Never577worker1 # [ 0.004867] CPU features: detected: Data independent timing control (DIT)578worker2 # [ 0.004920] CPU features: detected: PMUv3579client # [ 0.026542] printk: console [ttyAMA0] enabled580lb # [ 0.004976] CPU features: detected: PMUv3581worker1 # [ 0.004870] CPU features: detected: E0PD582worker2 # [ 0.004922] CPU features: detected: RAS Extension Support583lb # [ 0.004979] CPU features: detected: RAS Extension Support584worker1 # [ 0.004872] CPU features: detected: Enhanced Counter Virtualization585worker2 # [ 0.004925] CPU features: detected: RASv1p1 Extension Support586lb # [ 0.004982] CPU features: detected: RASv1p1 Extension Support587worker1 # [ 0.004876] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)588worker2 # [ 0.004927] CPU features: detected: Random Number Generator589lb # [ 0.004985] CPU features: detected: Random Number Generator590worker1 # [ 0.004879] CPU features: detected: Enhanced Virtualization Traps591worker2 # [ 0.004930] CPU features: detected: Speculation barrier (SB)592lb # [ 0.004987] CPU features: detected: Speculation barrier (SB)593worker1 # [ 0.004882] CPU features: detected: Fine Grained Traps594worker2 # [ 0.004933] CPU features: detected: Stage-2 Force Write-Back595lb # [ 0.004990] CPU features: detected: Stage-2 Force Write-Back596worker1 # [ 0.004886] CPU features: detected: Generic authentication (architected QARMA5 algorithm)597worker2 # [ 0.004936] CPU features: detected: TLB range maintenance instructions598lb # [ 0.004993] CPU features: detected: TLB range maintenance instructions599worker1 # [ 0.004891] CPU features: detected: RCpc load-acquire (LDAPR)600worker2 # [ 0.004940] CPU features: detected: Speculative Store Bypassing Safe (SSBS)601lb # [ 0.004999] CPU features: detected: Speculative Store Bypassing Safe (SSBS)602worker1 # [ 0.004894] CPU features: detected: LSE atomic instructions603worker2 # [ 0.004976] alternatives: applying system-wide alternatives604lb # [ 0.005036] alternatives: applying system-wide alternatives605worker1 # [ 0.004897] CPU features: detected: Privileged Access Never606lb # [ 0.008075] CPU features: detected: BBM Level 2 without TLB conflict abort607worker2 # [ 0.007952] CPU features: detected: BBM Level 2 without TLB conflict abort608worker1 # [ 0.004900] CPU features: detected: PMUv3609client # [ 0.144405] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages610worker1 # [ 0.004903] CPU features: detected: RAS Extension Support611client # [ 0.144422] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page612worker1 # [ 0.004905] CPU features: detected: RASv1p1 Extension Support613worker2 # [ 0.008128] Memory: 1406512K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123772K reserved, 32768K cma-reserved)614lb # [ 0.008231] Memory: 1406544K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123764K reserved, 32768K cma-reserved)615worker2 # [ 0.008495] devtmpfs: initialized616lb # [ 0.008613] devtmpfs: initialized617worker1 # [ 0.004908] CPU features: detected: Random Number Generator618client # [ 0.144427] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages619worker1 # [ 0.004911] CPU features: detected: Speculation barrier (SB)620client # [ 0.144432] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page621worker2 # [ 0.010491] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)622lb # [ 0.010463] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)623worker1 # [ 0.004914] CPU features: detected: Stage-2 Force Write-Back624client # [ 0.144436] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages625worker2 # [ 0.010513] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).626lb # [ 0.010485] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).627worker1 # [ 0.004917] CPU features: detected: TLB range maintenance instructions628client # [ 0.144440] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page629worker2 # [ 0.010696] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL630lb # [ 0.010681] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL631worker1 # [ 0.004922] CPU features: detected: Speculative Store Bypassing Safe (SSBS)632client # [ 0.144445] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages633worker1 # [ 0.004958] alternatives: applying system-wide alternatives634client # [ 0.144449] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page635worker1 # [ 0.007902] CPU features: detected: BBM Level 2 without TLB conflict abort636client # [ 0.151788] fbcon: Taking over console637worker1 # [ 0.008137] Memory: 1406504K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123772K reserved, 32768K cma-reserved)638client # [ 0.151804] ACPI: Interpreter disabled.639worker1 # [ 0.008545] devtmpfs: initialized640worker1 # [ 0.010256] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)641worker1 # [ 0.010279] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).642worker1 # [ 0.010484] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL643worker1 # [ 0.010488] 0 pages in range for non-PLT usage644worker1 # [ 0.010489] 508288 pages in range for PLT usage645worker1 # [ 0.010596] pinctrl core: initialized pinctrl subsystem646worker2 # [ 0.010701] 0 pages in range for non-PLT usage647lb # [ 0.010685] 0 pages in range for non-PLT usage648worker1 # [ 0.011331] DMI not present or invalid.649worker2 # [ 0.010702] 508288 pages in range for PLT usage650lb # [ 0.010686] 508288 pages in range for PLT usage651worker1 # [ 0.014552] NET: Registered PF_NETLINK/PF_ROUTE protocol family652worker2 # [ 0.010810] pinctrl core: initialized pinctrl subsystem653lb # [ 0.010804] pinctrl core: initialized pinctrl subsystem654worker2 # [ 0.011599] DMI not present or invalid.655lb # [ 0.011567] DMI not present or invalid.656worker1 # [ 0.016754] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations657worker2 # [ 0.014825] NET: Registered PF_NETLINK/PF_ROUTE protocol family658lb # [ 0.014929] NET: Registered PF_NETLINK/PF_ROUTE protocol family659worker1 # [ 0.016927] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations660worker2 # [ 0.017045] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations661lb # [ 0.017140] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations662worker1 # [ 0.017121] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations663client # [ 0.160854] iommu: Default domain type: Translated664worker2 # [ 0.017239] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations665lb # [ 0.017328] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations666worker1 # [ 0.017143] audit: initializing netlink subsys (disabled)667client # [ 0.160864] iommu: DMA domain TLB invalidation policy: strict mode668worker2 # [ 0.017436] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations669lb # [ 0.017531] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations670client # [ 0.161252] SCSI subsystem initialized671worker1 # [ 0.017717] thermal_sys: Registered thermal governor 'fair_share'672worker2 # [ 0.017460] audit: initializing netlink subsys (disabled)673lb # [ 0.017552] audit: initializing netlink subsys (disabled)674worker1 # [ 0.017718] thermal_sys: Registered thermal governor 'bang_bang'675worker2 # [ 0.018038] thermal_sys: Registered thermal governor 'fair_share'676lb # [ 0.018151] thermal_sys: Registered thermal governor 'fair_share'677client # [ 0.163200] usbcore: registered new interface driver usbfs678worker1 # [ 0.017722] thermal_sys: Registered thermal governor 'step_wise'679worker2 # [ 0.018039] thermal_sys: Registered thermal governor 'bang_bang'680lb # [ 0.018154] thermal_sys: Registered thermal governor 'bang_bang'681client # [ 0.163247] usbcore: registered new interface driver hub682worker1 # [ 0.017727] thermal_sys: Registered thermal governor 'user_space'683worker2 # [ 0.018043] thermal_sys: Registered thermal governor 'step_wise'684lb # [ 0.018158] thermal_sys: Registered thermal governor 'step_wise'685client # [ 0.163266] usbcore: registered new device driver usb686worker1 # [ 0.017730] thermal_sys: Registered thermal governor 'power_allocator'687client # [ 0.163537] pps_core: LinuxPPS API ver. 1 registered688worker2 # [ 0.018049] thermal_sys: Registered thermal governor 'user_space'689lb # [ 0.018163] thermal_sys: Registered thermal governor 'user_space'690worker1 # [ 0.017754] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1691worker2 # [ 0.018051] thermal_sys: Registered thermal governor 'power_allocator'692lb # [ 0.018165] thermal_sys: Registered thermal governor 'power_allocator'693worker1 # [ 0.017762] cpuidle: using governor ladder694client # [ 0.163543] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>695client # [ 0.163554] PTP clock support registered696worker2 # [ 0.018075] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1697lb # [ 0.018198] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1698client # [ 0.163605] EDAC MC: Ver: 3.0.0699worker2 # [ 0.018083] cpuidle: using governor ladder700lb # [ 0.018206] cpuidle: using governor ladder701worker2 # [ 0.018089] cpuidle: using governor menu702lb # [ 0.018212] cpuidle: using governor menu703client # [ 0.168099] scmi_core: SCMI protocol bus registered704worker2 # [ 0.018280] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.705lb # [ 0.018394] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.706client # [ 0.169086] FPGA manager framework707worker2 # [ 0.018294] ASID allocator initialised with 65536 entries708lb # [ 0.018409] ASID allocator initialised with 65536 entries709client # [ 0.170084] vgaarb: loaded710worker2 # [ 0.019437] Serial: AMBA PL011 UART driver711lb # [ 0.019590] Serial: AMBA PL011 UART driver712client # [ 0.170708] clocksource: Switched to clocksource arch_sys_counter713worker2 # [ 0.024485] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1714lb # [ 0.024891] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1715worker2 # [ 0.024605] printk: console [ttyAMA0] enabled716lb # [ 0.025017] printk: console [ttyAMA0] enabled717worker2 # [ 0.143326] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages718lb # [ 0.143558] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages719worker2 # [ 0.143346] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page720lb # [ 0.143578] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page721worker2 # [ 0.143350] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages722lb # [ 0.143584] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages723worker1 # [ 0.017767] cpuidle: using governor menu724worker2 # [ 0.143355] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page725lb # [ 0.143588] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page726client # [ 0.175595] VFS: Disk quotas dquot_6.6.0727worker1 # [ 0.017956] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.728worker2 # [ 0.143359] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages729lb # [ 0.143593] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages730client # [ 0.175631] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)731worker1 # [ 0.017971] ASID allocator initialised with 65536 entries732worker2 # [ 0.143363] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page733lb # [ 0.143597] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page734worker1 # [ 0.019138] Serial: AMBA PL011 UART driver735worker2 # [ 0.143367] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages736lb # [ 0.143601] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages737worker1 # [ 0.024272] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1738worker2 # [ 0.143371] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page739lb # [ 0.143606] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page740worker1 # [ 0.024421] printk: console [ttyAMA0] enabled741client # [ 0.179201] netfs: FS-Cache loaded742client # [ 0.179343] pnp: PnP ACPI: disabled743worker1 # [ 0.144120] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages744worker1 # [ 0.144136] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page745worker2 # [ 0.150627] fbcon: Taking over console746lb # [ 0.150894] fbcon: Taking over console747worker2 # [ 0.150641] ACPI: Interpreter disabled.748lb # [ 0.150908] ACPI: Interpreter disabled.749worker1 # [ 0.144141] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages750worker2 # [ 0.152405] iommu: Default domain type: Translated751lb # [ 0.152711] iommu: Default domain type: Translated752worker1 # [ 0.144146] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page753worker2 # [ 0.152414] iommu: DMA domain TLB invalidation policy: strict mode754lb # [ 0.152721] iommu: DMA domain TLB invalidation policy: strict mode755worker1 # [ 0.144150] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages756worker2 # [ 0.154069] SCSI subsystem initialized757lb # [ 0.154398] SCSI subsystem initialized758worker1 # [ 0.144154] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page759worker1 # [ 0.144158] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages760worker1 # [ 0.144162] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page761client # [ 0.183453] NET: Registered PF_INET protocol family762worker1 # [ 0.151483] fbcon: Taking over console763client # [ 0.183726] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)764worker1 # [ 0.151499] ACPI: Interpreter disabled.765worker1 # [ 0.153386] iommu: Default domain type: Translated766worker1 # [ 0.153394] iommu: DMA domain TLB invalidation policy: strict mode767worker1 # [ 0.155059] SCSI subsystem initialized768worker2 # [ 0.161502] usbcore: registered new interface driver usbfs769lb # [ 0.162009] usbcore: registered new interface driver usbfs770worker2 # [ 0.161534] usbcore: registered new interface driver hub771lb # [ 0.162042] usbcore: registered new interface driver hub772worker2 # [ 0.161552] usbcore: registered new device driver usb773lb # [ 0.162066] usbcore: registered new device driver usb774worker2 # [ 0.161828] pps_core: LinuxPPS API ver. 1 registered775lb # [ 0.162370] pps_core: LinuxPPS API ver. 1 registered776worker2 # [ 0.161834] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>777lb # [ 0.162376] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>778worker2 # [ 0.161843] PTP clock support registered779lb # [ 0.162386] PTP clock support registered780worker2 # [ 0.161890] EDAC MC: Ver: 3.0.0781lb # [ 0.162425] EDAC MC: Ver: 3.0.0782worker2 # [ 0.166611] scmi_core: SCMI protocol bus registered783worker1 # [ 0.160405] usbcore: registered new interface driver usbfs784lb # [ 0.167120] scmi_core: SCMI protocol bus registered785worker1 # [ 0.160436] usbcore: registered new interface driver hub786worker2 # [ 0.167551] FPGA manager framework787worker1 # [ 0.160454] usbcore: registered new device driver usb788lb # [ 0.168084] FPGA manager framework789worker2 # [ 0.168534] vgaarb: loaded790lb # [ 0.169070] vgaarb: loaded791worker1 # [ 0.160711] pps_core: LinuxPPS API ver. 1 registered792worker1 # [ 0.160717] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>793worker2 # [ 0.169157] clocksource: Switched to clocksource arch_sys_counter794worker1 # [ 0.160726] PTP clock support registered795worker2 # [ 0.169748] VFS: Disk quotas dquot_6.6.0796lb # [ 0.169694] clocksource: Switched to clocksource arch_sys_counter797worker1 # [ 0.160773] EDAC MC: Ver: 3.0.0798lb # [ 0.170515] VFS: Disk quotas dquot_6.6.0799worker2 # [ 0.169775] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)800worker1 # [ 0.165386] scmi_core: SCMI protocol bus registered801lb # [ 0.170554] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)802worker2 # [ 0.172128] netfs: FS-Cache loaded803worker2 # [ 0.172254] pnp: PnP ACPI: disabled804worker1 # [ 0.166345] FPGA manager framework805lb # [ 0.172918] netfs: FS-Cache loaded806worker1 # [ 0.167265] vgaarb: loaded807lb # [ 0.173019] pnp: PnP ACPI: disabled808worker1 # [ 0.167896] clocksource: Switched to clocksource arch_sys_counter809worker1 # [ 0.171542] VFS: Disk quotas dquot_6.6.0810worker1 # [ 0.171577] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)811worker2 # [ 0.178136] NET: Registered PF_INET protocol family812worker1 # [ 0.175243] netfs: FS-Cache loaded813lb # [ 0.178924] NET: Registered PF_INET protocol family814worker1 # [ 0.175379] pnp: PnP ACPI: disabled815worker2 # [ 0.178408] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)816lb # [ 0.179195] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)817client # [ 0.214157] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)818worker1 # [ 0.179124] NET: Registered PF_INET protocol family819worker1 # [ 0.179395] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)820client # [ 0.214206] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)821client # [ 0.214238] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)822client # [ 0.214308] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)823client # [ 0.214450] TCP: Hash tables configured (established 16384 bind 16384)824client # [ 0.214544] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)825client # [ 0.214609] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)826client # [ 0.214647] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)827client # [ 0.222138] NET: Registered PF_UNIX/PF_LOCAL protocol family828client # [ 0.222197] NET: Registered PF_XDP protocol family829client # [ 0.222217] PCI: CLS 0 bytes, default 64830client # [ 0.222466] Trying to unpack rootfs image as initramfs...831client # [ 0.233236] kvm [1]: HYP mode not available832lb # [ 0.209422] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)833lb # [ 0.209476] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)834lb # [ 0.209511] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)835lb # [ 0.209599] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)836worker2 # [ 0.207953] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)837worker2 # [ 0.208008] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)838worker2 # [ 0.208037] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)839worker2 # [ 0.208124] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)840worker2 # [ 0.208267] TCP: Hash tables configured (established 16384 bind 16384)841lb # [ 0.213722] TCP: Hash tables configured (established 16384 bind 16384)842worker2 # [ 0.208354] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)843worker1 # [ 0.209055] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)844lb # [ 0.213820] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)845worker2 # [ 0.208390] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)846worker1 # [ 0.209111] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)847lb # [ 0.213857] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)848worker2 # [ 0.208427] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)849lb # [ 0.213895] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)850worker1 # [ 0.209145] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)851worker2 # [ 0.208574] NET: Registered PF_UNIX/PF_LOCAL protocol family852lb # [ 0.214028] NET: Registered PF_UNIX/PF_LOCAL protocol family853worker2 # [ 0.208597] NET: Registered PF_XDP protocol family854worker1 # [ 0.209228] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)855worker2 # [ 0.208615] PCI: CLS 0 bytes, default 64856lb # [ 0.214051] NET: Registered PF_XDP protocol family857lb # [ 0.214069] PCI: CLS 0 bytes, default 64858worker1 # [ 0.209371] TCP: Hash tables configured (established 16384 bind 16384)859worker2 # [ 0.208880] Trying to unpack rootfs image as initramfs...860lb # [ 0.214349] Trying to unpack rootfs image as initramfs...861worker1 # [ 0.209455] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)862worker1 # [ 0.209491] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)863worker1 # [ 0.209527] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)864worker1 # [ 0.209633] NET: Registered PF_UNIX/PF_LOCAL protocol family865lb # [ 0.227295] kvm [1]: HYP mode not available866worker1 # [ 0.209675] NET: Registered PF_XDP protocol family867worker1 # [ 0.209694] PCI: CLS 0 bytes, default 64868worker1 # [ 0.209952] Trying to unpack rootfs image as initramfs...869worker2 # [ 0.226736] kvm [1]: HYP mode not available870worker1 # [ 0.225483] kvm [1]: HYP mode not available871client # [ 0.331304] Initialise system trusted keyrings872client # [ 0.332052] workingset: timestamp_bits=42 max_order=19 bucket_order=0873client # [ 0.333347] squashfs: version 4.0 (2009/01/31) Phillip Lougher874client # [ 0.334109] 9p: Installing v9fs 9p2000 file system support875worker2 # [ 0.323582] Initialise system trusted keyrings876worker2 # [ 0.324311] workingset: timestamp_bits=42 max_order=19 bucket_order=0877lb # [ 0.326802] Initialise system trusted keyrings878worker1 # [ 0.321239] Initialise system trusted keyrings879lb # [ 0.327526] workingset: timestamp_bits=42 max_order=19 bucket_order=0880worker1 # [ 0.322027] workingset: timestamp_bits=42 max_order=19 bucket_order=0881worker1 # [ 0.323279] squashfs: version 4.0 (2009/01/31) Phillip Lougher882worker2 # [ 0.325607] squashfs: version 4.0 (2009/01/31) Phillip Lougher883client # [ 0.362845] Key type asymmetric registered884worker2 # [ 0.325679] 9p: Installing v9fs 9p2000 file system support885lb # [ 0.334198] squashfs: version 4.0 (2009/01/31) Phillip Lougher886client # [ 0.362874] Asymmetric key parser 'x509' registered887client # [ 0.362955] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)888client # [ 0.365093] io scheduler mq-deadline registered889client # [ 0.365104] io scheduler kyber registered890lb # [ 0.334988] 9p: Installing v9fs 9p2000 file system support891worker1 # [ 0.324079] 9p: Installing v9fs 9p2000 file system support892worker2 # [ 0.345601] Key type asymmetric registered893worker2 # [ 0.345626] Asymmetric key parser 'x509' registered894client # [ 0.374836] pl061_gpio 9030000.pl061: PL061 GPIO chip registered895worker2 # [ 0.345711] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)896client # [ 0.376267] ledtrig-cpu: registered to indicate activity on CPUs897worker1 # [ 0.344036] Key type asymmetric registered898client # [ 0.376648] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:899worker1 # [ 0.344061] Asymmetric key parser 'x509' registered900client # [ 0.376666] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000901worker1 # [ 0.344126] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)902client # [ 0.376678] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000903worker2 # [ 0.353202] io scheduler mq-deadline registered904worker2 # [ 0.353222] io scheduler kyber registered905client # [ 0.376687] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000906client # [ 0.376706] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits907lb # [ 0.355671] Key type asymmetric registered908client # [ 0.376731] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]909lb # [ 0.355698] Asymmetric key parser 'x509' registered910client # [ 0.376808] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00911lb # [ 0.355771] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)912client # [ 0.376817] pci_bus 0000:00: root bus resource [bus 00-ff]913lb # [ 0.357960] io scheduler mq-deadline registered914worker1 # [ 0.351954] io scheduler mq-deadline registered915client # [ 0.376823] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]916lb # [ 0.357970] io scheduler kyber registered917worker1 # [ 0.351972] io scheduler kyber registered918client # [ 0.376829] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]919client # [ 0.376834] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]920client # [ 0.376933] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint921client # [ 0.377371] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint922worker2 # [ 0.361303] pl061_gpio 9030000.pl061: PL061 GPIO chip registered923client # [ 0.377555] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]924client # [ 0.377571] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]925client # [ 0.377600] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]926client # [ 0.377616] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]927worker2 # [ 0.362653] ledtrig-cpu: registered to indicate activity on CPUs928client # [ 0.378061] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint929worker2 # [ 0.363038] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:930client # [ 0.378241] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]931worker1 # [ 0.360048] pl061_gpio 9030000.pl061: PL061 GPIO chip registered932worker2 # [ 0.363054] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000933client # [ 0.378256] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]934client # [ 0.378285] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]935worker2 # [ 0.363066] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000936worker2 # [ 0.363075] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000937worker1 # [ 0.361360] ledtrig-cpu: registered to indicate activity on CPUs938worker2 # [ 0.363093] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits939worker1 # [ 0.361758] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:940worker2 # [ 0.363117] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]941worker1 # [ 0.361775] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000942worker2 # [ 0.363193] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00943worker2 # [ 0.363202] pci_bus 0000:00: root bus resource [bus 00-ff]944worker1 # [ 0.361788] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000945worker2 # [ 0.363208] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]946worker1 # [ 0.361796] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000947worker2 # [ 0.363213] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]948worker1 # [ 0.361817] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits949worker2 # [ 0.363218] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]950worker1 # [ 0.361840] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]951worker2 # [ 0.363273] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint952worker1 # [ 0.361913] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00953worker2 # [ 0.363708] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint954worker1 # [ 0.361923] pci_bus 0000:00: root bus resource [bus 00-ff]955worker2 # [ 0.363900] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]956worker1 # [ 0.361929] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]957worker2 # [ 0.363916] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]958worker1 # [ 0.361934] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]959worker2 # [ 0.363946] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]960worker2 # [ 0.363961] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]961worker1 # [ 0.361939] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]962worker1 # [ 0.361996] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint963worker2 # [ 0.364414] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint964lb # [ 0.373850] pl061_gpio 9030000.pl061: PL061 GPIO chip registered965client # [ 0.402754] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint966worker2 # [ 0.364597] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]967lb # [ 0.374545] ledtrig-cpu: registered to indicate activity on CPUs968worker1 # [ 0.362429] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint969client # [ 0.402953] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]970worker2 # [ 0.364613] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]971worker1 # [ 0.362616] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]972client # [ 0.402969] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]973lb # [ 0.374909] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:974worker2 # [ 0.364642] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]975worker1 # [ 0.362633] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]976client # [ 0.402999] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]977lb # [ 0.374926] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000978worker1 # [ 0.362662] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]979worker2 # [ 0.365093] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint980client # [ 0.403503] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint981worker1 # [ 0.362678] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]982lb # [ 0.374958] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000983client # [ 0.403687] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]984worker1 # [ 0.363144] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint985client # [ 0.403703] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]986lb # [ 0.374967] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000987worker1 # [ 0.363329] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]988client # [ 0.403736] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]989lb # [ 0.374989] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits990worker1 # [ 0.363344] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]991client # [ 0.404198] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint992worker1 # [ 0.363373] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]993lb # [ 0.375022] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]994client # [ 0.404384] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]995lb # [ 0.375101] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00996worker1 # [ 0.363823] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint997client # [ 0.404400] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]998lb # [ 0.375110] pci_bus 0000:00: root bus resource [bus 00-ff]999client # [ 0.404429] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1000lb # [ 0.375116] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1001client # [ 0.404886] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1002lb # [ 0.375121] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1003client # [ 0.405072] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1004lb # [ 0.375127] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1005client # [ 0.405087] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1006lb # [ 0.375245] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1007client # [ 0.405117] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1008lb # [ 0.375687] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1009client # [ 0.405572] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1010lb # [ 0.375872] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1011client # [ 0.405757] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1012lb # [ 0.375888] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1013client # [ 0.405773] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1014lb # [ 0.375917] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1015client # [ 0.405802] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1016lb # [ 0.375933] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1017client # [ 0.405818] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1018lb # [ 0.376391] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1019client # [ 0.406301] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1020lb # [ 0.376575] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1021client # [ 0.406492] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1022lb # [ 0.376590] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1023client # [ 0.406521] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1024lb # [ 0.376620] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1025client # [ 0.406990] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1026worker2 # [ 0.393350] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]1027lb # [ 0.377069] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1028worker2 # [ 0.393382] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1029client # [ 0.407179] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1030lb # [ 0.377251] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]1031worker2 # [ 0.393412] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1032client # [ 0.407215] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1033lb # [ 0.377267] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1034worker2 # [ 0.393922] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1035client # [ 0.407610] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1036lb # [ 0.377297] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1037worker2 # [ 0.394127] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1038client # [ 0.407795] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1039lb # [ 0.377776] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1040worker2 # [ 0.394142] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1041client # [ 0.408039] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1042lb # [ 0.377958] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1043worker2 # [ 0.394174] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1044worker1 # [ 0.392092] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]1045client # [ 0.408332] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1046lb # [ 0.377974] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1047client # [ 0.408350] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1048worker1 # [ 0.392122] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1049worker2 # [ 0.394639] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1050lb # [ 0.378006] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1051worker2 # [ 0.394827] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1052client # [ 0.408379] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1053worker1 # [ 0.392152] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1054lb # [ 0.378476] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1055worker2 # [ 0.394842] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1056worker1 # [ 0.392655] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1057lb # [ 0.378659] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1058client # [ 0.408852] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1059worker2 # [ 0.394872] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1060worker1 # [ 0.392843] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1061client # [ 0.409037] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1062lb # [ 0.378675] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1063client # [ 0.409052] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1064worker2 # [ 0.395335] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1065worker1 # [ 0.392859] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1066lb # [ 0.378705] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1067worker2 # [ 0.395525] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1068client # [ 0.409082] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1069worker1 # [ 0.392891] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1070lb # [ 0.379174] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1071worker2 # [ 0.395541] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1072client # [ 0.409658] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1073lb # [ 0.379355] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1074worker1 # [ 0.393351] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1075worker2 # [ 0.395570] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1076client # [ 0.409669] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1077lb # [ 0.379371] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1078worker1 # [ 0.393536] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1079worker1 # [ 0.393551] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1080worker2 # [ 0.396023] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1081lb # [ 0.379401] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1082client # [ 0.409675] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1083worker2 # [ 0.396206] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1084worker1 # [ 0.393580] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1085lb # [ 0.379847] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1086worker2 # [ 0.396221] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1087client # [ 0.409720] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1088worker1 # [ 0.394067] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1089lb # [ 0.380029] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1090worker2 # [ 0.396250] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1091client # [ 0.409767] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1092worker1 # [ 0.394251] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1093lb # [ 0.380044] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1094worker2 # [ 0.396266] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1095client # [ 0.409813] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1096worker1 # [ 0.394266] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1097lb # [ 0.380074] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1098worker2 # [ 0.396764] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1099worker1 # [ 0.394295] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1100lb # [ 0.380089] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1101client # [ 0.409859] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1102worker2 # [ 0.396951] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1103worker1 # [ 0.394746] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1104lb # [ 0.380572] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1105client # [ 0.409905] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1106worker2 # [ 0.396980] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1107worker1 # [ 0.394932] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1108lb # [ 0.380759] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1109client # [ 0.409953] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1110worker2 # [ 0.397445] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1111worker1 # [ 0.394947] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1112lb # [ 0.380789] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1113worker2 # [ 0.397632] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1114worker1 # [ 0.394976] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1115client # [ 0.409999] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1116lb # [ 0.381243] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1117worker2 # [ 0.397661] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1118worker1 # [ 0.394992] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1119client # [ 0.410046] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1120lb # [ 0.381429] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1121worker1 # [ 0.395453] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1122worker2 # [ 0.398059] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1123lb # [ 0.381458] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1124client # [ 0.410093] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1125worker2 # [ 0.398248] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1126worker1 # [ 0.395642] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1127client # [ 0.410164] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1128worker1 # [ 0.395672] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1129worker2 # [ 0.398491] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1130client # [ 0.410211] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1131worker2 # [ 0.398786] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1132worker1 # [ 0.396154] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1133client # [ 0.410232] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1134worker2 # [ 0.398803] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1135worker1 # [ 0.396342] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1136client # [ 0.410254] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1137worker2 # [ 0.398832] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1138worker1 # [ 0.396371] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1139client # [ 0.410275] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1140worker2 # [ 0.399294] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1141worker1 # [ 0.396757] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1142client # [ 0.410299] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1143worker2 # [ 0.399476] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1144worker1 # [ 0.396941] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1145worker2 # [ 0.399491] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1146client # [ 0.410320] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1147worker1 # [ 0.397190] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1148worker2 # [ 0.399521] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1149client # [ 0.410342] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1150worker1 # [ 0.397466] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1151worker2 # [ 0.400112] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1152client # [ 0.410364] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1153worker1 # [ 0.397483] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1154worker2 # [ 0.400122] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1155client # [ 0.410386] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1156worker1 # [ 0.397513] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1157client # [ 0.410408] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1158worker2 # [ 0.400127] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1159worker1 # [ 0.397983] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1160client # [ 0.410430] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1161worker2 # [ 0.400172] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1162worker1 # [ 0.398168] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1163client # [ 0.410453] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1164worker1 # [ 0.398183] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1165worker2 # [ 0.400219] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1166client # [ 0.410474] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1167worker1 # [ 0.398212] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1168worker2 # [ 0.400265] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1169client # [ 0.410496] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1170worker1 # [ 0.398791] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1171client # [ 0.410517] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1172worker2 # [ 0.400311] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1173worker1 # [ 0.398802] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1174client # [ 0.410538] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1175worker2 # [ 0.400356] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1176worker1 # [ 0.398807] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1177client # [ 0.410559] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1178lb # [ 0.429938] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1179worker2 # [ 0.400402] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1180client # [ 0.410581] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1181lb # [ 0.430140] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1182worker1 # [ 0.398852] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1183worker2 # [ 0.400447] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1184client # [ 0.410602] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1185worker1 # [ 0.398900] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1186lb # [ 0.430394] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1187client # [ 0.410624] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1188worker2 # [ 0.400493] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1189lb # [ 0.430672] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1190worker1 # [ 0.398946] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1191client # [ 0.410646] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1192lb # [ 0.430690] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1193worker2 # [ 0.400539] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1194client # [ 0.410672] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1195worker1 # [ 0.398993] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1196lb # [ 0.430719] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1197client # [ 0.410682] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1198worker2 # [ 0.400611] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1199worker1 # [ 0.399039] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1200lb # [ 0.431172] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1201client # [ 0.410686] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1202worker2 # [ 0.400657] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1203worker1 # [ 0.399090] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1204lb # [ 0.431352] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1205worker2 # [ 0.400679] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1206lb # [ 0.431368] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1207worker1 # [ 0.399137] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1208worker2 # [ 0.400700] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1209lb # [ 0.431398] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1210worker2 # [ 0.400722] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1211worker1 # [ 0.399183] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1212lb # [ 0.431971] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1213worker2 # [ 0.400745] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1214worker1 # [ 0.399231] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1215lb # [ 0.431982] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1216worker2 # [ 0.400767] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1217worker1 # [ 0.399301] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1218lb # [ 0.431988] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1219worker2 # [ 0.400788] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1220worker1 # [ 0.399348] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1221worker2 # [ 0.400810] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1222lb # [ 0.432032] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1223worker1 # [ 0.399370] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1224worker2 # [ 0.400831] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1225lb # [ 0.432079] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1226worker1 # [ 0.399392] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1227worker2 # [ 0.400853] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1228lb # [ 0.432124] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1229client # [ 0.475932] pci 0000:00:0a.0: enabling device (0000 -> 0002)1230worker1 # [ 0.399413] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1231worker2 # [ 0.400875] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1232worker1 # [ 0.399437] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1233lb # [ 0.432171] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1234worker2 # [ 0.400898] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1235worker1 # [ 0.399459] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1236lb # [ 0.432218] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1237worker2 # [ 0.400919] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1238worker1 # [ 0.399482] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1239worker2 # [ 0.400940] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1240lb # [ 0.432265] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1241worker2 # [ 0.400961] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1242worker1 # [ 0.399504] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1243lb # [ 0.432312] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1244worker2 # [ 0.400982] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1245worker1 # [ 0.399526] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1246lb # [ 0.432360] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1247worker2 # [ 0.401003] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1248worker1 # [ 0.399548] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1249worker2 # [ 0.401024] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1250lb # [ 0.432406] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1251worker1 # [ 0.399570] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1252worker2 # [ 0.401045] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1253lb # [ 0.432465] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1254worker1 # [ 0.399593] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1255worker2 # [ 0.401067] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1256worker1 # [ 0.399615] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1257lb # [ 0.432510] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1258worker2 # [ 0.401088] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1259worker1 # [ 0.399636] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1260lb # [ 0.432531] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1261worker2 # [ 0.401115] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1262worker1 # [ 0.399658] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1263lb # [ 0.432552] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1264worker2 # [ 0.401125] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1265worker1 # [ 0.399679] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1266worker2 # [ 0.401130] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1267lb # [ 0.432573] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1268worker1 # [ 0.399701] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1269lb # [ 0.432597] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1270worker1 # [ 0.399722] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1271lb # [ 0.432619] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1272worker1 # [ 0.399743] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1273lb # [ 0.432641] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1274worker1 # [ 0.399766] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1275lb # [ 0.432662] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1276worker1 # [ 0.399787] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1277lb # [ 0.432684] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1278worker1 # [ 0.399814] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1279worker1 # [ 0.399823] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1280lb # [ 0.432706] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1281worker1 # [ 0.399828] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1282lb # [ 0.432728] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1283lb # [ 0.432750] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1284lb # [ 0.432770] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1285lb # [ 0.432791] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1286lb # [ 0.432812] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1287lb # [ 0.432833] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1288lb # [ 0.432854] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1289worker2 # [ 0.466353] pci 0000:00:0a.0: enabling device (0000 -> 0002)1290lb # [ 0.432875] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1291lb # [ 0.432896] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1292lb # [ 0.432918] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1293lb # [ 0.432939] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1294lb # [ 0.432966] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1295lb # [ 0.432975] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1296worker1 # [ 0.465053] pci 0000:00:0a.0: enabling device (0000 -> 0002)1297lb # [ 0.432981] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1298client # [ 0.501800] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1299lb # [ 0.434160] pci 0000:00:0a.0: enabling device (0000 -> 0002)1300client # [ 0.508015] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1301client # [ 0.510114] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1302worker2 # [ 0.486914] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1303worker2 # [ 0.489056] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1304client # [ 0.520209] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1305lb # [ 0.493115] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1306worker1 # [ 0.485778] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1307client # [ 0.522152] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1308worker2 # [ 0.498946] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1309worker2 # [ 0.500921] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1310client # [ 0.533684] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1311lb # [ 0.503141] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1312worker1 # [ 0.495980] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1313worker1 # [ 0.499413] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1314client # [ 0.536734] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1315worker1 # [ 0.503124] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1316worker2 # [ 0.512127] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1317lb # [ 0.507326] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1318lb # [ 0.509376] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1319lb # [ 0.511471] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1320lb # [ 0.513654] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1321worker1 # [ 0.509224] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1322worker2 # [ 0.518494] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1323worker2 # [ 0.520771] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1324client # [ 0.546874] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1325client # [ 0.549226] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1326client # [ 0.551103] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1327client # [ 0.553512] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1328worker1 # [ 0.520477] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1329worker1 # [ 0.522710] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1330worker2 # [ 0.530652] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1331worker2 # [ 0.532942] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1332lb # [ 0.535283] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1333lb # [ 0.537673] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1334worker1 # [ 0.533832] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1335lb # [ 0.541642] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1336client # [ 0.572028] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1337worker2 # [ 0.544238] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1338worker1 # [ 0.536876] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1339worker1 # [ 0.538772] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1340lb # [ 0.544366] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1341client # [ 0.574765] msm_serial: driver initialized1342worker2 # [ 0.547417] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1343client # [ 0.574918] SuperH (H)SCI(F) driver initialized1344client # [ 0.574973] STM32 USART driver initialized1345lb # [ 0.555116] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1346worker1 # [ 0.549331] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1347worker2 # [ 0.560353] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1348worker2 # [ 0.562945] msm_serial: driver initialized1349worker1 # [ 0.558437] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1350lb # [ 0.564384] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1351worker2 # [ 0.563086] SuperH (H)SCI(F) driver initialized1352worker2 # [ 0.563138] STM32 USART driver initialized1353lb # [ 0.571050] msm_serial: driver initialized1354worker1 # [ 0.565148] msm_serial: driver initialized1355lb # [ 0.571203] SuperH (H)SCI(F) driver initialized1356worker1 # [ 0.565305] SuperH (H)SCI(F) driver initialized1357lb # [ 0.571261] STM32 USART driver initialized1358worker1 # [ 0.565356] STM32 USART driver initialized1359client # [ 0.609594] loop: module loaded1360client # [ 0.609786] virtio_blk virtio5: 1/0/0 default/read/poll queues1361client # [ 0.610535] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1362client # [ 0.619448] megasas: 07.734.00.00-rc11363client # [ 0.620191] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1364worker2 # [ 0.597799] loop: module loaded1365client # [ 0.622087] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001366worker2 # [ 0.598030] virtio_blk virtio5: 1/0/0 default/read/poll queues1367client # [ 0.622113] Intel/Sharp Extended Query Table at 0x00311368client # [ 0.623765] Using buffer write method1369worker2 # [ 0.598779] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1370worker1 # [ 0.594857] loop: module loaded1371client # [ 0.623825] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1372worker1 # [ 0.595051] virtio_blk virtio5: 1/0/0 default/read/poll queues1373worker1 # [ 0.595815] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1374client # [ 0.625481] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001375client # [ 0.625510] Intel/Sharp Extended Query Table at 0x00311376lb # [ 0.609953] loop: module loaded1377lb # [ 0.610179] virtio_blk virtio5: 1/0/0 default/read/poll queues1378worker2 # [ 0.605824] megasas: 07.734.00.00-rc11379lb # [ 0.611006] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1380worker2 # [ 0.606618] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1381client # [ 0.643801] Using buffer write method1382client # [ 0.643840] Concatenating MTD devices:1383client # [ 0.643844] (0): "0.flash"1384worker2 # [ 0.608789] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001385client # [ 0.643848] (1): "0.flash"1386client # [ 0.643852] into device "0.flash"1387worker2 # [ 0.608813] Intel/Sharp Extended Query Table at 0x00311388worker2 # [ 0.610423] Using buffer write method1389worker2 # [ 0.610480] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1390worker2 # [ 0.612134] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001391worker2 # [ 0.612186] Intel/Sharp Extended Query Table at 0x00311392lb # [ 0.615198] megasas: 07.734.00.00-rc11393worker1 # [ 0.608594] megasas: 07.734.00.00-rc11394lb # [ 0.615978] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1395worker1 # [ 0.609396] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1396lb # [ 0.619945] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001397worker1 # [ 0.611376] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001398lb # [ 0.619971] Intel/Sharp Extended Query Table at 0x00311399lb # [ 0.621677] Using buffer write method1400worker1 # [ 0.611400] Intel/Sharp Extended Query Table at 0x00311401worker1 # [ 0.613107] Using buffer write method1402worker2 # [ 0.625760] Using buffer write method1403worker2 # [ 0.625794] Concatenating MTD devices:1404worker2 # [ 0.625798] (0): "0.flash"1405worker2 # [ 0.625802] (1): "0.flash"1406worker1 # [ 0.613171] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1407worker2 # [ 0.625805] into device "0.flash"1408worker1 # [ 0.615032] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001409worker1 # [ 0.615093] Intel/Sharp Extended Query Table at 0x00311410lb # [ 0.629791] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1411worker1 # [ 0.628878] Using buffer write method1412lb # [ 0.631778] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001413worker1 # [ 0.628912] Concatenating MTD devices:1414worker1 # [ 0.628916] (0): "0.flash"1415lb # [ 0.631812] Intel/Sharp Extended Query Table at 0x00311416worker1 # [ 0.628920] (1): "0.flash"1417lb # [ 0.633673] Using buffer write method1418worker1 # [ 0.628923] into device "0.flash"1419lb # [ 0.641748] Concatenating MTD devices:1420lb # [ 0.641755] (0): "0.flash"1421lb # [ 0.641760] (1): "0.flash"1422lb # [ 0.641764] into device "0.flash"1423client # [ 0.886742] Freeing initrd memory: 26160K1424client # [ 0.892967] tun: Universal TUN/TAP device driver, 1.61425client # [ 0.896775] thunder_xcv, ver 1.01426client # [ 0.896821] thunder_bgx, ver 1.01427client # [ 0.896855] nicpf, ver 1.01428client # [ 0.897418] e1000: Intel(R) PRO/1000 Network Driver1429client # [ 0.897425] e1000: Copyright (c) 1999-2006 Intel Corporation.1430client # [ 0.897451] e1000e: Intel(R) PRO/1000 Network Driver1431client # [ 0.897460] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1432client # [ 0.897492] igb: Intel(R) Gigabit Ethernet Network Driver1433client # [ 0.897498] igb: Copyright (c) 2007-2014 Intel Corporation.1434client # [ 0.897522] igbvf: Intel(R) Gigabit Virtual Function Network Driver1435client # [ 0.897527] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1436client # [ 0.897664] sky2: driver version 1.301437client # [ 0.906495] usbcore: registered new interface driver usb-storage1438client # [ 0.906573] usbcore: registered new interface driver usbserial_generic1439client # [ 0.906589] usbserial: USB Serial support registered for generic1440worker2 # [ 0.880288] Freeing initrd memory: 26148K1441client # [ 0.907490] ehci-pci 0000:00:0a.0: EHCI Host Controller1442client # [ 0.907517] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11443client # [ 0.907795] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001444client # [ 0.912900] hv_vmbus: registering driver hyperv_keyboard1445client # [ 0.914491] rtc-pl031 9010000.pl031: registered as rtc01446worker2 # [ 0.886233] tun: Universal TUN/TAP device driver, 1.61447client # [ 0.914519] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T10:49:10 UTC (1790074150)1448client # [ 0.916571] i2c_dev: i2c /dev entries driver1449worker1 # [ 0.882004] Freeing initrd memory: 26148K1450client # [ 0.919887] sdhci: Secure Digital Host Controller Interface driver1451worker2 # [ 0.889956] thunder_xcv, ver 1.01452client # [ 0.919898] sdhci: Copyright(c) Pierre Ossman1453worker2 # [ 0.889998] thunder_bgx, ver 1.01454worker2 # [ 0.890021] nicpf, ver 1.01455client # [ 0.920171] Synopsys Designware Multimedia Card Interface Driver1456worker2 # [ 0.890551] e1000: Intel(R) PRO/1000 Network Driver1457client # [ 0.920566] sdhci-pltfm: SDHCI platform and OF driver helper1458worker1 # [ 0.889697] tun: Universal TUN/TAP device driver, 1.61459worker2 # [ 0.890558] e1000: Copyright (c) 1999-2006 Intel Corporation.1460client # [ 0.922312] hid: raw HID events driver (C) Jiri Kosina1461worker2 # [ 0.890581] e1000e: Intel(R) PRO/1000 Network Driver1462client # [ 0.922590] usbcore: registered new interface driver usbhid1463client # [ 0.922597] usbhid: USB HID core driver1464worker2 # [ 0.890590] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1465worker2 # [ 0.890617] igb: Intel(R) Gigabit Ethernet Network Driver1466worker2 # [ 0.890622] igb: Copyright (c) 2007-2014 Intel Corporation.1467client # [ 0.927323] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001468client # [ 0.927636] hub 1-0:1.0: USB hub found1469worker2 # [ 0.890644] igbvf: Intel(R) Gigabit Virtual Function Network Driver1470client # [ 0.927659] hub 1-0:1.0: 6 ports detected1471worker2 # [ 0.890649] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1472worker2 # [ 0.890778] sky2: driver version 1.301473lb # [ 0.901280] Freeing initrd memory: 26140K1474worker2 # [ 0.892282] usbcore: registered new interface driver usb-storage1475worker1 # [ 0.894183] thunder_xcv, ver 1.01476worker2 # [ 0.892347] usbcore: registered new interface driver usbserial_generic1477worker1 # [ 0.894251] thunder_bgx, ver 1.01478worker1 # [ 0.894285] nicpf, ver 1.01479worker2 # [ 0.892362] usbserial: USB Serial support registered for generic1480worker1 # [ 0.895069] e1000: Intel(R) PRO/1000 Network Driver1481worker2 # [ 0.892929] hv_vmbus: registering driver hyperv_keyboard1482client # [ 0.930732] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1483worker1 # [ 0.895078] e1000: Copyright (c) 1999-2006 Intel Corporation.1484worker2 # [ 0.893911] ehci-pci 0000:00:0a.0: EHCI Host Controller1485client # [ 0.932318] drop_monitor: Initializing network drop monitor service1486worker1 # [ 0.895114] e1000e: Intel(R) PRO/1000 Network Driver1487worker2 # [ 0.893989] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11488client # [ 0.932446] NET: Registered PF_INET6 protocol family1489worker1 # [ 0.895123] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1490worker2 # [ 0.894243] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001491worker1 # [ 0.895167] igb: Intel(R) Gigabit Ethernet Network Driver1492client # [ 0.935382] Segment Routing with IPv61493client # [ 0.935405] In-situ OAM (IOAM) with IPv61494worker1 # [ 0.895174] igb: Copyright (c) 2007-2014 Intel Corporation.1495worker2 # [ 0.906182] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001496client # [ 0.935434] NET: Registered PF_PACKET protocol family1497lb # [ 0.907297] tun: Universal TUN/TAP device driver, 1.61498worker2 # [ 0.906546] hub 1-0:1.0: USB hub found1499worker1 # [ 0.895210] igbvf: Intel(R) Gigabit Virtual Function Network Driver1500worker2 # [ 0.906565] hub 1-0:1.0: 6 ports detected1501worker1 # [ 0.895217] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1502worker1 # [ 0.895411] sky2: driver version 1.301503worker2 # [ 0.909140] rtc-pl031 9010000.pl031: registered as rtc01504client # [ 0.937086] 9pnet: Installing 9P2000 support1505worker2 # [ 0.909844] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T10:49:10 UTC (1790074150)1506worker2 # [ 0.910163] i2c_dev: i2c /dev entries driver1507worker1 # [ 0.904748] usbcore: registered new interface driver usb-storage1508worker1 # [ 0.904845] usbcore: registered new interface driver usbserial_generic1509client # [ 0.939884] Key type dns_resolver registered1510worker1 # [ 0.904859] usbserial: USB Serial support registered for generic1511lb # [ 0.911118] thunder_xcv, ver 1.01512worker1 # [ 0.905555] hv_vmbus: registering driver hyperv_keyboard1513lb # [ 0.911163] thunder_bgx, ver 1.01514lb # [ 0.911186] nicpf, ver 1.01515lb # [ 0.911767] e1000: Intel(R) PRO/1000 Network Driver1516lb # [ 0.911774] e1000: Copyright (c) 1999-2006 Intel Corporation.1517worker1 # [ 0.909955] ehci-pci 0000:00:0a.0: EHCI Host Controller1518lb # [ 0.911798] e1000e: Intel(R) PRO/1000 Network Driver1519worker2 # [ 0.914398] sdhci: Secure Digital Host Controller Interface driver1520client # [ 0.946357] registered taskstats version 11521worker1 # [ 0.909993] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11522lb # [ 0.911806] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1523worker2 # [ 0.914409] sdhci: Copyright(c) Pierre Ossman1524client # [ 0.946519] Loading compiled-in X.509 certificates1525worker1 # [ 0.910222] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001526lb # [ 0.911835] igb: Intel(R) Gigabit Ethernet Network Driver1527worker2 # [ 0.914661] Synopsys Designware Multimedia Card Interface Driver1528lb # [ 0.911840] igb: Copyright (c) 2007-2014 Intel Corporation.1529worker2 # [ 0.915019] sdhci-pltfm: SDHCI platform and OF driver helper1530lb # [ 0.911863] igbvf: Intel(R) Gigabit Virtual Function Network Driver1531worker2 # [ 0.916705] hid: raw HID events driver (C) Jiri Kosina1532worker1 # [ 0.912887] rtc-pl031 9010000.pl031: registered as rtc01533lb # [ 0.911869] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1534worker2 # [ 0.916943] usbcore: registered new interface driver usbhid1535lb # [ 0.912002] sky2: driver version 1.301536worker2 # [ 0.916950] usbhid: USB HID core driver1537worker1 # [ 0.912919] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T10:49:10 UTC (1790074150)1538lb # [ 0.913600] usbcore: registered new interface driver usb-storage1539worker1 # [ 0.913284] i2c_dev: i2c /dev entries driver1540lb # [ 0.913648] usbcore: registered new interface driver usbserial_generic1541lb # [ 0.913662] usbserial: USB Serial support registered for generic1542lb # [ 0.914473] ehci-pci 0000:00:0a.0: EHCI Host Controller1543worker2 # [ 0.922388] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1544lb # [ 0.914501] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11545lb # [ 0.914752] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001546worker2 # [ 0.923862] drop_monitor: Initializing network drop monitor service1547worker2 # [ 0.923998] NET: Registered PF_INET6 protocol family1548client # [ 0.955265] Demotion targets for Node 0: null1549lb # [ 0.926251] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001550worker1 # [ 0.918402] sdhci: Secure Digital Host Controller Interface driver1551client # [ 0.955378] Key type .fscrypt registered1552worker2 # [ 0.926997] Segment Routing with IPv61553lb # [ 0.926620] hub 1-0:1.0: USB hub found1554worker1 # [ 0.918413] sdhci: Copyright(c) Pierre Ossman1555worker2 # [ 0.927019] In-situ OAM (IOAM) with IPv61556client # [ 0.955384] Key type fscrypt-provisioning registered1557lb # [ 0.926636] hub 1-0:1.0: 6 ports detected1558worker2 # [ 0.927047] NET: Registered PF_PACKET protocol family1559worker1 # [ 0.918694] Synopsys Designware Multimedia Card Interface Driver1560client # [ 0.955484] ima: No TPM chip found, activating TPM-bypass!1561client # [ 0.955503] ima: Allocated hash algorithm: sha11562worker1 # [ 0.919122] sdhci-pltfm: SDHCI platform and OF driver helper1563client # [ 0.955523] ima: No architecture policies found1564worker2 # [ 0.928579] 9pnet: Installing 9P2000 support1565worker1 # [ 0.919949] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001566lb # [ 0.929200] hv_vmbus: registering driver hyperv_keyboard1567worker1 # [ 0.920269] hub 1-0:1.0: USB hub found1568worker1 # [ 0.920291] hub 1-0:1.0: 6 ports detected1569client # [ 0.959757] input: gpio-keys as /devices/platform/gpio-keys/input/input01570lb # [ 0.930722] rtc-pl031 9010000.pl031: registered as rtc01571lb # [ 0.930749] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T10:49:10 UTC (1790074150)1572worker2 # [ 0.931312] Key type dns_resolver registered1573lb # [ 0.931054] i2c_dev: i2c /dev entries driver1574worker1 # [ 0.926087] hid: raw HID events driver (C) Jiri Kosina1575worker1 # [ 0.926347] usbcore: registered new interface driver usbhid1576worker1 # [ 0.926355] usbhid: USB HID core driver1577lb # [ 0.935940] sdhci: Secure Digital Host Controller Interface driver1578worker1 # [ 0.929322] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1579lb # [ 0.935950] sdhci: Copyright(c) Pierre Ossman1580worker1 # [ 0.930920] drop_monitor: Initializing network drop monitor service1581lb # [ 0.936221] Synopsys Designware Multimedia Card Interface Driver1582worker1 # [ 0.931113] NET: Registered PF_INET6 protocol family1583lb # [ 0.936599] sdhci-pltfm: SDHCI platform and OF driver helper1584worker1 # [ 0.934166] Segment Routing with IPv61585worker1 # [ 0.934185] In-situ OAM (IOAM) with IPv61586worker1 # [ 0.934215] NET: Registered PF_PACKET protocol family1587worker2 # [ 0.937528] registered taskstats version 11588lb # [ 0.940968] hid: raw HID events driver (C) Jiri Kosina1589worker2 # [ 0.937670] Loading compiled-in X.509 certificates1590lb # [ 0.941213] usbcore: registered new interface driver usbhid1591lb # [ 0.941220] usbhid: USB HID core driver1592lb # [ 0.944122] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1593worker2 # [ 0.946235] Demotion targets for Node 0: null1594lb # [ 0.945632] drop_monitor: Initializing network drop monitor service1595worker2 # [ 0.946340] Key type .fscrypt registered1596lb # [ 0.947644] NET: Registered PF_INET6 protocol family1597worker2 # [ 0.946347] Key type fscrypt-provisioning registered1598worker2 # [ 0.946445] ima: No TPM chip found, activating TPM-bypass!1599lb # [ 0.948844] Segment Routing with IPv61600worker2 # [ 0.946464] ima: Allocated hash algorithm: sha11601lb # [ 0.948863] In-situ OAM (IOAM) with IPv61602client # [ 0.978080] clk: Disabling unused clocks1603worker2 # [ 0.946485] ima: No architecture policies found1604worker1 # [ 0.935933] 9pnet: Installing 9P2000 support1605lb # [ 0.948892] NET: Registered PF_PACKET protocol family1606client # [ 0.978105] PM: genpd: Disabling unused power domains1607worker1 # [ 0.938211] Key type dns_resolver registered1608worker2 # [ 0.950483] input: gpio-keys as /devices/platform/gpio-keys/input/input01609client # [ 0.982400] Freeing unused kernel memory: 4736K1610lb # [ 0.950700] 9pnet: Installing 9P2000 support1611client # [ 0.982598] Run /init as init process1612lb # [ 0.952875] Key type dns_resolver registered1613worker1 # [ 0.944973] registered taskstats version 11614worker1 # [ 0.945138] Loading compiled-in X.509 certificates1615worker1 # [ 0.953894] Demotion targets for Node 0: null1616worker1 # [ 0.954014] Key type .fscrypt registered1617worker1 # [ 0.954021] Key type fscrypt-provisioning registered1618lb # [ 0.959702] registered taskstats version 11619worker1 # [ 0.954126] ima: No TPM chip found, activating TPM-bypass!1620lb # [ 0.959878] Loading compiled-in X.509 certificates1621worker1 # [ 0.954146] ima: Allocated hash algorithm: sha11622worker1 # [ 0.954173] ima: No architecture policies found1623worker1 # [ 0.958245] input: gpio-keys as /devices/platform/gpio-keys/input/input01624client # [ 0.997014] systemd[1]: Successfully made /usr/ read-only.1625lb # [ 0.968322] Demotion targets for Node 0: null1626lb # [ 0.968433] Key type .fscrypt registered1627worker2 # [ 0.968364] clk: Disabling unused clocks1628lb # [ 0.968440] Key type fscrypt-provisioning registered1629worker2 # [ 0.968398] PM: genpd: Disabling unused power domains1630lb # [ 0.968534] ima: No TPM chip found, activating TPM-bypass!1631lb # [ 0.968554] ima: Allocated hash algorithm: sha11632lb # [ 0.968576] ima: No architecture policies found1633lb # [ 0.972639] input: gpio-keys as /devices/platform/gpio-keys/input/input01634worker2 # [ 0.972685] Freeing unused kernel memory: 4736K1635worker2 # [ 0.972884] Run /init as init process1636worker1 # [ 0.975733] clk: Disabling unused clocks1637worker1 # [ 0.975760] PM: genpd: Disabling unused power domains1638worker1 # [ 0.979960] Freeing unused kernel memory: 4736K1639worker1 # [ 0.980162] Run /init as init process1640worker2 # [ 0.987343] systemd[1]: Successfully made /usr/ read-only.1641lb # [ 0.991302] clk: Disabling unused clocks1642lb # [ 0.991326] PM: genpd: Disabling unused power domains1643lb # [ 0.995568] Freeing unused kernel memory: 4736K1644lb # [ 0.995780] Run /init as init process1645worker1 # [ 0.994191] systemd[1]: Successfully made /usr/ read-only.1646lb # [ 1.011961] systemd[1]: Successfully made /usr/ read-only.1647client # [ 1.178786] usb 1-1: new high-speed USB device number 2 using ehci-pci1648worker2 # [ 1.153243] usb 1-1: new high-speed USB device number 2 using ehci-pci1649lb # [ 1.173777] usb 1-1: new high-speed USB device number 2 using ehci-pci1650worker1 # [ 1.168015] usb 1-1: new high-speed USB device number 2 using ehci-pci1651client # [ 1.330443] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11652worker2 # [ 1.302820] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11653client # [ 1.334605] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1654client # [ 1.342843] systemd[1]: Detected virtualization qemu.1655client # [ 1.344154] systemd[1]: Detected architecture arm64.1656client # [ 1.345514] systemd[1]: Running in initrd.1657client # [ 1.346158] systemd[1]: Initializing machine ID from random generator.1658client # [ 1.346342] systemd[1]: Hostname set to <client>.1659worker2 # [ 1.321941] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1660worker1 # [ 1.317851] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11661worker2 # [ 1.326331] systemd[1]: Detected virtualization qemu.1662worker2 # [ 1.327045] systemd[1]: Detected architecture arm64.1663worker2 # [ 1.327726] systemd[1]: Running in initrd.1664lb # [ 1.327506] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11665worker2 # [ 1.328782] systemd[1]: Initializing machine ID from random generator.1666worker2 # [ 1.329966] systemd[1]: Hostname set to <worker2>.1667worker1 # [ 1.329053] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1668worker1 # [ 1.341546] systemd[1]: Detected virtualization qemu.1669worker1 # [ 1.343776] systemd[1]: Detected architecture arm64.1670worker1 # [ 1.345930] systemd[1]: Running in initrd.1671worker1 # [ 1.348686] systemd[1]: Initializing machine ID from random generator.1672lb # [ 1.346943] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1673worker1 # [ 1.351840] systemd[1]: Hostname set to <worker1>.1674lb # [ 1.358714] systemd[1]: Detected virtualization qemu.1675lb # [ 1.360524] systemd[1]: Detected architecture arm64.1676lb # [ 1.362385] systemd[1]: Running in initrd.1677lb # [ 1.364837] systemd[1]: Initializing machine ID from random generator.1678lb # [ 1.367528] systemd[1]: Hostname set to <lb>.1679client # [ 1.410909] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01680worker2 # [ 1.381440] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01681lb # [ 1.410011] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01682worker1 # [ 1.408180] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01683worker2 # [ 1.501208] usb 1-2: new high-speed USB device number 3 using ehci-pci1684client # [ 1.530831] usb 1-2: new high-speed USB device number 3 using ehci-pci1685lb # [ 1.533757] usb 1-2: new high-speed USB device number 3 using ehci-pci1686worker1 # [ 1.528010] usb 1-2: new high-speed USB device number 3 using ehci-pci1687client # [ 1.621715] systemd[1]: bpf-restrict-fs: LSM BPF program attached1688worker2 # [ 1.612802] systemd[1]: bpf-restrict-fs: LSM BPF program attached1689worker1 # [ 1.647744] systemd[1]: bpf-restrict-fs: LSM BPF program attached1690worker2 # [ 1.659114] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21691client # [ 1.689567] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21692client # [ 1.689761] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01693worker2 # [ 1.663334] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01694client # [ 1.711387] systemd[1]: Queued start job for default target Initrd Default Target.1695worker1 # [ 1.682141] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21696client # [ 1.721317] systemd[1]: Created slice Slice /system/modprobe.1697worker1 # [ 1.684872] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01698client # [ 1.722643] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1699client # [ 1.724359] systemd[1]: Expecting device /dev/disk/by-label/nixos...1700client # [ 1.724403] systemd[1]: Reached target Path Units.1701client # [ 1.724426] systemd[1]: Reached target Slice Units.1702client # [ 1.724449] systemd[1]: Reached target Swaps.1703client # [ 1.724472] systemd[1]: Reached target Timer Units.1704lb # [ 1.697395] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21705client # [ 1.724688] systemd[1]: Listening on D-Bus System Message Bus Socket.1706client # [ 1.724862] systemd[1]: Listening on Journal Socket (/dev/log).1707client # [ 1.725027] systemd[1]: Listening on Journal Sockets.1708client # [ 1.725177] systemd[1]: Listening on udev Control Socket.1709client # [ 1.725318] systemd[1]: Listening on udev Kernel Socket.1710lb # [ 1.703259] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01711client # [ 1.725343] systemd[1]: Reached target Socket Units.1712client # [ 1.737713] systemd[1]: Starting Create List of Static Device Nodes...1713lb # [ 1.708085] systemd[1]: bpf-restrict-fs: LSM BPF program attached1714client # [ 1.746920] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1715worker2 # [ 1.718164] systemd[1]: Queued start job for default target Initrd Default Target.1716client # [ 1.748127] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1717client # [ 1.753716] systemd[1]: Mounting Kernel Configuration File System...1718worker2 # [ 1.727691] systemd[1]: Created slice Slice /system/modprobe.1719worker2 # [ 1.728970] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1720worker2 # [ 1.730475] systemd[1]: Expecting device /dev/disk/by-label/nixos...1721worker2 # [ 1.731581] systemd[1]: Reached target Path Units.1722worker2 # [ 1.732428] systemd[1]: Reached target Slice Units.1723worker2 # [ 1.733332] systemd[1]: Reached target Swaps.1724worker2 # [ 1.734174] systemd[1]: Reached target Timer Units.1725worker2 # [ 1.735233] systemd[1]: Listening on D-Bus System Message Bus Socket.1726worker1 # [ 1.729807] systemd[1]: Queued start job for default target Initrd Default Target.1727worker2 # [ 1.736532] systemd[1]: Listening on Journal Socket (/dev/log).1728worker2 # [ 1.737763] systemd[1]: Listening on Journal Sockets.1729worker1 # [ 1.734086] systemd[1]: Created slice Slice /system/modprobe.1730worker2 # [ 1.737911] systemd[1]: Listening on udev Control Socket.1731worker2 # [ 1.738054] systemd[1]: Listening on udev Kernel Socket.1732client # [ 1.769643] systemd[1]: Starting Journal Service...1733worker1 # [ 1.735186] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1734worker2 # [ 1.738078] systemd[1]: Reached target Socket Units.1735worker1 # [ 1.736455] systemd[1]: Expecting device /dev/disk/by-label/nixos...1736worker1 # [ 1.737422] systemd[1]: Reached target Path Units.1737worker2 # [ 1.743369] systemd[1]: Starting Create List of Static Device Nodes...1738worker1 # [ 1.738250] systemd[1]: Reached target Slice Units.1739worker1 # [ 1.738991] systemd[1]: Reached target Swaps.1740worker1 # [ 1.739656] systemd[1]: Reached target Timer Units.1741worker1 # [ 1.740680] systemd[1]: Listening on D-Bus System Message Bus Socket.1742worker1 # [ 1.741796] systemd[1]: Listening on Journal Socket (/dev/log).1743worker1 # [ 1.742831] systemd[1]: Listening on Journal Sockets.1744worker1 # [ 1.743735] systemd[1]: Listening on udev Control Socket.1745worker1 # [ 1.744744] systemd[1]: Listening on udev Kernel Socket.1746worker2 # [ 1.750573] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1747worker1 # [ 1.745536] systemd[1]: Reached target Socket Units.1748worker2 # [ 1.751785] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1749worker1 # [ 1.747846] systemd[1]: Starting Create List of Static Device Nodes...1750worker2 # [ 1.761368] systemd[1]: Mounting Kernel Configuration File System...1751client # [ 1.790917] systemd[1]: Starting Load Kernel Modules...1752worker1 # [ 1.756135] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1753client # [ 1.791055] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1754worker1 # [ 1.756227] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1755worker2 # [ 1.769350] systemd[1]: Starting Journal Service...1756worker1 # [ 1.764235] systemd[1]: Mounting Kernel Configuration File System...1757client # [ 1.806906] systemd[1]: Starting Coldplug All udev Devices...1758worker1 # [ 1.784232] systemd[1]: Starting Journal Service...1759lb # [ 1.798055] systemd[1]: Queued start job for default target Initrd Default Target.1760worker1 # [ 1.795663] systemd[1]: Starting Load Kernel Modules...1761client # [ 1.831713] systemd-journald[73]: Collecting audit messages is disabled.1762worker1 # [ 1.796587] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1763lb # [ 1.808110] systemd[1]: Created slice Slice /system/modprobe.1764lb # [ 1.809232] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1765client # [ 1.838910] systemd[1]: Finished Create List of Static Device Nodes.1766worker2 # [ 1.809476] systemd[1]: Starting Load Kernel Modules...1767lb # [ 1.810552] systemd[1]: Expecting device /dev/disk/by-label/nixos...1768lb # [ 1.811551] systemd[1]: Reached target Path Units.1769worker2 # [ 1.809605] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1770client # [ 1.840769] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1771lb # [ 1.812304] systemd[1]: Reached target Slice Units.1772lb # [ 1.813073] systemd[1]: Reached target Swaps.1773lb # [ 1.813789] systemd[1]: Reached target Timer Units.1774lb # [ 1.814763] systemd[1]: Listening on D-Bus System Message Bus Socket.1775lb # [ 1.815912] systemd[1]: Listening on Journal Socket (/dev/log).1776lb # [ 1.816968] systemd[1]: Listening on Journal Sockets.1777lb # [ 1.818030] systemd[1]: Listening on udev Control Socket.1778lb # [ 1.818994] systemd[1]: Listening on udev Kernel Socket.1779worker2 # [ 1.817628] systemd[1]: Starting Coldplug All udev Devices...1780lb # [ 1.819828] systemd[1]: Reached target Socket Units.1781worker1 # [ 1.812332] systemd[1]: Starting Coldplug All udev Devices...1782lb # [ 1.822293] systemd[1]: Starting Create List of Static Device Nodes...1783client # [ 1.851156] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1784client # [ 1.855098] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1785lb # [ 1.829891] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1786worker2 # [ 1.830482] systemd-journald[73]: Collecting audit messages is disabled.1787worker2 # [ 1.831562] systemd[1]: Finished Create List of Static Device Nodes.1788lb # [ 1.830995] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1789worker1 # [ 1.824171] systemd[1]: Finished Create List of Static Device Nodes.1790worker1 # [ 1.825144] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1791worker2 # [ 1.834137] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1792worker1 # [ 1.825433] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1793worker1 # [ 1.825659] systemd[1]: Mounted Kernel Configuration File System.1794lb # [ 1.837373] systemd[1]: Mounting Kernel Configuration File System...1795worker1 # [ 1.838328] systemd-journald[73]: Collecting audit messages is disabled.1796worker2 # [ 1.845665] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1797worker2 # [ 1.846081] systemd[1]: Mounted Kernel Configuration File System.1798client # [ 1.878836] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1799client # [ 1.881055] systemd[1]: Mounted Kernel Configuration File System.1800worker2 # [ 1.852086] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1801lb # [ 1.855644] systemd[1]: Starting Journal Service...1802worker1 # [ 1.852025] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1803worker1 # [ 1.854102] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1804worker2 # [ 1.857793] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1805worker2 # [ 1.865286] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1806client # [ 1.895162] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1807worker1 # [ 1.864713] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1808client # [ 1.902622] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01809worker2 # [ 1.873340] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01810worker2 # [ 1.873587] [drm] features: -virgl +edid -resource_blob -host_visible1811worker2 # [ 1.873599] [drm] features: -context_init1812worker2 # [ 1.874330] [drm] number of scanouts: 11813worker2 # [ 1.874348] [drm] number of cap sets: 01814lb # [ 1.878273] systemd[1]: Starting Load Kernel Modules...1815lb # [ 1.882060] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1816worker1 # [ 1.876113] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01817worker1 # [ 1.876354] [drm] features: -virgl +edid -resource_blob -host_visible1818worker1 # [ 1.876363] [drm] features: -context_init1819worker1 # [ 1.877098] [drm] number of scanouts: 11820worker1 # [ 1.877116] [drm] number of cap sets: 01821client # [ 1.919000] [drm] features: -virgl +edid -resource_blob -host_visible1822client # [ 1.919022] [drm] features: -context_init1823client # [ 1.919826] [drm] number of scanouts: 11824client # [ 1.919845] [drm] number of cap sets: 01825lb # [ 1.893967] systemd[1]: Starting Coldplug All udev Devices...1826worker2 # [ 1.893892] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1827worker2 # [ 1.893939] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01828worker1 # [ 1.893141] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1829worker1 # [ 1.895670] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1830worker1 # [ 1.895693] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01831client # [ 1.927061] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1832client # [ 1.927080] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01833worker1 # [ 1.904398] systemd[1]: Starting Create Static Device Nodes in /dev...1834worker2 # [ 1.917870] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1835client # [ 1.947479] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1836worker2 # [ 1.920575] systemd[1]: Starting Create Static Device Nodes in /dev...1837client # [ 1.950081] systemd[1]: Starting Create Static Device Nodes in /dev...1838lb # [ 1.918553] systemd-journald[73]: Collecting audit messages is disabled.1839lb # [ 1.922851] systemd[1]: Finished Create List of Static Device Nodes.1840lb # [ 1.924214] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1841lb # [ 1.924724] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1842lb # [ 1.925183] systemd[1]: Mounted Kernel Configuration File System.1843client # [ 1.955143] Console: switching to colour frame buffer device 160x501844client # [ 1.962205] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1845worker1 # [ 1.924612] Console: switching to colour frame buffer device 160x501846worker2 # [ 1.933480] Console: switching to colour frame buffer device 160x501847worker2 # [ 1.940467] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1848lb # [ 1.943664] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1849worker1 # [ 1.956360] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1850client # [ 1.991486] systemd[1]: Finished Load Kernel Modules.1851client # [ 1.994846] systemd[1]: Starting Apply Kernel Variables...1852worker2 # [ 1.966073] systemd[1]: Finished Load Kernel Modules.1853worker1 # [ 1.960598] systemd[1]: Finished Create Static Device Nodes in /dev.1854worker1 # [ 1.960981] systemd[1]: Reached target Preparation for Local File Systems.1855worker2 # [ 1.969284] systemd[1]: Starting Apply Kernel Variables...1856worker1 # [ 1.961019] systemd[1]: Reached target Local File Systems.1857worker1 # [ 1.969038] systemd[1]: Starting Rule-based Manager for Device Events and Files...1858client # [ 2.005994] systemd[1]: Started Journal Service.1859lb # [ 1.971646] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1860client # [ 2.004950] systemd-modules-load[74]: Inserted module 'dm_mod'1861client # [ 2.008221] systemd-modules-load[74]: Module 'virtio_balloon' is built in1862lb # [ 1.993803] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1863client # [ 2.009361] systemd-modules-load[74]: Module 'virtio_console' is built in1864client # [ 2.010436] systemd-modules-load[74]: Inserted module 'virtio_gpu'1865worker1 # [ 1.988097] systemd[1]: Finished Load Kernel Modules.1866client # [ 2.011400] systemd-modules-load[74]: Module 'virtio_rng' is built in1867lb # [ 1.998426] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1868worker1 # [ 1.992263] systemd[1]: Starting Apply Kernel Variables...1869worker2 # [ 1.997528] systemd[1]: Finished Create Static Device Nodes in /dev.1870worker2 # [ 1.998003] systemd[1]: Reached target Preparation for Local File Systems.1871worker2 # [ 1.998041] systemd[1]: Reached target Local File Systems.1872worker2 # [ 2.005609] systemd[1]: Starting Rule-based Manager for Device Events and Files...1873lb # [ 2.008683] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01874client # [ 2.024550] systemd[1]: Finished Create Static Device Nodes in /dev.1875lb # [ 2.008938] [drm] features: -virgl +edid -resource_blob -host_visible1876lb # [ 2.008951] [drm] features: -context_init1877client # [ 2.025684] systemd[1]: Reached target Preparation for Local File Systems.1878lb # [ 2.009653] [drm] number of scanouts: 11879lb # [ 2.009673] [drm] number of cap sets: 01880client # [ 2.026747] systemd[1]: Reached target Local File Systems.1881client # [ 2.027641] systemd[1]: Starting Create System Files and Directories...1882lb # [ 2.013317] systemd[1]: Starting Create Static Device Nodes in /dev...1883client # [ 2.041420] systemd[1]: Starting Rule-based Manager for Device Events and Files...1884client # [ 2.045048] systemd[1]: Finished Apply Kernel Variables.1885worker1 # [ 2.008508] systemd-modules-load[74]: Inserted module 'dm_mod'1886worker1 # [ 2.025548] systemd[1]: Started Journal Service.1887worker1 # [ 2.013039] systemd-modules-load[74]: Module 'virtio_balloon' is built in1888worker1 # [ 2.014182] systemd-modules-load[74]: Module 'virtio_console' is built in1889worker1 # [ 2.015227] systemd-modules-load[74]: Inserted module 'virtio_gpu'1890worker2 # [ 2.037457] systemd[1]: Finished Apply Kernel Variables.1891lb # [ 2.034688] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1892worker2 # [ 2.039873] systemd[1]: Started Journal Service.1893lb # [ 2.034722] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01894worker2 # [ 2.024989] systemd-modules-load[75]: Inserted module 'dm_mod'1895worker2 # [ 2.033186] systemd-modules-load[75]: Module 'virtio_balloon' is built in1896worker2 # [ 2.034308] systemd-modules-load[75]: Module 'virtio_console' is built in1897worker2 # [ 2.035361] systemd-modules-load[75]: Inserted module 'virtio_gpu'1898worker1 # [ 2.032450] systemd-modules-load[74]: Module 'virtio_rng' is built in1899worker1 # [ 2.033683] systemd[1]: Starting Create System Files and Directories...1900worker2 # [ 2.048436] systemd-modules-load[75]: Module 'virtio_rng' is built in1901worker2 # [ 2.049491] systemd[1]: Starting Create System Files and Directories...1902lb # [ 2.058606] Console: switching to colour frame buffer device 160x501903worker1 # [ 2.045941] systemd[1]: Finished Apply Kernel Variables.1904client # [ 2.084947] systemd[1]: Finished Create System Files and Directories.1905lb # [ 2.078480] systemd[1]: Finished Create Static Device Nodes in /dev.1906lb # [ 2.078868] systemd[1]: Reached target Preparation for Local File Systems.1907lb # [ 2.078893] systemd[1]: Reached target Local File Systems.1908lb # [ 2.086124] systemd[1]: Starting Rule-based Manager for Device Events and Files...1909worker1 # [ 2.069825] systemd-udevd[80]: Using default interface naming scheme 'v261'.1910lb # [ 2.094387] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1911client # [ 2.109959] systemd-udevd[81]: Using default interface naming scheme 'v261'.1912worker1 # [ 2.084553] systemd[1]: Finished Create System Files and Directories.1913worker2 # [ 2.093703] systemd-udevd[81]: Using default interface naming scheme 'v261'.1914lb # [ 2.104579] systemd-modules-load[75]: Inserted module 'dm_mod'1915worker2 # [ 2.105564] systemd[1]: Finished Create System Files and Directories.1916lb # [ 2.119515] systemd[1]: Finished Load Kernel Modules.1917lb # [ 2.121617] systemd[1]: Started Journal Service.1918client # [ 2.138310] systemd[1]: Started Rule-based Manager for Device Events and Files.1919lb # [ 2.110969] systemd-modules-load[75]: Module 'virtio_balloon' is built in1920lb # [ 2.112257] systemd-modules-load[75]: Module 'virtio_console' is built in1921worker1 # [ 2.105977] systemd[1]: Started Rule-based Manager for Device Events and Files.1922lb # [ 2.115638] systemd-modules-load[75]: Inserted module 'virtio_gpu'1923worker2 # [ 2.126519] systemd[1]: Started Rule-based Manager for Device Events and Files.1924lb # [ 2.128472] systemd-modules-load[75]: Module 'virtio_rng' is built in1925lb # [ 2.129549] systemd[1]: Starting Apply Kernel Variables...1926lb # [ 2.144223] systemd[1]: Starting Create System Files and Directories...1927lb # [ 2.174095] systemd-udevd[79]: Using default interface naming scheme 'v261'.1928client # [ 2.204170] systemd[1]: Starting Virtual Console Setup...1929worker1 # [ 2.168082] systemd[1]: Starting Virtual Console Setup...1930worker2 # [ 2.184095] systemd[1]: Starting Virtual Console Setup...1931lb # [ 2.192924] systemd[1]: Finished Apply Kernel Variables.1932lb # [ 2.201858] systemd[1]: Finished Create System Files and Directories.1933lb # [ 2.218718] systemd[1]: Started Rule-based Manager for Device Events and Files.1934client # [ 2.248521] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1935client # [ 2.252211] systemd[1]: Finished Virtual Console Setup.1936worker1 # [ 2.220530] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1937worker1 # [ 2.224071] systemd[1]: Finished Virtual Console Setup.1938worker2 # [ 2.236537] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1939worker2 # [ 2.239944] systemd[1]: Finished Virtual Console Setup.1940lb # [ 2.278929] systemd[1]: Starting Virtual Console Setup...1941lb # [ 2.336517] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1942lb # [ 2.339789] systemd[1]: Finished Virtual Console Setup.1943client # [ 2.825589] systemd[1]: Finished Coldplug All udev Devices.1944client # [ 2.826565] systemd[1]: Reached target System Initialization.1945client # [ 2.828155] systemd[1]: Reached target Basic System.1946worker2 # [ 2.814350] systemd[1]: Finished Coldplug All udev Devices.1947worker2 # [ 2.815921] systemd[1]: Reached target System Initialization.1948worker2 # [ 2.820415] systemd[1]: Reached target Basic System.1949worker1 # [ 2.825439] systemd[1]: Finished Coldplug All udev Devices.1950worker1 # [ 2.826500] systemd[1]: Reached target System Initialization.1951worker1 # [ 2.827318] systemd[1]: Reached target Basic System.1952client # [ 2.956193] (udev-worker)[94]: Network interface NamePolicy= disabled on kernel command line.1953lb # [ 2.929367] systemd[1]: Finished Coldplug All udev Devices.1954lb # [ 2.930459] systemd[1]: Reached target System Initialization.1955lb # [ 2.931400] systemd[1]: Reached target Basic System.1956client # [ 2.998587] (udev-worker)[98]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1957client # [ 3.004267] (udev-worker)[98]: Network interface NamePolicy= disabled on kernel command line.1958worker2 # [ 2.977614] (udev-worker)[95]: Network interface NamePolicy= disabled on kernel command line.1959worker1 # [ 2.981260] (udev-worker)[101]: Network interface NamePolicy= disabled on kernel command line.1960worker2 # [ 2.995421] (udev-worker)[102]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1961worker1 # [ 2.992182] (udev-worker)[95]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1962worker2 # [ 3.002159] (udev-worker)[102]: Network interface NamePolicy= disabled on kernel command line.1963worker1 # [ 2.998441] (udev-worker)[95]: Network interface NamePolicy= disabled on kernel command line.1964client # [ 3.064575] systemd[1]: Found device /dev/disk/by-label/nixos.1965client # [ 3.067106] systemd[1]: Reached target Initrd Root Device.1966client # [ 3.070969] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1967worker1 # [ 3.061267] systemd[1]: Found device /dev/disk/by-label/nixos.1968worker1 # [ 3.063720] systemd[1]: Reached target Initrd Root Device.1969worker1 # [ 3.066152] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1970worker2 # [ 3.079746] systemd[1]: Found device /dev/disk/by-label/nixos.1971worker2 # [ 3.082444] systemd[1]: Reached target Initrd Root Device.1972worker2 # [ 3.085447] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1973client # [ 3.121886] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks1974lb # [ 3.096491] (udev-worker)[98]: Network interface NamePolicy= disabled on kernel command line.1975lb # [ 3.099073] (udev-worker)[106]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1976client # [ 3.130157] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1977lb # [ 3.104294] (udev-worker)[106]: Network interface NamePolicy= disabled on kernel command line.1978client # [ 3.139887] systemd[1]: Mounting /sysroot...1979worker2 # [ 3.127481] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks1980worker1 # [ 3.122484] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks1981worker2 # [ 3.136157] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1982worker1 # [ 3.127620] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1983worker2 # [ 3.141039] systemd[1]: Mounting /sysroot...1984worker1 # [ 3.139128] systemd[1]: Mounting /sysroot...1985client # [ 3.189991] EXT4-fs (vda): mounted filesystem d4dd8969-73c0-43cf-ab04-047d2883e63d r/w with ordered data mode. Quota mode: none.1986client # [ 3.181397] systemd[1]: Mounted /sysroot.1987client # [ 3.182387] systemd[1]: Reached target Initrd Root File System.1988client # [ 3.188131] systemd[1]: Starting Mountpoints Configured in the Real Root...1989worker2 # [ 3.186317] EXT4-fs (vda): mounted filesystem 47eff782-01e5-4cab-8c65-8fec002074a1 r/w with ordered data mode. Quota mode: none.1990worker2 # [ 3.179682] systemd[1]: Mounted /sysroot.1991lb # [ 3.180626] systemd[1]: Found device /dev/disk/by-label/nixos.1992worker2 # [ 3.181239] systemd[1]: Reached target Initrd Root File System.1993worker1 # [ 3.187805] EXT4-fs (vda): mounted filesystem bf9ef386-e4ac-4152-8954-4575120e86c0 r/w with ordered data mode. Quota mode: none.1994worker1 # [ 3.177472] systemd[1]: Mounted /sysroot.1995worker2 # [ 3.186423] systemd[1]: Starting Mountpoints Configured in the Real Root...1996lb # [ 3.187319] systemd[1]: Reached target Initrd Root Device.1997client # [ 3.216231] systemd-sysroot-fstab-check[120]: /sysroot should be mounted in the initrd, will request daemon-reload.1998worker1 # [ 3.180810] systemd[1]: Reached target Initrd Root File System.1999worker1 # [ 3.185211] systemd[1]: Starting Mountpoints Configured in the Real Root...2000lb # [ 3.193751] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...
loading 1,859 hidden lines…
3860worker2 # [ 31.807094] nix-grpc-daemon[780]: ts=2026-09-22T10:49:41Z level=info event=scheduler_connected addr=lb:500513861worker2 # [ 31.813181] nix-grpc-daemon[780]: ts=2026-09-22T10:49:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3862worker1 # [ 31.846198] nix-grpc-daemon[774]: ts=2026-09-22T10:49:41Z level=info event=scheduler_connected addr=lb:500513863worker1 # [ 31.852620] nix-grpc-daemon[774]: ts=2026-09-22T10:49:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3864worker2 # [ 32.014822] nix-grpc-daemon[780]: ts=2026-09-22T10:49:41Z level=info event=scheduler_connected addr=lb:500513865worker2 # [ 32.020563] nix-grpc-daemon[780]: ts=2026-09-22T10:49:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3866worker1 # [ 32.053856] nix-grpc-daemon[774]: ts=2026-09-22T10:49:41Z level=info event=scheduler_connected addr=lb:500513867lb: must succeed: curl -sf localhost:9901/clusters3868worker1 # [ 32.059986] nix-grpc-daemon[774]: ts=2026-09-22T10:49:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3869lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)3870worker2 # [ 32.221972] nix-grpc-daemon[780]: ts=2026-09-22T10:49:41Z level=info event=scheduler_connected addr=lb:500513871worker2 # [ 32.228435] nix-grpc-daemon[780]: ts=2026-09-22T10:49:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3872worker1 # [ 32.263999] nix-grpc-daemon[774]: ts=2026-09-22T10:49:41Z level=info event=scheduler_connected addr=lb:500513873worker1 # [ 32.268412] nix-grpc-daemon[774]: ts=2026-09-22T10:49:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3874worker2 # [ 32.430694] nix-grpc-daemon[780]: ts=2026-09-22T10:49:42Z level=info event=scheduler_connected addr=lb:500513875worker2 # [ 32.434047] nix-grpc-daemon[780]: ts=2026-09-22T10:49:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3876worker1 # [ 32.464522] nix-grpc-daemon[774]: ts=2026-09-22T10:49:42Z level=info event=scheduler_connected addr=lb:500513877worker1 # [ 32.470045] nix-grpc-daemon[774]: ts=2026-09-22T10:49:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3878worker2 # [ 32.632364] nix-grpc-daemon[780]: ts=2026-09-22T10:49:42Z level=info event=scheduler_connected addr=lb:500513879worker2 # [ 32.637503] nix-grpc-daemon[780]: ts=2026-09-22T10:49:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3880worker1 # [ 32.670714] nix-grpc-daemon[774]: ts=2026-09-22T10:49:42Z level=info event=scheduler_connected addr=lb:500513881worker1 # [ 32.675083] nix-grpc-daemon[774]: ts=2026-09-22T10:49:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3882worker2 # [ 32.838647] nix-grpc-daemon[780]: ts=2026-09-22T10:49:42Z level=info event=scheduler_connected addr=lb:500513883worker2 # [ 32.844147] nix-grpc-daemon[780]: ts=2026-09-22T10:49:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3884worker1 # [ 32.877476] nix-grpc-daemon[774]: ts=2026-09-22T10:49:42Z level=info event=scheduler_connected addr=lb:500513885worker1 # [ 32.881963] nix-grpc-daemon[774]: ts=2026-09-22T10:49:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3886worker2 # [ 33.045966] nix-grpc-daemon[780]: ts=2026-09-22T10:49:42Z level=info event=scheduler_connected addr=lb:500513887worker2 # [ 33.051571] nix-grpc-daemon[780]: ts=2026-09-22T10:49:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3888worker1 # [ 33.078826] nix-grpc-daemon[774]: ts=2026-09-22T10:49:42Z level=info event=scheduler_connected addr=lb:500513889worker1 # [ 33.084767] nix-grpc-daemon[774]: ts=2026-09-22T10:49:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3890lb: must succeed: curl -sf localhost:9901/clusters3891worker2 # [ 33.253706] nix-grpc-daemon[780]: ts=2026-09-22T10:49:42Z level=info event=scheduler_connected addr=lb:500513892worker2 # [ 33.259915] nix-grpc-daemon[780]: ts=2026-09-22T10:49:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3893lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)3894worker1 # [ 33.289284] nix-grpc-daemon[774]: ts=2026-09-22T10:49:42Z level=info event=scheduler_connected addr=lb:500513895worker1 # [ 33.293562] nix-grpc-daemon[774]: ts=2026-09-22T10:49:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3896worker2 # [ 33.463875] nix-grpc-daemon[780]: ts=2026-09-22T10:49:43Z level=info event=scheduler_connected addr=lb:500513897worker2 # [ 33.468172] nix-grpc-daemon[780]: ts=2026-09-22T10:49:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3898worker1 # [ 33.489672] nix-grpc-daemon[774]: ts=2026-09-22T10:49:43Z level=info event=scheduler_connected addr=lb:500513899worker1 # [ 33.495415] nix-grpc-daemon[774]: ts=2026-09-22T10:49:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3900worker2 # [ 33.664773] nix-grpc-daemon[780]: ts=2026-09-22T10:49:43Z level=info event=scheduler_connected addr=lb:500513901worker2 # [ 33.670347] nix-grpc-daemon[780]: ts=2026-09-22T10:49:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3902worker1 # [ 33.696682] nix-grpc-daemon[774]: ts=2026-09-22T10:49:43Z level=info event=scheduler_connected addr=lb:500513903worker1 # [ 33.702217] nix-grpc-daemon[774]: ts=2026-09-22T10:49:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3904worker2 # [ 33.872085] nix-grpc-daemon[780]: ts=2026-09-22T10:49:43Z level=info event=scheduler_connected addr=lb:500513905worker2 # [ 33.877755] nix-grpc-daemon[780]: ts=2026-09-22T10:49:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3906worker1 # [ 33.903099] nix-grpc-daemon[774]: ts=2026-09-22T10:49:43Z level=info event=scheduler_connected addr=lb:500513907worker1 # [ 33.908928] nix-grpc-daemon[774]: ts=2026-09-22T10:49:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3908worker2 # [ 34.080470] nix-grpc-daemon[780]: ts=2026-09-22T10:49:43Z level=info event=scheduler_connected addr=lb:500513909worker2 # [ 34.085683] nix-grpc-daemon[780]: ts=2026-09-22T10:49:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3910worker1 # [ 34.110196] nix-grpc-daemon[774]: ts=2026-09-22T10:49:43Z level=info event=scheduler_connected addr=lb:500513911worker1 # [ 34.115942] nix-grpc-daemon[774]: ts=2026-09-22T10:49:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3912lb: must succeed: curl -sf localhost:9901/clusters3913worker2 # [ 34.287148] nix-grpc-daemon[780]: ts=2026-09-22T10:49:43Z level=info event=scheduler_connected addr=lb:500513914worker2 # [ 34.293436] nix-grpc-daemon[780]: ts=2026-09-22T10:49:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3915worker1 # [ 34.318611] nix-grpc-daemon[774]: ts=2026-09-22T10:49:43Z level=info event=scheduler_connected addr=lb:500513916worker1 # [ 34.324104] nix-grpc-daemon[774]: ts=2026-09-22T10:49:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3917lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.11 seconds)3918worker2 # [ 34.497483] nix-grpc-daemon[780]: ts=2026-09-22T10:49:44Z level=info event=scheduler_connected addr=lb:500513919worker2 # [ 34.501901] nix-grpc-daemon[780]: ts=2026-09-22T10:49:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3920worker1 # [ 34.525953] nix-grpc-daemon[774]: ts=2026-09-22T10:49:44Z level=info event=scheduler_connected addr=lb:500513921worker1 # [ 34.531661] nix-grpc-daemon[774]: ts=2026-09-22T10:49:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3922worker2 # [ 34.698890] nix-grpc-daemon[780]: ts=2026-09-22T10:49:44Z level=info event=scheduler_connected addr=lb:500513923worker2 # [ 34.705102] nix-grpc-daemon[780]: ts=2026-09-22T10:49:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3924worker1 # [ 34.732730] nix-grpc-daemon[774]: ts=2026-09-22T10:49:44Z level=info event=scheduler_connected addr=lb:500513925worker1 # [ 34.738307] nix-grpc-daemon[774]: ts=2026-09-22T10:49:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3926worker2 # [ 34.906817] nix-grpc-daemon[780]: ts=2026-09-22T10:49:44Z level=info event=scheduler_connected addr=lb:500513927worker2 # [ 34.912676] nix-grpc-daemon[780]: ts=2026-09-22T10:49:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3928worker1 # [ 34.939788] nix-grpc-daemon[774]: ts=2026-09-22T10:49:44Z level=info event=scheduler_connected addr=lb:500513929worker1 # [ 34.945474] nix-grpc-daemon[774]: ts=2026-09-22T10:49:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3930worker2 # [ 35.113994] nix-grpc-daemon[780]: ts=2026-09-22T10:49:44Z level=info event=scheduler_connected addr=lb:500513931worker2 # [ 35.119773] nix-grpc-daemon[780]: ts=2026-09-22T10:49:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3932worker1 # [ 35.146312] nix-grpc-daemon[774]: ts=2026-09-22T10:49:44Z level=info event=scheduler_connected addr=lb:500513933worker1 # [ 35.151797] nix-grpc-daemon[774]: ts=2026-09-22T10:49:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3934worker2 # [ 35.321045] nix-grpc-daemon[780]: ts=2026-09-22T10:49:44Z level=info event=scheduler_connected addr=lb:500513935worker2 # [ 35.326624] nix-grpc-daemon[780]: ts=2026-09-22T10:49:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3936worker1 # [ 35.355746] nix-grpc-daemon[774]: ts=2026-09-22T10:49:44Z level=info event=scheduler_connected addr=lb:500513937lb: must succeed: curl -sf localhost:9901/clusters3938worker1 # [ 35.360620] nix-grpc-daemon[774]: ts=2026-09-22T10:49:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3939lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)3940worker2 # [ 35.531145] nix-grpc-daemon[780]: ts=2026-09-22T10:49:45Z level=info event=scheduler_connected addr=lb:500513941worker2 # [ 35.535521] nix-grpc-daemon[780]: ts=2026-09-22T10:49:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3942worker1 # [ 35.556622] nix-grpc-daemon[774]: ts=2026-09-22T10:49:45Z level=info event=scheduler_connected addr=lb:500513943worker1 # [ 35.562013] nix-grpc-daemon[774]: ts=2026-09-22T10:49:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3944worker2 # [ 35.733102] nix-grpc-daemon[780]: ts=2026-09-22T10:49:45Z level=info event=scheduler_connected addr=lb:500513945worker2 # [ 35.736403] nix-grpc-daemon[780]: ts=2026-09-22T10:49:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3946worker1 # [ 35.763176] nix-grpc-daemon[774]: ts=2026-09-22T10:49:45Z level=info event=scheduler_connected addr=lb:500513947worker1 # [ 35.769331] nix-grpc-daemon[774]: ts=2026-09-22T10:49:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3948worker2 # [ 35.938273] nix-grpc-daemon[780]: ts=2026-09-22T10:49:45Z level=info event=scheduler_connected addr=lb:500513949worker2 # [ 35.943904] nix-grpc-daemon[780]: ts=2026-09-22T10:49:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3950worker1 # [ 35.969899] nix-grpc-daemon[774]: ts=2026-09-22T10:49:45Z level=info event=scheduler_connected addr=lb:500513951worker1 # [ 35.973040] nix-grpc-daemon[774]: ts=2026-09-22T10:49:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3952worker2 # [ 35.998126] nix-grpc-daemon[780]: ts=2026-09-22T10:49:45Z level=info event=scheduler_take_over3953lb # [ 36.003614] niks3-server[991]: 2026/09/22 10:49:45 INFO lead: acquired remote=192.168.1.4:357183954worker2 # [ 36.145046] nix-grpc-daemon[780]: ts=2026-09-22T10:49:45Z level=info event=scheduler_connected addr=lb:500513955worker2 # [ 36.176204] nix-grpc-daemon[780]: ts=2026-09-22T10:49:45Z level=info event=oidc_jwks provider=mock keys=13956worker1 # [ 36.174028] nix-grpc-daemon[774]: ts=2026-09-22T10:49:45Z level=info event=scheduler_connected addr=lb:500513957worker2 # [ 36.181639] nix-grpc-daemon[780]: ts=2026-09-22T10:49:45Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:454543958worker2 # [ 36.188809] nix-grpc-daemon[780]: ts=2026-09-22T10:49:45Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=03959worker2 # [ 36.194487] nix-grpc-daemon[780]: ts=2026-09-22T10:49:45Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:454543960worker2 # [ 36.199417] nix-grpc-daemon[780]: ts=2026-09-22T10:49:45Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=03961lb: must succeed: curl -sf localhost:9901/clusters3962lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.11 seconds)3963worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3964worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.10 seconds)3965lb: must succeed: curl -sf localhost:9901/clusters3966lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)3967lb: must succeed: curl -sf localhost:9901/clusters3968lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)3969lb: must succeed: curl -sf localhost:9901/clusters3970lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)3971lb: must succeed: curl -sf localhost:9901/clusters3972lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)3973worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'3974worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.07 seconds)3975worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3976worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)3977lb: must succeed: curl -sf localhost:9901/clusters3978lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)3979lb: must succeed: curl -sf localhost:9901/clusters3980lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)3981worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics3982worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.11 seconds)3983worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3984worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)3985worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3986worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)3987worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3988worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)3989(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 10.89 seconds)3990subtest: balancer auth: client cert, bearer token, nothing, unknown CN3991worker1 # [ 37.987625] nix-grpc-daemon[774]: ts=2026-09-22T10:49:47Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:342783992worker1 # [ 38.009936] systemd[1]: Started Nix Daemon.3993worker1 # [ 38.349875] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)3994worker1 # [ 38.377790] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 8 ms (attempt 1/5)3995worker1 # [ 38.385811] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 64 ms (attempt 2/5)3996worker1 # [ 38.453042] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 243 ms (attempt 3/5)3997worker1 # [ 38.696625] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 90 ms (attempt 4/5)3998worker1 # [ 38.796441] nix-grpc-daemon[774]: disabling binary cache 'https://cache.nixos.org' for 60 seconds3999worker1 # [ 38.802611] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4000worker1 # [ 38.823308] nix-grpc-daemon[774]: ts=2026-09-22T10:49:48Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:342784001worker1 # [ 38.841558] nix-grpc-daemon[774]: ts=2026-09-22T10:49:48Z level=debug event=path_info_miss path=5252da3xcqz5fxzqz01xd82cxl855ak1-nixos-system-client-test error="error: path '/nix/store/5252da3xcqz5fxzqz01xd82cxl855ak1-nixos-system-client-test' is required, but there is no substituter that can build it"4002worker1 # [ 38.854902] nix-grpc-daemon[774]: ts=2026-09-22T10:49:48Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:342784003worker1 # [ 38.867756] nix-grpc-daemon[774]: ts=2026-09-22T10:49:48Z level=debug event=path_info_miss path=5252da3xcqz5fxzqz01xd82cxl855ak1-nixos-system-client-test error="error: path '/nix/store/5252da3xcqz5fxzqz01xd82cxl855ak1-nixos-system-client-test' is required, but there is no substituter that can build it"4004client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4005client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.10 seconds)4006client: must succeed: install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4007client: (finished: must succeed: install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.05 seconds)4008worker1 # [ 39.360432] nix-grpc-daemon[774]: ts=2026-09-22T10:49:48Z level=info event=oidc_jwks provider=mock keys=14009worker1 # [ 39.366661] nix-grpc-daemon[774]: ts=2026-09-22T10:49:48Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:342784010worker1 # [ 39.375422] nix-grpc-daemon[774]: ts=2026-09-22T10:49:48Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:342784011worker1 # [ 39.383267] nix-grpc-daemon[774]: ts=2026-09-22T10:49:48Z level=debug event=path_info_miss path=5252da3xcqz5fxzqz01xd82cxl855ak1-nixos-system-client-test error="error: path '/nix/store/5252da3xcqz5fxzqz01xd82cxl855ak1-nixos-system-client-test' is required, but there is no substituter that can build it"4012worker2 # [ 39.420849] nix-grpc-daemon[780]: ts=2026-09-22T10:49:49Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:454664013worker2 # [ 39.441721] systemd[1]: Started Nix Daemon.4014worker2 # [ 39.765854] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4015worker2 # [ 39.790510] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 33 ms (attempt 1/5)4016worker2 # [ 39.826626] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 84 ms (attempt 2/5)4017worker2 # [ 39.911007] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 346 ms (attempt 3/5)4018worker2 # [ 40.257515] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 178 ms (attempt 4/5)4019worker2 # [ 40.439562] nix-grpc-daemon[780]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4020worker2 # [ 40.444827] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4021worker2 # [ 40.458121] nix-grpc-daemon[780]: ts=2026-09-22T10:49:50Z level=debug event=path_info_miss path=5252da3xcqz5fxzqz01xd82cxl855ak1-nixos-system-client-test error="error: path '/nix/store/5252da3xcqz5fxzqz01xd82cxl855ak1-nixos-system-client-test' is required, but there is no substituter that can build it"4022client: must succeed: rm -r /var/lib/nix-grpc-store4023client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.05 seconds)4024worker1 # [ 40.800697] nix-grpc-daemon[774]: ts=2026-09-22T10:49:50Z level=info event=denied method=QueryMissing cn=- role=none4025worker2 # [ 41.074377] nix-grpc-daemon[780]: ts=2026-09-22T10:49:50Z level=info event=denied method=QueryMissing cn=stranger role=none4026(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.54 seconds)4027subtest: DAG build is scheduled across workers and lands in the cache4028client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&24029worker2 # [ 41.343207] nix-grpc-daemon[780]: ts=2026-09-22T10:49:50Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:454664030client # don't know how to build these paths:4031client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4032worker1 # [ 41.348238] nix-grpc-daemon[774]: ts=2026-09-22T10:49:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:342784033worker2 # [ 41.398636] nix-grpc-daemon[780]: ts=2026-09-22T10:49:51Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=04034worker2 # [ 41.406330] nix-grpc-daemon[780]: ts=2026-09-22T10:49:51Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.4:50051 clients=14035worker2 # [ 41.453431] nix-grpc-daemon[780]: ts=2026-09-22T10:49:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454664036worker2 # [ 41.473886] nix-grpc-daemon[780]: ts=2026-09-22T10:49:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:454664037client # copying 1 paths to 'lb:50051'...4038worker2 # [ 41.490407] nix-grpc-daemon[780]: ts=2026-09-22T10:49:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:454664039client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'...4040worker2 # [ 41.522844] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4041worker2 # [ 41.559970] nix-grpc-daemon[780]: ts=2026-09-22T10:49:51Z level=info event=niks3_push_started pid=8804042lb # [ 41.575295] envoy[927]: conn peer=[2001:db8:1::1]:48890 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=715 tx=365 ms=9474043lb # [ 41.580828] envoy[927]: conn peer=[2001:db8:1::1]:48906 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2630 tx=422 ms=11224044lb # [ 41.585548] envoy[927]: conn peer=[2001:db8:1::1]:43760 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=124045lb # [ 41.590087] envoy[927]: conn peer=[2001:db8:1::1]:43762 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=194046worker2 # [ 41.851764] nix-grpc-daemon[880]: time=2026-09-22T10:49:51.453Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""4047worker2 # [ 42.048888] nix-daemon[866]: accepted connection from pid 893, user nix-grpc-daemon (trusted)4048worker2 # [ 42.069711] nix-daemon[866]: reaped child process 896, status = succeeded4049worker2 # [ 42.257576] nix-daemon[866]: accepted connection from pid 898, user nix-grpc-daemon (trusted)4050worker2 # [ 42.279091] nix-daemon[866]: reaped child process 901, status = succeeded4051lb # [ 42.301531] niks3-server[991]: 2026/09/22 10:49:51 INFO Received uploads request method=POST path=/api/pending_closures4052worker2 # [ 42.328217] nix-grpc-daemon[880]: time=2026-09-22T10:49:51.931Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4053worker2 # [ 42.330867] nix-grpc-daemon[880]: time=2026-09-22T10:49:51.933Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)"4054lb # [ 42.380886] niks3-server[991]: 2026/09/22 10:49:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4055lb # [ 42.391403] niks3-server[991]: 2026/09/22 10:49:51 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst4056lb # [ 42.418515] niks3-server[991]: 2026/09/22 10:49:52 INFO Signed narinfos id=1 count=14057worker2 # [ 42.423215] nix-grpc-daemon[880]: time=2026-09-22T10:49:52.026Z level=INFO msg="Uploading 1 narinfos"4058lb # [ 42.438368] niks3-server[991]: 2026/09/22 10:49:52 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4059lb # [ 42.445268] niks3-server[991]: 2026/09/22 10:49:52 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls4060lb # [ 42.452554] niks3-server[991]: 2026/09/22 10:49:52 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo4061lb # [ 42.456355] niks3-server[991]: 2026/09/22 10:49:52 INFO Completed upload id=14062worker2 # [ 42.457236] nix-grpc-daemon[880]: time=2026-09-22T10:49:52.060Z level=INFO msg="Upload complete. (598ms)"4063worker2 # [ 42.469452] nix-grpc-daemon[780]: ts=2026-09-22T10:49:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=0 paths=1 nar_bytes_in=5044064worker2 # [ 42.487626] nix-grpc-daemon[780]: ts=2026-09-22T10:49:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454664065worker2 # [ 42.498287] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4066worker2 # [ 42.506032] nix-daemon[866]: reaped child process 878, status = succeeded4067client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051...4068client # farm-a> worker2: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4069worker2 # [ 42.514544] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4070client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4071client # farm-a> LOG-farm-a-t14072worker2 # [ 44.138086] nix-daemon[866]: reaped child process 906, status = succeeded4073worker2 # [ 44.301807] nix-daemon[866]: accepted connection from pid 922, user nix-grpc-daemon (trusted)4074worker2 # [ 44.318706] nix-daemon[866]: reaped child process 925, status = succeeded4075lb # [ 44.330349] niks3-server[991]: 2026/09/22 10:49:53 INFO Received uploads request method=POST path=/api/pending_closures4076worker2 # [ 44.344399] nix-grpc-daemon[880]: time=2026-09-22T10:49:53.946Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4077worker2 # [ 44.345865] nix-grpc-daemon[880]: time=2026-09-22T10:49:53.946Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)"4078lb # [ 44.378166] niks3-server[991]: 2026/09/22 10:49:53 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4079lb # [ 44.391220] niks3-server[991]: 2026/09/22 10:49:53 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4080lb # [ 44.403816] niks3-server[991]: 2026/09/22 10:49:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4081worker2 # [ 44.411861] nix-grpc-daemon[880]: time=2026-09-22T10:49:54.013Z level=INFO msg="Uploading 1 narinfos"4082lb # [ 44.412648] niks3-server[991]: 2026/09/22 10:49:53 INFO Signed narinfos id=2 count=14083lb # [ 44.416858] niks3-server[991]: 2026/09/22 10:49:53 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls4084lb # [ 44.430039] niks3-server[991]: 2026/09/22 10:49:54 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4085lb # [ 44.439881] niks3-server[991]: 2026/09/22 10:49:54 INFO Completed upload id=24086worker2 # [ 44.440537] nix-grpc-daemon[880]: time=2026-09-22T10:49:54.043Z level=INFO msg="Upload complete. (312ms)"4087lb # [ 44.443139] niks3-server[991]: 2026/09/22 10:49:54 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo4088worker2 # [ 44.464715] nix-grpc-daemon[780]: ts=2026-09-22T10:49:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=14089worker2 # [ 44.478092] nix-grpc-daemon[780]: ts=2026-09-22T10:49:54Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=04090worker2 # [ 44.481626] nix-grpc-daemon[780]: ts=2026-09-22T10:49:54Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.4:50051 clients=14091worker2 # [ 44.485298] nix-daemon[866]: reaped child process 904, status = succeeded4092worker2 # [ 44.509047] nix-grpc-daemon[780]: ts=2026-09-22T10:49:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454664093worker2 # [ 44.524447] nix-grpc-daemon[780]: ts=2026-09-22T10:49:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:454664094worker2 # [ 44.539764] nix-grpc-daemon[880]: time=2026-09-22T10:49:54.142Z level=INFO msg="All 1 paths already cached"4095client # copying 1 paths to 'lb:50051'...4096worker2 # [ 44.550025] nix-grpc-daemon[780]: ts=2026-09-22T10:49:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:454664097client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'...4098worker2 # [ 44.574401] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4099worker2 # [ 44.782543] nix-daemon[866]: accepted connection from pid 931, user nix-grpc-daemon (trusted)4100worker2 # [ 44.799969] nix-daemon[866]: reaped child process 934, status = succeeded4101worker2 # [ 44.971918] nix-daemon[866]: accepted connection from pid 936, user nix-grpc-daemon (trusted)4102worker2 # [ 44.988665] nix-daemon[866]: reaped child process 939, status = succeeded4103lb # [ 44.999769] niks3-server[991]: 2026/09/22 10:49:54 INFO Received uploads request method=POST path=/api/pending_closures4104worker2 # [ 45.011250] nix-grpc-daemon[880]: time=2026-09-22T10:49:54.614Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4105worker2 # [ 45.012875] nix-grpc-daemon[880]: time=2026-09-22T10:49:54.614Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)"4106lb # [ 45.028450] niks3-server[991]: 2026/09/22 10:49:54 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst4107lb # [ 45.042450] niks3-server[991]: 2026/09/22 10:49:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4108worker2 # [ 45.048966] nix-grpc-daemon[880]: time=2026-09-22T10:49:54.652Z level=INFO msg="Uploading 1 narinfos"4109lb # [ 45.054049] niks3-server[991]: 2026/09/22 10:49:54 INFO Signed narinfos id=3 count=14110lb # [ 45.061313] niks3-server[991]: 2026/09/22 10:49:54 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls4111lb # [ 45.069575] niks3-server[991]: 2026/09/22 10:49:54 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4112lb # [ 45.079401] niks3-server[991]: 2026/09/22 10:49:54 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo4113lb # [ 45.083762] niks3-server[991]: 2026/09/22 10:49:54 INFO Completed upload id=34114worker2 # [ 45.084428] nix-grpc-daemon[880]: time=2026-09-22T10:49:54.687Z level=INFO msg="Upload complete. (492ms)"4115worker2 # [ 45.093208] nix-grpc-daemon[780]: ts=2026-09-22T10:49:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=0 paths=1 nar_bytes_in=6244116worker2 # [ 45.107099] nix-grpc-daemon[780]: ts=2026-09-22T10:49:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454664117worker2 # [ 45.113986] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4118worker2 # [ 45.118583] nix-daemon[866]: reaped child process 929, status = succeeded4119client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051...4120client # farm-b> worker2: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4121worker2 # [ 45.132118] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4122client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4123client # farm-b> LOG-farm-b-t14124worker2 # [ 45.966981] nix-daemon[866]: reaped child process 943, status = succeeded4125worker2 # [ 46.134156] nix-daemon[866]: accepted connection from pid 960, user nix-grpc-daemon (trusted)4126worker2 # [ 46.153676] nix-daemon[866]: reaped child process 963, status = succeeded4127lb # [ 46.165516] niks3-server[991]: 2026/09/22 10:49:55 INFO Received uploads request method=POST path=/api/pending_closures4128worker2 # [ 46.172430] nix-grpc-daemon[880]: time=2026-09-22T10:49:55.774Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4129worker2 # [ 46.173921] nix-grpc-daemon[880]: time=2026-09-22T10:49:55.774Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)"4130lb # [ 46.190174] niks3-server[991]: 2026/09/22 10:49:55 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4131lb # [ 46.194546] niks3-server[991]: 2026/09/22 10:49:55 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst4132lb # [ 46.197822] niks3-server[991]: 2026/09/22 10:49:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4133worker2 # [ 46.200635] nix-grpc-daemon[880]: time=2026-09-22T10:49:55.803Z level=INFO msg="Uploading 1 narinfos"4134lb # [ 46.202606] niks3-server[991]: 2026/09/22 10:49:55 INFO Signed narinfos id=4 count=14135lb # [ 46.205445] niks3-server[991]: 2026/09/22 10:49:55 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls4136lb # [ 46.209471] niks3-server[991]: 2026/09/22 10:49:55 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4137lb # [ 46.213268] niks3-server[991]: 2026/09/22 10:49:55 INFO Completed upload id=44138worker2 # [ 46.213875] nix-grpc-daemon[880]: time=2026-09-22T10:49:55.817Z level=INFO msg="Upload complete. (271ms)"4139lb # [ 46.214614] niks3-server[991]: 2026/09/22 10:49:55 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo4140worker2 # [ 46.223830] nix-grpc-daemon[780]: ts=2026-09-22T10:49:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=14141worker2 # [ 46.234166] nix-grpc-daemon[780]: ts=2026-09-22T10:49:55Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=04142worker2 # [ 46.236524] nix-grpc-daemon[780]: ts=2026-09-22T10:49:55Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.4:50051 clients=14143worker2 # [ 46.239073] nix-daemon[866]: reaped child process 941, status = succeeded4144worker2 # [ 46.267546] nix-grpc-daemon[780]: ts=2026-09-22T10:49:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454664145worker2 # [ 46.278122] nix-grpc-daemon[780]: ts=2026-09-22T10:49:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:454664146worker2 # [ 46.287797] nix-grpc-daemon[880]: time=2026-09-22T10:49:55.891Z level=INFO msg="All 2 paths already cached"4147client # copying 1 paths to 'lb:50051'...4148client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'...4149worker2 # [ 46.295937] nix-grpc-daemon[780]: ts=2026-09-22T10:49:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:454664150worker2 # [ 46.312360] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4151worker2 # [ 46.497911] nix-daemon[866]: accepted connection from pid 969, user nix-grpc-daemon (trusted)4152worker2 # [ 46.517436] nix-daemon[866]: reaped child process 972, status = succeeded4153worker2 # [ 46.691799] nix-daemon[866]: accepted connection from pid 974, user nix-grpc-daemon (trusted)4154worker2 # [ 46.709778] nix-daemon[866]: reaped child process 977, status = succeeded4155lb # [ 46.721401] niks3-server[991]: 2026/09/22 10:49:56 INFO Received uploads request method=POST path=/api/pending_closures4156worker2 # [ 46.732140] nix-grpc-daemon[880]: time=2026-09-22T10:49:56.335Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4157worker2 # [ 46.734070] nix-grpc-daemon[880]: time=2026-09-22T10:49:56.336Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)"4158lb # [ 46.751368] niks3-server[991]: 2026/09/22 10:49:56 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst4159lb # [ 46.765984] niks3-server[991]: 2026/09/22 10:49:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4160worker2 # [ 46.773351] nix-grpc-daemon[880]: time=2026-09-22T10:49:56.376Z level=INFO msg="Uploading 1 narinfos"4161lb # [ 46.777667] niks3-server[991]: 2026/09/22 10:49:56 INFO Signed narinfos id=5 count=14162lb # [ 46.783912] niks3-server[991]: 2026/09/22 10:49:56 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls4163lb # [ 46.793403] niks3-server[991]: 2026/09/22 10:49:56 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4164lb # [ 46.800616] niks3-server[991]: 2026/09/22 10:49:56 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo4165lb # [ 46.805040] niks3-server[991]: 2026/09/22 10:49:56 INFO Completed upload id=54166worker2 # [ 46.805786] nix-grpc-daemon[880]: time=2026-09-22T10:49:56.408Z level=INFO msg="Upload complete. (480ms)"4167worker2 # [ 46.816340] nix-grpc-daemon[780]: ts=2026-09-22T10:49:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=0 paths=1 nar_bytes_in=7524168worker2 # [ 46.831905] nix-grpc-daemon[780]: ts=2026-09-22T10:49:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454664169worker2 # [ 46.838131] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4170worker2 # [ 46.842706] nix-daemon[866]: reaped child process 967, status = succeeded4171client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051...4172worker2 # [ 46.853986] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4173client # farm-top> worker2: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4174client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4175client # farm-top> LOG-farm-top-t14176worker2 # [ 48.498362] nix-daemon[866]: reaped child process 981, status = succeeded4177worker2 # [ 48.672605] nix-daemon[866]: accepted connection from pid 997, user nix-grpc-daemon (trusted)4178worker2 # [ 48.689523] nix-daemon[866]: reaped child process 1000, status = succeeded4179lb # [ 48.703269] niks3-server[991]: 2026/09/22 10:49:58 INFO Received uploads request method=POST path=/api/pending_closures4180worker2 # [ 48.717435] nix-grpc-daemon[880]: time=2026-09-22T10:49:58.318Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4181worker2 # [ 48.718917] nix-grpc-daemon[880]: time=2026-09-22T10:49:58.318Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)"4182lb # [ 48.738274] niks3-server[991]: 2026/09/22 10:49:58 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4183lb # [ 48.752253] niks3-server[991]: 2026/09/22 10:49:58 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst4184lb # [ 48.765684] niks3-server[991]: 2026/09/22 10:49:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4185worker2 # [ 48.770545] nix-grpc-daemon[880]: time=2026-09-22T10:49:58.373Z level=INFO msg="Uploading 1 narinfos"4186lb # [ 48.774261] niks3-server[991]: 2026/09/22 10:49:58 INFO Signed narinfos id=6 count=14187lb # [ 48.779639] niks3-server[991]: 2026/09/22 10:49:58 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls4188lb # [ 48.787440] niks3-server[991]: 2026/09/22 10:49:58 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4189worker2 # [ 48.793507] nix-grpc-daemon[880]: time=2026-09-22T10:49:58.396Z level=INFO msg="Upload complete. (326ms)"4190lb # [ 48.795545] niks3-server[991]: 2026/09/22 10:49:58 INFO Completed upload id=64191lb # [ 48.799445] niks3-server[991]: 2026/09/22 10:49:58 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo4192worker2 # [ 48.806545] nix-grpc-daemon[780]: ts=2026-09-22T10:49:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=14193worker2 # [ 48.823261] nix-daemon[866]: reaped child process 979, status = succeeded4194client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2, in 7.75 seconds)4195client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath4196client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath, in 0.24 seconds)4197client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4198client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.02 seconds)4199client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4200client # copying 3 paths...4201client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'...4202client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'...4203client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'...4204client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.28 seconds)4205(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 8.40 seconds)4206subtest: repeat is answered Cached by the scheduler without building4207worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4208worker1 # finding garbage collector roots...4209worker1 # deleting specified paths...4210worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4211worker1 # deleting unused links...4212worker1 # note: hard linking is currently saving 0.0 KiB4213worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.30 seconds)4214worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4215worker2 # finding garbage collector roots...4216worker2 # deleting specified paths...4217worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4218worker2 # deleting unused links...4219worker2 # note: hard linking is currently saving 0.0 KiB4220worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.28 seconds)4221client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&24222worker2 # [ 50.340952] nix-grpc-daemon[780]: ts=2026-09-22T10:49:59Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:454664223client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4224client # /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4225worker1 # [ 50.353323] nix-grpc-daemon[774]: ts=2026-09-22T10:49:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:342784226worker2 # [ 50.388870] nix-grpc-daemon[780]: ts=2026-09-22T10:49:59Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=14227client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2, in 0.34 seconds)4228client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath4229client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath, in 0.21 seconds)4230lb: must succeed: curl -sf localhost:9901/clusters4231lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)4232lb: must succeed: curl -sf localhost:9901/clusters4233lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4234worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4235worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4236(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.45 seconds)4237subtest: inputs already in the cache are not sent to the scheduler4238lb: must succeed: curl -sf localhost:9901/clusters4239lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4240lb: must succeed: curl -sf localhost:9901/clusters4241lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4242worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4243worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.07 seconds)4244client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b >&24245client # don't know how to build these paths:4246client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4247worker2 # [ 51.423883] nix-grpc-daemon[780]: ts=2026-09-22T10:50:01Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:454664248worker1 # [ 51.419761] nix-grpc-daemon[774]: ts=2026-09-22T10:50:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:342784249worker2 # [ 51.444116] nix-grpc-daemon[780]: ts=2026-09-22T10:50:01Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=04250worker2 # [ 51.450368] nix-grpc-daemon[780]: ts=2026-09-22T10:50:01Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.4:50051 clients=14251worker2 # [ 51.496784] nix-grpc-daemon[780]: ts=2026-09-22T10:50:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454664252worker2 # [ 51.512179] nix-grpc-daemon[780]: ts=2026-09-22T10:50:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:454664253client # copying 1 paths to 'lb:50051'...4254worker2 # [ 51.528551] nix-grpc-daemon[880]: time=2026-09-22T10:50:01.130Z level=INFO msg="All 2 paths already cached"4255client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'...4256worker2 # [ 51.533738] nix-grpc-daemon[780]: ts=2026-09-22T10:50:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:454664257worker2 # [ 51.548563] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4258lb # [ 51.580538] envoy[927]: conn peer=[2001:db8:1::1]:43776 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9549 tx=4104 ms=74934259worker2 # [ 51.750930] nix-daemon[866]: accepted connection from pid 1029, user nix-grpc-daemon (trusted)4260worker2 # [ 51.767996] nix-daemon[866]: reaped child process 1032, status = succeeded4261worker2 # [ 51.941168] nix-daemon[866]: accepted connection from pid 1034, user nix-grpc-daemon (trusted)4262worker2 # [ 51.957744] nix-daemon[866]: reaped child process 1037, status = succeeded4263lb # [ 51.585531] envoy[927]: conn peer=[2001:db8:1::1]:43790 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=480 ms=694264lb # [ 51.969973] niks3-server[991]: 2026/09/22 10:50:01 INFO Received uploads request method=POST path=/api/pending_closures4265worker2 # [ 51.983971] nix-grpc-daemon[880]: time=2026-09-22T10:50:01.586Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4266worker2 # [ 51.985722] nix-grpc-daemon[880]: time=2026-09-22T10:50:01.586Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)"4267lb # [ 52.004127] niks3-server[991]: 2026/09/22 10:50:01 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst4268lb # [ 52.016311] niks3-server[991]: 2026/09/22 10:50:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4269worker2 # [ 52.024243] nix-grpc-daemon[880]: time=2026-09-22T10:50:01.627Z level=INFO msg="Uploading 1 narinfos"4270lb # [ 52.023701] niks3-server[991]: 2026/09/22 10:50:01 INFO Signed narinfos id=7 count=14271lb # [ 52.030485] niks3-server[991]: 2026/09/22 10:50:01 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls4272lb # [ 52.044772] niks3-server[991]: 2026/09/22 10:50:01 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4273worker2 # [ 52.051882] nix-grpc-daemon[880]: time=2026-09-22T10:50:01.655Z level=INFO msg="Upload complete. (485ms)"4274lb # [ 52.052075] niks3-server[991]: 2026/09/22 10:50:01 INFO Completed upload id=74275lb # [ 52.056100] niks3-server[991]: 2026/09/22 10:50:01 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo4276worker2 # [ 52.060286] nix-grpc-daemon[780]: ts=2026-09-22T10:50:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=0 paths=1 nar_bytes_in=7524277worker2 # [ 52.075229] nix-grpc-daemon[780]: ts=2026-09-22T10:50:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454664278worker2 # [ 52.081622] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4279worker2 # [ 52.086901] nix-daemon[866]: reaped child process 1027, status = succeeded4280client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051...4281client # farm-top> worker2: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4282worker2 # [ 52.098876] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4283client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4284worker2 # [ 52.720878] nix-daemon[866]: reaped child process 1041, status = succeeded4285client # farm-top> LOG-farm-top-t1b4286worker2 # [ 52.874515] nix-daemon[866]: accepted connection from pid 1057, user nix-grpc-daemon (trusted)4287worker2 # [ 52.891914] nix-daemon[866]: reaped child process 1060, status = succeeded4288lb # [ 52.906146] niks3-server[991]: 2026/09/22 10:50:02 INFO Received uploads request method=POST path=/api/pending_closures4289worker2 # [ 52.918839] nix-grpc-daemon[880]: time=2026-09-22T10:50:02.519Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4290worker2 # [ 52.920393] nix-grpc-daemon[880]: time=2026-09-22T10:50:02.519Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)"4291lb # [ 52.940104] niks3-server[991]: 2026/09/22 10:50:02 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst4292lb # [ 52.955687] niks3-server[991]: 2026/09/22 10:50:02 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4293lb # [ 52.965251] niks3-server[991]: 2026/09/22 10:50:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4294worker2 # [ 52.972194] nix-grpc-daemon[880]: time=2026-09-22T10:50:02.575Z level=INFO msg="Uploading 1 narinfos"4295lb # [ 52.974446] niks3-server[991]: 2026/09/22 10:50:02 INFO Signed narinfos id=8 count=14296lb # [ 52.980410] niks3-server[991]: 2026/09/22 10:50:02 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls4297lb # [ 52.991057] niks3-server[991]: 2026/09/22 10:50:02 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4298lb # [ 52.996288] niks3-server[991]: 2026/09/22 10:50:02 INFO Completed upload id=84299worker2 # [ 52.996915] nix-grpc-daemon[880]: time=2026-09-22T10:50:02.600Z level=INFO msg="Upload complete. (287ms)"4300lb # [ 52.999084] niks3-server[991]: 2026/09/22 10:50:02 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo4301worker2 # [ 53.009004] nix-grpc-daemon[780]: ts=2026-09-22T10:50:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=0 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=14302worker2 # [ 53.023421] nix-daemon[866]: reaped child process 1039, status = succeeded4303client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b >&2, in 1.89 seconds)4304client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b outPath4305client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.23 seconds)4306lb: must succeed: curl -sf localhost:9901/clusters4307lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4308lb: must succeed: curl -sf localhost:9901/clusters4309lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4310worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4311worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4312(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.53 seconds)4313subtest: two clients wanting the same drv build it once4314client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup4315client # [ 53.590746] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup.4316client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup, in 0.10 seconds)4317client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup4318worker2 # [ 53.829447] nix-grpc-daemon[780]: ts=2026-09-22T10:50:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:454664319client # Running as unit: dedup1.service; invocation ID: 2ca2d044966f41369129059539af8f454320client # [ 53.865218] nix[881]: don't know how to build these paths:4321client # [ 53.867135] nix[881]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4322worker2 # [ 53.850557] nix-grpc-daemon[780]: ts=2026-09-22T10:50:03Z level=debug event=want client=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04323worker2 # [ 53.856714] nix-grpc-daemon[780]: ts=2026-09-22T10:50:03Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.3:50051 clients=14324worker1 # [ 53.890548] nix-grpc-daemon[774]: ts=2026-09-22T10:50:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:342784325worker1 # [ 53.908417] nix-grpc-daemon[774]: ts=2026-09-22T10:50:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:342784326worker1 # [ 53.923275] nix-grpc-daemon[774]: ts=2026-09-22T10:50:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:342784327client # [ 53.963797] nix[881]: copying 1 paths to 'lb:50051'...4328client # [ 53.966136] nix[881]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'...4329worker1 # [ 53.945871] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4330worker1 # [ 53.976133] nix-grpc-daemon[774]: ts=2026-09-22T10:50:03Z level=info event=niks3_push_started pid=8674331worker1 # [ 54.244854] nix-grpc-daemon[867]: time=2026-09-22T10:50:03.846Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""4332worker1 # [ 54.440125] nix-daemon[839]: accepted connection from pid 880, user nix-grpc-daemon (trusted)4333worker1 # [ 54.457738] nix-daemon[839]: reaped child process 883, status = succeeded4334worker1 # [ 54.630076] nix-daemon[839]: accepted connection from pid 885, user nix-grpc-daemon (trusted)4335worker1 # [ 54.645551] nix-daemon[839]: reaped child process 888, status = succeeded4336lb # [ 54.666343] niks3-server[991]: 2026/09/22 10:50:04 INFO Received uploads request method=POST path=/api/pending_closures4337worker1 # [ 54.667521] nix-grpc-daemon[867]: time=2026-09-22T10:50:04.270Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4338worker1 # [ 54.670889] nix-grpc-daemon[867]: time=2026-09-22T10:50:04.272Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)"4339client # [ 54.710726] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup.4340client # Running as unit: dedup2.service; invocation ID: 271fa0e3bb11494085c21e6e85abdb5b4341client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup, in 1.13 seconds)4342worker1 # [ 54.710736] nix-grpc-daemon[867]: time=2026-09-22T10:50:04.313Z level=INFO msg="Uploading 1 narinfos"4343lb # [ 54.710095] niks3-server[991]: 2026/09/22 10:50:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4344lb # [ 54.721680] niks3-server[991]: 2026/09/22 10:50:04 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst4345lb # [ 54.732621] niks3-server[991]: 2026/09/22 10:50:04 INFO Signed narinfos id=9 count=14346lb # [ 54.734973] niks3-server[991]: 2026/09/22 10:50:04 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls4347lb # [ 54.743260] niks3-server[991]: 2026/09/22 10:50:04 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4348lb # [ 54.752221] niks3-server[991]: 2026/09/22 10:50:04 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo4349lb # [ 54.757095] niks3-server[991]: 2026/09/22 10:50:04 INFO Completed upload id=94350worker1 # [ 54.749498] nix-grpc-daemon[867]: time=2026-09-22T10:50:04.352Z level=INFO msg="Upload complete. (496ms)"4351worker1 # [ 54.753929] nix-grpc-daemon[774]: ts=2026-09-22T10:50:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=0 paths=1 nar_bytes_in=4324352worker1 # [ 54.763195] nix-grpc-daemon[774]: ts=2026-09-22T10:50:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:342784353worker1 # [ 54.768681] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4354worker1 # [ 54.771522] nix-daemon[839]: reaped child process 865, status = succeeded4355client # [ 54.818881] nix[881]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4356worker1 # [ 54.784238] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4357worker1 # [ 55.049338] nix-grpc-daemon[774]: ts=2026-09-22T10:50:04Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:342784358client # [ 55.099623] nix[897]: this derivation will be built:4359worker2 # [ 55.076615] nix-grpc-daemon[780]: ts=2026-09-22T10:50:04Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:454664360worker2 # [ 55.095593] nix-grpc-daemon[780]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' from 'http://lb:5751'...4361client # [ 55.145753] nix[897]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4362worker2 # [ 55.133260] nix-grpc-daemon[780]: ts=2026-09-22T10:50:04Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04363worker1 # [ 55.133682] nix-grpc-daemon[774]: ts=2026-09-22T10:50:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:342784364client # [ 55.177255] nix[897]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4365client: must succeed: sleep 24366client: (finished: must succeed: sleep 2, in 2.07 seconds)4367worker1 # [ 59.412442] nix-daemon[839]: reaped child process 896, status = succeeded4368client: waiting for success: ! systemctl is-active dedup1 dedup24369worker1 # [ 59.573593] nix-daemon[839]: accepted connection from pid 926, user nix-grpc-daemon (trusted)4370worker1 # [ 59.590442] nix-daemon[839]: reaped child process 929, status = succeeded4371lb # [ 59.611788] niks3-server[991]: 2026/09/22 10:50:09 INFO Received uploads request method=POST path=/api/pending_closures4372worker1 # [ 59.618233] nix-grpc-daemon[867]: time=2026-09-22T10:50:09.219Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4373worker1 # [ 59.620106] nix-grpc-daemon[867]: time=2026-09-22T10:50:09.219Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)"4374lb # [ 59.656111] niks3-server[991]: 2026/09/22 10:50:09 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4375lb # [ 59.677605] niks3-server[991]: 2026/09/22 10:50:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4376worker1 # [ 59.678209] nix-grpc-daemon[867]: time=2026-09-22T10:50:09.279Z level=INFO msg="Uploading 1 narinfos"4377lb # [ 59.688985] niks3-server[991]: 2026/09/22 10:50:09 INFO Signed narinfos id=10 count=14378lb # [ 59.694099] niks3-server[991]: 2026/09/22 10:50:09 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4379lb # [ 59.704358] niks3-server[991]: 2026/09/22 10:50:09 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls4380lb # [ 59.712491] niks3-server[991]: 2026/09/22 10:50:09 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4381lb # [ 59.718881] niks3-server[991]: 2026/09/22 10:50:09 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo4382lb # [ 59.724793] niks3-server[991]: 2026/09/22 10:50:09 INFO Completed upload id=104383worker1 # [ 59.716731] nix-grpc-daemon[867]: time=2026-09-22T10:50:09.319Z level=INFO msg="Upload complete. (319ms)"4384worker1 # [ 59.729198] nix-grpc-daemon[774]: ts=2026-09-22T10:50:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14385worker1 # [ 59.746907] nix-grpc-daemon[774]: ts=2026-09-22T10:50:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14386worker1 # [ 59.754003] nix-daemon[839]: reaped child process 894, status = succeeded4387client # [ 59.807694] systemd[1]: dedup2.service: Deactivated successfully.4388client # [ 59.813735] systemd[1]: dedup2.service: Consumed 225ms CPU time over 5.096s wall clock time, 33.2M memory peak, 6.5K incoming IP traffic, 5.8K outgoing IP traffic.4389client # [ 59.857544] systemd[1]: dedup1.service: Deactivated successfully.4390client # [ 59.859779] systemd[1]: dedup1.service: Consumed 177ms CPU time over 6.269s wall clock time, 41.3M memory peak, 7.2K incoming IP traffic, 8.5K outgoing IP traffic.4391client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.14 seconds)4392client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4393client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.06 seconds)4394worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4395worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.08 seconds)4396worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4397worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4398worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4399worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4400worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4401worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4402(finished: subtest: two clients wanting the same drv build it once, in 7.45 seconds)4403subtest: the first client leaving does not fail the second4404client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share4405client # [ 61.030864] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share.4406client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share, in 0.09 seconds)4407client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share4408client # Running as unit: share1.service; invocation ID: 350a011f71db4cbaa2916fe0152fbed14409client # [ 61.302125] nix[936]: don't know how to build these paths:4410client # [ 61.303903] nix[936]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4411worker2 # [ 61.273083] nix-grpc-daemon[780]: ts=2026-09-22T10:50:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:454664412worker2 # [ 61.287761] nix-grpc-daemon[780]: ts=2026-09-22T10:50:10Z level=debug event=want client=6 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=04413worker2 # [ 61.295100] nix-grpc-daemon[780]: ts=2026-09-22T10:50:10Z level=debug event=assigned drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker=192.168.1.4:50051 clients=14414worker2 # [ 61.333266] nix-grpc-daemon[780]: ts=2026-09-22T10:50:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454664415worker2 # [ 61.349476] nix-grpc-daemon[780]: ts=2026-09-22T10:50:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:454664416client # [ 61.392234] nix[936]: copying 1 paths to 'lb:50051'...4417worker2 # [ 61.364684] nix-grpc-daemon[780]: ts=2026-09-22T10:50:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:454664418client # [ 61.394544] nix[936]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' to 'lb:50051'...4419worker2 # [ 61.381889] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4420lb # [ 61.581609] envoy[927]: conn peer=[2001:db8:1::1]:56508 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4142 tx=1791 ms=15974421lb # [ 61.586598] envoy[927]: conn peer=[2001:db8:1::1]:56522 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1579 tx=1487 ms=46934422worker2 # [ 61.591052] nix-daemon[866]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)4423lb # [ 61.591573] envoy[927]: conn peer=[2001:db8:1::1]:56512 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3005 tx=1451 ms=59684424worker2 # [ 61.611418] nix-daemon[866]: reaped child process 1104, status = succeeded4425worker2 # [ 61.782167] nix-daemon[866]: accepted connection from pid 1106, user nix-grpc-daemon (trusted)4426worker2 # [ 61.799120] nix-daemon[866]: reaped child process 1109, status = succeeded4427lb # [ 61.810799] niks3-server[991]: 2026/09/22 10:50:11 INFO Received uploads request method=POST path=/api/pending_closures4428worker2 # [ 61.820988] nix-grpc-daemon[880]: time=2026-09-22T10:50:11.423Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4429worker2 # [ 61.822446] nix-grpc-daemon[880]: time=2026-09-22T10:50:11.423Z level=INFO msg="Uploading rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv (432B)"4430lb # [ 61.838702] niks3-server[991]: 2026/09/22 10:50:11 INFO Registered completed upload object_key=nar/0chdjpc9an2ckdblh08lnyrmzvw3h8qs76cn5ydwjwvgwkrm731l.nar.zst4431worker2 # [ 61.862697] nix-grpc-daemon[880]: time=2026-09-22T10:50:11.465Z level=INFO msg="Uploading 1 narinfos"4432lb # [ 61.853301] niks3-server[991]: 2026/09/22 10:50:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4433lb # [ 61.867006] niks3-server[991]: 2026/09/22 10:50:11 INFO Signed narinfos id=11 count=14434lb # [ 61.873860] niks3-server[991]: 2026/09/22 10:50:11 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.ls4435worker2 # [ 61.885493] nix-grpc-daemon[880]: time=2026-09-22T10:50:11.488Z level=INFO msg="Upload complete. (486ms)"4436lb # [ 61.886571] niks3-server[991]: 2026/09/22 10:50:11 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4437lb # [ 61.892326] niks3-server[991]: 2026/09/22 10:50:11 INFO Completed upload id=114438worker2 # [ 61.892368] nix-grpc-daemon[780]: ts=2026-09-22T10:50:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=0 paths=1 nar_bytes_in=4324439lb # [ 61.895801] niks3-server[991]: 2026/09/22 10:50:11 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.narinfo4440worker2 # [ 61.900659] nix-grpc-daemon[780]: ts=2026-09-22T10:50:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454664441worker2 # [ 61.912359] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4442worker2 # [ 61.917630] nix-daemon[866]: reaped child process 1099, status = succeeded4443client # [ 61.954789] nix[936]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051...4444worker2 # [ 61.929924] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4445client # [ 62.153485] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share.4446client # Running as unit: share2.service; invocation ID: e7f3e5ee91bc4398ac7b3436435c8b6a4447client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share, in 1.13 seconds)4448worker1 # [ 62.387701] nix-grpc-daemon[774]: ts=2026-09-22T10:50:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:342784449client # [ 62.432489] nix[952]: don't know how to build these paths:4450client # [ 62.433432] nix[952]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4451worker2 # [ 62.417895] nix-grpc-daemon[780]: ts=2026-09-22T10:50:12Z level=debug event=want client=7 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=04452client # [ 62.453732] nix[952]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051...4453worker2 # [ 62.425477] nix-grpc-daemon[780]: ts=2026-09-22T10:50:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454664454client: must succeed: sleep 2; systemctl kill -s INT share14455client # [ 65.582801] systemd[1]: share1.service: Sent signal SIGINT to main process 936 (nix) on client request.4456client # [ 65.591310] nix[936]: shutting downrxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)4457client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.10 seconds)4458client: waiting for success: ! systemctl is-active share14459client # [ 65.607859] nix[936]: error: interrupted by the user4460client # [ 65.669968] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE4461client # [ 65.674680] systemd[1]: share1.service: Failed with result 'exit-code'.4462client # [ 65.680454] systemd[1]: share1.service: Consumed 181ms CPU time over 4.642s wall clock time, 41.3M memory peak, 6.3K incoming IP traffic, 8.2K outgoing IP traffic.4463client: (finished: waiting for success: ! systemctl is-active share1, in 0.09 seconds)4464client: waiting for success: ! systemctl is-active share24465worker2 # [ 66.180653] nix-daemon[866]: reaped child process 1113, status = succeeded4466worker2 # [ 66.364827] nix-daemon[866]: accepted connection from pid 1149, user nix-grpc-daemon (trusted)4467worker2 # [ 66.381880] nix-daemon[866]: reaped child process 1152, status = succeeded4468lb # [ 66.395284] niks3-server[991]: 2026/09/22 10:50:15 INFO Received uploads request method=POST path=/api/pending_closures4469worker2 # [ 66.408395] nix-grpc-daemon[880]: time=2026-09-22T10:50:16.008Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4470worker2 # [ 66.409876] nix-grpc-daemon[880]: time=2026-09-22T10:50:16.008Z level=INFO msg="Uploading fjww0fd4rwk1s35jav3cyal62l24n356-slow-share (120B)"4471lb # [ 66.436127] niks3-server[991]: 2026/09/22 10:50:16 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst4472lb # [ 66.448536] niks3-server[991]: 2026/09/22 10:50:16 INFO Registered completed upload object_key=log/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4473worker2 # [ 66.456282] nix-grpc-daemon[880]: time=2026-09-22T10:50:16.059Z level=INFO msg="Uploading 1 narinfos"4474lb # [ 66.460565] niks3-server[991]: 2026/09/22 10:50:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4475lb # [ 66.463985] niks3-server[991]: 2026/09/22 10:50:16 INFO Signed narinfos id=12 count=14476lb # [ 66.473289] niks3-server[991]: 2026/09/22 10:50:16 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.ls4477worker2 # [ 66.473729] nix-grpc-daemon[880]: time=2026-09-22T10:50:16.076Z level=INFO msg="Upload complete. (331ms)"4478lb # [ 66.478466] niks3-server[991]: 2026/09/22 10:50:16 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4479lb # [ 66.482574] niks3-server[991]: 2026/09/22 10:50:16 INFO Completed upload id=124480lb # [ 66.484484] niks3-server[991]: 2026/09/22 10:50:16 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.narinfo4481worker2 # [ 66.485027] nix-grpc-daemon[780]: ts=2026-09-22T10:50:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=4 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=6 outputs=14482worker2 # [ 66.496260] nix-grpc-daemon[780]: ts=2026-09-22T10:50:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=4 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=6 outputs=14483worker2 # [ 66.502947] nix-daemon[866]: reaped child process 1111, status = succeeded4484client # [ 66.597520] systemd[1]: share2.service: Deactivated successfully.4485client # [ 66.601789] systemd[1]: share2.service: Consumed 174ms CPU time over 4.448s wall clock time, 31.2M memory peak, 5.9K incoming IP traffic, 5.8K outgoing IP traffic.4486client: (finished: waiting for success: ! systemctl is-active share2, in 1.14 seconds)4487client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }4488client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.06 seconds)4489(finished: subtest: the first client leaving does not fail the second, in 6.21 seconds)4490subtest: build hook: nix-daemon with builders = grpc://lb4491client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag hook 2>/tmp/hook.log4492client # [ 67.377709] systemd[1]: Started Nix Daemon.4493client # [ 67.569533] nix-daemon[999]: accepted connection from pid 996, user root (trusted)4494worker1 # [ 67.747086] nix-grpc-daemon[774]: ts=2026-09-22T10:50:17Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:342784495worker2 # [ 67.761428] nix-grpc-daemon[780]: ts=2026-09-22T10:50:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:454664496worker2 # [ 67.775381] nix-grpc-daemon[780]: ts=2026-09-22T10:50:17Z level=debug event=want client=8 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=04497worker2 # [ 67.781358] nix-grpc-daemon[780]: ts=2026-09-22T10:50:17Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.3:50051 clients=14498worker1 # [ 67.812298] nix-grpc-daemon[774]: ts=2026-09-22T10:50:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:342784499worker1 # [ 67.827868] nix-grpc-daemon[774]: ts=2026-09-22T10:50:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:342784500worker1 # [ 67.843082] nix-grpc-daemon[774]: ts=2026-09-22T10:50:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:342784501worker1 # [ 67.858106] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4502worker1 # [ 68.063637] nix-daemon[839]: accepted connection from pid 958, user nix-grpc-daemon (trusted)4503worker1 # [ 68.083337] nix-daemon[839]: reaped child process 961, status = succeeded4504worker1 # [ 68.253315] nix-daemon[839]: accepted connection from pid 963, user nix-grpc-daemon (trusted)4505worker1 # [ 68.271755] nix-daemon[839]: reaped child process 966, status = succeeded4506lb # [ 68.292910] niks3-server[991]: 2026/09/22 10:50:17 INFO Received uploads request method=POST path=/api/pending_closures4507worker1 # [ 68.294306] nix-grpc-daemon[867]: time=2026-09-22T10:50:17.896Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4508worker1 # [ 68.295782] nix-grpc-daemon[867]: time=2026-09-22T10:50:17.896Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)"4509lb # [ 68.322021] niks3-server[991]: 2026/09/22 10:50:17 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst4510lb # [ 68.336592] niks3-server[991]: 2026/09/22 10:50:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4511worker1 # [ 68.335535] nix-grpc-daemon[867]: time=2026-09-22T10:50:17.938Z level=INFO msg="Uploading 1 narinfos"4512lb # [ 68.346635] niks3-server[991]: 2026/09/22 10:50:17 INFO Signed narinfos id=13 count=14513lb # [ 68.352754] niks3-server[991]: 2026/09/22 10:50:17 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls4514lb # [ 68.364178] niks3-server[991]: 2026/09/22 10:50:17 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4515worker1 # [ 68.363051] nix-grpc-daemon[867]: time=2026-09-22T10:50:17.965Z level=INFO msg="Upload complete. (488ms)"4516lb # [ 68.371152] niks3-server[991]: 2026/09/22 10:50:17 INFO Completed upload id=134517lb # [ 68.374838] niks3-server[991]: 2026/09/22 10:50:17 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo4518worker1 # [ 68.372319] nix-grpc-daemon[774]: ts=2026-09-22T10:50:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=0 paths=1 nar_bytes_in=5124519worker1 # [ 68.383787] nix-grpc-daemon[774]: ts=2026-09-22T10:50:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:342784520worker1 # [ 68.393890] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4521worker1 # [ 68.397057] nix-daemon[839]: reaped child process 956, status = succeeded4522worker1 # [ 68.410713] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4523worker1 # [ 69.638801] nix-daemon[839]: reaped child process 970, status = succeeded4524worker1 # [ 69.801240] nix-daemon[839]: accepted connection from pid 986, user nix-grpc-daemon (trusted)4525worker1 # [ 69.817889] nix-daemon[839]: reaped child process 989, status = succeeded4526lb # [ 69.838311] niks3-server[991]: 2026/09/22 10:50:19 INFO Received uploads request method=POST path=/api/pending_closures4527worker1 # [ 69.840522] nix-grpc-daemon[867]: time=2026-09-22T10:50:19.442Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4528worker1 # [ 69.841995] nix-grpc-daemon[867]: time=2026-09-22T10:50:19.442Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)"4529lb # [ 69.874942] niks3-server[991]: 2026/09/22 10:50:19 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv4530lb # [ 69.886641] niks3-server[991]: 2026/09/22 10:50:19 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4531lb # [ 69.897293] niks3-server[991]: 2026/09/22 10:50:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4532worker1 # [ 69.893787] nix-grpc-daemon[867]: time=2026-09-22T10:50:19.496Z level=INFO msg="Uploading 1 narinfos"4533lb # [ 69.904660] niks3-server[991]: 2026/09/22 10:50:19 INFO Signed narinfos id=14 count=14534lb # [ 69.906765] niks3-server[991]: 2026/09/22 10:50:19 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls4535lb # [ 69.916074] niks3-server[991]: 2026/09/22 10:50:19 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4536lb # [ 69.922161] niks3-server[991]: 2026/09/22 10:50:19 INFO Completed upload id=144537worker1 # [ 69.914114] nix-grpc-daemon[867]: time=2026-09-22T10:50:19.516Z level=INFO msg="Upload complete. (289ms)"4538lb # [ 69.924552] niks3-server[991]: 2026/09/22 10:50:19 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo4539worker1 # [ 69.924912] nix-grpc-daemon[774]: ts=2026-09-22T10:50:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=7 outputs=14540worker1 # [ 69.940390] nix-grpc-daemon[774]: ts=2026-09-22T10:50:19Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:342784541worker1 # [ 69.944509] nix-daemon[839]: reaped child process 968, status = succeeded4542worker1 # [ 69.964125] nix-grpc-daemon[774]: ts=2026-09-22T10:50:19Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:342784543worker1 # [ 69.970765] nix-grpc-daemon[774]: ts=2026-09-22T10:50:19Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=0 paths=1 nar_bytes_out=1284544worker1 # [ 70.290670] nix-grpc-daemon[774]: ts=2026-09-22T10:50:19Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:342784545worker1 # [ 70.298034] nix-grpc-daemon[774]: ts=2026-09-22T10:50:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:342784546worker1 # [ 70.304606] nix-grpc-daemon[867]: time=2026-09-22T10:50:19.906Z level=INFO msg="All 1 paths already cached"4547worker2 # [ 70.328524] nix-grpc-daemon[780]: ts=2026-09-22T10:50:19Z level=debug event=want client=9 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=04548worker1 # [ 70.323303] nix-grpc-daemon[774]: ts=2026-09-22T10:50:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:342784549worker2 # [ 70.329014] nix-grpc-daemon[780]: ts=2026-09-22T10:50:19Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.3:50051 clients=14550worker1 # [ 70.339004] nix-grpc-daemon[774]: ts=2026-09-22T10:50:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:342784551worker1 # [ 70.353021] nix-grpc-daemon[867]: time=2026-09-22T10:50:19.955Z level=INFO msg="All 1 paths already cached"4552worker1 # [ 70.362354] nix-grpc-daemon[774]: ts=2026-09-22T10:50:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:342784553worker1 # [ 70.376133] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4554worker1 # [ 70.572634] nix-daemon[839]: accepted connection from pid 993, user nix-grpc-daemon (trusted)4555worker1 # [ 70.589925] nix-daemon[839]: reaped child process 996, status = succeeded4556worker1 # [ 70.762973] nix-daemon[839]: accepted connection from pid 998, user nix-grpc-daemon (trusted)4557worker1 # [ 70.780209] nix-daemon[839]: reaped child process 1001, status = succeeded4558lb # [ 70.799425] niks3-server[991]: 2026/09/22 10:50:20 INFO Received uploads request method=POST path=/api/pending_closures4559worker1 # [ 70.801956] nix-grpc-daemon[867]: time=2026-09-22T10:50:20.404Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4560worker1 # [ 70.803437] nix-grpc-daemon[867]: time=2026-09-22T10:50:20.404Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)"4561lb # [ 70.827124] niks3-server[991]: 2026/09/22 10:50:20 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst4562lb # [ 70.843615] niks3-server[991]: 2026/09/22 10:50:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4563worker1 # [ 70.842651] nix-grpc-daemon[867]: time=2026-09-22T10:50:20.444Z level=INFO msg="Uploading 1 narinfos"4564lb # [ 70.857550] niks3-server[991]: 2026/09/22 10:50:20 INFO Signed narinfos id=15 count=14565worker1 # [ 70.857796] nix-grpc-daemon[867]: time=2026-09-22T10:50:20.460Z level=INFO msg="Upload complete. (464ms)"4566lb # [ 70.867299] niks3-server[991]: 2026/09/22 10:50:20 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls4567lb # [ 70.872147] niks3-server[991]: 2026/09/22 10:50:20 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4568worker1 # [ 70.864129] nix-grpc-daemon[774]: ts=2026-09-22T10:50:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=0 paths=1 nar_bytes_in=6244569lb # [ 70.876576] niks3-server[991]: 2026/09/22 10:50:20 INFO Completed upload id=154570lb # [ 70.878519] niks3-server[991]: 2026/09/22 10:50:20 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo4571worker1 # [ 70.876370] nix-grpc-daemon[774]: ts=2026-09-22T10:50:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:342784572worker1 # [ 70.879620] nix-daemon[839]: reaped child process 991, status = succeeded4573worker1 # [ 70.886669] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4574worker1 # [ 70.899233] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4575lb # [ 71.583319] envoy[927]: conn peer=[2001:db8:1::1]:51364 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2977 tx=814 ms=43204576lb # [ 71.588542] envoy[927]: conn peer=[2001:db8:1::1]:51374 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1443 tx=1085 ms=41444577lb # [ 71.593784] envoy[927]: conn peer=[2001:db8:1::1]:51380 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3413 tx=1717 ms=22824578lb # [ 71.599443] envoy[927]: conn peer=[2001:db8:1::1]:51394 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=333 ms=744579worker1 # [ 71.944181] nix-daemon[839]: reaped child process 1005, status = succeeded4580worker1 # [ 72.112115] nix-daemon[839]: accepted connection from pid 1021, user nix-grpc-daemon (trusted)4581worker1 # [ 72.128526] nix-daemon[839]: reaped child process 1024, status = succeeded4582lb # [ 72.153928] niks3-server[991]: 2026/09/22 10:50:21 INFO Received uploads request method=POST path=/api/pending_closures4583worker1 # [ 72.155417] nix-grpc-daemon[867]: time=2026-09-22T10:50:21.756Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4584worker1 # [ 72.157366] nix-grpc-daemon[867]: time=2026-09-22T10:50:21.756Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)"4585lb # [ 72.183760] niks3-server[991]: 2026/09/22 10:50:21 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst4586lb # [ 72.199420] niks3-server[991]: 2026/09/22 10:50:21 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv4587lb # [ 72.208861] niks3-server[991]: 2026/09/22 10:50:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4588worker1 # [ 72.207617] nix-grpc-daemon[867]: time=2026-09-22T10:50:21.810Z level=INFO msg="Uploading 1 narinfos"4589lb # [ 72.218686] niks3-server[991]: 2026/09/22 10:50:21 INFO Signed narinfos id=16 count=14590lb # [ 72.224254] niks3-server[991]: 2026/09/22 10:50:21 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls4591lb # [ 72.233655] niks3-server[991]: 2026/09/22 10:50:21 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4592lb # [ 72.243191] niks3-server[991]: 2026/09/22 10:50:21 INFO Completed upload id=164593worker1 # [ 72.234026] nix-grpc-daemon[867]: time=2026-09-22T10:50:21.836Z level=INFO msg="Upload complete. (324ms)"4594lb # [ 72.246334] niks3-server[991]: 2026/09/22 10:50:21 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo4595worker1 # [ 72.245259] nix-grpc-daemon[774]: ts=2026-09-22T10:50:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=8 outputs=14596worker1 # [ 72.265292] nix-grpc-daemon[774]: ts=2026-09-22T10:50:21Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:342784597worker1 # [ 72.273671] nix-daemon[839]: reaped child process 1003, status = succeeded4598worker1 # [ 72.281644] nix-grpc-daemon[774]: ts=2026-09-22T10:50:21Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:342784599worker1 # [ 72.287412] nix-grpc-daemon[774]: ts=2026-09-22T10:50:21Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=0 paths=1 nar_bytes_out=1764600worker1 # [ 72.626013] nix-grpc-daemon[774]: ts=2026-09-22T10:50:22Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:342784601worker2 # [ 72.640804] nix-grpc-daemon[780]: ts=2026-09-22T10:50:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:454664602worker2 # [ 72.677849] nix-grpc-daemon[780]: ts=2026-09-22T10:50:22Z level=debug event=want client=10 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=04603worker2 # [ 72.684499] nix-grpc-daemon[780]: ts=2026-09-22T10:50:22Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.3:50051 clients=14604worker1 # [ 72.679460] nix-grpc-daemon[774]: ts=2026-09-22T10:50:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:342784605worker1 # [ 72.695570] nix-grpc-daemon[774]: ts=2026-09-22T10:50:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:342784606worker1 # [ 72.713165] nix-grpc-daemon[867]: time=2026-09-22T10:50:22.315Z level=INFO msg="All 2 paths already cached"4607worker1 # [ 72.723276] nix-grpc-daemon[774]: ts=2026-09-22T10:50:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:342784608worker1 # [ 72.737777] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4609worker1 # [ 72.952679] nix-daemon[839]: accepted connection from pid 1028, user nix-grpc-daemon (trusted)4610worker1 # [ 72.974157] nix-daemon[839]: reaped child process 1031, status = succeeded4611worker1 # [ 73.164263] nix-daemon[839]: accepted connection from pid 1033, user nix-grpc-daemon (trusted)4612worker1 # [ 73.183654] nix-daemon[839]: reaped child process 1036, status = succeeded4613lb # [ 73.205603] niks3-server[991]: 2026/09/22 10:50:22 INFO Received uploads request method=POST path=/api/pending_closures4614worker1 # [ 73.248591] nix-grpc-daemon[867]: time=2026-09-22T10:50:22.850Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4615worker1 # [ 73.250461] nix-grpc-daemon[867]: time=2026-09-22T10:50:22.850Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)"4616lb # [ 73.281665] niks3-server[991]: 2026/09/22 10:50:22 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst4617lb # [ 73.299588] niks3-server[991]: 2026/09/22 10:50:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4618worker1 # [ 73.298906] nix-grpc-daemon[867]: time=2026-09-22T10:50:22.901Z level=INFO msg="Uploading 1 narinfos"4619lb # [ 73.312141] niks3-server[991]: 2026/09/22 10:50:22 INFO Signed narinfos id=17 count=14620lb # [ 73.322233] niks3-server[991]: 2026/09/22 10:50:22 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls4621lb # [ 73.339386] niks3-server[991]: 2026/09/22 10:50:22 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4622lb # [ 73.349654] niks3-server[991]: 2026/09/22 10:50:22 INFO Completed upload id=174623worker1 # [ 73.340571] nix-grpc-daemon[867]: time=2026-09-22T10:50:22.943Z level=INFO msg="Upload complete. (584ms)"4624lb # [ 73.353108] niks3-server[991]: 2026/09/22 10:50:22 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo4625worker1 # [ 73.353449] nix-grpc-daemon[774]: ts=2026-09-22T10:50:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=0 paths=1 nar_bytes_in=7524626worker1 # [ 73.368724] nix-grpc-daemon[774]: ts=2026-09-22T10:50:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:342784627worker1 # [ 73.388254] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4628worker1 # [ 73.400254] nix-daemon[839]: reaped child process 1026, status = succeeded4629worker1 # [ 73.416552] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4630worker1 # [ 74.689869] nix-daemon[839]: reaped child process 1040, status = succeeded4631worker1 # [ 74.906896] nix-daemon[839]: accepted connection from pid 1056, user nix-grpc-daemon (trusted)4632worker1 # [ 74.930001] nix-daemon[839]: reaped child process 1059, status = succeeded4633lb # [ 74.955100] niks3-server[991]: 2026/09/22 10:50:24 INFO Received uploads request method=POST path=/api/pending_closures4634worker1 # [ 74.958544] nix-grpc-daemon[867]: time=2026-09-22T10:50:24.561Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4635worker1 # [ 74.963624] nix-grpc-daemon[867]: time=2026-09-22T10:50:24.563Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)"4636lb # [ 75.021074] niks3-server[991]: 2026/09/22 10:50:24 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst4637lb # [ 75.031726] niks3-server[991]: 2026/09/22 10:50:24 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv4638lb # [ 75.041116] niks3-server[991]: 2026/09/22 10:50:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign4639worker1 # [ 75.038912] nix-grpc-daemon[867]: time=2026-09-22T10:50:24.640Z level=INFO msg="Uploading 1 narinfos"4640lb # [ 75.051398] niks3-server[991]: 2026/09/22 10:50:24 INFO Signed narinfos id=18 count=14641lb # [ 75.060793] niks3-server[991]: 2026/09/22 10:50:24 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls4642lb # [ 75.072096] niks3-server[991]: 2026/09/22 10:50:24 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete4643lb # [ 75.083532] niks3-server[991]: 2026/09/22 10:50:24 INFO Completed upload id=184644worker1 # [ 75.075542] nix-grpc-daemon[867]: time=2026-09-22T10:50:24.678Z level=INFO msg="Upload complete. (441ms)"4645lb # [ 75.086744] niks3-server[991]: 2026/09/22 10:50:24 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo4646worker1 # [ 75.089761] nix-grpc-daemon[774]: ts=2026-09-22T10:50:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=9 outputs=14647worker1 # [ 75.115262] nix-grpc-daemon[774]: ts=2026-09-22T10:50:24Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:342784648worker1 # [ 75.120685] nix-daemon[839]: reaped child process 1038, status = succeeded4649worker2 # [ 75.152721] nix-grpc-daemon[780]: ts=2026-09-22T10:50:24Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:454664650worker2 # [ 75.180699] nix-grpc-daemon[780]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'...4651worker2 # [ 75.206395] nix-grpc-daemon[780]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'...4652worker2 # [ 75.228675] nix-grpc-daemon[780]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'...4653worker2 # [ 75.253394] nix-grpc-daemon[780]: ts=2026-09-22T10:50:24Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=0 paths=1 nar_bytes_out=2324654client # [ 75.436287] nix-daemon[999]: reaped child process 1001, status = succeeded4655client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag hook 2>/tmp/hook.log, in 8.28 seconds)4656client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top4657client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.05 seconds)4658client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4659client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds)4660client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4661client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.04 seconds)4662(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 8.42 seconds)4663subtest: an input only one worker has still reaches the builder4664client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag w1only4665client # this derivation will be built:4666client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv4667client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'...4668client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag w1only, in 0.50 seconds)4669client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure4670client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.22 seconds)4671worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4672worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.25 seconds)4673worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only4674worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.04 seconds)4675client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt a >&24676client # [ 76.825781] nix-daemon[999]: accepted connection from pid 1082, user root (trusted)4677client # this derivation will be built:4678client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4679worker1 # [ 77.044537] nix-grpc-daemon[774]: ts=2026-09-22T10:50:26Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:342784680client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'...4681worker2 # [ 77.058915] nix-grpc-daemon[780]: ts=2026-09-22T10:50:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:454664682client # copying 1 paths...4683client # copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' to 'grpc://lb:50051'...4684worker1 # [ 77.067966] nix-grpc-daemon[774]: ts=2026-09-22T10:50:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:342784685worker1 # [ 77.083738] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4686worker1 # [ 77.289983] nix-daemon[839]: accepted connection from pid 1068, user nix-grpc-daemon (trusted)4687worker1 # [ 77.308748] nix-daemon[839]: reaped child process 1071, status = succeeded4688worker1 # [ 77.334827] nix-grpc-daemon[867]: time=2026-09-22T10:50:26.937Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4689worker1 # [ 77.336415] nix-grpc-daemon[867]: time=2026-09-22T10:50:26.937Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)"4690lb # [ 77.331893] niks3-server[991]: 2026/09/22 10:50:26 INFO Received uploads request method=POST path=/api/pending_closures4691lb # [ 77.361701] niks3-server[991]: 2026/09/22 10:50:26 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4692lb # [ 77.376995] niks3-server[991]: 2026/09/22 10:50:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign4693worker1 # [ 77.374826] nix-grpc-daemon[867]: time=2026-09-22T10:50:26.977Z level=INFO msg="Uploading 1 narinfos"4694lb # [ 77.389932] niks3-server[991]: 2026/09/22 10:50:26 INFO Signed narinfos id=19 count=14695lb # [ 77.397383] niks3-server[991]: 2026/09/22 10:50:26 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls4696lb # [ 77.408668] niks3-server[991]: 2026/09/22 10:50:26 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete4697lb # [ 77.413753] niks3-server[991]: 2026/09/22 10:50:26 INFO Completed upload id=194698lb # [ 77.416881] niks3-server[991]: 2026/09/22 10:50:26 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo4699worker1 # [ 77.405340] nix-grpc-daemon[867]: time=2026-09-22T10:50:27.008Z level=INFO msg="Upload complete. (306ms)"4700worker1 # [ 77.416537] nix-grpc-daemon[774]: ts=2026-09-22T10:50:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=0 paths=1 nar_bytes_in=1204701worker1 # [ 77.429938] nix-daemon[839]: reaped child process 1066, status = succeeded4702worker2 # [ 77.439929] nix-grpc-daemon[780]: ts=2026-09-22T10:50:27Z level=debug event=want client=11 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=04703worker2 # [ 77.447457] nix-grpc-daemon[780]: ts=2026-09-22T10:50:27Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=14704worker2 # [ 77.484919] nix-grpc-daemon[780]: ts=2026-09-22T10:50:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454664705worker2 # [ 77.500823] nix-grpc-daemon[780]: ts=2026-09-22T10:50:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:454664706client # copying 2 paths to 'lb:50051'...4707client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'...4708worker2 # [ 77.521575] nix-grpc-daemon[780]: ts=2026-09-22T10:50:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:454664709client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'...4710worker2 # [ 77.539576] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4711worker2 # [ 77.757519] nix-daemon[866]: accepted connection from pid 1170, user nix-grpc-daemon (trusted)4712worker2 # [ 77.775422] nix-daemon[866]: reaped child process 1173, status = succeeded4713worker2 # [ 77.947893] nix-daemon[866]: accepted connection from pid 1175, user nix-grpc-daemon (trusted)4714worker2 # [ 77.963655] nix-daemon[866]: reaped child process 1178, status = succeeded4715lb # [ 77.977990] niks3-server[991]: 2026/09/22 10:50:27 INFO Received uploads request method=POST path=/api/pending_closures4716lb # [ 77.987478] niks3-server[991]: 2026/09/22 10:50:27 INFO Received uploads request method=POST path=/api/pending_closures4717worker2 # [ 77.997460] nix-grpc-daemon[880]: time=2026-09-22T10:50:27.599Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4718worker2 # [ 77.998941] nix-grpc-daemon[880]: time=2026-09-22T10:50:27.599Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)"4719worker2 # [ 78.000808] nix-grpc-daemon[880]: time=2026-09-22T10:50:27.599Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)"4720lb # [ 78.017129] niks3-server[991]: 2026/09/22 10:50:27 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst4721lb # [ 78.032974] niks3-server[991]: 2026/09/22 10:50:27 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst4722lb # [ 78.046717] niks3-server[991]: 2026/09/22 10:50:27 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls4723lb # [ 78.052949] niks3-server[991]: 2026/09/22 10:50:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign4724lb # [ 78.060375] niks3-server[991]: 2026/09/22 10:50:27 INFO Signed narinfos id=20 count=14725worker2 # [ 78.059660] nix-grpc-daemon[880]: time=2026-09-22T10:50:27.661Z level=INFO msg="Uploading 3 narinfos"4726lb # [ 78.065216] niks3-server[991]: 2026/09/22 10:50:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign4727lb # [ 78.071084] niks3-server[991]: 2026/09/22 10:50:27 INFO Signed narinfos id=21 count=24728lb # [ 78.075084] niks3-server[991]: 2026/09/22 10:50:27 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls4729lb # [ 78.088094] niks3-server[991]: 2026/09/22 10:50:27 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete4730lb # [ 78.098568] niks3-server[991]: 2026/09/22 10:50:27 INFO Completed upload id=204731lb # [ 78.101106] niks3-server[991]: 2026/09/22 10:50:27 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete4732lb # [ 78.103851] niks3-server[991]: 2026/09/22 10:50:27 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo4733lb # [ 78.108662] niks3-server[991]: 2026/09/22 10:50:27 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4734worker2 # [ 78.109025] nix-grpc-daemon[880]: time=2026-09-22T10:50:27.712Z level=INFO msg="Upload complete. (547ms)"4735lb # [ 78.112098] niks3-server[991]: 2026/09/22 10:50:27 INFO Completed upload id=214736lb # [ 78.113738] niks3-server[991]: 2026/09/22 10:50:27 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4737worker2 # [ 78.121821] nix-grpc-daemon[780]: ts=2026-09-22T10:50:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=0 paths=2 nar_bytes_in=9844738worker2 # [ 78.136496] nix-grpc-daemon[780]: ts=2026-09-22T10:50:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454664739worker2 # [ 78.149067] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4740worker2 # [ 78.152725] nix-daemon[866]: reaped child process 1168, status = succeeded4741worker2 # [ 78.167888] nix-grpc-daemon[780]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'...4742client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051...4743worker2 # [ 78.184427] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4744client # uses-input-w1onlya> worker2: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4745client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds4746worker2 # [ 79.011379] nix-daemon[866]: reaped child process 1183, status = succeeded4747worker2 # [ 79.181240] nix-daemon[866]: accepted connection from pid 1199, user nix-grpc-daemon (trusted)4748worker2 # [ 79.202534] nix-daemon[866]: reaped child process 1202, status = succeeded4749lb # [ 79.216856] niks3-server[991]: 2026/09/22 10:50:28 INFO Received uploads request method=POST path=/api/pending_closures4750worker2 # [ 79.316257] nix-grpc-daemon[880]: time=2026-09-22T10:50:28.918Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4751lb # [ 79.344580] niks3-server[991]: 2026/09/22 10:50:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign4752worker2 # [ 79.353019] nix-grpc-daemon[880]: time=2026-09-22T10:50:28.955Z level=INFO msg="Uploading 1 narinfos"4753lb # [ 79.356722] niks3-server[991]: 2026/09/22 10:50:28 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls4754lb # [ 79.365943] niks3-server[991]: 2026/09/22 10:50:28 INFO Signed narinfos id=22 count=14755lb # [ 79.373693] niks3-server[991]: 2026/09/22 10:50:28 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4756lb # [ 79.385171] niks3-server[991]: 2026/09/22 10:50:28 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete4757lb # [ 79.390934] niks3-server[991]: 2026/09/22 10:50:28 INFO Completed upload id=224758worker2 # [ 79.391524] nix-grpc-daemon[880]: time=2026-09-22T10:50:28.994Z level=INFO msg="Upload complete. (391ms)"4759lb # [ 79.396381] niks3-server[991]: 2026/09/22 10:50:28 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo4760client # copying 1 paths...4761worker2 # [ 79.408941] nix-grpc-daemon[780]: ts=2026-09-22T10:50:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=10 outputs=14762client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'...4763worker2 # [ 79.427769] nix-grpc-daemon[780]: ts=2026-09-22T10:50:29Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:454664764worker2 # [ 79.433711] nix-daemon[866]: reaped child process 1180, status = succeeded4765worker1 # [ 79.442057] nix-grpc-daemon[774]: ts=2026-09-22T10:50:29Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:342784766worker1 # [ 79.458897] nix-grpc-daemon[774]: copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'http://lb:5751'...4767worker1 # [ 79.479680] nix-grpc-daemon[774]: ts=2026-09-22T10:50:29Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=0 paths=1 nar_bytes_out=1204768client # [ 79.646176] nix-daemon[999]: reaped child process 1085, status = succeeded4769client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt a >&2, in 3.05 seconds)4770client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt b >&24771client # [ 79.863118] nix-daemon[999]: accepted connection from pid 1104, user root (trusted)4772client # this derivation will be built:4773client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4774worker2 # [ 80.046107] nix-grpc-daemon[780]: ts=2026-09-22T10:50:29Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:454664775client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'...4776worker2 # [ 80.053954] nix-grpc-daemon[780]: ts=2026-09-22T10:50:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:454664777worker2 # [ 80.061763] nix-grpc-daemon[880]: time=2026-09-22T10:50:29.662Z level=INFO msg="All 1 paths already cached"4778worker2 # [ 80.074266] nix-grpc-daemon[780]: ts=2026-09-22T10:50:29Z level=debug event=want client=12 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=04779worker2 # [ 80.081784] nix-grpc-daemon[780]: ts=2026-09-22T10:50:29Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=14780worker1 # [ 80.076061] nix-grpc-daemon[774]: ts=2026-09-22T10:50:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:342784781worker1 # [ 80.092607] nix-grpc-daemon[774]: ts=2026-09-22T10:50:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:342784782client # copying 1 paths to 'lb:50051'...4783client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'...4784worker1 # [ 80.113465] nix-grpc-daemon[774]: ts=2026-09-22T10:50:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:342784785worker1 # [ 80.127976] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4786worker1 # [ 80.151474] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 9 ms (attempt 1/5)4787worker1 # [ 80.167258] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 8 ms (attempt 2/5)4788worker1 # [ 80.182906] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 67 ms (attempt 3/5)4789worker1 # [ 80.257435] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 264 ms (attempt 4/5)4790worker1 # [ 80.530205] nix-grpc-daemon[774]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4791worker1 # [ 80.537173] nix-grpc-daemon[774]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4792worker1 # [ 80.544725] nix-grpc-daemon[774]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'...4793worker1 # [ 80.761605] nix-daemon[839]: accepted connection from pid 1081, user nix-grpc-daemon (trusted)4794worker1 # [ 80.779291] nix-daemon[839]: reaped child process 1084, status = succeeded4795worker1 # [ 80.951127] nix-daemon[839]: accepted connection from pid 1086, user nix-grpc-daemon (trusted)4796worker1 # [ 80.970047] nix-daemon[839]: reaped child process 1089, status = succeeded4797lb # [ 80.990947] niks3-server[991]: 2026/09/22 10:50:30 INFO Received uploads request method=POST path=/api/pending_closures4798worker1 # [ 80.995351] nix-grpc-daemon[867]: time=2026-09-22T10:50:30.597Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4799worker1 # [ 80.997330] nix-grpc-daemon[867]: time=2026-09-22T10:50:30.599Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)"4800lb # [ 81.024197] niks3-server[991]: 2026/09/22 10:50:30 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst4801worker1 # [ 81.030038] nix-grpc-daemon[867]: time=2026-09-22T10:50:30.632Z level=INFO msg="Uploading 1 narinfos"4802lb # [ 81.040608] niks3-server[991]: 2026/09/22 10:50:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign4803lb # [ 81.048721] niks3-server[991]: 2026/09/22 10:50:30 INFO Signed narinfos id=23 count=14804lb # [ 81.051051] niks3-server[991]: 2026/09/22 10:50:30 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls4805lb # [ 81.060848] niks3-server[991]: 2026/09/22 10:50:30 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete4806lb # [ 81.065349] niks3-server[991]: 2026/09/22 10:50:30 INFO Completed upload id=234807worker1 # [ 81.056298] nix-grpc-daemon[867]: time=2026-09-22T10:50:30.659Z level=INFO msg="Upload complete. (491ms)"4808lb # [ 81.067615] niks3-server[991]: 2026/09/22 10:50:30 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo4809worker1 # [ 81.064816] nix-grpc-daemon[774]: ts=2026-09-22T10:50:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=0 paths=1 nar_bytes_in=5684810worker1 # [ 81.080406] nix-grpc-daemon[774]: ts=2026-09-22T10:50:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:342784811worker1 # [ 81.088638] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4812worker1 # [ 81.093923] nix-daemon[839]: reaped child process 1074, status = succeeded4813worker1 # [ 81.105466] nix-daemon[839]: accepted connection from pid 774, user nix-grpc-daemon (trusted)4814client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051...4815client # uses-input-w1onlyb> node-a: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4816lb # [ 81.586177] envoy[927]: conn peer=[2001:db8:1::1]:51410 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3886 tx=1955 ms=20544817lb # [ 81.591291] envoy[927]: conn peer=[2001:db8:1::1]:36640 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=375 ms=674818lb # [ 81.596114] envoy[927]: conn peer=[2001:db8:1::1]:36658 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=438 tx=407 ms=1814819lb # [ 81.600666] envoy[927]: conn peer=[2001:db8:1::1]:36646 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4354 tx=2203 ms=26844820lb # [ 81.606173] envoy[927]: conn peer=[2001:db8:1::1]:36674 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=329 ms=984821lb # [ 81.610912] envoy[927]: conn peer=[2001:db8:1::1]:36662 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4280 tx=1850 ms=24934822client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds4823worker1 # [ 82.133797] nix-daemon[839]: reaped child process 1093, status = succeeded4824worker1 # [ 82.292512] nix-daemon[839]: accepted connection from pid 1109, user nix-grpc-daemon (trusted)4825worker1 # [ 82.310728] nix-daemon[839]: reaped child process 1112, status = succeeded4826lb # [ 82.331695] niks3-server[991]: 2026/09/22 10:50:31 INFO Received uploads request method=POST path=/api/pending_closures4827worker1 # [ 82.334470] nix-grpc-daemon[867]: time=2026-09-22T10:50:31.935Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4828lb # [ 82.360817] niks3-server[991]: 2026/09/22 10:50:31 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4829worker1 # [ 82.362132] nix-grpc-daemon[867]: time=2026-09-22T10:50:31.964Z level=INFO msg="Uploading 1 narinfos"4830lb # [ 82.373620] niks3-server[991]: 2026/09/22 10:50:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign4831lb # [ 82.384412] niks3-server[991]: 2026/09/22 10:50:31 INFO Signed narinfos id=24 count=14832lb # [ 82.389136] niks3-server[991]: 2026/09/22 10:50:31 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls4833lb # [ 82.396715] niks3-server[991]: 2026/09/22 10:50:31 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete4834lb # [ 82.400964] niks3-server[991]: 2026/09/22 10:50:31 INFO Completed upload id=244835worker1 # [ 82.392391] nix-grpc-daemon[867]: time=2026-09-22T10:50:31.995Z level=INFO msg="Upload complete. (274ms)"4836lb # [ 82.403647] niks3-server[991]: 2026/09/22 10:50:31 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo4837client # copying 1 paths...4838worker1 # [ 82.404676] nix-grpc-daemon[774]: ts=2026-09-22T10:50:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=11 outputs=14839worker2 # [ 82.417811] nix-grpc-daemon[780]: ts=2026-09-22T10:50:32Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:454664840worker1 # [ 82.413875] nix-daemon[839]: reaped child process 1091, status = succeeded4841worker2 # [ 82.431025] nix-grpc-daemon[780]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'...4842client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'...4843worker1 # [ 82.468117] nix-grpc-daemon[774]: ts=2026-09-22T10:50:32Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:342784844worker1 # [ 82.473839] nix-grpc-daemon[774]: ts=2026-09-22T10:50:32Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=0 paths=1 nar_bytes_out=1204845client # [ 82.595656] nix-daemon[999]: reaped child process 1107, status = succeeded4846client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt b >&2, in 2.95 seconds)4847client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt c >&24848client # [ 82.806007] nix-daemon[999]: accepted connection from pid 1126, user root (trusted)4849client # this derivation will be built:4850client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4851worker2 # [ 82.987590] nix-grpc-daemon[780]: ts=2026-09-22T10:50:32Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:454664852client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'...4853worker1 # [ 82.983647] nix-grpc-daemon[774]: ts=2026-09-22T10:50:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:342784854worker1 # [ 82.993268] nix-grpc-daemon[867]: time=2026-09-22T10:50:32.596Z level=INFO msg="All 1 paths already cached"4855worker2 # [ 83.019132] nix-grpc-daemon[780]: ts=2026-09-22T10:50:32Z level=debug event=want client=13 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=04856worker2 # [ 83.026925] nix-grpc-daemon[780]: ts=2026-09-22T10:50:32Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=14857worker2 # [ 83.036640] nix-grpc-daemon[780]: ts=2026-09-22T10:50:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454664858worker2 # [ 83.047383] nix-grpc-daemon[780]: ts=2026-09-22T10:50:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:454664859client # copying 1 paths to 'lb:50051'...4860worker2 # [ 83.063623] nix-grpc-daemon[880]: time=2026-09-22T10:50:32.664Z level=INFO msg="All 1 paths already cached"4861client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'...4862worker2 # [ 83.069302] nix-grpc-daemon[780]: ts=2026-09-22T10:50:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:454664863worker2 # [ 83.085114] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4864worker2 # [ 83.294433] nix-daemon[866]: accepted connection from pid 1211, user nix-grpc-daemon (trusted)4865worker2 # [ 83.313154] nix-daemon[866]: reaped child process 1214, status = succeeded4866worker2 # [ 83.489309] nix-daemon[866]: accepted connection from pid 1216, user nix-grpc-daemon (trusted)4867worker2 # [ 83.506822] nix-daemon[866]: reaped child process 1219, status = succeeded4868lb # [ 83.519090] niks3-server[991]: 2026/09/22 10:50:33 INFO Received uploads request method=POST path=/api/pending_closures4869worker2 # [ 83.530719] nix-grpc-daemon[880]: time=2026-09-22T10:50:33.133Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4870worker2 # [ 83.532396] nix-grpc-daemon[880]: time=2026-09-22T10:50:33.133Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)"4871lb # [ 83.548704] niks3-server[991]: 2026/09/22 10:50:33 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst4872worker2 # [ 83.562637] nix-grpc-daemon[880]: time=2026-09-22T10:50:33.165Z level=INFO msg="Uploading 1 narinfos"4873lb # [ 83.563104] niks3-server[991]: 2026/09/22 10:50:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign4874lb # [ 83.574998] niks3-server[991]: 2026/09/22 10:50:33 INFO Signed narinfos id=25 count=14875worker2 # [ 83.584601] nix-grpc-daemon[880]: time=2026-09-22T10:50:33.187Z level=INFO msg="Upload complete. (483ms)"4876lb # [ 83.584784] niks3-server[991]: 2026/09/22 10:50:33 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls4877lb # [ 83.589420] niks3-server[991]: 2026/09/22 10:50:33 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete4878lb # [ 83.593813] niks3-server[991]: 2026/09/22 10:50:33 INFO Completed upload id=254879lb # [ 83.595843] niks3-server[991]: 2026/09/22 10:50:33 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo4880worker2 # [ 83.590784] nix-grpc-daemon[780]: ts=2026-09-22T10:50:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=0 paths=1 nar_bytes_in=5684881worker2 # [ 83.603387] nix-grpc-daemon[780]: ts=2026-09-22T10:50:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:454664882worker2 # [ 83.611742] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4883worker2 # [ 83.614474] nix-daemon[866]: reaped child process 1209, status = succeeded4884client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051...4885worker2 # [ 83.626721] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4886client # uses-input-w1onlyc> worker2: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4887client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds4888worker2 # [ 84.870220] nix-daemon[866]: reaped child process 1223, status = succeeded4889worker2 # [ 85.053600] nix-daemon[866]: accepted connection from pid 1239, user nix-grpc-daemon (trusted)4890worker2 # [ 85.070647] nix-daemon[866]: reaped child process 1242, status = succeeded4891lb # [ 85.082539] niks3-server[991]: 2026/09/22 10:50:34 INFO Received uploads request method=POST path=/api/pending_closures4892worker2 # [ 85.096073] nix-grpc-daemon[880]: time=2026-09-22T10:50:34.695Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4893lb # [ 85.114577] niks3-server[991]: 2026/09/22 10:50:34 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls4894worker2 # [ 85.122776] nix-grpc-daemon[880]: time=2026-09-22T10:50:34.725Z level=INFO msg="Uploading 1 narinfos"4895lb # [ 85.126668] niks3-server[991]: 2026/09/22 10:50:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign4896lb # [ 85.136729] niks3-server[991]: 2026/09/22 10:50:34 INFO Signed narinfos id=26 count=14897lb # [ 85.139363] niks3-server[991]: 2026/09/22 10:50:34 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4898lb # [ 85.147571] niks3-server[991]: 2026/09/22 10:50:34 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete4899lb # [ 85.153823] niks3-server[991]: 2026/09/22 10:50:34 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo4900worker2 # [ 85.158415] nix-grpc-daemon[880]: time=2026-09-22T10:50:34.761Z level=INFO msg="Upload complete. (322ms)"4901lb # [ 85.158488] niks3-server[991]: 2026/09/22 10:50:34 INFO Completed upload id=264902worker2 # [ 85.165812] nix-grpc-daemon[780]: ts=2026-09-22T10:50:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45466 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=12 outputs=14903worker2 # [ 85.175430] nix-daemon[866]: reaped child process 1221, status = succeeded4904client # copying 1 paths...4905worker2 # [ 85.214078] nix-grpc-daemon[780]: ts=2026-09-22T10:50:34Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:454664906client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'...4907worker1 # [ 85.238038] nix-grpc-daemon[774]: ts=2026-09-22T10:50:34Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:342784908worker1 # [ 85.251361] nix-grpc-daemon[774]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'...4909worker1 # [ 85.268227] nix-grpc-daemon[774]: ts=2026-09-22T10:50:34Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34278 duration_s=0 paths=1 nar_bytes_out=1204910client # [ 85.439841] nix-daemon[999]: reaped child process 1129, status = succeeded4911client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.85 seconds)4912(finished: subtest: an input only one worker has still reaches the builder, in 9.86 seconds)4913subtest: upload whose reference the worker lacks is completed from the cache4914client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag viacache4915client # this derivation will be built:4916client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv4917client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'...4918client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag viacache, in 0.45 seconds)4919client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A referrer --argstr tag viacache4920client # this derivation will be built:4921client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv4922client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'...4923client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A referrer --argstr tag viacache, in 0.43 seconds)4924client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure4925client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.20 seconds)4926worker1: must succeed: nix-store --import < /tmp/shared/ref.closure4927worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.25 seconds)4928worker2 # [ 86.832647] systemd[1]: Stopping Nix worker-protocol over gRPC...4929worker2 # [ 87.065346] nix-grpc-daemon[780]: ts=2026-09-22T10:50:36Z level=info event=signal_exit4930lb # [ 87.338714] niks3-server[991]: 2026/09/22 10:50:36 INFO lead: released remote=192.168.1.4:357184931worker2 # [ 88.337320] nix-grpc-daemon[780]: ts=2026-09-22T10:50:37Z level=info event=scheduler_restarting4932worker1 # [ 88.330413] nix-grpc-daemon[774]: ts=2026-09-22T10:50:37Z level=info event=scheduler_restarting addr=lb:500514933worker2 # [ 88.350310] nix-grpc-daemon[780]: ts=2026-09-22T10:50:37Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:454544934worker2 # [ 88.356202] nix-grpc-daemon[780]: ts=2026-09-22T10:50:37Z level=info event=worker_gone addr=192.168.1.3:500514935worker2 # [ 88.360861] nix-grpc-daemon[780]: ts=2026-09-22T10:50:37Z level=info event=scheduler_restarting addr=lb:500514936worker2 # [ 88.365771] nix-grpc-daemon[780]: ts=2026-09-22T10:50:37Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:454544937worker2 # [ 88.370596] nix-grpc-daemon[780]: ts=2026-09-22T10:50:37Z level=info event=worker_gone addr=192.168.1.4:500514938worker2 # [ 88.397734] nix-daemon[866]: reaped child process 868, status = succeeded4939worker1 # [ 88.530592] nix-grpc-daemon[774]: ts=2026-09-22T10:50:38Z level=info event=scheduler_connected addr=lb:500514940worker1 # [ 88.535970] nix-grpc-daemon[774]: ts=2026-09-22T10:50:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4941worker2 # [ 88.574063] nix-grpc-daemon[780]: ts=2026-09-22T10:50:38Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4942worker1 # [ 88.737681] nix-grpc-daemon[774]: ts=2026-09-22T10:50:38Z level=info event=scheduler_connected addr=lb:500514943worker1 # [ 88.742842] nix-grpc-daemon[774]: ts=2026-09-22T10:50:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4944worker1 # [ 88.943683] nix-grpc-daemon[774]: ts=2026-09-22T10:50:38Z level=info event=scheduler_connected addr=lb:500514945worker1 # [ 88.948963] nix-grpc-daemon[774]: ts=2026-09-22T10:50:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4946worker2 # [ 89.011991] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4947worker2 # [ 89.015342] systemd[1]: Stopped Nix worker-protocol over gRPC.4948worker2 # [ 89.026232] systemd[1]: nix-grpc-daemon.service: Consumed 3.301s CPU time over 1min 7.272s wall clock time, 66.3M memory peak, 91.8K incoming IP traffic, 233.6K outgoing IP traffic.4949worker2 # [ 89.036520] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.4950worker2 # [ 89.038444] systemd[1]: Closed Nix worker-protocol over gRPC.4951worker2 # [ 89.041096] systemd[1]: nix-grpc-daemon.socket: Consumed 579us CPU time over 1min 18.108s wall clock time, 256K memory peak, 92.3K incoming IP traffic, 75.4K outgoing IP traffic.4952lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy4953lb: must succeed: curl -sf localhost:9901/clusters4954lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4955lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.08 seconds)4956client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&24957worker1 # [ 89.149671] nix-grpc-daemon[774]: ts=2026-09-22T10:50:38Z level=info event=scheduler_connected addr=lb:500514958worker1 # [ 89.154800] nix-grpc-daemon[774]: ts=2026-09-22T10:50:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4959worker1 # [ 89.357405] nix-grpc-daemon[774]: ts=2026-09-22T10:50:38Z level=info event=scheduler_connected addr=lb:500514960worker1 # [ 89.361473] nix-grpc-daemon[774]: ts=2026-09-22T10:50:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4961worker1 # [ 89.404941] nix-grpc-daemon[774]: ts=2026-09-22T10:50:39Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:342784962worker1 # [ 89.414398] nix-grpc-daemon[774]: ts=2026-09-22T10:50:39Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:342784963client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4964client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.31 seconds)4965worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4966worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.04 seconds)4967worker2 # [ 89.562472] systemd[1]: Listening on Nix worker-protocol over gRPC.4968worker1 # [ 89.558219] nix-grpc-daemon[774]: ts=2026-09-22T10:50:39Z level=info event=scheduler_connected addr=lb:500514969worker1 # [ 89.563338] nix-grpc-daemon[774]: ts=2026-09-22T10:50:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4970worker1 # [ 89.633778] systemd[1]: Stopping Nix worker-protocol over gRPC...4971worker1 # [ 89.764290] nix-grpc-daemon[774]: ts=2026-09-22T10:50:39Z level=info event=scheduler_connected addr=lb:500514972worker1 # [ 89.769441] nix-grpc-daemon[774]: ts=2026-09-22T10:50:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4973worker1 # [ 89.954739] nix-grpc-daemon[774]: ts=2026-09-22T10:50:39Z level=info event=signal_exit4974worker1 # [ 89.970202] nix-grpc-daemon[774]: ts=2026-09-22T10:50:39Z level=info event=scheduler_connected addr=lb:500514975worker1 # [ 89.975183] nix-grpc-daemon[774]: ts=2026-09-22T10:50:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4976worker1 # [ 90.175993] nix-grpc-daemon[774]: ts=2026-09-22T10:50:39Z level=info event=scheduler_connected addr=lb:500514977worker1 # [ 90.181036] nix-grpc-daemon[774]: ts=2026-09-22T10:50:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4978worker2 # [ 90.211896] systemd[1]: Starting scheduler-token.service...4979worker2 # [ 90.317652] systemd[1]: scheduler-token.service: Deactivated successfully.4980worker2 # [ 90.324222] systemd[1]: Finished scheduler-token.service.4981worker2 # [ 90.325843] systemd[1]: scheduler-token.service: Consumed 51ms CPU time over 110ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4982worker2 # [ 90.333885] systemd[1]: Starting Nix worker-protocol over gRPC...4983worker1 # [ 90.383559] nix-grpc-daemon[774]: ts=2026-09-22T10:50:39Z level=info event=scheduler_connected addr=lb:500514984worker1 # [ 90.387703] nix-grpc-daemon[774]: ts=2026-09-22T10:50:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4985worker2 # [ 90.523732] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:40Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4986worker2 # [ 90.526423] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:40Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4987worker2 # [ 90.530596] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:40Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=14988worker2 # [ 90.539473] nix-grpc-daemon[1266]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR4989worker2 # [ 90.553712] nix-grpc-daemon[1266]: E0000 00:00:1790074240.140480 1266 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4990lb # [ 90.557644] niks3-server[991]: 2026/09/22 10:50:40 INFO lead: acquired remote=192.168.1.4:604164991worker2 # [ 90.562529] nix-grpc-daemon[1266]: E0000 00:00:1790074240.142532 1266 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4992worker2 # [ 90.591861] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:40Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4993worker1 # [ 90.584521] nix-grpc-daemon[774]: ts=2026-09-22T10:50:40Z level=info event=scheduler_connected addr=lb:500514994worker2 # [ 90.595759] nix-grpc-daemon[1266]: E0000 00:00:1790074240.159815 1308 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4995worker2 # [ 90.598586] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:40Z level=info event=scheduler_take_over4996worker2 # [ 90.599769] nix-grpc-daemon[1266]: E0000 00:00:1790074240.180212 1308 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4997worker2 # [ 90.608240] nix-grpc-daemon[1266]: E0000 00:00:1790074240.180362 1308 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4998worker2 # [ 90.611010] nix-grpc-daemon[1266]: E0000 00:00:1790074240.194948 1308 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4999worker2 # [ 90.613985] systemd[1]: Started Nix worker-protocol over gRPC.5000worker2 # [ 90.615130] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:40Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:538885001worker2 # [ 90.618613] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:40Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05002worker2 # [ 90.620511] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:40Z level=info event=scheduler_connected addr=lb:500515003worker2 # [ 90.621766] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:40Z level=info event=oidc_jwks provider=mock keys=15004worker2 # [ 90.622979] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:40Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:538885005worker2 # [ 90.624686] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:40Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05006worker1 # [ 91.205724] nix-grpc-daemon[774]: ts=2026-09-22T10:50:40Z level=info event=scheduler_restarting5007worker2 # [ 91.223249] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:40Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:538885008worker2 # [ 91.223613] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:40Z level=info event=worker_gone addr=192.168.1.3:500515009worker1 # [ 91.248803] nix-grpc-daemon[774]: ts=2026-09-22T10:50:40Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5010worker1 # [ 91.265007] nix-daemon[839]: reaped child process 841, status = succeeded5011lb # [ 91.589044] envoy[927]: conn peer=[2001:db8:1::1]:36684 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3739 tx=2013 ms=24525012worker1 # [ 92.067540] nix-grpc-daemon[774]: E0000 00:00:1790074241.669027 819 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true5013worker1 # [ 92.123565] nix-grpc-daemon[774]: E0000 00:00:1790074241.726278 819 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5014lb # [ 91.594110] envoy[927]: conn peer=[2001:db8:1::1]:49634 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=328 ms=275015lb # [ 91.594242] envoy[927]: conn peer=[2001:db8:1::1]:49642 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3738 tx=1874 ms=23495016lb # [ 91.594340] envoy[927]: conn peer=[2001:db8:1::1]:49656 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=329 ms=965017lb # [ 91.594438] envoy[927]: conn peer=[2001:db8:1::4]:53772 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=54305 tx=3161 ms=664395018lb # [ 91.594533] envoy[927]: conn peer=[2001:db8:1::1]:49658 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=275019lb # [ 91.594628] envoy[927]: conn peer=[2001:db8:1::3]:34896 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=12291 tx=3268 ms=692335020worker1 # [ 92.906491] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5021worker1 # [ 92.911230] systemd[1]: Stopped Nix worker-protocol over gRPC.5022worker1 # [ 92.916632] systemd[1]: nix-grpc-daemon.service: Consumed 2.693s CPU time over 1min 11.336s wall clock time, 55.9M memory peak, 69K incoming IP traffic, 125.9K outgoing IP traffic.5023worker1 # [ 92.933182] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5024worker1 # [ 92.938300] systemd[1]: Closed Nix worker-protocol over gRPC.5025worker1 # [ 92.940421] systemd[1]: nix-grpc-daemon.socket: Consumed 837us CPU time over 1min 21.517s wall clock time, 256K memory peak, 76.1K incoming IP traffic, 64.9K outgoing IP traffic.5026lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy5027lb: must succeed: curl -sf localhost:9901/clusters5028lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5029lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.08 seconds)5030client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&25031worker2 # [ 93.311361] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389185032worker2 # [ 93.327013] nix-daemon[866]: accepted connection from pid 1266, user nix-grpc-daemon (trusted)5033worker2 # [ 93.354949] nix-grpc-daemon[1266]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 52 ms (attempt 1/5)5034worker2 # [ 93.414326] nix-grpc-daemon[1266]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 98 ms (attempt 2/5)5035worker2 # [ 93.519384] nix-grpc-daemon[1266]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 117 ms (attempt 3/5)5036worker2 # [ 93.644261] nix-grpc-daemon[1266]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 556 ms (attempt 4/5)5037worker2 # [ 94.201168] nix-grpc-daemon[1266]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5038worker2 # [ 94.206751] nix-grpc-daemon[1266]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5039client # copying 2 paths...5040client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'...5041worker2 # [ 94.230853] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389185042client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'...5043worker2 # [ 94.247823] nix-daemon[866]: accepted connection from pid 1266, user nix-grpc-daemon (trusted)5044worker2 # [ 94.276705] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:43Z level=info event=niks3_push_started pid=13265045worker2 # [ 94.491210] nix-grpc-daemon[1326]: time=2026-09-22T10:50:44.094Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""5046worker2 # [ 94.668663] nix-daemon[866]: accepted connection from pid 1339, user nix-grpc-daemon (trusted)5047worker2 # [ 94.685781] nix-daemon[866]: reaped child process 1342, status = succeeded5048lb # [ 94.701675] niks3-server[991]: 2026/09/22 10:50:44 INFO Received uploads request method=POST path=/api/pending_closures5049lb # [ 94.710379] niks3-server[991]: 2026/09/22 10:50:44 INFO Received uploads request method=POST path=/api/pending_closures5050worker2 # [ 94.720077] nix-grpc-daemon[1326]: time=2026-09-22T10:50:44.322Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5051worker2 # [ 94.721624] nix-grpc-daemon[1326]: time=2026-09-22T10:50:44.322Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)"5052worker2 # [ 94.725489] nix-grpc-daemon[1326]: time=2026-09-22T10:50:44.322Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)"5053lb # [ 94.758528] niks3-server[991]: 2026/09/22 10:50:44 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst5054lb # [ 94.771322] niks3-server[991]: 2026/09/22 10:50:44 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5055lb # [ 94.783843] niks3-server[991]: 2026/09/22 10:50:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5056worker2 # [ 94.791394] nix-grpc-daemon[1326]: time=2026-09-22T10:50:44.393Z level=INFO msg="Uploading 3 narinfos"5057lb # [ 94.791879] niks3-server[991]: 2026/09/22 10:50:44 INFO Signed narinfos id=27 count=15058lb # [ 94.798503] niks3-server[991]: 2026/09/22 10:50:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5059lb # [ 94.805025] niks3-server[991]: 2026/09/22 10:50:44 INFO Signed narinfos id=28 count=25060lb # [ 94.807032] niks3-server[991]: 2026/09/22 10:50:44 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls5061lb # [ 94.815048] niks3-server[991]: 2026/09/22 10:50:44 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls5062lb # [ 94.828059] niks3-server[991]: 2026/09/22 10:50:44 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5063lb # [ 94.836992] niks3-server[991]: 2026/09/22 10:50:44 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5064lb # [ 94.844223] niks3-server[991]: 2026/09/22 10:50:44 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5065lb # [ 94.851295] niks3-server[991]: 2026/09/22 10:50:44 INFO Completed upload id=275066worker2 # [ 94.850982] nix-grpc-daemon[1326]: time=2026-09-22T10:50:44.454Z level=INFO msg="Upload complete. (355ms)"5067lb # [ 94.854775] niks3-server[991]: 2026/09/22 10:50:44 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5068lb # [ 94.858642] niks3-server[991]: 2026/09/22 10:50:44 INFO Completed upload id=285069worker2 # [ 94.865207] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38918 duration_s=0 paths=2 nar_bytes_in=3125070worker2 # [ 94.880232] nix-daemon[866]: reaped child process 1324, status = succeeded5071client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.87 seconds)5072worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache5073worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.03 seconds)5074worker1 # [ 95.016795] systemd[1]: Listening on Nix worker-protocol over gRPC.5075worker1 # [ 95.023543] systemd[1]: Starting Nix worker-protocol over gRPC...5076worker1 # [ 95.220453] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:44Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5077worker1 # [ 95.223367] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:44Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5078worker1 # [ 95.227511] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:44Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15079worker1 # [ 95.236068] nix-grpc-daemon[1136]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5080worker1 # [ 95.237497] nix-grpc-daemon[1136]: E0000 00:00:1790074244.836798 1136 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5081worker1 # [ 95.239755] nix-grpc-daemon[1136]: E0000 00:00:1790074244.838896 1136 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5082worker1 # [ 95.248045] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:44Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5083worker1 # [ 95.251412] systemd[1]: Started Nix worker-protocol over gRPC.5084lb: waiting until exactly no endpoints of aarch64-linux are unhealthy5085lb: must succeed: curl -sf localhost:9901/clusters5086worker1 # [ 95.256778] nix-grpc-daemon[1136]: E0000 00:00:1790074244.859049 1175 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5087worker1 # [ 95.284185] nix-grpc-daemon[1136]: E0000 00:00:1790074244.876493 1175 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5088worker1 # [ 95.286458] nix-grpc-daemon[1136]: E0000 00:00:1790074244.876667 1175 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5089worker1 # [ 95.294000] nix-grpc-daemon[1136]: E0000 00:00:1790074244.895898 1175 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5090worker1 # [ 95.310672] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:44Z level=info event=scheduler_connected addr=lb:500515091worker2 # [ 95.321433] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:44Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:538885092worker2 # [ 95.327102] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:44Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05093lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)5094lb: must succeed: curl -sf localhost:9901/clusters5095lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5096lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.17 seconds)5097(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.01 seconds)5098subtest: a niks3 restart does not cost the leader its role5099lb: must succeed: curl -sf localhost:9901/clusters5100lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5101lb: must succeed: curl -sf localhost:9901/clusters5102lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5103lb # [ 96.581060] niks3-server[991]: 2026/09/22 10:50:46 INFO Shutdown signal received, draining in-flight requests timeout=10s5104lb # [ 96.585047] systemd[1]: Stopping niks3 server...5105worker1 # [ 96.579898] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:46Z level=info event=scheduler_lock_lost status=2005106worker2 # [ 96.589306] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:46Z level=info event=scheduler_lock_lost status=2005107lb # [ 96.603318] niks3-server[991]: 2026/09/22 10:50:46 INFO lead: released remote=192.168.1.4:604165108lb # [ 96.607824] systemd[1]: niks3.service: Deactivated successfully.5109lb # [ 96.609516] systemd[1]: Stopped niks3 server.5110lb # [ 96.610339] systemd[1]: niks3.service: Consumed 566ms CPU time over 1min 10.997s wall clock time, 22.5M memory peak, 80.9K incoming IP traffic, 113.6K outgoing IP traffic.5111lb # [ 96.622013] systemd[1]: Starting niks3 server...5112lb # [ 96.695775] niks3-server[1130]: 2026/09/22 10:50:46 goose: no migrations to run. current version: 202609200000005113lb # [ 96.702122] niks3-server[1130]: 2026/09/22 10:50:46 OK 1_commit_pending_closure.sql (6.86ms)5114lb # [ 96.705550] niks3-server[1130]: 2026/09/22 10:50:46 OK 2_object_stats_trigger.sql (3.11ms)5115lb # [ 96.706931] niks3-server[1130]: 2026/09/22 10:50:46 goose: up to current file version: 25116lb # [ 96.708907] niks3-server[1130]: 2026/09/22 10:50:46 INFO Loaded signing key name=farm-test-1 path=/nix/store/8lzz3wcf17v05vfdp1zcfaa9hipw3f70-key5117lb # [ 96.719969] niks3-server[1130]: 2026/09/22 10:50:46 INFO Read proxy enabled — serving cache objects from S35118lb # [ 96.721317] niks3-server[1130]: 2026/09/22 10:50:46 INFO Using socket-activated listener address=0.0.0.0:57515119lb # [ 96.725685] niks3-server[1130]: 2026/09/22 10:50:46 INFO systemd watchdog enabled interval=15s5120lb # [ 96.726953] niks3-server[1130]: 2026/09/22 10:50:46 INFO Starting HTTP server address=0.0.0.0:57515121lb # [ 96.728345] systemd[1]: Started niks3 server.5122lb: waiting for unit niks3.service5123lb: (finished: waiting for unit niks3.service, in 0.04 seconds)5124lb: must succeed: curl -sf localhost:9901/clusters5125lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5126worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5127worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5128lb: must succeed: curl -sf localhost:9901/clusters5129lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5130lb: must succeed: curl -sf localhost:9901/clusters5131lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5132worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5133worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.07 seconds)5134client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag after-niks3-restart >&25135worker2 # [ 97.339261] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:389185136client # don't know how to build these paths:5137client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5138worker1 # [ 97.345309] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:327785139worker1 # [ 97.357172] nix-daemon[839]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)5140worker1 # [ 97.391674] nix-grpc-daemon[1136]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 49 ms (attempt 1/5)5141worker1 # [ 97.447850] nix-grpc-daemon[1136]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 41 ms (attempt 2/5)5142worker1 # [ 97.496381] nix-grpc-daemon[1136]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 315 ms (attempt 3/5)5143lb # [ 97.601350] niks3-server[1130]: 2026/09/22 10:50:47 INFO lead: acquired remote=192.168.1.4:574225144worker1 # [ 97.819991] nix-grpc-daemon[1136]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 589 ms (attempt 4/5)5145worker1 # [ 98.410192] nix-grpc-daemon[1136]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5146worker1 # [ 98.416160] nix-grpc-daemon[1136]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5147worker2 # [ 98.474954] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:48Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=05148worker2 # [ 98.475254] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:48Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.4:50051 clients=15149worker2 # [ 98.517726] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389185150worker2 # [ 98.534040] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389185151client # copying 1 paths to 'lb:50051'...5152client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'...5153worker2 # [ 98.548104] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389185154worker2 # [ 98.565876] nix-daemon[866]: accepted connection from pid 1266, user nix-grpc-daemon (trusted)5155worker2 # [ 98.783157] nix-daemon[866]: accepted connection from pid 1361, user nix-grpc-daemon (trusted)5156worker2 # [ 98.799880] nix-daemon[866]: reaped child process 1364, status = succeeded5157worker2 # [ 98.974725] nix-daemon[866]: accepted connection from pid 1366, user nix-grpc-daemon (trusted)5158worker2 # [ 98.991156] nix-daemon[866]: reaped child process 1369, status = succeeded5159lb # [ 99.003701] niks3-server[1130]: 2026/09/22 10:50:48 INFO Received uploads request method=POST path=/api/pending_closures5160worker2 # [ 99.020430] nix-grpc-daemon[1326]: time=2026-09-22T10:50:48.622Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5161worker2 # [ 99.021933] nix-grpc-daemon[1326]: time=2026-09-22T10:50:48.623Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)"5162lb # [ 99.049211] niks3-server[1130]: 2026/09/22 10:50:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5163lb # [ 99.063262] niks3-server[1130]: 2026/09/22 10:50:48 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst5164lb # [ 99.085330] niks3-server[1130]: 2026/09/22 10:50:48 INFO Signed narinfos id=29 count=15165worker2 # [ 99.088652] nix-grpc-daemon[1326]: time=2026-09-22T10:50:48.691Z level=INFO msg="Uploading 1 narinfos"5166lb # [ 99.098438] niks3-server[1130]: 2026/09/22 10:50:48 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5167lb # [ 99.108636] niks3-server[1130]: 2026/09/22 10:50:48 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls5168lb # [ 99.113392] niks3-server[1130]: 2026/09/22 10:50:48 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.narinfo5169lb # [ 99.116388] niks3-server[1130]: 2026/09/22 10:50:48 INFO Completed upload id=295170worker2 # [ 99.117167] nix-grpc-daemon[1326]: time=2026-09-22T10:50:48.720Z level=INFO msg="Upload complete. (534ms)"5171worker2 # [ 99.128310] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38918 duration_s=0 paths=1 nar_bytes_in=5445172worker2 # [ 99.144487] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389185173worker2 # [ 99.152701] nix-daemon[866]: accepted connection from pid 1266, user nix-grpc-daemon (trusted)5174worker2 # [ 99.163536] nix-daemon[866]: reaped child process 1359, status = succeeded5175client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051...5176client # farm-a> worker2: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5177worker2 # [ 99.171526] nix-daemon[866]: accepted connection from pid 1266, user nix-grpc-daemon (trusted)5178client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5179client # farm-a> LOG-farm-a-after-niks3-restart5180worker2 # [ 100.396335] nix-daemon[866]: reaped child process 1373, status = succeeded5181worker2 # [ 100.562670] nix-daemon[866]: accepted connection from pid 1389, user nix-grpc-daemon (trusted)5182worker2 # [ 100.580828] nix-daemon[866]: reaped child process 1392, status = succeeded5183lb # [ 100.595661] niks3-server[1130]: 2026/09/22 10:50:50 INFO Received uploads request method=POST path=/api/pending_closures5184worker2 # [ 100.611292] nix-grpc-daemon[1326]: time=2026-09-22T10:50:50.212Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5185worker2 # [ 100.612962] nix-grpc-daemon[1326]: time=2026-09-22T10:50:50.212Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)"5186lb # [ 100.641330] niks3-server[1130]: 2026/09/22 10:50:50 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5187lb # [ 100.652426] niks3-server[1130]: 2026/09/22 10:50:50 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5188worker2 # [ 100.662596] nix-grpc-daemon[1326]: time=2026-09-22T10:50:50.264Z level=INFO msg="Uploading 1 narinfos"5189lb # [ 100.661272] niks3-server[1130]: 2026/09/22 10:50:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5190lb # [ 100.666152] niks3-server[1130]: 2026/09/22 10:50:50 INFO Signed narinfos id=30 count=15191lb # [ 100.670158] niks3-server[1130]: 2026/09/22 10:50:50 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls5192lb # [ 100.676471] niks3-server[1130]: 2026/09/22 10:50:50 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5193lb # [ 100.685590] niks3-server[1130]: 2026/09/22 10:50:50 INFO Completed upload id=305194worker2 # [ 100.686272] nix-grpc-daemon[1326]: time=2026-09-22T10:50:50.289Z level=INFO msg="Upload complete. (302ms)"5195lb # [ 100.688531] niks3-server[1130]: 2026/09/22 10:50:50 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.narinfo5196worker2 # [ 100.704552] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38918 duration_s=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=15197worker2 # [ 100.718882] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:50Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=05198worker2 # [ 100.722521] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:50Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.4:50051 clients=15199worker2 # [ 100.726190] nix-daemon[866]: reaped child process 1371, status = succeeded5200worker2 # [ 100.748741] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389185201worker2 # [ 100.763496] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389185202worker2 # [ 100.779048] nix-grpc-daemon[1326]: time=2026-09-22T10:50:50.382Z level=INFO msg="All 1 paths already cached"5203client # copying 1 paths to 'lb:50051'...5204client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'...5205worker2 # [ 100.789574] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389185206worker2 # [ 100.803231] nix-daemon[866]: accepted connection from pid 1266, user nix-grpc-daemon (trusted)5207worker2 # [ 101.001450] nix-daemon[866]: accepted connection from pid 1398, user nix-grpc-daemon (trusted)5208worker2 # [ 101.018102] nix-daemon[866]: reaped child process 1401, status = succeeded5209worker2 # [ 101.188895] nix-daemon[866]: accepted connection from pid 1403, user nix-grpc-daemon (trusted)5210worker2 # [ 101.206562] nix-daemon[866]: reaped child process 1406, status = succeeded5211lb # [ 101.217940] niks3-server[1130]: 2026/09/22 10:50:50 INFO Received uploads request method=POST path=/api/pending_closures5212worker2 # [ 101.232090] nix-grpc-daemon[1326]: time=2026-09-22T10:50:50.834Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5213worker2 # [ 101.233564] nix-grpc-daemon[1326]: time=2026-09-22T10:50:50.834Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)"5214lb # [ 101.252634] niks3-server[1130]: 2026/09/22 10:50:50 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst5215worker2 # [ 101.265868] nix-grpc-daemon[1326]: time=2026-09-22T10:50:50.868Z level=INFO msg="Uploading 1 narinfos"5216lb # [ 101.265687] niks3-server[1130]: 2026/09/22 10:50:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5217lb # [ 101.271274] niks3-server[1130]: 2026/09/22 10:50:50 INFO Signed narinfos id=31 count=15218lb # [ 101.279662] niks3-server[1130]: 2026/09/22 10:50:50 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5219lb # [ 101.291426] niks3-server[1130]: 2026/09/22 10:50:50 INFO Completed upload id=315220worker2 # [ 101.292045] nix-grpc-daemon[1326]: time=2026-09-22T10:50:50.895Z level=INFO msg="Upload complete. (473ms)"5221lb # [ 101.294072] niks3-server[1130]: 2026/09/22 10:50:50 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls5222lb # [ 101.298306] niks3-server[1130]: 2026/09/22 10:50:50 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo5223worker2 # [ 101.298976] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38918 duration_s=0 paths=1 nar_bytes_in=6565224worker2 # [ 101.311700] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389185225worker2 # [ 101.318243] nix-daemon[866]: accepted connection from pid 1266, user nix-grpc-daemon (trusted)5226worker2 # [ 101.322963] nix-daemon[866]: reaped child process 1396, status = succeeded5227client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051...5228worker2 # [ 101.335368] nix-daemon[866]: accepted connection from pid 1266, user nix-grpc-daemon (trusted)5229client # farm-b> worker2: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5230lb # [ 101.585227] envoy[927]: conn peer=[2001:db8:1::1]:36956 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1006 tx=304 ms=15675231client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5232client # farm-b> LOG-farm-b-after-niks3-restart5233worker2 # [ 102.170445] nix-daemon[866]: reaped child process 1410, status = succeeded5234worker2 # [ 102.359022] nix-daemon[866]: accepted connection from pid 1426, user nix-grpc-daemon (trusted)5235worker2 # [ 102.376201] nix-daemon[866]: reaped child process 1429, status = succeeded5236lb # [ 102.389742] niks3-server[1130]: 2026/09/22 10:50:51 INFO Received uploads request method=POST path=/api/pending_closures5237worker2 # [ 102.401695] nix-grpc-daemon[1326]: time=2026-09-22T10:50:52.004Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5238worker2 # [ 102.403262] nix-grpc-daemon[1326]: time=2026-09-22T10:50:52.004Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)"5239lb # [ 102.429794] niks3-server[1130]: 2026/09/22 10:50:52 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst5240worker2 # [ 102.449700] nix-grpc-daemon[1326]: time=2026-09-22T10:50:52.052Z level=INFO msg="Uploading 1 narinfos"5241lb # [ 102.441192] niks3-server[1130]: 2026/09/22 10:50:52 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5242lb # [ 102.452877] niks3-server[1130]: 2026/09/22 10:50:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5243lb # [ 102.461417] niks3-server[1130]: 2026/09/22 10:50:52 INFO Signed narinfos id=32 count=15244lb # [ 102.465458] niks3-server[1130]: 2026/09/22 10:50:52 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls5245worker2 # [ 102.474284] nix-grpc-daemon[1326]: time=2026-09-22T10:50:52.077Z level=INFO msg="Upload complete. (325ms)"5246lb # [ 102.475317] niks3-server[1130]: 2026/09/22 10:50:52 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5247lb # [ 102.481255] niks3-server[1130]: 2026/09/22 10:50:52 INFO Completed upload id=325248lb # [ 102.484234] niks3-server[1130]: 2026/09/22 10:50:52 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.narinfo5249worker2 # [ 102.489885] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38918 duration_s=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=15250worker2 # [ 102.509639] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:52Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=05251worker2 # [ 102.514464] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:52Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.4:50051 clients=15252worker2 # [ 102.519473] nix-daemon[866]: reaped child process 1408, status = succeeded5253worker2 # [ 102.548945] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389185254worker2 # [ 102.565023] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:389185255worker2 # [ 102.579269] nix-grpc-daemon[1326]: time=2026-09-22T10:50:52.182Z level=INFO msg="All 2 paths already cached"5256client # copying 1 paths to 'lb:50051'...5257client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'...5258worker2 # [ 102.590341] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:389185259worker2 # [ 102.604395] nix-daemon[866]: accepted connection from pid 1266, user nix-grpc-daemon (trusted)5260worker2 # [ 102.802605] nix-daemon[866]: accepted connection from pid 1435, user nix-grpc-daemon (trusted)5261worker2 # [ 102.821209] nix-daemon[866]: reaped child process 1438, status = succeeded5262worker2 # [ 102.993585] nix-daemon[866]: accepted connection from pid 1440, user nix-grpc-daemon (trusted)5263worker2 # [ 103.009959] nix-daemon[866]: reaped child process 1443, status = succeeded5264lb # [ 103.020792] niks3-server[1130]: 2026/09/22 10:50:52 INFO Received uploads request method=POST path=/api/pending_closures5265worker2 # [ 103.031024] nix-grpc-daemon[1326]: time=2026-09-22T10:50:52.633Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5266worker2 # [ 103.032603] nix-grpc-daemon[1326]: time=2026-09-22T10:50:52.634Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)"5267lb # [ 103.051448] niks3-server[1130]: 2026/09/22 10:50:52 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst5268lb # [ 103.061873] niks3-server[1130]: 2026/09/22 10:50:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5269worker2 # [ 103.070625] nix-grpc-daemon[1326]: time=2026-09-22T10:50:52.673Z level=INFO msg="Uploading 1 narinfos"5270lb # [ 103.072709] niks3-server[1130]: 2026/09/22 10:50:52 INFO Signed narinfos id=33 count=15271lb # [ 103.075898] niks3-server[1130]: 2026/09/22 10:50:52 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls5272lb # [ 103.091761] niks3-server[1130]: 2026/09/22 10:50:52 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5273worker2 # [ 103.097236] nix-grpc-daemon[1326]: time=2026-09-22T10:50:52.700Z level=INFO msg="Upload complete. (475ms)"5274lb # [ 103.097498] niks3-server[1130]: 2026/09/22 10:50:52 INFO Completed upload id=335275lb # [ 103.099800] niks3-server[1130]: 2026/09/22 10:50:52 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo5276worker2 # [ 103.104180] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38918 duration_s=0 paths=1 nar_bytes_in=7845277worker2 # [ 103.116615] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:389185278worker2 # [ 103.125926] nix-daemon[866]: accepted connection from pid 1266, user nix-grpc-daemon (trusted)5279worker2 # [ 103.128915] nix-daemon[866]: reaped child process 1433, status = succeeded5280client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051...5281client # farm-top> worker2: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5282worker2 # [ 103.142623] nix-daemon[866]: accepted connection from pid 1266, user nix-grpc-daemon (trusted)5283client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5284client # farm-top> LOG-farm-top-after-niks3-restart5285worker2 # [ 104.190008] nix-daemon[866]: reaped child process 1447, status = succeeded5286worker2 # [ 104.361058] nix-daemon[866]: accepted connection from pid 1463, user nix-grpc-daemon (trusted)5287worker2 # [ 104.378180] nix-daemon[866]: reaped child process 1466, status = succeeded5288lb # [ 104.391232] niks3-server[1130]: 2026/09/22 10:50:53 INFO Received uploads request method=POST path=/api/pending_closures5289worker2 # [ 104.404415] nix-grpc-daemon[1326]: time=2026-09-22T10:50:54.006Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5290worker2 # [ 104.406102] nix-grpc-daemon[1326]: time=2026-09-22T10:50:54.006Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)"5291lb # [ 104.430001] niks3-server[1130]: 2026/09/22 10:50:54 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst5292lb # [ 104.446369] niks3-server[1130]: 2026/09/22 10:50:54 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5293lb # [ 104.454624] niks3-server[1130]: 2026/09/22 10:50:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5294worker2 # [ 104.460696] nix-grpc-daemon[1326]: time=2026-09-22T10:50:54.063Z level=INFO msg="Uploading 1 narinfos"5295lb # [ 104.463116] niks3-server[1130]: 2026/09/22 10:50:54 INFO Signed narinfos id=34 count=15296lb # [ 104.468635] niks3-server[1130]: 2026/09/22 10:50:54 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls5297lb # [ 104.475127] niks3-server[1130]: 2026/09/22 10:50:54 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5298lb # [ 104.481657] niks3-server[1130]: 2026/09/22 10:50:54 INFO Completed upload id=345299worker2 # [ 104.482325] nix-grpc-daemon[1326]: time=2026-09-22T10:50:54.085Z level=INFO msg="Upload complete. (329ms)"5300lb # [ 104.484285] niks3-server[1130]: 2026/09/22 10:50:54 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo5301worker2 # [ 104.497131] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38918 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=15302worker2 # [ 104.512264] nix-daemon[866]: reaped child process 1445, status = succeeded5303client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag after-niks3-restart >&2, in 7.47 seconds)5304client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag after-niks3-restart outPath5305client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag after-niks3-restart outPath, in 0.23 seconds)5306(finished: subtest: a niks3 restart does not cost the leader its role, in 8.34 seconds)5307subtest: scheduler down: the other node takes the lock and keeps it5308lb: must succeed: curl -sf localhost:9901/clusters5309lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5310lb: must succeed: curl -sf localhost:9901/clusters5311lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5312lb: must succeed: curl -sf localhost:9901/clusters5313lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5314lb: must succeed: curl -sf localhost:9901/clusters5315lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5316worker2 # [ 105.093906] systemd[1]: Stopping Nix worker-protocol over gRPC...5317worker2 # [ 105.559494] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:55Z level=info event=signal_exit5318lb # [ 105.813024] niks3-server[1130]: 2026/09/22 10:50:55 INFO lead: released remote=192.168.1.4:574225319worker1 # [ 106.805193] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:56Z level=info event=scheduler_restarting addr=lb:500515320worker2 # [ 106.817040] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:56Z level=info event=scheduler_restarting5321worker2 # [ 106.817436] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:56Z level=info event=scheduler_restarting addr=lb:500515322worker2 # [ 106.817574] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:56Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:538885323worker2 # [ 106.817686] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:56Z level=info event=worker_gone addr=192.168.1.4:500515324worker2 # [ 106.817808] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:56Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:538885325worker2 # [ 106.817910] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:56Z level=info event=worker_gone addr=192.168.1.3:500515326worker2 # [ 106.865547] nix-daemon[866]: reaped child process 1318, status = succeeded5327worker1 # [ 107.004540] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:56Z level=info event=scheduler_connected addr=lb:500515328worker1 # [ 107.009783] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5329worker2 # [ 107.033070] nix-grpc-daemon[1266]: ts=2026-09-22T10:50:56Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5330worker1 # [ 107.210622] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:56Z level=info event=scheduler_connected addr=lb:500515331worker1 # [ 107.215854] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5332worker1 # [ 107.416735] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:57Z level=info event=scheduler_connected addr=lb:500515333worker1 # [ 107.421929] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5334worker1 # [ 107.585937] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:57Z level=info event=scheduler_take_over5335lb # [ 107.594991] niks3-server[1130]: 2026/09/22 10:50:57 INFO lead: acquired remote=192.168.1.3:497005336worker2 # [ 107.601226] nix-grpc-daemon[1266]: E0000 00:00:1790074257.204145 1311 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5337worker2 # [ 107.607969] nix-grpc-daemon[1266]: E0000 00:00:1790074257.204373 1311 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5338worker1 # [ 107.622720] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:57Z level=info event=scheduler_connected addr=lb:500515339worker1 # [ 107.627903] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5340worker1 # [ 107.830557] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:57Z level=info event=scheduler_connected addr=lb:500515341worker1 # [ 107.834653] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5342worker1 # [ 108.031040] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:57Z level=info event=scheduler_connected addr=lb:500515343worker1 # [ 108.040132] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:57Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:445785344worker1 # [ 108.045392] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:57Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05345worker2 # [ 108.947637] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5346worker2 # [ 108.951198] systemd[1]: Stopped Nix worker-protocol over gRPC.5347worker2 # [ 108.956390] systemd[1]: nix-grpc-daemon.service: Consumed 1.522s CPU time over 18.606s wall clock time, 36.4M memory peak, 45.5K incoming IP traffic, 76.4K outgoing IP traffic.5348worker2 # [ 108.967798] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5349worker2 # [ 108.973278] systemd[1]: Closed Nix worker-protocol over gRPC.5350worker2 # [ 108.976882] systemd[1]: nix-grpc-daemon.socket: Consumed 729us CPU time over 19.414s wall clock time, 256K memory peak, 39.4K incoming IP traffic, 29.2K outgoing IP traffic.5351worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5352worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.09 seconds)5353worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5354worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5355client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby >&25356worker1 # [ 109.378965] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:327785357client # don't know how to build these paths:5358client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5359worker1 # [ 109.390720] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:327785360worker1 # [ 109.417795] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:59Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=05361worker1 # [ 109.424886] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:59Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.3:50051 clients=15362worker1 # [ 109.464134] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:327785363worker1 # [ 109.480403] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:327785364client # copying 1 paths to 'lb:50051'...5365client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'...5366worker1 # [ 109.496109] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:327785367worker1 # [ 109.511403] nix-daemon[839]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)5368worker1 # [ 109.538782] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:59Z level=info event=niks3_push_started pid=12095369worker1 # [ 109.756985] nix-grpc-daemon[1209]: time=2026-09-22T10:50:59.359Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""5370worker1 # [ 109.938879] nix-daemon[839]: accepted connection from pid 1222, user nix-grpc-daemon (trusted)5371worker1 # [ 109.960293] nix-daemon[839]: reaped child process 1225, status = succeeded5372worker1 # [ 110.147513] nix-daemon[839]: accepted connection from pid 1227, user nix-grpc-daemon (trusted)5373worker1 # [ 110.164996] nix-daemon[839]: reaped child process 1230, status = succeeded5374lb # [ 110.189429] niks3-server[1130]: 2026/09/22 10:50:59 INFO Received uploads request method=POST path=/api/pending_closures5375worker1 # [ 110.194012] nix-grpc-daemon[1209]: time=2026-09-22T10:50:59.795Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5376worker1 # [ 110.195556] nix-grpc-daemon[1209]: time=2026-09-22T10:50:59.795Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)"5377lb # [ 110.232494] niks3-server[1130]: 2026/09/22 10:50:59 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst5378lb # [ 110.248561] niks3-server[1130]: 2026/09/22 10:50:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5379worker1 # [ 110.250425] nix-grpc-daemon[1209]: time=2026-09-22T10:50:59.852Z level=INFO msg="Uploading 1 narinfos"5380lb # [ 110.262611] niks3-server[1130]: 2026/09/22 10:50:59 INFO Signed narinfos id=35 count=15381lb # [ 110.273141] niks3-server[1130]: 2026/09/22 10:50:59 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls5382lb # [ 110.288831] niks3-server[1130]: 2026/09/22 10:50:59 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5383lb # [ 110.298448] niks3-server[1130]: 2026/09/22 10:50:59 INFO Completed upload id=355384worker1 # [ 110.289649] nix-grpc-daemon[1209]: time=2026-09-22T10:50:59.892Z level=INFO msg="Upload complete. (528ms)"5385lb # [ 110.301691] niks3-server[1130]: 2026/09/22 10:50:59 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo5386worker1 # [ 110.300489] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32778 duration_s=0 paths=1 nar_bytes_in=5205387worker1 # [ 110.320913] nix-grpc-daemon[1136]: ts=2026-09-22T10:50:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:327785388worker1 # [ 110.329760] nix-daemon[839]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)5389worker1 # [ 110.335383] nix-daemon[839]: reaped child process 1207, status = succeeded5390client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051...5391client # farm-a> node-a: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5392worker1 # [ 110.351103] nix-daemon[839]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)5393client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5394lb # [ 111.587515] envoy[927]: conn peer=[2001:db8:1::1]:36970 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9710 tx=4139 ms=71665395lb # [ 111.593042] envoy[927]: conn peer=[2001:db8:1::4]:37642 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=1691 tx=425 ms=164185396client # farm-a> LOG-farm-a-on-standby5397worker1 # [ 111.806658] nix-daemon[839]: reaped child process 1234, status = succeeded5398worker1 # [ 111.984960] nix-daemon[839]: accepted connection from pid 1250, user nix-grpc-daemon (trusted)5399worker1 # [ 112.001568] nix-daemon[839]: reaped child process 1253, status = succeeded5400lb # [ 112.023788] niks3-server[1130]: 2026/09/22 10:51:01 INFO Received uploads request method=POST path=/api/pending_closures5401worker1 # [ 112.027841] nix-grpc-daemon[1209]: time=2026-09-22T10:51:01.627Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5402worker1 # [ 112.030066] nix-grpc-daemon[1209]: time=2026-09-22T10:51:01.627Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)"5403lb # [ 112.064371] niks3-server[1130]: 2026/09/22 10:51:01 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5404lb # [ 112.080735] niks3-server[1130]: 2026/09/22 10:51:01 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5405lb # [ 112.092065] niks3-server[1130]: 2026/09/22 10:51:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5406lb # [ 112.095698] niks3-server[1130]: 2026/09/22 10:51:01 INFO Signed narinfos id=36 count=15407worker1 # [ 112.091730] nix-grpc-daemon[1209]: time=2026-09-22T10:51:01.693Z level=INFO msg="Uploading 1 narinfos"5408lb # [ 112.107642] niks3-server[1130]: 2026/09/22 10:51:01 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls5409lb # [ 112.118297] niks3-server[1130]: 2026/09/22 10:51:01 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5410lb # [ 112.123138] niks3-server[1130]: 2026/09/22 10:51:01 INFO Completed upload id=365411worker1 # [ 112.114381] nix-grpc-daemon[1209]: time=2026-09-22T10:51:01.717Z level=INFO msg="Upload complete. (344ms)"5412lb # [ 112.126420] niks3-server[1130]: 2026/09/22 10:51:01 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo5413worker1 # [ 112.137441] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32778 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=15414worker1 # [ 112.152278] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:01Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=05415worker1 # [ 112.155692] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:01Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.3:50051 clients=15416worker1 # [ 112.160825] nix-daemon[839]: reaped child process 1232, status = succeeded5417worker1 # [ 112.179544] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:327785418worker1 # [ 112.190827] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:327785419client # copying 1 paths to 'lb:50051'...5420worker1 # [ 112.204004] nix-grpc-daemon[1209]: time=2026-09-22T10:51:01.806Z level=INFO msg="All 1 paths already cached"5421client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'...5422worker1 # [ 112.209860] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:327785423worker1 # [ 112.223190] nix-daemon[839]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)5424worker1 # [ 112.421882] nix-daemon[839]: accepted connection from pid 1259, user nix-grpc-daemon (trusted)5425worker1 # [ 112.439999] nix-daemon[839]: reaped child process 1262, status = succeeded5426worker1 # [ 112.615455] nix-daemon[839]: accepted connection from pid 1264, user nix-grpc-daemon (trusted)5427worker1 # [ 112.630956] nix-daemon[839]: reaped child process 1267, status = succeeded5428lb # [ 112.650980] niks3-server[1130]: 2026/09/22 10:51:02 INFO Received uploads request method=POST path=/api/pending_closures5429worker1 # [ 112.652676] nix-grpc-daemon[1209]: time=2026-09-22T10:51:02.255Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5430worker1 # [ 112.654130] nix-grpc-daemon[1209]: time=2026-09-22T10:51:02.255Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)"5431lb # [ 112.677918] niks3-server[1130]: 2026/09/22 10:51:02 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst5432worker1 # [ 112.692670] nix-grpc-daemon[1209]: time=2026-09-22T10:51:02.295Z level=INFO msg="Uploading 1 narinfos"5433lb # [ 112.694090] niks3-server[1130]: 2026/09/22 10:51:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5434lb # [ 112.706353] niks3-server[1130]: 2026/09/22 10:51:02 INFO Signed narinfos id=37 count=15435lb # [ 112.713456] niks3-server[1130]: 2026/09/22 10:51:02 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls5436lb # [ 112.723343] niks3-server[1130]: 2026/09/22 10:51:02 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5437worker1 # [ 112.720634] nix-grpc-daemon[1209]: time=2026-09-22T10:51:02.323Z level=INFO msg="Upload complete. (482ms)"5438lb # [ 112.729771] niks3-server[1130]: 2026/09/22 10:51:02 INFO Completed upload id=375439lb # [ 112.733360] niks3-server[1130]: 2026/09/22 10:51:02 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo5440worker1 # [ 112.728160] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32778 duration_s=0 paths=1 nar_bytes_in=6405441worker1 # [ 112.741773] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:327785442worker1 # [ 112.747822] nix-daemon[839]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)5443worker1 # [ 112.753106] nix-daemon[839]: reaped child process 1257, status = succeeded5444client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051...5445client # farm-b> node-a: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5446worker1 # [ 112.766885] nix-daemon[839]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)5447client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5448client # farm-b> LOG-farm-b-on-standby5449worker1 # [ 113.585782] nix-daemon[839]: reaped child process 1271, status = succeeded5450worker1 # [ 113.740003] nix-daemon[839]: accepted connection from pid 1287, user nix-grpc-daemon (trusted)5451worker1 # [ 113.758232] nix-daemon[839]: reaped child process 1290, status = succeeded5452lb # [ 113.779545] niks3-server[1130]: 2026/09/22 10:51:03 INFO Received uploads request method=POST path=/api/pending_closures5453worker1 # [ 113.781317] nix-grpc-daemon[1209]: time=2026-09-22T10:51:03.383Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5454worker1 # [ 113.782762] nix-grpc-daemon[1209]: time=2026-09-22T10:51:03.383Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)"5455lb # [ 113.813503] niks3-server[1130]: 2026/09/22 10:51:03 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst5456lb # [ 113.825390] niks3-server[1130]: 2026/09/22 10:51:03 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5457lb # [ 113.835366] niks3-server[1130]: 2026/09/22 10:51:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign5458lb # [ 113.841427] niks3-server[1130]: 2026/09/22 10:51:03 INFO Signed narinfos id=38 count=15459worker1 # [ 113.832698] nix-grpc-daemon[1209]: time=2026-09-22T10:51:03.435Z level=INFO msg="Uploading 1 narinfos"5460lb # [ 113.847015] niks3-server[1130]: 2026/09/22 10:51:03 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls5461lb # [ 113.856255] niks3-server[1130]: 2026/09/22 10:51:03 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete5462lb # [ 113.860845] niks3-server[1130]: 2026/09/22 10:51:03 INFO Completed upload id=385463worker1 # [ 113.852790] nix-grpc-daemon[1209]: time=2026-09-22T10:51:03.455Z level=INFO msg="Upload complete. (276ms)"5464lb # [ 113.863168] niks3-server[1130]: 2026/09/22 10:51:03 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo5465worker1 # [ 113.867635] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32778 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=15466worker1 # [ 113.881037] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:03Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=05467worker1 # [ 113.884424] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:03Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.3:50051 clients=15468worker1 # [ 113.887804] nix-daemon[839]: reaped child process 1269, status = succeeded5469worker1 # [ 113.912155] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:327785470worker1 # [ 113.926338] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:327785471worker1 # [ 113.940573] nix-grpc-daemon[1209]: time=2026-09-22T10:51:03.543Z level=INFO msg="All 2 paths already cached"5472client # copying 1 paths to 'lb:50051'...5473client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'...5474worker1 # [ 113.950314] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:327785475worker1 # [ 113.963850] nix-daemon[839]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)5476worker1 # [ 114.155578] nix-daemon[839]: accepted connection from pid 1296, user nix-grpc-daemon (trusted)5477worker1 # [ 114.173967] nix-daemon[839]: reaped child process 1299, status = succeeded5478worker1 # [ 114.347875] nix-daemon[839]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)5479worker1 # [ 114.364535] nix-daemon[839]: reaped child process 1304, status = succeeded5480lb # [ 114.384357] niks3-server[1130]: 2026/09/22 10:51:03 INFO Received uploads request method=POST path=/api/pending_closures5481worker1 # [ 114.385174] nix-grpc-daemon[1209]: time=2026-09-22T10:51:03.987Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5482worker1 # [ 114.386647] nix-grpc-daemon[1209]: time=2026-09-22T10:51:03.987Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)"5483lb # [ 114.414257] niks3-server[1130]: 2026/09/22 10:51:03 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst5484lb # [ 114.426320] niks3-server[1130]: 2026/09/22 10:51:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign5485worker1 # [ 114.425494] nix-grpc-daemon[1209]: time=2026-09-22T10:51:04.028Z level=INFO msg="Uploading 1 narinfos"5486lb # [ 114.436119] niks3-server[1130]: 2026/09/22 10:51:04 INFO Signed narinfos id=39 count=15487lb # [ 114.442068] niks3-server[1130]: 2026/09/22 10:51:04 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls5488lb # [ 114.450539] niks3-server[1130]: 2026/09/22 10:51:04 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete5489lb # [ 114.456620] niks3-server[1130]: 2026/09/22 10:51:04 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo5490lb # [ 114.461728] niks3-server[1130]: 2026/09/22 10:51:04 INFO Completed upload id=395491worker1 # [ 114.453610] nix-grpc-daemon[1209]: time=2026-09-22T10:51:04.056Z level=INFO msg="Upload complete. (474ms)"5492worker1 # [ 114.461665] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32778 duration_s=0 paths=1 nar_bytes_in=7685493worker1 # [ 114.474944] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:327785494worker1 # [ 114.481220] nix-daemon[839]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)5495worker1 # [ 114.485898] nix-daemon[839]: reaped child process 1294, status = succeeded5496client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051...5497client # farm-top> node-a: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5498worker1 # [ 114.498960] nix-daemon[839]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)5499client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5500client # farm-top> LOG-farm-top-on-standby5501worker1 # [ 115.521451] nix-daemon[839]: reaped child process 1308, status = succeeded5502worker1 # [ 115.678244] nix-daemon[839]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)5503worker1 # [ 115.695733] nix-daemon[839]: reaped child process 1327, status = succeeded5504lb # [ 115.716765] niks3-server[1130]: 2026/09/22 10:51:05 INFO Received uploads request method=POST path=/api/pending_closures5505worker1 # [ 115.719402] nix-grpc-daemon[1209]: time=2026-09-22T10:51:05.320Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5506worker1 # [ 115.720992] nix-grpc-daemon[1209]: time=2026-09-22T10:51:05.320Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)"5507lb # [ 115.758076] niks3-server[1130]: 2026/09/22 10:51:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign5508lb # [ 115.775664] niks3-server[1130]: 2026/09/22 10:51:05 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst5509lb # [ 115.784234] niks3-server[1130]: 2026/09/22 10:51:05 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5510worker1 # [ 115.783249] nix-grpc-daemon[1209]: time=2026-09-22T10:51:05.385Z level=INFO msg="Uploading 1 narinfos"5511lb # [ 115.791009] niks3-server[1130]: 2026/09/22 10:51:05 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls5512lb # [ 115.794909] niks3-server[1130]: 2026/09/22 10:51:05 INFO Signed narinfos id=40 count=15513lb # [ 115.801841] niks3-server[1130]: 2026/09/22 10:51:05 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete5514lb # [ 115.811655] niks3-server[1130]: 2026/09/22 10:51:05 INFO Completed upload id=405515worker1 # [ 115.803579] nix-grpc-daemon[1209]: time=2026-09-22T10:51:05.406Z level=INFO msg="Upload complete. (295ms)"5516lb # [ 115.820936] niks3-server[1130]: 2026/09/22 10:51:05 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo5517worker1 # [ 115.816940] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32778 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=15518worker1 # [ 115.828879] nix-daemon[839]: reaped child process 1306, status = succeeded5519client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby >&2, in 6.75 seconds)5520client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby outPath5521client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby outPath, in 0.24 seconds)5522worker2 # [ 116.204813] systemd[1]: Listening on Nix worker-protocol over gRPC.5523worker2 # [ 116.210777] systemd[1]: Starting scheduler-token.service...5524worker2 # [ 116.309324] systemd[1]: scheduler-token.service: Deactivated successfully.5525worker2 # [ 116.316826] systemd[1]: Finished scheduler-token.service.5526worker2 # [ 116.318337] systemd[1]: scheduler-token.service: Consumed 39ms CPU time over 101ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5527worker2 # [ 116.324466] systemd[1]: Starting Nix worker-protocol over gRPC...5528worker2 # [ 116.508209] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:06Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5529worker2 # [ 116.510804] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:06Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5530worker2 # [ 116.514930] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:06Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15531worker2 # [ 116.522370] nix-grpc-daemon[1487]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5532worker2 # [ 116.523833] nix-grpc-daemon[1487]: E0000 00:00:1790074266.124743 1487 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5533worker2 # [ 116.535401] nix-grpc-daemon[1487]: E0000 00:00:1790074266.130933 1487 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5534lb: must succeed: curl -sf localhost:9901/clusters5535worker2 # [ 116.539811] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:06Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5536worker2 # [ 116.573782] systemd[1]: Started Nix worker-protocol over gRPC.5537worker2 # [ 116.574744] nix-grpc-daemon[1487]: E0000 00:00:1790074266.143482 1527 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5538worker2 # [ 116.577206] nix-grpc-daemon[1487]: E0000 00:00:1790074266.158569 1527 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5539worker2 # [ 116.579442] nix-grpc-daemon[1487]: E0000 00:00:1790074266.158750 1527 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5540worker2 # [ 116.581971] nix-grpc-daemon[1487]: E0000 00:00:1790074266.173936 1527 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5541worker2 # [ 116.591136] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:06Z level=info event=scheduler_connected addr=lb:500515542worker1 # [ 116.592162] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:06Z level=info event=oidc_jwks provider=mock keys=15543worker1 # [ 116.597832] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:06Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:445785544worker1 # [ 116.603224] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:06Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05545lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5546worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5547worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5548lb: must succeed: curl -sf localhost:9901/clusters5549lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5550lb: must succeed: curl -sf localhost:9901/clusters5551lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5552worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5553worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5554(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 12.15 seconds)5555subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5556client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs5557client # [ 117.024730] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs.5558client # Running as unit: rs.service; invocation ID: a9a3804f980042408e66ba196b2cc5565559client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs, in 0.10 seconds)5560worker2 # [ 117.258473] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:387805561worker2 # [ 117.269810] nix-daemon[866]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)5562client # [ 117.318687] nix[1252]: don't know how to build these paths:5563client # [ 117.321631] nix[1252]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5564worker1 # [ 117.297537] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:06Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=05565worker1 # [ 117.297831] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:06Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.3:50051 clients=15566worker1 # [ 117.340325] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:327785567worker1 # [ 117.355204] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:327785568client # [ 117.404110] nix[1252]: copying 1 paths to 'lb:50051'...5569client # [ 117.407966] nix[1252]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'...5570worker1 # [ 117.372106] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:327785571worker1 # [ 117.387927] nix-daemon[839]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)5572worker1 # [ 117.586108] nix-daemon[839]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5573worker1 # [ 117.604336] nix-daemon[839]: reaped child process 1343, status = succeeded5574worker1 # [ 117.777405] nix-daemon[839]: accepted connection from pid 1345, user nix-grpc-daemon (trusted)5575worker1 # [ 117.795563] nix-daemon[839]: reaped child process 1348, status = succeeded5576lb # [ 117.815741] niks3-server[1130]: 2026/09/22 10:51:07 INFO Received uploads request method=POST path=/api/pending_closures5577worker1 # [ 117.820376] nix-grpc-daemon[1209]: time=2026-09-22T10:51:07.422Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5578worker1 # [ 117.821810] nix-grpc-daemon[1209]: time=2026-09-22T10:51:07.422Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)"5579lb # [ 117.848072] niks3-server[1130]: 2026/09/22 10:51:07 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst5580lb # [ 117.860906] niks3-server[1130]: 2026/09/22 10:51:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign5581worker1 # [ 117.858818] nix-grpc-daemon[1209]: time=2026-09-22T10:51:07.461Z level=INFO msg="Uploading 1 narinfos"5582lb # [ 117.871302] niks3-server[1130]: 2026/09/22 10:51:07 INFO Signed narinfos id=41 count=15583lb # [ 117.876570] niks3-server[1130]: 2026/09/22 10:51:07 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls5584lb # [ 117.883891] niks3-server[1130]: 2026/09/22 10:51:07 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete5585lb # [ 117.890348] niks3-server[1130]: 2026/09/22 10:51:07 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo5586lb # [ 117.894334] niks3-server[1130]: 2026/09/22 10:51:07 INFO Completed upload id=415587worker1 # [ 117.885912] nix-grpc-daemon[1209]: time=2026-09-22T10:51:07.488Z level=INFO msg="Upload complete. (481ms)"5588worker1 # [ 117.893975] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:32778 duration_s=0 paths=1 nar_bytes_in=4165589worker1 # [ 117.907415] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:327785590worker1 # [ 117.913965] nix-daemon[839]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)5591worker1 # [ 117.918413] nix-daemon[839]: reaped child process 1338, status = succeeded5592worker1 # [ 117.928849] nix-daemon[839]: accepted connection from pid 1136, user nix-grpc-daemon (trusted)5593client # [ 117.963972] nix[1252]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051...5594lb: must succeed: curl -sf localhost:9901/clusters5595lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5596lb: must succeed: curl -sf localhost:9901/clusters5597lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5598lb: must succeed: curl -sf localhost:9901/clusters5599lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5600lb: must succeed: curl -sf localhost:9901/clusters5601lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5602worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5603worker2 # [ 119.940094] systemd-journald[267]: Received client request to rotate journal, rotating.5604worker2 # [ 119.958935] systemd-journald[267]: Vacuuming done, freed 0B of archived journals from /var/log/journal/773f8e80e43b44ba9aa84949c58ca24f.5605worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.09 seconds)5606worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service5607worker1 # [ 120.010617] systemd[1]: Stopping Nix worker-protocol over gRPC...5608worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.05 seconds)5609client: must succeed: sleep 5.05610worker1 # [ 120.255396] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:09Z level=info event=draining5611lb # [ 121.586635] envoy[927]: conn peer=[2001:db8:1::1]:50644 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9649 tx=4094 ms=64495612client: (finished: must succeed: sleep 5.0, in 5.06 seconds)5613worker1 # [ 125.359782] nix-daemon[839]: reaped child process 1352, status = succeeded5614worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5615worker1 # [ 125.517957] nix-daemon[839]: accepted connection from pid 1393, user nix-grpc-daemon (trusted)5616worker1 # [ 125.535781] nix-daemon[839]: reaped child process 1396, status = succeeded5617lb # [ 125.560980] niks3-server[1130]: 2026/09/22 10:51:15 INFO Received uploads request method=POST path=/api/pending_closures5618worker1 # [ 125.564379] nix-grpc-daemon[1209]: time=2026-09-22T10:51:15.165Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5619worker1 # [ 125.565825] nix-grpc-daemon[1209]: time=2026-09-22T10:51:15.165Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)"5620lb # [ 125.602951] niks3-server[1130]: 2026/09/22 10:51:15 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5621lb # [ 125.621929] niks3-server[1130]: 2026/09/22 10:51:15 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5622worker1 # [ 125.621870] nix-grpc-daemon[1209]: time=2026-09-22T10:51:15.224Z level=INFO msg="Uploading 1 narinfos"5623lb # [ 125.630124] niks3-server[1130]: 2026/09/22 10:51:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign5624lb # [ 125.635395] niks3-server[1130]: 2026/09/22 10:51:15 INFO Signed narinfos id=42 count=15625lb # [ 125.641229] niks3-server[1130]: 2026/09/22 10:51:15 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls5626lb # [ 125.648949] niks3-server[1130]: 2026/09/22 10:51:15 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete5627lb # [ 125.653570] niks3-server[1130]: 2026/09/22 10:51:15 INFO Completed upload id=425628worker1 # [ 125.644837] nix-grpc-daemon[1209]: time=2026-09-22T10:51:15.247Z level=INFO msg="Upload complete. (298ms)"5629lb # [ 125.655819] niks3-server[1130]: 2026/09/22 10:51:15 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo5630worker1 # [ 125.657083] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:32778 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=15631worker1 # [ 125.672938] nix-daemon[839]: reaped child process 1350, status = succeeded5632client # [ 125.733165] systemd[1]: rs.service: Deactivated successfully.5633client # [ 125.737532] systemd[1]: rs.service: Consumed 179ms CPU time over 8.712s wall clock time, 41.2M memory peak, 7.1K incoming IP traffic, 8.4K outgoing IP traffic.5634worker1 # [ 126.256831] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:15Z level=info event=signal_exit5635lb # [ 126.828634] niks3-server[1130]: 2026/09/22 10:51:16 INFO lead: released remote=192.168.1.3:497005636worker2 # [ 127.824384] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:17Z level=info event=scheduler_restarting addr=lb:500515637worker1 # [ 127.819089] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:17Z level=info event=scheduler_restarting5638worker1 # [ 127.824790] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:17Z level=info event=scheduler_restarting addr=lb:500515639worker1 # [ 127.829674] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:17Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:445785640worker1 # [ 127.835216] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:17Z level=info event=worker_gone addr=192.168.1.4:500515641worker1 # [ 127.839762] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:17Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:445785642worker1 # [ 127.845036] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:17Z level=info event=worker_gone addr=192.168.1.3:500515643worker1 # [ 127.869109] nix-daemon[839]: reaped child process 1188, status = succeeded5644worker1 # [ 127.984835] nix-grpc-daemon[1136]: E0000 00:00:1790074277.587433 1181 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5645worker1 # [ 127.992718] nix-grpc-daemon[1136]: E0000 00:00:1790074277.587648 1181 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true5646worker2 # [ 128.024744] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:17Z level=info event=scheduler_connected addr=lb:500515647worker2 # [ 128.030851] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5648worker1 # [ 128.038459] nix-grpc-daemon[1136]: ts=2026-09-22T10:51:17Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5649worker2 # [ 128.231644] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:17Z level=info event=scheduler_connected addr=lb:500515650worker2 # [ 128.236936] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5651worker2 # [ 128.437753] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:18Z level=info event=scheduler_connected addr=lb:500515652worker2 # [ 128.442769] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5653worker2 # [ 128.643741] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:18Z level=info event=scheduler_connected addr=lb:500515654worker2 # [ 128.649199] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5655worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.28 seconds)5656client: waiting for success: ! systemctl is-active rs5657client: (finished: waiting for success: ! systemctl is-active rs, in 0.07 seconds)5658client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5659worker1 # [ 128.760929] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5660worker1 # [ 128.764461] systemd[1]: Stopped Nix worker-protocol over gRPC.5661client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds)5662client: must fail: journalctl -u rs | grep -q 'reconnecting'5663worker1 # [ 128.766867] systemd[1]: nix-grpc-daemon.service: Consumed 1.772s CPU time over 33.735s wall clock time, 37.2M memory peak, 46.8K incoming IP traffic, 87.9K outgoing IP traffic.5664worker1 # [ 128.787370] systemd[1]: Starting Nix worker-protocol over gRPC...5665client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5666lb: must succeed: curl -sf localhost:9901/clusters5667worker2 # [ 128.851413] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:18Z level=info event=scheduler_connected addr=lb:500515668worker2 # [ 128.855554] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5669lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5670worker1 # [ 128.981688] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:18Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5671worker1 # [ 128.984248] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:18Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5672worker1 # [ 128.988283] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:18Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15673worker1 # [ 128.996070] nix-grpc-daemon[1403]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5674worker1 # [ 128.997512] nix-grpc-daemon[1403]: E0000 00:00:1790074278.597372 1403 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5675worker1 # [ 129.004361] nix-grpc-daemon[1403]: E0000 00:00:1790074278.603562 1403 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5676worker1 # [ 129.008391] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:18Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5677worker1 # [ 129.011514] systemd[1]: Started Nix worker-protocol over gRPC.5678lb # [ 129.019003] niks3-server[1130]: 2026/09/22 10:51:18 INFO lead: acquired remote=192.168.1.3:395045679worker1 # [ 129.014843] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:18Z level=info event=scheduler_take_over5680worker1 # [ 129.031723] nix-grpc-daemon[1403]: E0000 00:00:1790074278.619496 1445 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5681worker2 # [ 129.052069] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:18Z level=info event=scheduler_connected addr=lb:500515682worker1 # [ 129.049949] nix-grpc-daemon[1403]: E0000 00:00:1790074278.634503 1445 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5683worker2 # [ 129.057000] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5684worker1 # [ 129.052808] nix-grpc-daemon[1403]: E0000 00:00:1790074278.637231 1445 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5685worker1 # [ 129.055262] nix-grpc-daemon[1403]: E0000 00:00:1790074278.651923 1445 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5686worker1 # [ 129.062855] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5687worker2 # [ 129.257906] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:18Z level=info event=scheduler_connected addr=lb:500515688worker2 # [ 129.262902] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5689worker1 # [ 129.264215] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:18Z level=info event=scheduler_connected addr=lb:500515690worker1 # [ 129.269160] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5691worker2 # [ 129.463933] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:19Z level=info event=scheduler_connected addr=lb:500515692worker2 # [ 129.468996] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5693worker1 # [ 129.469885] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:19Z level=info event=scheduler_connected addr=lb:500515694worker1 # [ 129.474987] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5695worker2 # [ 129.670003] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:19Z level=info event=scheduler_connected addr=lb:500515696worker2 # [ 129.675106] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5697worker1 # [ 129.675853] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:19Z level=info event=scheduler_connected addr=lb:500515698worker1 # [ 129.680985] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5699worker2 # [ 129.877625] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:19Z level=info event=scheduler_connected addr=lb:500515700lb: must succeed: curl -sf localhost:9901/clusters5701worker1 # [ 129.880151] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:19Z level=info event=oidc_jwks provider=mock keys=15702worker1 # [ 129.886863] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:19Z level=info event=scheduler_connected addr=lb:500515703worker1 # [ 129.891014] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:19Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:380745704worker1 # [ 129.896367] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:19Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05705worker1 # [ 129.901849] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:19Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:380745706worker1 # [ 129.906596] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:19Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05707lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5708worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5709worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5710(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 13.12 seconds)5711subtest: scheduler restart mid-build: build finishes, no double build5712client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid5713client # [ 130.147483] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid.5714client # Running as unit: mid.service; invocation ID: d54ccbca82f8404bb81cca76605be3e15715client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid, in 0.09 seconds)5716worker1 # [ 130.365965] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:380785717worker1 # [ 130.381721] nix-daemon[839]: accepted connection from pid 1403, user nix-grpc-daemon (trusted)5718client # [ 130.435237] nix[1288]: don't know how to build these paths:5719client # [ 130.437294] nix[1288]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5720worker1 # [ 130.410598] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:20Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05721worker1 # [ 130.416569] nix-grpc-daemon[1403]: ts=2026-09-22T10:51:20Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=15722worker2 # [ 130.465564] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:387805723worker2 # [ 130.484226] nix-grpc-daemon[1487]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 16 ms (attempt 1/5)5724worker2 # [ 130.507617] nix-grpc-daemon[1487]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 33 ms (attempt 2/5)5725worker2 # [ 130.548388] nix-grpc-daemon[1487]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 41 ms (attempt 3/5)5726worker2 # [ 130.596781] nix-grpc-daemon[1487]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 17 ms (attempt 4/5)5727worker2 # [ 130.621332] nix-grpc-daemon[1487]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5728worker2 # [ 130.627372] nix-grpc-daemon[1487]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5729worker2 # [ 130.646017] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:387805730client # [ 130.688264] nix[1288]: copying 1 paths to 'lb:50051'...5731worker2 # [ 130.660751] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:387805732client # [ 130.690485] nix[1288]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'...5733worker2 # [ 130.678027] nix-daemon[866]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)5734worker2 # [ 130.704998] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:20Z level=info event=niks3_push_started pid=16075735worker2 # [ 130.917521] nix-grpc-daemon[1607]: time=2026-09-22T10:51:20.519Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""5736worker2 # [ 131.091386] nix-daemon[866]: accepted connection from pid 1620, user nix-grpc-daemon (trusted)5737worker2 # [ 131.107497] nix-daemon[866]: reaped child process 1623, status = succeeded5738worker2 # [ 131.278427] nix-daemon[866]: accepted connection from pid 1625, user nix-grpc-daemon (trusted)5739worker2 # [ 131.294118] nix-daemon[866]: reaped child process 1628, status = succeeded5740lb # [ 131.307615] niks3-server[1130]: 2026/09/22 10:51:20 INFO Received uploads request method=POST path=/api/pending_closures5741worker2 # [ 131.319760] nix-grpc-daemon[1607]: time=2026-09-22T10:51:20.921Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5742worker2 # [ 131.321463] nix-grpc-daemon[1607]: time=2026-09-22T10:51:20.921Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)"5743lb # [ 131.343594] niks3-server[1130]: 2026/09/22 10:51:20 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst5744worker2 # [ 131.354947] nix-grpc-daemon[1607]: time=2026-09-22T10:51:20.957Z level=INFO msg="Uploading 1 narinfos"5745lb # [ 131.353859] niks3-server[1130]: 2026/09/22 10:51:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign5746lb # [ 131.362797] niks3-server[1130]: 2026/09/22 10:51:20 INFO Signed narinfos id=43 count=15747lb # [ 131.371589] niks3-server[1130]: 2026/09/22 10:51:20 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls5748lb # [ 131.380187] niks3-server[1130]: 2026/09/22 10:51:20 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete5749lb # [ 131.391702] niks3-server[1130]: 2026/09/22 10:51:20 INFO Completed upload id=435750worker2 # [ 131.392746] nix-grpc-daemon[1607]: time=2026-09-22T10:51:20.995Z level=INFO msg="Upload complete. (471ms)"5751lb # [ 131.395089] niks3-server[1130]: 2026/09/22 10:51:20 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo5752worker2 # [ 131.408770] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38780 duration_s=0 paths=1 nar_bytes_in=4245753worker2 # [ 131.422220] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:387805754worker2 # [ 131.430910] nix-daemon[866]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)5755worker2 # [ 131.444607] nix-daemon[866]: reaped child process 1605, status = succeeded5756worker2 # [ 131.449543] nix-daemon[866]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)5757client # [ 131.477370] nix[1288]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5758lb # [ 131.591944] envoy[927]: conn peer=[2001:db8:1::1]:39220 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=1428 ms=84215759lb # [ 131.598075] envoy[927]: conn peer=[2001:db8:1::3]:42722 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2143 tx=780 ms=327185760lb: must succeed: curl -sf localhost:9901/clusters5761lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5762lb: must succeed: curl -sf localhost:9901/clusters5763lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5764worker1: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5765worker1 # [ 132.977223] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1403 (nix-grpc-daemon) on client request.5766worker1 # [ 132.994712] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5767lb # [ 133.009024] niks3-server[1130]: 2026/09/22 10:51:22 INFO lead: released remote=192.168.1.3:395045768worker2 # [ 133.015995] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:22Z level=info event=scheduler_disconnected addr=lb:50051 error="Stream removed (RST_STREAM (Received RST_STREAM with error code 2))"5769client # [ 133.044818] nix[1288]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5770worker1 # [ 133.034144] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5771worker1 # [ 133.048368] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5772worker1 # [ 133.050247] systemd[1]: nix-grpc-daemon.service: Consumed 215ms CPU time over 4.245s wall clock time, 14.3M memory peak, 9.8K incoming IP traffic, 11.5K outgoing IP traffic.5773worker1: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.13 seconds)5774client: must succeed: sleep 5.05775worker1 # [ 133.054777] nix-daemon[839]: reaped child process 1462, status = succeeded5776worker2 # [ 133.216327] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:22Z level=info event=scheduler_connected addr=lb:500515777worker1 # [ 133.211691] systemd[1]: nix-grpc-daemon.service: Scheduled restart job, restart counter is at 1.5778worker1 # [ 133.224536] systemd[1]: Starting Nix worker-protocol over gRPC...5779worker1 # [ 133.436440] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:23Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5780worker1 # [ 133.439283] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:23Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5781worker1 # [ 133.443301] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:23Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15782worker1 # [ 133.450612] nix-grpc-daemon[1486]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5783worker1 # [ 133.460621] nix-grpc-daemon[1486]: E0000 00:00:1790074283.052710 1486 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5784lb # [ 133.470628] niks3-server[1130]: 2026/09/22 10:51:23 INFO lead: acquired remote=192.168.1.3:591905785worker1 # [ 133.464180] nix-grpc-daemon[1486]: E0000 00:00:1790074283.056226 1486 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5786worker1 # [ 133.466733] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:23Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5787worker1 # [ 133.468526] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:23Z level=info event=scheduler_take_over5788worker1 # [ 133.470237] systemd[1]: Started Nix worker-protocol over gRPC.5789worker1 # [ 133.476062] nix-grpc-daemon[1486]: E0000 00:00:1790074283.076650 1528 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5790worker1 # [ 133.504052] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:23Z level=info event=oidc_jwks provider=mock keys=15791worker1 # [ 133.505350] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:23Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:370625792worker1 # [ 133.506992] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:23Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=15793worker1 # [ 133.513857] nix-grpc-daemon[1486]: E0000 00:00:1790074283.096307 1528 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5794worker1 # [ 133.516826] nix-grpc-daemon[1486]: E0000 00:00:1790074283.096437 1528 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5795worker1 # [ 133.519348] nix-grpc-daemon[1486]: E0000 00:00:1790074283.116056 1528 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5796worker1 # [ 133.526603] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:23Z level=info event=scheduler_connected addr=lb:500515797worker1 # [ 133.528468] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:23Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:370625798worker1 # [ 133.530034] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:23Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05799client: (finished: must succeed: sleep 5.0, in 5.06 seconds)5800client: waiting for success: ! systemctl is-active mid5801worker2 # [ 138.479988] nix-daemon[866]: reaped child process 1636, status = succeeded5802worker2 # [ 138.640325] nix-daemon[866]: accepted connection from pid 1669, user nix-grpc-daemon (trusted)5803worker2 # [ 138.657551] nix-daemon[866]: reaped child process 1672, status = succeeded5804lb # [ 138.669630] niks3-server[1130]: 2026/09/22 10:51:28 INFO Received uploads request method=POST path=/api/pending_closures5805worker2 # [ 138.684101] nix-grpc-daemon[1607]: time=2026-09-22T10:51:28.284Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5806worker2 # [ 138.685531] nix-grpc-daemon[1607]: time=2026-09-22T10:51:28.284Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)"5807lb # [ 138.705044] niks3-server[1130]: 2026/09/22 10:51:28 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5808lb # [ 138.721062] niks3-server[1130]: 2026/09/22 10:51:28 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst5809lb # [ 138.735694] niks3-server[1130]: 2026/09/22 10:51:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign5810worker2 # [ 138.737948] nix-grpc-daemon[1607]: time=2026-09-22T10:51:28.339Z level=INFO msg="Uploading 1 narinfos"5811lb # [ 138.744870] niks3-server[1130]: 2026/09/22 10:51:28 INFO Signed narinfos id=44 count=15812lb # [ 138.747395] niks3-server[1130]: 2026/09/22 10:51:28 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls5813worker2 # [ 138.765013] nix-grpc-daemon[1607]: time=2026-09-22T10:51:28.368Z level=INFO msg="Upload complete. (297ms)"5814lb # [ 138.757856] niks3-server[1130]: 2026/09/22 10:51:28 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete5815lb # [ 138.768083] niks3-server[1130]: 2026/09/22 10:51:28 INFO Completed upload id=445816worker2 # [ 138.775051] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38780 duration_s=7 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=15817worker2 # [ 138.790717] nix-daemon[866]: reaped child process 1633, status = succeeded5818client # [ 138.849166] systemd[1]: mid.service: Deactivated successfully.5819client # [ 138.853992] systemd[1]: mid.service: Consumed 176ms CPU time over 8.705s wall clock time, 41.2M memory peak, 7.6K incoming IP traffic, 8.9K outgoing IP traffic.5820client: (finished: waiting for success: ! systemctl is-active mid, in 1.14 seconds)5821client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }5822client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.05 seconds)5823(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 9.55 seconds)5824subtest: interrupting the client stops the build on the worker5825client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr5826client # [ 139.670603] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr.5827client # Running as unit: intr.service; invocation ID: 7537de620a6641698dee0f372f2618465828client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr, in 0.05 seconds)5829worker1 # [ 139.871381] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:29Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:370845830worker1 # [ 139.885333] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)5831client # [ 139.939387] nix[1324]: don't know how to build these paths:5832client # [ 139.940760] nix[1324]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5833worker1 # [ 139.911771] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:29Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=05834worker1 # [ 139.918908] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:29Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.3:50051 clients=15835worker1 # [ 139.960178] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370845836worker1 # [ 139.976119] nix-grpc-daemon[1486]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 12 ms (attempt 1/5)5837worker1 # [ 139.995486] nix-grpc-daemon[1486]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 128 ms (attempt 2/5)5838worker1 # [ 140.130771] nix-grpc-daemon[1486]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 210 ms (attempt 3/5)5839worker1 # [ 140.348362] nix-grpc-daemon[1486]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 400 ms (attempt 4/5)5840worker1 # [ 140.749341] nix-grpc-daemon[1486]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5841worker1 # [ 140.754678] nix-grpc-daemon[1486]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5842worker1 # [ 140.773421] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370845843client # [ 140.824316] nix[1324]: copying 1 paths to 'lb:50051'...5844worker1 # [ 140.788398] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:370845845client # [ 140.824631] nix[1324]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'...5846worker1 # [ 140.807707] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)5847worker1 # [ 140.845898] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:30Z level=info event=niks3_push_started pid=15635848worker1 # [ 141.062327] nix-grpc-daemon[1563]: time=2026-09-22T10:51:30.664Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""5849worker1 # [ 141.232676] nix-daemon[839]: accepted connection from pid 1577, user nix-grpc-daemon (trusted)5850worker1 # [ 141.250769] nix-daemon[839]: reaped child process 1580, status = succeeded5851worker1 # [ 141.421095] nix-daemon[839]: accepted connection from pid 1582, user nix-grpc-daemon (trusted)5852worker1 # [ 141.438735] nix-daemon[839]: reaped child process 1585, status = succeeded5853lb # [ 141.460648] niks3-server[1130]: 2026/09/22 10:51:31 INFO Received uploads request method=POST path=/api/pending_closures5854worker1 # [ 141.466810] nix-grpc-daemon[1563]: time=2026-09-22T10:51:31.068Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5855worker1 # [ 141.468596] nix-grpc-daemon[1563]: time=2026-09-22T10:51:31.068Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)"5856lb # [ 141.509735] niks3-server[1130]: 2026/09/22 10:51:31 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst5857worker1 # [ 141.513057] nix-grpc-daemon[1563]: time=2026-09-22T10:51:31.115Z level=INFO msg="Uploading 1 narinfos"5858lb # [ 141.524161] niks3-server[1130]: 2026/09/22 10:51:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign5859lb # [ 141.533722] niks3-server[1130]: 2026/09/22 10:51:31 INFO Signed narinfos id=45 count=15860worker1 # [ 141.535636] nix-grpc-daemon[1563]: time=2026-09-22T10:51:31.138Z level=INFO msg="Upload complete. (469ms)"5861lb # [ 141.539750] niks3-server[1130]: 2026/09/22 10:51:31 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls5862lb # [ 141.549605] niks3-server[1130]: 2026/09/22 10:51:31 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete5863lb # [ 141.554220] niks3-server[1130]: 2026/09/22 10:51:31 INFO Completed upload id=455864worker1 # [ 141.544928] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=0 paths=1 nar_bytes_in=4245865lb # [ 141.556442] niks3-server[1130]: 2026/09/22 10:51:31 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo5866worker1 # [ 141.559500] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370845867worker1 # [ 141.567890] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)5868worker1 # [ 141.570710] nix-daemon[839]: reaped child process 1558, status = succeeded5869worker1 # [ 141.581402] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)5870client # [ 141.617484] nix[1324]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051...5871lb # [ 141.591103] envoy[927]: conn peer=[2001:db8:1::3]:43162 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=854 tx=362 ms=39535872lb # [ 141.596682] envoy[927]: conn peer=[2001:db8:1::1]:56316 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3091 tx=1496 ms=84205873client: must succeed: systemctl kill -s INT intr5874client # [ 143.438167] systemd[1]: intr.service: Sent signal SIGINT to main process 1324 (nix) on client request.5875worker1 # [ 143.409282] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:33Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5876client: (finished: must succeed: systemctl kill -s INT intr, in 0.07 seconds)5877client # [ 143.447551] nix[1324]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5878client # [ 143.452693] nix[1324]: error: interrupted by the user5879client # [ 143.491039] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE5880client # [ 143.496175] systemd[1]: intr.service: Failed with result 'exit-code'.5881client # [ 143.498336] systemd[1]: intr.service: Consumed 155ms CPU time over 3.824s wall clock time, 41.2M memory peak, 6.4K incoming IP traffic, 8.3K outgoing IP traffic.5882worker1 # [ 143.607930] nix-daemon[839]: reaped child process 1587, status = succeeded5883worker1 # [ 143.612970] nix-daemon[1589]: unexpected Nix daemon error: error: interrupted by the user5884worker1 # [ 143.619806] nix-daemon[839]: reaped child process 1589, status = failed with exit code 15885(finished: subtest: interrupting the client stops the build on the worker, in 5.13 seconds)5886subtest: a deploy mid-build drains: the build finishes, the new generation takes over5887client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw5888client # [ 144.837270] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw.5889client # Running as unit: sw.service; invocation ID: 86bf105c0b044a9baca404e6eb441f765890client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw, in 0.09 seconds)5891client # [ 145.102537] nix[1345]: don't know how to build these paths:5892client # [ 145.103580] nix[1345]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv5893worker2 # [ 145.073375] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:34Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:387805894worker1 # [ 145.081398] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:34Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=05895worker1 # [ 145.087431] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:34Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.4:50051 clients=15896worker2 # [ 145.136655] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:387805897worker2 # [ 145.152965] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:387805898client # [ 145.192558] nix[1345]: copying 1 paths to 'lb:50051'...5899client # [ 145.198193] nix[1345]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'...5900worker2 # [ 145.170269] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:387805901worker2 # [ 145.185388] nix-daemon[866]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)5902worker2 # [ 145.391060] nix-daemon[866]: accepted connection from pid 1704, user nix-grpc-daemon (trusted)5903worker2 # [ 145.409015] nix-daemon[866]: reaped child process 1707, status = succeeded5904worker2 # [ 145.581765] nix-daemon[866]: accepted connection from pid 1709, user nix-grpc-daemon (trusted)5905worker2 # [ 145.597974] nix-daemon[866]: reaped child process 1712, status = succeeded5906lb # [ 145.609985] niks3-server[1130]: 2026/09/22 10:51:35 INFO Received uploads request method=POST path=/api/pending_closures5907worker2 # [ 145.621342] nix-grpc-daemon[1607]: time=2026-09-22T10:51:35.223Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5908worker2 # [ 145.622891] nix-grpc-daemon[1607]: time=2026-09-22T10:51:35.223Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)"5909lb # [ 145.638530] niks3-server[1130]: 2026/09/22 10:51:35 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst5910lb # [ 145.651686] niks3-server[1130]: 2026/09/22 10:51:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign5911worker2 # [ 145.658988] nix-grpc-daemon[1607]: time=2026-09-22T10:51:35.261Z level=INFO msg="Uploading 1 narinfos"5912lb # [ 145.661381] niks3-server[1130]: 2026/09/22 10:51:35 INFO Signed narinfos id=46 count=15913lb # [ 145.670122] niks3-server[1130]: 2026/09/22 10:51:35 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls5914lb # [ 145.682337] niks3-server[1130]: 2026/09/22 10:51:35 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete5915lb # [ 145.687728] niks3-server[1130]: 2026/09/22 10:51:35 INFO Completed upload id=465916worker2 # [ 145.688425] nix-grpc-daemon[1607]: time=2026-09-22T10:51:35.291Z level=INFO msg="Upload complete. (486ms)"5917lb # [ 145.691857] niks3-server[1130]: 2026/09/22 10:51:35 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo5918worker2 # [ 145.700236] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38780 duration_s=0 paths=1 nar_bytes_in=4165919worker2 # [ 145.709632] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:387805920worker2 # [ 145.718777] nix-daemon[866]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)5921worker2 # [ 145.721214] nix-daemon[866]: reaped child process 1702, status = succeeded5922client # [ 145.758390] nix[1345]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051...5923worker2 # [ 145.734342] nix-daemon[866]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)5924worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&25925worker2 # Checking switch inhibitors... done5926worker2 # activating the configuration...5927worker2 # [ 148.271788] nixos[1748]: switching to system configuration /nix/store/h60cql6xyl0psr1b1smnni8r3597yviy-nixos-system-worker2-test5928worker2 # [ 148.274323] systemd[1]: Stopped target Remote File Systems.5929worker2 # [ 148.276937] systemd[1]: Stopped target Local File Systems.5930worker2 # setting up /etc...5931worker2 # [ 149.035343] systemd[1]: Reload requested from client PID 1748 ('.switch-to-conf') (unit backdoor.service)...5932worker2 # [ 149.045252] systemd[1]: Reloading...5933worker2 # [ 149.912529] systemd[1]: Reloading finished in 869 ms.5934worker2 # restarting sysinit-reactivation.target5935worker2 # [ 149.989968] systemd[1]: Starting Re-setup tmpfiles on a system that is already running....5936worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service5937worker2 # [ 150.151086] systemd[1]: Finished Re-setup tmpfiles on a system that is already running..5938worker2 # [ 150.164330] systemd[1]: Reached target Reactivate sysinit units.5939worker2 # [ 150.165192] systemd[1]: Reloading D-Bus System Message Bus...5940worker2 # [ 150.166242] dbus-broker-launch[494]: Caught SIGHUP, trigger reload.5941worker2 # [ 150.167094] systemd[1]: Reloading Nix worker-protocol over gRPC...5942worker2 # [ 150.183222] dbus-broker-launch[494]: Looking up NSS user entry for 'systemd-timesync'...5943worker2 # [ 150.185689] dbus-broker-launch[494]: NSS returned no entry for 'systemd-timesync'5944worker2 # [ 150.187568] dbus-broker-launch[494]: Invalid user-name in /nix/store/602n2r0g7ngz2pj9sxi9qamwpihwymx6-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"5945worker2 # [ 150.214793] systemd[1]: Reloaded Nix worker-protocol over gRPC.5946worker2 # [ 150.238356] systemd[1]: Reloaded D-Bus System Message Bus.5947worker2 # [ 150.246991] systemd[1]: Reached target Local File Systems.5948worker2 # [ 150.250209] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore5949worker2 # [ 150.260427] systemd[1]: Starting Create SUID/SGID Wrappers...5950worker2 # [ 150.272140] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met.5951worker2 # [ 150.273383] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc675952worker2 # [ 150.275431] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore5953worker2 # [ 150.284130] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os5954worker2 # [ 150.285401] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os5955worker2 # [ 150.287219] systemd[1]: Reached target Remote File Systems.5956worker2 # [ 150.289186] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os5957worker2 # [ 150.292428] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki5958worker2 # [ 150.540716] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:40Z level=info event=draining5959worker2 # [ 150.762837] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully.5960worker2 # [ 150.765592] systemd[1]: Finished Create SUID/SGID Wrappers.5961worker2 # [ 150.768469] systemd[1]: Starting linger-users.service...5962worker2 # [ 150.818518] systemd[1]: linger-users.service: Deactivated successfully.5963worker2 # [ 150.819493] systemd[1]: Finished linger-users.service.5964worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service5965worker2 # [ 151.072144] nixos[1748]: finished switching to system configuration /nix/store/h60cql6xyl0psr1b1smnni8r3597yviy-nixos-system-worker2-test5966worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 3.63 seconds)5967worker2: must succeed: pgrep -f 'read -t [9]0 x'5968worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.06 seconds)5969client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain >&25970client # don't know how to build these paths:5971client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5972worker1 # [ 151.399079] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:370845973worker1 # [ 151.407316] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370845974worker1 # [ 151.432739] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:41Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=05975worker1 # [ 151.439494] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:41Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.3:50051 clients=15976worker1 # [ 151.484779] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370845977worker1 # [ 151.499437] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370845978client # copying 1 paths to 'lb:50051'...5979worker1 # [ 151.514882] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:370845980client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'...5981worker1 # [ 151.539280] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)5982lb # [ 151.589961] envoy[927]: conn peer=[2001:db8:1::1]:40050 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2987 tx=826 ms=35695983worker1 # [ 151.745117] nix-daemon[839]: accepted connection from pid 1644, user nix-grpc-daemon (trusted)5984worker1 # [ 151.761918] nix-daemon[839]: reaped child process 1647, status = succeeded5985worker1 # [ 151.935373] nix-daemon[839]: accepted connection from pid 1649, user nix-grpc-daemon (trusted)5986worker1 # [ 151.951571] nix-daemon[839]: reaped child process 1652, status = succeeded5987lb # [ 151.973841] niks3-server[1130]: 2026/09/22 10:51:41 INFO Received uploads request method=POST path=/api/pending_closures5988worker1 # [ 151.974043] nix-grpc-daemon[1563]: time=2026-09-22T10:51:41.576Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5989worker1 # [ 151.975481] nix-grpc-daemon[1563]: time=2026-09-22T10:51:41.576Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)"5990worker1 # [ 152.006905] nix-grpc-daemon[1563]: time=2026-09-22T10:51:41.609Z level=INFO msg="Uploading 1 narinfos"5991lb # [ 152.002931] niks3-server[1130]: 2026/09/22 10:51:41 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst5992lb # [ 152.017667] niks3-server[1130]: 2026/09/22 10:51:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign5993lb # [ 152.028792] niks3-server[1130]: 2026/09/22 10:51:41 INFO Signed narinfos id=47 count=15994lb # [ 152.033991] niks3-server[1130]: 2026/09/22 10:51:41 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls5995worker1 # [ 152.036869] nix-grpc-daemon[1563]: time=2026-09-22T10:51:41.639Z level=INFO msg="Upload complete. (480ms)"5996lb # [ 152.043987] niks3-server[1130]: 2026/09/22 10:51:41 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete5997lb # [ 152.051676] niks3-server[1130]: 2026/09/22 10:51:41 INFO Completed upload id=475998worker1 # [ 152.044109] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=0 paths=1 nar_bytes_in=5285999lb # [ 152.055573] niks3-server[1130]: 2026/09/22 10:51:41 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo6000worker1 # [ 152.055065] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846001worker1 # [ 152.063747] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6002worker1 # [ 152.066634] nix-daemon[839]: reaped child process 1642, status = succeeded6003client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051...6004worker1 # [ 152.077867] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6005client # farm-a> node-a: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv6006client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6007client # farm-a> LOG-farm-a-during-drain6008worker1 # [ 153.303867] nix-daemon[839]: reaped child process 1656, status = succeeded6009worker1 # [ 153.465156] nix-daemon[839]: accepted connection from pid 1672, user nix-grpc-daemon (trusted)6010worker1 # [ 153.484146] nix-daemon[839]: reaped child process 1675, status = succeeded6011lb # [ 153.506138] niks3-server[1130]: 2026/09/22 10:51:43 INFO Received uploads request method=POST path=/api/pending_closures6012worker1 # [ 153.508593] nix-grpc-daemon[1563]: time=2026-09-22T10:51:43.109Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6013worker1 # [ 153.510070] nix-grpc-daemon[1563]: time=2026-09-22T10:51:43.110Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)"6014lb # [ 153.543589] niks3-server[1130]: 2026/09/22 10:51:43 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv6015lb # [ 153.557796] niks3-server[1130]: 2026/09/22 10:51:43 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6016lb # [ 153.569465] niks3-server[1130]: 2026/09/22 10:51:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6017worker1 # [ 153.567163] nix-grpc-daemon[1563]: time=2026-09-22T10:51:43.169Z level=INFO msg="Uploading 1 narinfos"6018lb # [ 153.575619] niks3-server[1130]: 2026/09/22 10:51:43 INFO Signed narinfos id=48 count=16019lb # [ 153.581497] niks3-server[1130]: 2026/09/22 10:51:43 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls6020lb # [ 153.591660] niks3-server[1130]: 2026/09/22 10:51:43 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6021lb # [ 153.598756] niks3-server[1130]: 2026/09/22 10:51:43 INFO Completed upload id=486022worker1 # [ 153.590021] nix-grpc-daemon[1563]: time=2026-09-22T10:51:43.192Z level=INFO msg="Upload complete. (300ms)"6023lb # [ 153.601498] niks3-server[1130]: 2026/09/22 10:51:43 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo6024worker1 # [ 153.602413] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=3 outputs=16025worker1 # [ 153.620166] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:43Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=06026worker1 # [ 153.623769] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:43Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.3:50051 clients=16027worker1 # [ 153.628173] nix-daemon[839]: reaped child process 1654, status = succeeded6028worker1 # [ 153.666199] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846029worker1 # [ 153.681371] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370846030worker1 # [ 153.698106] nix-grpc-daemon[1563]: time=2026-09-22T10:51:43.300Z level=INFO msg="All 1 paths already cached"6031client # copying 1 paths to 'lb:50051'...6032worker1 # [ 153.708305] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:370846033client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'...6034worker1 # [ 153.732291] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6035worker1 # [ 153.936403] nix-daemon[839]: accepted connection from pid 1681, user nix-grpc-daemon (trusted)6036worker1 # [ 153.953776] nix-daemon[839]: reaped child process 1684, status = succeeded6037worker1 # [ 154.127397] nix-daemon[839]: accepted connection from pid 1686, user nix-grpc-daemon (trusted)6038worker1 # [ 154.146349] nix-daemon[839]: reaped child process 1689, status = succeeded6039lb # [ 154.165870] niks3-server[1130]: 2026/09/22 10:51:43 INFO Received uploads request method=POST path=/api/pending_closures6040worker1 # [ 154.166697] nix-grpc-daemon[1563]: time=2026-09-22T10:51:43.769Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6041worker1 # [ 154.168854] nix-grpc-daemon[1563]: time=2026-09-22T10:51:43.771Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)"6042lb # [ 154.194906] niks3-server[1130]: 2026/09/22 10:51:43 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst6043lb # [ 154.210048] niks3-server[1130]: 2026/09/22 10:51:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6044worker1 # [ 154.208221] nix-grpc-daemon[1563]: time=2026-09-22T10:51:43.811Z level=INFO msg="Uploading 1 narinfos"6045lb # [ 154.217574] niks3-server[1130]: 2026/09/22 10:51:43 INFO Signed narinfos id=49 count=16046lb # [ 154.224427] niks3-server[1130]: 2026/09/22 10:51:43 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls6047lb # [ 154.238000] niks3-server[1130]: 2026/09/22 10:51:43 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6048worker1 # [ 154.238927] nix-grpc-daemon[1563]: time=2026-09-22T10:51:43.841Z level=INFO msg="Upload complete. (489ms)"6049lb # [ 154.247195] niks3-server[1130]: 2026/09/22 10:51:43 INFO Completed upload id=496050lb # [ 154.250880] niks3-server[1130]: 2026/09/22 10:51:43 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo6051worker1 # [ 154.248282] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=0 paths=1 nar_bytes_in=6406052worker1 # [ 154.259975] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846053worker1 # [ 154.267380] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6054worker1 # [ 154.273047] nix-daemon[839]: reaped child process 1679, status = succeeded6055client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051...6056client # farm-b> node-a: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv6057worker1 # [ 154.287426] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6058client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6059client # farm-b> LOG-farm-b-during-drain6060worker1 # [ 155.933435] nix-daemon[839]: reaped child process 1693, status = succeeded6061worker1 # [ 156.114719] nix-daemon[839]: accepted connection from pid 1709, user nix-grpc-daemon (trusted)6062worker1 # [ 156.131056] nix-daemon[839]: reaped child process 1712, status = succeeded6063lb # [ 156.152585] niks3-server[1130]: 2026/09/22 10:51:45 INFO Received uploads request method=POST path=/api/pending_closures6064worker1 # [ 156.156869] nix-grpc-daemon[1563]: time=2026-09-22T10:51:45.758Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6065worker1 # [ 156.158435] nix-grpc-daemon[1563]: time=2026-09-22T10:51:45.758Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)"6066lb # [ 156.188985] niks3-server[1130]: 2026/09/22 10:51:45 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv6067lb # [ 156.202403] niks3-server[1130]: 2026/09/22 10:51:45 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst6068lb # [ 156.213122] niks3-server[1130]: 2026/09/22 10:51:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6069worker1 # [ 156.209239] nix-grpc-daemon[1563]: time=2026-09-22T10:51:45.811Z level=INFO msg="Uploading 1 narinfos"6070lb # [ 156.220423] niks3-server[1130]: 2026/09/22 10:51:45 INFO Signed narinfos id=50 count=16071lb # [ 156.225121] niks3-server[1130]: 2026/09/22 10:51:45 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls6072lb # [ 156.231380] niks3-server[1130]: 2026/09/22 10:51:45 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6073lb # [ 156.241560] niks3-server[1130]: 2026/09/22 10:51:45 INFO Completed upload id=506074lb # [ 156.244782] niks3-server[1130]: 2026/09/22 10:51:45 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo6075worker1 # [ 156.233567] nix-grpc-daemon[1563]: time=2026-09-22T10:51:45.836Z level=INFO msg="Upload complete. (335ms)"6076worker1 # [ 156.253035] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=4 outputs=16077worker1 # [ 156.268129] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:45Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=06078worker1 # [ 156.271739] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:45Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.3:50051 clients=16079worker1 # [ 156.276137] nix-daemon[839]: reaped child process 1691, status = succeeded6080worker1 # [ 156.296285] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846081worker1 # [ 156.311120] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370846082client # copying 1 paths to 'lb:50051'...6083worker1 # [ 156.325759] nix-grpc-daemon[1563]: time=2026-09-22T10:51:45.928Z level=INFO msg="All 2 paths already cached"6084client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'...6085worker1 # [ 156.332112] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:370846086worker1 # [ 156.351018] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6087worker1 # [ 156.551199] nix-daemon[839]: accepted connection from pid 1718, user nix-grpc-daemon (trusted)6088worker1 # [ 156.567653] nix-daemon[839]: reaped child process 1721, status = succeeded6089worker1 # [ 156.739493] nix-daemon[839]: accepted connection from pid 1723, user nix-grpc-daemon (trusted)6090worker1 # [ 156.755343] nix-daemon[839]: reaped child process 1726, status = succeeded6091lb # [ 156.776817] niks3-server[1130]: 2026/09/22 10:51:46 INFO Received uploads request method=POST path=/api/pending_closures6092worker1 # [ 156.777031] nix-grpc-daemon[1563]: time=2026-09-22T10:51:46.379Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6093worker1 # [ 156.778499] nix-grpc-daemon[1563]: time=2026-09-22T10:51:46.379Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)"6094lb # [ 156.800373] niks3-server[1130]: 2026/09/22 10:51:46 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst6095lb # [ 156.811708] niks3-server[1130]: 2026/09/22 10:51:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6096worker1 # [ 156.807978] nix-grpc-daemon[1563]: time=2026-09-22T10:51:46.410Z level=INFO msg="Uploading 1 narinfos"6097lb # [ 156.818823] niks3-server[1130]: 2026/09/22 10:51:46 INFO Signed narinfos id=51 count=16098lb # [ 156.823467] niks3-server[1130]: 2026/09/22 10:51:46 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls6099worker1 # [ 156.824217] nix-grpc-daemon[1563]: time=2026-09-22T10:51:46.427Z level=INFO msg="Upload complete. (454ms)"6100lb # [ 156.831632] niks3-server[1130]: 2026/09/22 10:51:46 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6101lb # [ 156.836243] niks3-server[1130]: 2026/09/22 10:51:46 INFO Completed upload id=516102lb # [ 156.838513] niks3-server[1130]: 2026/09/22 10:51:46 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo6103worker1 # [ 156.830707] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=0 paths=1 nar_bytes_in=7686104worker1 # [ 156.838297] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846105worker1 # [ 156.843689] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6106worker1 # [ 156.847486] nix-daemon[839]: reaped child process 1716, status = succeeded6107worker1 # [ 156.856120] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6108client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051...6109client # farm-top> node-a: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6110client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6111client # farm-top> LOG-farm-top-during-drain6112worker1 # [ 158.101394] nix-daemon[839]: reaped child process 1730, status = succeeded6113worker1 # [ 158.285985] nix-daemon[839]: accepted connection from pid 1746, user nix-grpc-daemon (trusted)6114worker1 # [ 158.302787] nix-daemon[839]: reaped child process 1749, status = succeeded6115lb # [ 158.323674] niks3-server[1130]: 2026/09/22 10:51:47 INFO Received uploads request method=POST path=/api/pending_closures6116worker1 # [ 158.327256] nix-grpc-daemon[1563]: time=2026-09-22T10:51:47.928Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6117worker1 # [ 158.328827] nix-grpc-daemon[1563]: time=2026-09-22T10:51:47.928Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)"6118lb # [ 158.357889] niks3-server[1130]: 2026/09/22 10:51:47 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6119lb # [ 158.372490] niks3-server[1130]: 2026/09/22 10:51:47 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst6120worker1 # [ 158.376630] nix-grpc-daemon[1563]: time=2026-09-22T10:51:47.979Z level=INFO msg="Uploading 1 narinfos"6121lb # [ 158.384613] niks3-server[1130]: 2026/09/22 10:51:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6122lb # [ 158.393702] niks3-server[1130]: 2026/09/22 10:51:47 INFO Signed narinfos id=52 count=16123lb # [ 158.400419] niks3-server[1130]: 2026/09/22 10:51:47 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls6124lb # [ 158.407694] niks3-server[1130]: 2026/09/22 10:51:47 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6125lb # [ 158.414789] niks3-server[1130]: 2026/09/22 10:51:47 INFO Completed upload id=526126worker1 # [ 158.406697] nix-grpc-daemon[1563]: time=2026-09-22T10:51:48.009Z level=INFO msg="Upload complete. (339ms)"6127lb # [ 158.417660] niks3-server[1130]: 2026/09/22 10:51:48 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo6128worker1 # [ 158.421101] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=5 outputs=16129worker1 # [ 158.436996] nix-daemon[839]: reaped child process 1728, status = succeeded6130client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain >&2, in 7.33 seconds)6131client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain outPath6132client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain outPath, in 0.23 seconds)6133worker2: must succeed: pkill -f 'read -t [9]0 x'6134worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.06 seconds)6135client: waiting for success: ! systemctl is-active sw6136worker2 # [ 158.994842] nix-daemon[866]: reaped child process 1716, status = succeeded6137worker2 # [ 159.176696] nix-daemon[866]: accepted connection from pid 1888, user nix-grpc-daemon (trusted)6138worker2 # [ 159.195339] nix-daemon[866]: reaped child process 1891, status = succeeded6139lb # [ 159.206490] niks3-server[1130]: 2026/09/22 10:51:48 INFO Received uploads request method=POST path=/api/pending_closures6140worker2 # [ 159.216436] nix-grpc-daemon[1607]: time=2026-09-22T10:51:48.819Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6141worker2 # [ 159.217896] nix-grpc-daemon[1607]: time=2026-09-22T10:51:48.819Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)"6142lb # [ 159.251526] niks3-server[1130]: 2026/09/22 10:51:48 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv6143worker2 # [ 159.261641] nix-grpc-daemon[1607]: time=2026-09-22T10:51:48.864Z level=INFO msg="Uploading 1 narinfos"6144lb # [ 159.266656] niks3-server[1130]: 2026/09/22 10:51:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6145lb # [ 159.275024] niks3-server[1130]: 2026/09/22 10:51:48 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6146lb # [ 159.283825] niks3-server[1130]: 2026/09/22 10:51:48 INFO Signed narinfos id=53 count=16147worker2 # [ 159.287053] nix-grpc-daemon[1607]: time=2026-09-22T10:51:48.890Z level=INFO msg="Upload complete. (329ms)"6148lb # [ 159.287144] niks3-server[1130]: 2026/09/22 10:51:48 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls6149lb # [ 159.292511] niks3-server[1130]: 2026/09/22 10:51:48 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6150lb # [ 159.296230] niks3-server[1130]: 2026/09/22 10:51:48 INFO Completed upload id=536151worker2 # [ 159.296851] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38780 duration_s=13 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=2 outputs=16152lb # [ 159.298864] niks3-server[1130]: 2026/09/22 10:51:48 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo6153worker2 # [ 159.309118] nix-daemon[866]: reaped child process 1714, status = succeeded6154client # [ 159.372573] systemd[1]: sw.service: Deactivated successfully.6155client # [ 159.377923] systemd[1]: sw.service: Consumed 183ms CPU time over 14.539s wall clock time, 41.2M memory peak, 7.1K incoming IP traffic, 8.4K outgoing IP traffic.6156worker2 # [ 159.543370] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:49Z level=info event=signal_exit6157client: (finished: waiting for success: ! systemctl is-active sw, in 1.12 seconds)6158client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6159client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.05 seconds)6160worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6161worker2 # [ 160.027237] systemd[1]: Starting scheduler-token.service...6162worker2 # [ 160.114931] systemd[1]: scheduler-token.service: Deactivated successfully.6163worker2 # [ 160.117875] systemd[1]: Finished scheduler-token.service.6164worker2 # [ 160.118964] systemd[1]: scheduler-token.service: Consumed 34ms CPU time over 90ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6165worker1 # [ 160.774973] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:50Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:370626166worker2 # [ 160.784920] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:50Z level=info event=scheduler_restarting6167worker2 # [ 160.809885] nix-grpc-daemon[1487]: ts=2026-09-22T10:51:50Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6168worker2 # [ 160.833560] nix-daemon[866]: reaped child process 1546, status = succeeded6169worker2 # [ 160.998026] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6170worker2 # [ 161.003841] systemd[1]: nix-grpc-daemon.service: Consumed 1.120s CPU time over 44.672s wall clock time, 34.5M memory peak, 27K incoming IP traffic, 46.5K outgoing IP traffic.6171worker2 # [ 161.022059] systemd[1]: Starting Nix worker-protocol over gRPC...6172worker2 # [ 161.228385] nix-grpc-daemon[1908]: ts=2026-09-22T10:51:50Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6173worker2 # [ 161.230943] nix-grpc-daemon[1908]: ts=2026-09-22T10:51:50Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6174worker2 # [ 161.234959] nix-grpc-daemon[1908]: ts=2026-09-22T10:51:50Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16175worker2 # [ 161.242297] nix-grpc-daemon[1908]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6176worker2 # [ 161.243757] nix-grpc-daemon[1908]: E0000 00:00:1790074310.844656 1908 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6177worker2 # [ 161.259802] nix-grpc-daemon[1908]: E0000 00:00:1790074310.850586 1908 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6178worker2 # [ 161.280173] nix-grpc-daemon[1908]: ts=2026-09-22T10:51:50Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6179worker2 # [ 161.300746] systemd[1]: Started Nix worker-protocol over gRPC.6180worker2 # [ 161.301596] nix-grpc-daemon[1908]: E0000 00:00:1790074310.867906 1948 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6181worker2 # [ 161.303897] nix-grpc-daemon[1908]: E0000 00:00:1790074310.885598 1948 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6182worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 1.37 seconds)6183lb: waiting until exactly no endpoints of aarch64-linux are unhealthy6184lb: must succeed: curl -sf localhost:9901/clusters6185worker2 # [ 161.309415] nix-grpc-daemon[1908]: E0000 00:00:1790074310.885748 1948 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6186worker2 # [ 161.313666] nix-grpc-daemon[1908]: E0000 00:00:1790074310.901991 1948 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6187worker2 # [ 161.316237] nix-grpc-daemon[1908]: ts=2026-09-22T10:51:50Z level=info event=scheduler_connected addr=lb:500516188worker1 # [ 160.779707] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:50Z level=info event=worker_gone addr=192.168.1.4:500516189worker1 # [ 161.307391] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:50Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:370626190worker1 # [ 161.316850] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:50Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=06191lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6192lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.08 seconds)6193lb: must succeed: curl -sf localhost:9901/clusters6194lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6195worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6196worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6197(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 16.78 seconds)6198subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6199client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16200lb # [ 161.587952] envoy[927]: conn peer=[2001:db8:1::1]:56726 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9655 tx=4145 ms=70366201lb # [ 161.588262] envoy[927]: conn peer=[2001:db8:1::1]:46256 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2995 tx=1424 ms=142436202lb # [ 161.588356] envoy[927]: conn peer=[2001:db8:1::4]:58122 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=9957 tx=743 ms=442086203worker2 # [ 161.786716] nix-grpc-daemon[1908]: ts=2026-09-22T10:51:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:406626204worker2 # [ 161.800227] nix-daemon[866]: accepted connection from pid 1908, user nix-grpc-daemon (trusted)6205worker1 # [ 161.811578] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370846206worker1 # [ 161.854157] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:51Z level=debug event=want client=5 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=06207worker1 # [ 161.861033] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:51Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=16208worker1 # [ 161.867628] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:51Z level=debug event=want client=5 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=06209worker1 # [ 161.873125] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:51Z level=debug event=want client=5 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=06210worker1 # [ 161.899832] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846211worker1 # [ 161.914532] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370846212worker1 # [ 161.928396] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:370846213worker1 # [ 161.943248] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6214worker1 # [ 162.144266] nix-daemon[839]: accepted connection from pid 1758, user nix-grpc-daemon (trusted)6215worker1 # [ 162.162967] nix-daemon[839]: reaped child process 1761, status = succeeded6216worker1 # [ 162.332884] nix-daemon[839]: accepted connection from pid 1763, user nix-grpc-daemon (trusted)6217worker1 # [ 162.350411] nix-daemon[839]: reaped child process 1766, status = succeeded6218lb # [ 162.373930] niks3-server[1130]: 2026/09/22 10:51:51 INFO Received uploads request method=POST path=/api/pending_closures6219worker1 # [ 162.375543] nix-grpc-daemon[1563]: time=2026-09-22T10:51:51.977Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6220worker1 # [ 162.377098] nix-grpc-daemon[1563]: time=2026-09-22T10:51:51.977Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)"6221lb # [ 162.399644] niks3-server[1130]: 2026/09/22 10:51:51 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst6222lb # [ 162.415205] niks3-server[1130]: 2026/09/22 10:51:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6223worker1 # [ 162.413273] nix-grpc-daemon[1563]: time=2026-09-22T10:51:52.015Z level=INFO msg="Uploading 1 narinfos"6224lb # [ 162.427418] niks3-server[1130]: 2026/09/22 10:51:52 INFO Signed narinfos id=54 count=16225lb # [ 162.437545] niks3-server[1130]: 2026/09/22 10:51:52 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls6226worker1 # [ 162.435006] nix-grpc-daemon[1563]: time=2026-09-22T10:51:52.037Z level=INFO msg="Upload complete. (475ms)"6227lb # [ 162.443953] niks3-server[1130]: 2026/09/22 10:51:52 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6228lb # [ 162.448957] niks3-server[1130]: 2026/09/22 10:51:52 INFO Completed upload id=546229worker1 # [ 162.440761] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=0 paths=1 nar_bytes_in=5126230lb # [ 162.452117] niks3-server[1130]: 2026/09/22 10:51:52 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo6231worker1 # [ 162.454178] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846232worker1 # [ 162.460584] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6233worker1 # [ 162.464892] nix-daemon[839]: reaped child process 1756, status = succeeded6234worker1 # [ 162.477860] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6235worker1 # [ 163.321576] nix-daemon[839]: reaped child process 1770, status = succeeded6236worker1 # [ 163.492566] nix-daemon[839]: accepted connection from pid 1786, user nix-grpc-daemon (trusted)6237worker1 # [ 163.507713] nix-daemon[839]: reaped child process 1789, status = succeeded6238lb # [ 163.533887] niks3-server[1130]: 2026/09/22 10:51:53 INFO Received uploads request method=POST path=/api/pending_closures6239worker1 # [ 163.535122] nix-grpc-daemon[1563]: time=2026-09-22T10:51:53.135Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6240worker1 # [ 163.536847] nix-grpc-daemon[1563]: time=2026-09-22T10:51:53.135Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)"6241lb # [ 163.566681] niks3-server[1130]: 2026/09/22 10:51:53 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6242lb # [ 163.579944] niks3-server[1130]: 2026/09/22 10:51:53 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv6243worker1 # [ 163.580113] nix-grpc-daemon[1563]: time=2026-09-22T10:51:53.182Z level=INFO msg="Uploading 1 narinfos"6244lb # [ 163.593181] niks3-server[1130]: 2026/09/22 10:51:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6245lb # [ 163.600535] niks3-server[1130]: 2026/09/22 10:51:53 INFO Signed narinfos id=55 count=16246lb # [ 163.606578] niks3-server[1130]: 2026/09/22 10:51:53 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls6247lb # [ 163.610138] niks3-server[1130]: 2026/09/22 10:51:53 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6248lb # [ 163.614869] niks3-server[1130]: 2026/09/22 10:51:53 INFO Completed upload id=556249worker1 # [ 163.606733] nix-grpc-daemon[1563]: time=2026-09-22T10:51:53.209Z level=INFO msg="Upload complete. (319ms)"6250lb # [ 163.617847] niks3-server[1130]: 2026/09/22 10:51:53 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo6251worker1 # [ 163.623388] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=6 outputs=16252worker1 # [ 163.640258] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:53Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=16253worker1 # [ 163.652771] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846254worker1 # [ 163.657388] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:53Z level=debug event=want client=5 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=06255worker1 # [ 163.666910] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370846256worker1 # [ 163.675324] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:370846257worker1 # [ 163.680805] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6258worker1 # [ 163.684197] nix-daemon[839]: reaped child process 1768, status = succeeded6259worker1 # [ 163.868727] nix-daemon[839]: accepted connection from pid 1795, user nix-grpc-daemon (trusted)6260worker1 # [ 163.885724] nix-daemon[839]: reaped child process 1798, status = succeeded6261worker1 # [ 164.057454] nix-daemon[839]: accepted connection from pid 1800, user nix-grpc-daemon (trusted)6262worker1 # [ 164.074608] nix-daemon[839]: reaped child process 1803, status = succeeded6263lb # [ 164.094674] niks3-server[1130]: 2026/09/22 10:51:53 INFO Received uploads request method=POST path=/api/pending_closures6264worker1 # [ 164.094712] nix-grpc-daemon[1563]: time=2026-09-22T10:51:53.697Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6265worker1 # [ 164.096359] nix-grpc-daemon[1563]: time=2026-09-22T10:51:53.697Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)"6266lb # [ 164.118601] niks3-server[1130]: 2026/09/22 10:51:53 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst6267worker1 # [ 164.124405] nix-grpc-daemon[1563]: time=2026-09-22T10:51:53.727Z level=INFO msg="Uploading 1 narinfos"6268lb # [ 164.133454] niks3-server[1130]: 2026/09/22 10:51:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6269lb # [ 164.142734] niks3-server[1130]: 2026/09/22 10:51:53 INFO Signed narinfos id=56 count=16270lb # [ 164.153069] niks3-server[1130]: 2026/09/22 10:51:53 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6271lb # [ 164.163159] niks3-server[1130]: 2026/09/22 10:51:53 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls6272lb # [ 164.170386] niks3-server[1130]: 2026/09/22 10:51:53 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo6273lb # [ 164.183059] niks3-server[1130]: 2026/09/22 10:51:53 INFO Completed upload id=566274worker1 # [ 164.174983] nix-grpc-daemon[1563]: time=2026-09-22T10:51:53.777Z level=INFO msg="Upload complete. (488ms)"6275worker1 # [ 164.182340] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=0 paths=1 nar_bytes_in=5126276worker1 # [ 164.196290] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846277worker1 # [ 164.202805] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6278worker1 # [ 164.207938] nix-daemon[839]: reaped child process 1793, status = succeeded6279worker1 # [ 164.221716] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6280worker1 # [ 165.062497] nix-daemon[839]: reaped child process 1807, status = succeeded6281worker1 # [ 165.239739] nix-daemon[839]: accepted connection from pid 1823, user nix-grpc-daemon (trusted)6282worker1 # [ 165.255467] nix-daemon[839]: reaped child process 1826, status = succeeded6283lb # [ 165.279774] niks3-server[1130]: 2026/09/22 10:51:54 INFO Received uploads request method=POST path=/api/pending_closures6284worker1 # [ 165.287218] nix-grpc-daemon[1563]: time=2026-09-22T10:51:54.886Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6285worker1 # [ 165.288793] nix-grpc-daemon[1563]: time=2026-09-22T10:51:54.886Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)"6286lb # [ 165.321574] niks3-server[1130]: 2026/09/22 10:51:54 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv6287lb # [ 165.332421] niks3-server[1130]: 2026/09/22 10:51:54 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6288worker1 # [ 165.336114] nix-grpc-daemon[1563]: time=2026-09-22T10:51:54.938Z level=INFO msg="Uploading 1 narinfos"6289lb # [ 165.343992] niks3-server[1130]: 2026/09/22 10:51:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6290lb # [ 165.349346] niks3-server[1130]: 2026/09/22 10:51:54 INFO Signed narinfos id=57 count=16291lb # [ 165.351684] niks3-server[1130]: 2026/09/22 10:51:54 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls6292lb # [ 165.362289] niks3-server[1130]: 2026/09/22 10:51:54 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6293lb # [ 165.369625] niks3-server[1130]: 2026/09/22 10:51:54 INFO Completed upload id=576294worker1 # [ 165.360931] nix-grpc-daemon[1563]: time=2026-09-22T10:51:54.963Z level=INFO msg="Upload complete. (329ms)"6295lb # [ 165.372412] niks3-server[1130]: 2026/09/22 10:51:54 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo6296worker1 # [ 165.373091] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=7 outputs=16297worker1 # [ 165.387754] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:54Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=16298worker1 # [ 165.392513] nix-daemon[839]: reaped child process 1805, status = succeeded6299worker1 # [ 165.414027] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846300worker1 # [ 165.420874] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:55Z level=debug event=want client=5 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=06301worker1 # [ 165.430862] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370846302worker1 # [ 165.443519] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:370846303worker1 # [ 165.456109] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6304worker1 # [ 165.647870] nix-daemon[839]: accepted connection from pid 1832, user nix-grpc-daemon (trusted)6305worker1 # [ 165.663221] nix-daemon[839]: reaped child process 1835, status = succeeded6306worker1 # [ 165.840706] nix-daemon[839]: accepted connection from pid 1837, user nix-grpc-daemon (trusted)6307worker1 # [ 165.855632] nix-daemon[839]: reaped child process 1840, status = succeeded6308lb # [ 165.877763] niks3-server[1130]: 2026/09/22 10:51:55 INFO Received uploads request method=POST path=/api/pending_closures6309worker1 # [ 165.878600] nix-grpc-daemon[1563]: time=2026-09-22T10:51:55.481Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6310worker1 # [ 165.880190] nix-grpc-daemon[1563]: time=2026-09-22T10:51:55.481Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)"6311lb # [ 165.907948] niks3-server[1130]: 2026/09/22 10:51:55 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst6312lb # [ 165.919677] niks3-server[1130]: 2026/09/22 10:51:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6313worker1 # [ 165.916887] nix-grpc-daemon[1563]: time=2026-09-22T10:51:55.519Z level=INFO msg="Uploading 1 narinfos"6314lb # [ 165.928822] niks3-server[1130]: 2026/09/22 10:51:55 INFO Signed narinfos id=58 count=16315lb # [ 165.935248] niks3-server[1130]: 2026/09/22 10:51:55 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls6316lb # [ 165.942712] niks3-server[1130]: 2026/09/22 10:51:55 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6317lb # [ 165.949013] niks3-server[1130]: 2026/09/22 10:51:55 INFO Completed upload id=586318worker1 # [ 165.940278] nix-grpc-daemon[1563]: time=2026-09-22T10:51:55.543Z level=INFO msg="Upload complete. (470ms)"6319lb # [ 165.951555] niks3-server[1130]: 2026/09/22 10:51:55 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo6320worker1 # [ 165.948117] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=0 paths=1 nar_bytes_in=5126321worker1 # [ 165.959099] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846322worker1 # [ 165.965377] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6323worker1 # [ 165.969874] nix-daemon[839]: reaped child process 1830, status = succeeded6324worker1 # [ 165.982797] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6325worker1 # [ 167.421067] nix-daemon[839]: reaped child process 1844, status = succeeded6326worker1 # [ 167.603370] nix-daemon[839]: accepted connection from pid 1860, user nix-grpc-daemon (trusted)6327worker1 # [ 167.621030] nix-daemon[839]: reaped child process 1863, status = succeeded6328lb # [ 167.641890] niks3-server[1130]: 2026/09/22 10:51:57 INFO Received uploads request method=POST path=/api/pending_closures6329worker1 # [ 167.644367] nix-grpc-daemon[1563]: time=2026-09-22T10:51:57.246Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6330worker1 # [ 167.645794] nix-grpc-daemon[1563]: time=2026-09-22T10:51:57.246Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)"6331lb # [ 167.683277] niks3-server[1130]: 2026/09/22 10:51:57 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv6332lb # [ 167.693381] niks3-server[1130]: 2026/09/22 10:51:57 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6333worker1 # [ 167.693269] nix-grpc-daemon[1563]: time=2026-09-22T10:51:57.295Z level=INFO msg="Uploading 1 narinfos"6334lb # [ 167.704725] niks3-server[1130]: 2026/09/22 10:51:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6335lb # [ 167.712700] niks3-server[1130]: 2026/09/22 10:51:57 INFO Signed narinfos id=59 count=16336lb # [ 167.714859] niks3-server[1130]: 2026/09/22 10:51:57 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls6337lb # [ 167.720325] niks3-server[1130]: 2026/09/22 10:51:57 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6338lb # [ 167.726599] niks3-server[1130]: 2026/09/22 10:51:57 INFO Completed upload id=596339lb # [ 167.728973] niks3-server[1130]: 2026/09/22 10:51:57 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo6340worker1 # [ 167.718505] nix-grpc-daemon[1563]: time=2026-09-22T10:51:57.321Z level=INFO msg="Upload complete. (325ms)"6341worker1 # [ 167.734357] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=8 outputs=16342worker1 # [ 167.752781] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:57Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=16343worker1 # [ 167.757206] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846344worker1 # [ 167.762115] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:57Z level=debug event=want client=5 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=06345worker1 # [ 167.766532] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370846346worker1 # [ 167.769816] nix-daemon[839]: reaped child process 1842, status = succeeded6347worker1 # [ 167.772650] nix-grpc-daemon[1563]: time=2026-09-22T10:51:57.374Z level=INFO msg="All 1 paths already cached"6348worker1 # [ 167.777457] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:370846349worker1 # [ 167.785504] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6350worker1 # [ 167.968771] nix-daemon[839]: accepted connection from pid 1869, user nix-grpc-daemon (trusted)6351worker1 # [ 167.985529] nix-daemon[839]: reaped child process 1872, status = succeeded6352worker1 # [ 168.155578] nix-daemon[839]: accepted connection from pid 1874, user nix-grpc-daemon (trusted)6353worker1 # [ 168.172852] nix-daemon[839]: reaped child process 1877, status = succeeded6354lb # [ 168.193662] niks3-server[1130]: 2026/09/22 10:51:57 INFO Received uploads request method=POST path=/api/pending_closures6355worker1 # [ 168.194448] nix-grpc-daemon[1563]: time=2026-09-22T10:51:57.797Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6356worker1 # [ 168.195915] nix-grpc-daemon[1563]: time=2026-09-22T10:51:57.797Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)"6357lb # [ 168.224424] niks3-server[1130]: 2026/09/22 10:51:57 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst6358worker1 # [ 168.222421] nix-grpc-daemon[1563]: time=2026-09-22T10:51:57.825Z level=INFO msg="Uploading 1 narinfos"6359lb # [ 168.233480] niks3-server[1130]: 2026/09/22 10:51:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6360lb # [ 168.244653] niks3-server[1130]: 2026/09/22 10:51:57 INFO Signed narinfos id=60 count=16361lb # [ 168.247594] niks3-server[1130]: 2026/09/22 10:51:57 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls6362lb # [ 168.255604] niks3-server[1130]: 2026/09/22 10:51:57 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6363lb # [ 168.262418] niks3-server[1130]: 2026/09/22 10:51:57 INFO Completed upload id=606364worker1 # [ 168.254224] nix-grpc-daemon[1563]: time=2026-09-22T10:51:57.857Z level=INFO msg="Upload complete. (458ms)"6365lb # [ 168.265058] niks3-server[1130]: 2026/09/22 10:51:57 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo6366worker1 # [ 168.260205] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=0 paths=1 nar_bytes_in=6246367worker1 # [ 168.272573] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846368worker1 # [ 168.280278] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6369worker1 # [ 168.283201] nix-daemon[839]: reaped child process 1867, status = succeeded6370worker1 # [ 168.293614] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6371worker1 # [ 169.134029] nix-daemon[839]: reaped child process 1881, status = succeeded6372worker1 # [ 169.316727] nix-daemon[839]: accepted connection from pid 1897, user nix-grpc-daemon (trusted)6373worker1 # [ 169.334433] nix-daemon[839]: reaped child process 1900, status = succeeded6374lb # [ 169.355274] niks3-server[1130]: 2026/09/22 10:51:58 INFO Received uploads request method=POST path=/api/pending_closures6375worker1 # [ 169.356444] nix-grpc-daemon[1563]: time=2026-09-22T10:51:58.958Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6376worker1 # [ 169.358313] nix-grpc-daemon[1563]: time=2026-09-22T10:51:58.958Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)"6377lb # [ 169.398891] niks3-server[1130]: 2026/09/22 10:51:58 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst6378worker1 # [ 169.401262] nix-grpc-daemon[1563]: time=2026-09-22T10:51:59.003Z level=INFO msg="Uploading 1 narinfos"6379lb # [ 169.413913] niks3-server[1130]: 2026/09/22 10:51:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6380lb # [ 169.423066] niks3-server[1130]: 2026/09/22 10:51:58 INFO Signed narinfos id=61 count=16381lb # [ 169.426389] niks3-server[1130]: 2026/09/22 10:51:58 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv6382lb # [ 169.433815] niks3-server[1130]: 2026/09/22 10:51:59 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls6383lb # [ 169.437279] niks3-server[1130]: 2026/09/22 10:51:59 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6384lb # [ 169.443103] niks3-server[1130]: 2026/09/22 10:51:59 INFO Completed upload id=616385worker1 # [ 169.433254] nix-grpc-daemon[1563]: time=2026-09-22T10:51:59.036Z level=INFO msg="Upload complete. (329ms)"6386lb # [ 169.445274] niks3-server[1130]: 2026/09/22 10:51:59 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo6387worker1 # [ 169.450455] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=9 outputs=16388worker1 # [ 169.464403] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:59Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=16389worker1 # [ 169.476099] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846390worker1 # [ 169.480731] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:59Z level=debug event=want client=5 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=06391worker1 # [ 169.485458] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370846392worker1 # [ 169.488944] nix-daemon[839]: reaped child process 1879, status = succeeded6393worker1 # [ 169.498610] nix-grpc-daemon[1563]: time=2026-09-22T10:51:59.082Z level=INFO msg="All 1 paths already cached"6394worker1 # [ 169.505579] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6395worker1 # [ 169.507628] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:370846396worker1 # [ 169.688463] nix-daemon[839]: accepted connection from pid 1906, user nix-grpc-daemon (trusted)6397worker1 # [ 169.707344] nix-daemon[839]: reaped child process 1909, status = succeeded6398worker1 # [ 169.887482] nix-daemon[839]: accepted connection from pid 1911, user nix-grpc-daemon (trusted)6399worker1 # [ 169.904302] nix-daemon[839]: reaped child process 1914, status = succeeded6400lb # [ 169.927215] niks3-server[1130]: 2026/09/22 10:51:59 INFO Received uploads request method=POST path=/api/pending_closures6401worker1 # [ 169.928075] nix-grpc-daemon[1563]: time=2026-09-22T10:51:59.530Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6402worker1 # [ 169.929547] nix-grpc-daemon[1563]: time=2026-09-22T10:51:59.530Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)"6403lb # [ 169.956609] niks3-server[1130]: 2026/09/22 10:51:59 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst6404lb # [ 169.970831] niks3-server[1130]: 2026/09/22 10:51:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6405worker1 # [ 169.969220] nix-grpc-daemon[1563]: time=2026-09-22T10:51:59.571Z level=INFO msg="Uploading 1 narinfos"6406lb # [ 169.983341] niks3-server[1130]: 2026/09/22 10:51:59 INFO Signed narinfos id=62 count=16407lb # [ 169.990032] niks3-server[1130]: 2026/09/22 10:51:59 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls6408worker1 # [ 169.990110] nix-grpc-daemon[1563]: time=2026-09-22T10:51:59.592Z level=INFO msg="Upload complete. (476ms)"6409lb # [ 169.998856] niks3-server[1130]: 2026/09/22 10:51:59 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6410lb # [ 170.004369] niks3-server[1130]: 2026/09/22 10:51:59 INFO Completed upload id=626411lb # [ 170.007594] niks3-server[1130]: 2026/09/22 10:51:59 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo6412worker1 # [ 170.000155] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=0 paths=1 nar_bytes_in=6246413worker1 # [ 170.010062] nix-grpc-daemon[1486]: ts=2026-09-22T10:51:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846414worker1 # [ 170.015729] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6415worker1 # [ 170.020396] nix-daemon[839]: reaped child process 1904, status = succeeded6416worker1 # [ 170.030739] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6417worker1 # [ 170.660580] nix-daemon[839]: reaped child process 1918, status = succeeded6418worker1 # [ 170.828569] nix-daemon[839]: accepted connection from pid 1934, user nix-grpc-daemon (trusted)6419worker1 # [ 170.849205] nix-daemon[839]: reaped child process 1937, status = succeeded6420lb # [ 170.870231] niks3-server[1130]: 2026/09/22 10:52:00 INFO Received uploads request method=POST path=/api/pending_closures6421worker1 # [ 170.872384] nix-grpc-daemon[1563]: time=2026-09-22T10:52:00.474Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6422worker1 # [ 170.873898] nix-grpc-daemon[1563]: time=2026-09-22T10:52:00.474Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)"6423lb # [ 170.897558] niks3-server[1130]: 2026/09/22 10:52:00 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv6424lb # [ 170.919031] niks3-server[1130]: 2026/09/22 10:52:00 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst6425lb # [ 170.928384] niks3-server[1130]: 2026/09/22 10:52:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6426worker1 # [ 170.925467] nix-grpc-daemon[1563]: time=2026-09-22T10:52:00.528Z level=INFO msg="Uploading 1 narinfos"6427lb # [ 170.935696] niks3-server[1130]: 2026/09/22 10:52:00 INFO Signed narinfos id=63 count=16428lb # [ 170.940502] niks3-server[1130]: 2026/09/22 10:52:00 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls6429lb # [ 170.947542] niks3-server[1130]: 2026/09/22 10:52:00 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6430lb # [ 170.954106] niks3-server[1130]: 2026/09/22 10:52:00 INFO Completed upload id=636431worker1 # [ 170.945258] nix-grpc-daemon[1563]: time=2026-09-22T10:52:00.548Z level=INFO msg="Upload complete. (304ms)"6432lb # [ 170.956899] niks3-server[1130]: 2026/09/22 10:52:00 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo6433worker1 # [ 170.956878] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=0 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=10 outputs=16434worker1 # [ 170.970711] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:00Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=16435worker1 # [ 170.974775] nix-daemon[839]: reaped child process 1916, status = succeeded6436worker1 # [ 171.001468] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846437worker1 # [ 171.011752] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:00Z level=debug event=want client=5 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=06438worker1 # [ 171.022022] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370846439worker1 # [ 171.035022] nix-grpc-daemon[1563]: time=2026-09-22T10:52:00.637Z level=INFO msg="All 1 paths already cached"6440worker1 # [ 171.043976] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:370846441worker1 # [ 171.057377] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6442worker1 # [ 171.248671] nix-daemon[839]: accepted connection from pid 1943, user nix-grpc-daemon (trusted)6443worker1 # [ 171.267372] nix-daemon[839]: reaped child process 1946, status = succeeded6444worker1 # [ 171.441649] nix-daemon[839]: accepted connection from pid 1948, user nix-grpc-daemon (trusted)6445worker1 # [ 171.457854] nix-daemon[839]: reaped child process 1951, status = succeeded6446lb # [ 171.481857] niks3-server[1130]: 2026/09/22 10:52:01 INFO Received uploads request method=POST path=/api/pending_closures6447worker1 # [ 171.482368] nix-grpc-daemon[1563]: time=2026-09-22T10:52:01.085Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6448worker1 # [ 171.483858] nix-grpc-daemon[1563]: time=2026-09-22T10:52:01.085Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)"6449lb # [ 171.506982] niks3-server[1130]: 2026/09/22 10:52:01 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst6450lb # [ 171.520843] niks3-server[1130]: 2026/09/22 10:52:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6451worker1 # [ 171.519584] nix-grpc-daemon[1563]: time=2026-09-22T10:52:01.122Z level=INFO msg="Uploading 1 narinfos"6452lb # [ 171.532425] niks3-server[1130]: 2026/09/22 10:52:01 INFO Signed narinfos id=64 count=16453lb # [ 171.535306] niks3-server[1130]: 2026/09/22 10:52:01 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls6454lb # [ 171.546906] niks3-server[1130]: 2026/09/22 10:52:01 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6455lb # [ 171.551967] niks3-server[1130]: 2026/09/22 10:52:01 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo6456lb # [ 171.556259] niks3-server[1130]: 2026/09/22 10:52:01 INFO Completed upload id=646457worker1 # [ 171.548060] nix-grpc-daemon[1563]: time=2026-09-22T10:52:01.150Z level=INFO msg="Upload complete. (477ms)"6458worker1 # [ 171.556277] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=0 paths=1 nar_bytes_in=6246459worker1 # [ 171.567448] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846460worker1 # [ 171.573409] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6461worker1 # [ 171.577977] nix-daemon[839]: reaped child process 1941, status = succeeded6462worker1 # [ 171.590603] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6463worker1 # [ 172.633178] nix-daemon[839]: reaped child process 1955, status = succeeded6464worker1 # [ 172.810772] nix-daemon[839]: accepted connection from pid 1971, user nix-grpc-daemon (trusted)6465worker1 # [ 172.826461] nix-daemon[839]: reaped child process 1974, status = succeeded6466lb # [ 172.850119] niks3-server[1130]: 2026/09/22 10:52:02 INFO Received uploads request method=POST path=/api/pending_closures6467worker1 # [ 172.852420] nix-grpc-daemon[1563]: time=2026-09-22T10:52:02.452Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6468worker1 # [ 172.853892] nix-grpc-daemon[1563]: time=2026-09-22T10:52:02.452Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)"6469lb # [ 172.889407] niks3-server[1130]: 2026/09/22 10:52:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6470worker1 # [ 172.890993] nix-grpc-daemon[1563]: time=2026-09-22T10:52:02.493Z level=INFO msg="Uploading 1 narinfos"6471lb # [ 172.905977] niks3-server[1130]: 2026/09/22 10:52:02 INFO Signed narinfos id=65 count=16472lb # [ 172.913351] niks3-server[1130]: 2026/09/22 10:52:02 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst6473lb # [ 172.926719] niks3-server[1130]: 2026/09/22 10:52:02 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls6474lb # [ 172.931921] niks3-server[1130]: 2026/09/22 10:52:02 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6475lb # [ 172.936199] niks3-server[1130]: 2026/09/22 10:52:02 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv6476worker1 # [ 172.927921] nix-grpc-daemon[1563]: time=2026-09-22T10:52:02.530Z level=INFO msg="Upload complete. (328ms)"6477lb # [ 172.939956] niks3-server[1130]: 2026/09/22 10:52:02 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo6478lb # [ 172.944266] niks3-server[1130]: 2026/09/22 10:52:02 INFO Completed upload id=656479worker1 # [ 172.940467] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=11 outputs=16480worker1 # [ 172.960821] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:02Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=16481worker1 # [ 172.967716] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846482worker1 # [ 172.973315] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:02Z level=debug event=want client=5 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=06483worker1 # [ 172.977316] nix-daemon[839]: reaped child process 1953, status = succeeded6484worker1 # [ 172.995776] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370846485worker1 # [ 173.008957] nix-grpc-daemon[1563]: time=2026-09-22T10:52:02.611Z level=INFO msg="All 2 paths already cached"6486worker1 # [ 173.017197] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:370846487worker1 # [ 173.029585] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6488worker1 # [ 173.231585] nix-daemon[839]: accepted connection from pid 1980, user nix-grpc-daemon (trusted)6489worker1 # [ 173.247192] nix-daemon[839]: reaped child process 1983, status = succeeded6490worker1 # [ 173.427637] nix-daemon[839]: accepted connection from pid 1985, user nix-grpc-daemon (trusted)6491worker1 # [ 173.441875] nix-daemon[839]: reaped child process 1988, status = succeeded6492lb # [ 173.465999] niks3-server[1130]: 2026/09/22 10:52:03 INFO Received uploads request method=POST path=/api/pending_closures6493worker1 # [ 173.466295] nix-grpc-daemon[1563]: time=2026-09-22T10:52:03.068Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6494worker1 # [ 173.467780] nix-grpc-daemon[1563]: time=2026-09-22T10:52:03.069Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)"6495lb # [ 173.489829] niks3-server[1130]: 2026/09/22 10:52:03 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst6496worker1 # [ 173.494470] nix-grpc-daemon[1563]: time=2026-09-22T10:52:03.097Z level=INFO msg="Uploading 1 narinfos"6497lb # [ 173.504213] niks3-server[1130]: 2026/09/22 10:52:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign6498lb # [ 173.514841] niks3-server[1130]: 2026/09/22 10:52:03 INFO Signed narinfos id=66 count=16499lb # [ 173.523859] niks3-server[1130]: 2026/09/22 10:52:03 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls6500worker1 # [ 173.521893] nix-grpc-daemon[1563]: time=2026-09-22T10:52:03.124Z level=INFO msg="Upload complete. (477ms)"6501lb # [ 173.530344] niks3-server[1130]: 2026/09/22 10:52:03 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete6502lb # [ 173.535274] niks3-server[1130]: 2026/09/22 10:52:03 INFO Completed upload id=666503worker1 # [ 173.528243] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=0 paths=1 nar_bytes_in=7526504lb # [ 173.539584] niks3-server[1130]: 2026/09/22 10:52:03 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo6505worker1 # [ 173.537965] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846506worker1 # [ 173.542911] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6507worker1 # [ 173.546831] nix-daemon[839]: reaped child process 1978, status = succeeded6508worker1 # [ 173.558586] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6509worker1 # [ 175.000567] nix-daemon[839]: reaped child process 1992, status = succeeded6510worker1 # [ 175.181291] nix-daemon[839]: accepted connection from pid 2008, user nix-grpc-daemon (trusted)6511worker1 # [ 175.197339] nix-daemon[839]: reaped child process 2011, status = succeeded6512lb # [ 175.219232] niks3-server[1130]: 2026/09/22 10:52:04 INFO Received uploads request method=POST path=/api/pending_closures6513worker1 # [ 175.222566] nix-grpc-daemon[1563]: time=2026-09-22T10:52:04.824Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6514worker1 # [ 175.224080] nix-grpc-daemon[1563]: time=2026-09-22T10:52:04.824Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)"6515worker1 # [ 175.258728] nix-grpc-daemon[1563]: time=2026-09-22T10:52:04.861Z level=INFO msg="Uploading 1 narinfos"6516lb # [ 175.256238] niks3-server[1130]: 2026/09/22 10:52:04 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst6517lb # [ 175.270882] niks3-server[1130]: 2026/09/22 10:52:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign6518lb # [ 175.281122] niks3-server[1130]: 2026/09/22 10:52:04 INFO Signed narinfos id=67 count=16519lb # [ 175.286349] niks3-server[1130]: 2026/09/22 10:52:04 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv6520worker1 # [ 175.285975] nix-grpc-daemon[1563]: time=2026-09-22T10:52:04.888Z level=INFO msg="Upload complete. (318ms)"6521lb # [ 175.295378] niks3-server[1130]: 2026/09/22 10:52:04 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls6522lb # [ 175.300549] niks3-server[1130]: 2026/09/22 10:52:04 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete6523lb # [ 175.304540] niks3-server[1130]: 2026/09/22 10:52:04 INFO Completed upload id=676524lb # [ 175.307145] niks3-server[1130]: 2026/09/22 10:52:04 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo6525worker1 # [ 175.297434] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=12 outputs=16526worker1 # [ 175.311148] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:04Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=16527worker1 # [ 175.315766] nix-daemon[839]: reaped child process 1990, status = succeeded6528worker1 # [ 175.339771] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846529worker1 # [ 175.354207] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370846530worker1 # [ 175.367873] nix-grpc-daemon[1563]: time=2026-09-22T10:52:04.970Z level=INFO msg="All 2 paths already cached"6531worker1 # [ 175.376591] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:370846532worker1 # [ 175.389868] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6533worker1 # [ 175.588261] nix-daemon[839]: accepted connection from pid 2015, user nix-grpc-daemon (trusted)6534worker1 # [ 175.603545] nix-daemon[839]: reaped child process 2018, status = succeeded6535worker1 # [ 175.779546] nix-daemon[839]: accepted connection from pid 2020, user nix-grpc-daemon (trusted)6536worker1 # [ 175.794087] nix-daemon[839]: reaped child process 2023, status = succeeded6537lb # [ 175.818192] niks3-server[1130]: 2026/09/22 10:52:05 INFO Received uploads request method=POST path=/api/pending_closures6538worker1 # [ 175.818301] nix-grpc-daemon[1563]: time=2026-09-22T10:52:05.420Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6539worker1 # [ 175.819766] nix-grpc-daemon[1563]: time=2026-09-22T10:52:05.421Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)"6540lb # [ 175.843162] niks3-server[1130]: 2026/09/22 10:52:05 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst6541worker1 # [ 175.848732] nix-grpc-daemon[1563]: time=2026-09-22T10:52:05.451Z level=INFO msg="Uploading 1 narinfos"6542lb # [ 175.859913] niks3-server[1130]: 2026/09/22 10:52:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign6543lb # [ 175.870357] niks3-server[1130]: 2026/09/22 10:52:05 INFO Signed narinfos id=68 count=16544lb # [ 175.876209] niks3-server[1130]: 2026/09/22 10:52:05 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls6545lb # [ 175.884644] niks3-server[1130]: 2026/09/22 10:52:05 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete6546worker1 # [ 175.880741] nix-grpc-daemon[1563]: time=2026-09-22T10:52:05.483Z level=INFO msg="Upload complete. (475ms)"6547lb # [ 175.889642] niks3-server[1130]: 2026/09/22 10:52:05 INFO Completed upload id=686548lb # [ 175.893429] niks3-server[1130]: 2026/09/22 10:52:05 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo6549worker1 # [ 175.888142] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=0 paths=1 nar_bytes_in=7526550worker1 # [ 175.898862] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846551worker1 # [ 175.904765] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6552worker1 # [ 175.909318] nix-daemon[839]: reaped child process 2013, status = succeeded6553worker1 # [ 175.921631] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6554worker1 # [ 177.562702] nix-daemon[839]: reaped child process 2027, status = succeeded6555worker1 # [ 177.748702] nix-daemon[839]: accepted connection from pid 2043, user nix-grpc-daemon (trusted)6556worker1 # [ 177.765154] nix-daemon[839]: reaped child process 2046, status = succeeded6557lb # [ 177.790332] niks3-server[1130]: 2026/09/22 10:52:07 INFO Received uploads request method=POST path=/api/pending_closures6558worker1 # [ 177.792431] nix-grpc-daemon[1563]: time=2026-09-22T10:52:07.394Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6559worker1 # [ 177.797268] nix-grpc-daemon[1563]: time=2026-09-22T10:52:07.394Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)"6560lb # [ 177.825466] niks3-server[1130]: 2026/09/22 10:52:07 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv6561lb # [ 177.837487] niks3-server[1130]: 2026/09/22 10:52:07 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst6562lb # [ 177.848389] niks3-server[1130]: 2026/09/22 10:52:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign6563worker1 # [ 177.845460] nix-grpc-daemon[1563]: time=2026-09-22T10:52:07.448Z level=INFO msg="Uploading 1 narinfos"6564lb # [ 177.857608] niks3-server[1130]: 2026/09/22 10:52:07 INFO Signed narinfos id=69 count=16565lb # [ 177.863120] niks3-server[1130]: 2026/09/22 10:52:07 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls6566lb # [ 177.871059] niks3-server[1130]: 2026/09/22 10:52:07 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete6567lb # [ 177.879774] niks3-server[1130]: 2026/09/22 10:52:07 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo6568lb # [ 177.885489] niks3-server[1130]: 2026/09/22 10:52:07 INFO Completed upload id=696569worker1 # [ 177.877502] nix-grpc-daemon[1563]: time=2026-09-22T10:52:07.480Z level=INFO msg="Upload complete. (344ms)"6570worker1 # [ 177.893166] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=13 outputs=16571worker1 # [ 177.906684] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:07Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=16572worker1 # [ 177.911591] nix-daemon[839]: reaped child process 2025, status = succeeded6573worker1 # [ 177.934006] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846574worker1 # [ 177.945438] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370846575worker1 # [ 177.960281] nix-grpc-daemon[1563]: time=2026-09-22T10:52:07.563Z level=INFO msg="All 2 paths already cached"6576worker1 # [ 177.969113] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:370846577worker1 # [ 177.983214] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6578worker1 # [ 178.178965] nix-daemon[839]: accepted connection from pid 2050, user nix-grpc-daemon (trusted)6579worker1 # [ 178.197112] nix-daemon[839]: reaped child process 2053, status = succeeded6580worker1 # [ 178.370808] nix-daemon[839]: accepted connection from pid 2055, user nix-grpc-daemon (trusted)6581worker1 # [ 178.386442] nix-daemon[839]: reaped child process 2058, status = succeeded6582lb # [ 178.407735] niks3-server[1130]: 2026/09/22 10:52:07 INFO Received uploads request method=POST path=/api/pending_closures6583worker1 # [ 178.408108] nix-grpc-daemon[1563]: time=2026-09-22T10:52:08.010Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6584worker1 # [ 178.409546] nix-grpc-daemon[1563]: time=2026-09-22T10:52:08.010Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)"6585lb # [ 178.435292] niks3-server[1130]: 2026/09/22 10:52:08 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst6586worker1 # [ 178.440216] nix-grpc-daemon[1563]: time=2026-09-22T10:52:08.042Z level=INFO msg="Uploading 1 narinfos"6587lb # [ 178.448783] niks3-server[1130]: 2026/09/22 10:52:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign6588lb # [ 178.459352] niks3-server[1130]: 2026/09/22 10:52:08 INFO Signed narinfos id=70 count=16589lb # [ 178.465067] niks3-server[1130]: 2026/09/22 10:52:08 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls6590worker1 # [ 178.464471] nix-grpc-daemon[1563]: time=2026-09-22T10:52:08.067Z level=INFO msg="Upload complete. (464ms)"6591lb # [ 178.473431] niks3-server[1130]: 2026/09/22 10:52:08 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete6592lb # [ 178.477939] niks3-server[1130]: 2026/09/22 10:52:08 INFO Completed upload id=706593lb # [ 178.480195] niks3-server[1130]: 2026/09/22 10:52:08 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo6594worker1 # [ 178.472116] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=0 paths=1 nar_bytes_in=7526595worker1 # [ 178.482981] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:370846596worker1 # [ 178.488865] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6597worker1 # [ 178.493100] nix-daemon[839]: reaped child process 2048, status = succeeded6598worker1 # [ 178.505116] nix-daemon[839]: accepted connection from pid 1486, user nix-grpc-daemon (trusted)6599worker1 # [ 179.743375] nix-daemon[839]: reaped child process 2062, status = succeeded6600worker1 # [ 179.918921] nix-daemon[839]: accepted connection from pid 2078, user nix-grpc-daemon (trusted)6601worker1 # [ 179.935780] nix-daemon[839]: reaped child process 2081, status = succeeded6602lb # [ 179.957347] niks3-server[1130]: 2026/09/22 10:52:09 INFO Received uploads request method=POST path=/api/pending_closures6603worker1 # [ 179.960423] nix-grpc-daemon[1563]: time=2026-09-22T10:52:09.562Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6604worker1 # [ 179.962090] nix-grpc-daemon[1563]: time=2026-09-22T10:52:09.562Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)"6605lb # [ 179.996094] niks3-server[1130]: 2026/09/22 10:52:09 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst6606lb # [ 180.007729] niks3-server[1130]: 2026/09/22 10:52:09 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv6607lb # [ 180.016300] niks3-server[1130]: 2026/09/22 10:52:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign6608worker1 # [ 180.014400] nix-grpc-daemon[1563]: time=2026-09-22T10:52:09.616Z level=INFO msg="Uploading 1 narinfos"6609lb # [ 180.022235] niks3-server[1130]: 2026/09/22 10:52:09 INFO Signed narinfos id=71 count=16610lb # [ 180.026803] niks3-server[1130]: 2026/09/22 10:52:09 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls6611lb # [ 180.037915] niks3-server[1130]: 2026/09/22 10:52:09 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete6612lb # [ 180.045073] niks3-server[1130]: 2026/09/22 10:52:09 INFO Completed upload id=716613worker1 # [ 180.036438] nix-grpc-daemon[1563]: time=2026-09-22T10:52:09.639Z level=INFO msg="Upload complete. (321ms)"6614lb # [ 180.048050] niks3-server[1130]: 2026/09/22 10:52:09 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo6615worker1 # [ 180.048841] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37084 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=14 outputs=16616worker1 # [ 180.061977] nix-daemon[839]: reaped child process 2060, status = succeeded6617client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 18.65 seconds)6618worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6619worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6620client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features '["gpu"]'6621client # [ 180.351899] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6622client # Running as unit: gpu.service; invocation ID: 5659e64a170b4dad97b693f2cdb23b9d6623client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds)6624lb: must succeed: curl -sf localhost:9901/clusters6625lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6626lb: must succeed: curl -sf localhost:9901/clusters6627lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6628worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6629worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6630worker1 # [ 180.562071] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:370846631client # [ 180.603744] nix[1423]: don't know how to build these paths:6632client # [ 180.604898] nix[1423]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv6633worker2 # [ 180.582587] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:406626634worker2 # [ 180.599814] nix-grpc-daemon[1908]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 56 ms (attempt 1/5)6635worker2 # [ 180.663561] nix-grpc-daemon[1908]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 115 ms (attempt 2/5)6636worker2 # [ 180.786110] nix-grpc-daemon[1908]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 49 ms (attempt 3/5)6637worker2 # [ 180.842626] nix-grpc-daemon[1908]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 353 ms (attempt 4/5)6638worker2 # [ 181.203909] nix-grpc-daemon[1908]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6639worker2 # [ 181.207484] nix-grpc-daemon[1908]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6640worker1 # [ 181.230977] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:10Z level=debug event=want client=6 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=06641client # [ 181.267546] nix[1423]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, waiting up to 30s for a worker6642lb: must succeed: curl -sf localhost:9901/clusters6643lb # [ 181.591736] envoy[927]: conn peer=[2001:db8:1::1]:60016 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27200 tx=11445 ms=183316644lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)6645lb: must succeed: curl -sf localhost:9901/clusters6646lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6647worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6648worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6649client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6650client # [ 211.269094] nix[1423]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}6651client # [ 211.278597] nix[1423]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed6652client # [ 211.326855] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6653client # [ 211.333940] systemd[1]: gpu.service: Failed with result 'exit-code'.6654client # [ 211.337206] systemd[1]: gpu.service: Consumed 178ms CPU time over 30.979s wall clock time, 35.4M memory peak, 5.4K incoming IP traffic, 5.1K outgoing IP traffic.6655lb # [ 211.589635] envoy[927]: conn peer=[2001:db8:1::1]:57576 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=914 tx=540 ms=306956656client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 30.31 seconds)6657client: must succeed: journalctl -u gpu -o cat6658client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)6659lb: must succeed: curl -sf localhost:9901/clusters6660lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6661lb: must succeed: curl -sf localhost:9901/clusters6662lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6663worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6664worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6665(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 50.84 seconds)6666subtest: low disk drains a worker and builds go to the other6667worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6668worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.11 seconds)6669worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6670worker1 # [ 213.494529] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:43Z level=info event=unhealthy reason=min_free6671worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 0.05 seconds)6672client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain >&26673client # don't know how to build these paths:6674client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6675worker2 # [ 213.780415] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:43Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:406626676worker1 # [ 213.774968] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:370846677worker1 # [ 213.814560] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:43Z level=debug event=want client=7 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=06678worker1 # [ 213.821804] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:43Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=16679worker2 # [ 213.869554] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:406626680worker2 # [ 213.886634] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:406626681client # copying 1 paths to 'lb:50051'...6682client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'...6683worker2 # [ 213.904117] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:406626684worker2 # [ 213.920777] nix-daemon[866]: accepted connection from pid 1908, user nix-grpc-daemon (trusted)6685worker2 # [ 213.950972] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:43Z level=info event=niks3_push_started pid=19686686worker2 # [ 214.174942] nix-grpc-daemon[1968]: time=2026-09-22T10:52:43.777Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""6687worker2 # [ 214.358953] nix-daemon[866]: accepted connection from pid 1981, user nix-grpc-daemon (trusted)6688worker2 # [ 214.377304] nix-daemon[866]: reaped child process 1984, status = succeeded6689worker2 # [ 214.557577] nix-daemon[866]: accepted connection from pid 1986, user nix-grpc-daemon (trusted)6690worker2 # [ 214.576540] nix-daemon[866]: reaped child process 1989, status = succeeded6691lb # [ 214.593406] niks3-server[1130]: 2026/09/22 10:52:44 INFO Received uploads request method=POST path=/api/pending_closures6692worker2 # [ 214.604075] nix-grpc-daemon[1968]: time=2026-09-22T10:52:44.206Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6693worker2 # [ 214.606722] nix-grpc-daemon[1968]: time=2026-09-22T10:52:44.206Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)"6694lb # [ 214.630956] niks3-server[1130]: 2026/09/22 10:52:44 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst6695worker2 # [ 214.643092] nix-grpc-daemon[1968]: time=2026-09-22T10:52:44.245Z level=INFO msg="Uploading 1 narinfos"6696lb # [ 214.641980] niks3-server[1130]: 2026/09/22 10:52:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign6697lb # [ 214.651771] niks3-server[1130]: 2026/09/22 10:52:44 INFO Signed narinfos id=72 count=16698lb # [ 214.661490] niks3-server[1130]: 2026/09/22 10:52:44 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls6699lb # [ 214.672462] niks3-server[1130]: 2026/09/22 10:52:44 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete6700lb # [ 214.680781] niks3-server[1130]: 2026/09/22 10:52:44 INFO Completed upload id=726701worker2 # [ 214.680741] nix-grpc-daemon[1968]: time=2026-09-22T10:52:44.283Z level=INFO msg="Upload complete. (501ms)"6702lb # [ 214.683565] niks3-server[1130]: 2026/09/22 10:52:44 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo6703worker2 # [ 214.688150] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40662 duration_s=0 paths=1 nar_bytes_in=5126704worker2 # [ 214.703434] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:406626705worker2 # [ 214.709947] nix-daemon[866]: accepted connection from pid 1908, user nix-grpc-daemon (trusted)6706worker2 # [ 214.714935] nix-daemon[866]: reaped child process 1966, status = succeeded6707client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051...6708client # farm-a> worker2-next: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6709worker2 # [ 214.729842] nix-daemon[866]: accepted connection from pid 1908, user nix-grpc-daemon (trusted)6710client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6711client # farm-a> LOG-farm-a-drain6712worker2 # [ 216.169722] nix-daemon[866]: reaped child process 1993, status = succeeded6713worker2 # [ 216.353144] nix-daemon[866]: accepted connection from pid 2009, user nix-grpc-daemon (trusted)6714worker2 # [ 216.370180] nix-daemon[866]: reaped child process 2012, status = succeeded6715lb # [ 216.383863] niks3-server[1130]: 2026/09/22 10:52:45 INFO Received uploads request method=POST path=/api/pending_closures6716worker2 # [ 216.396395] nix-grpc-daemon[1968]: time=2026-09-22T10:52:45.997Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6717worker2 # [ 216.397917] nix-grpc-daemon[1968]: time=2026-09-22T10:52:45.997Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)"6718lb # [ 216.416727] niks3-server[1130]: 2026/09/22 10:52:45 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6719lb # [ 216.435821] niks3-server[1130]: 2026/09/22 10:52:46 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6720lb # [ 216.446687] niks3-server[1130]: 2026/09/22 10:52:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign6721worker2 # [ 216.452143] nix-grpc-daemon[1968]: time=2026-09-22T10:52:46.054Z level=INFO msg="Uploading 1 narinfos"6722lb # [ 216.452696] niks3-server[1130]: 2026/09/22 10:52:46 INFO Signed narinfos id=73 count=16723lb # [ 216.458055] niks3-server[1130]: 2026/09/22 10:52:46 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls6724lb # [ 216.466593] niks3-server[1130]: 2026/09/22 10:52:46 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete6725lb # [ 216.473415] niks3-server[1130]: 2026/09/22 10:52:46 INFO Completed upload id=736726worker2 # [ 216.473348] nix-grpc-daemon[1968]: time=2026-09-22T10:52:46.076Z level=INFO msg="Upload complete. (332ms)"6727lb # [ 216.475942] niks3-server[1130]: 2026/09/22 10:52:46 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo6728worker1 # [ 216.481201] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:46Z level=debug event=want client=7 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=06729worker2 # [ 216.484998] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40662 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=15 outputs=16730worker1 # [ 216.486812] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:46Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=16731worker2 # [ 216.501143] nix-daemon[866]: reaped child process 1991, status = succeeded6732worker2 # [ 216.541248] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:406626733worker2 # [ 216.556391] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:406626734worker2 # [ 216.569736] nix-grpc-daemon[1968]: time=2026-09-22T10:52:46.172Z level=INFO msg="All 1 paths already cached"6735client # copying 1 paths to 'lb:50051'...6736client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'...6737worker2 # [ 216.579544] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:406626738worker2 # [ 216.594034] nix-daemon[866]: accepted connection from pid 1908, user nix-grpc-daemon (trusted)6739worker2 # [ 216.793240] nix-daemon[866]: accepted connection from pid 2016, user nix-grpc-daemon (trusted)6740worker2 # [ 216.809998] nix-daemon[866]: reaped child process 2019, status = succeeded6741worker2 # [ 216.981639] nix-daemon[866]: accepted connection from pid 2021, user nix-grpc-daemon (trusted)6742worker2 # [ 216.997598] nix-daemon[866]: reaped child process 2024, status = succeeded6743lb # [ 217.008400] niks3-server[1130]: 2026/09/22 10:52:46 INFO Received uploads request method=POST path=/api/pending_closures6744worker2 # [ 217.017616] nix-grpc-daemon[1968]: time=2026-09-22T10:52:46.620Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6745worker2 # [ 217.019097] nix-grpc-daemon[1968]: time=2026-09-22T10:52:46.620Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)"6746lb # [ 217.035548] niks3-server[1130]: 2026/09/22 10:52:46 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst6747lb # [ 217.047566] niks3-server[1130]: 2026/09/22 10:52:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign6748worker2 # [ 217.054492] nix-grpc-daemon[1968]: time=2026-09-22T10:52:46.657Z level=INFO msg="Uploading 1 narinfos"6749lb # [ 217.058301] niks3-server[1130]: 2026/09/22 10:52:46 INFO Signed narinfos id=74 count=16750lb # [ 217.065222] niks3-server[1130]: 2026/09/22 10:52:46 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls6751lb # [ 217.072691] niks3-server[1130]: 2026/09/22 10:52:46 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete6752lb # [ 217.078483] niks3-server[1130]: 2026/09/22 10:52:46 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo6753lb # [ 217.083484] niks3-server[1130]: 2026/09/22 10:52:46 INFO Completed upload id=746754worker2 # [ 217.084159] nix-grpc-daemon[1968]: time=2026-09-22T10:52:46.687Z level=INFO msg="Upload complete. (473ms)"6755worker2 # [ 217.096271] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40662 duration_s=0 paths=1 nar_bytes_in=6326756worker2 # [ 217.105246] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:406626757worker2 # [ 217.112432] nix-daemon[866]: accepted connection from pid 1908, user nix-grpc-daemon (trusted)6758worker2 # [ 217.119136] nix-daemon[866]: reaped child process 2014, status = succeeded6759client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051...6760worker2 # [ 217.125997] nix-daemon[866]: accepted connection from pid 1908, user nix-grpc-daemon (trusted)6761client # farm-b> worker2-next: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6762client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6763client # farm-b> LOG-farm-b-drain6764worker2 # [ 218.146910] nix-daemon[866]: reaped child process 2028, status = succeeded6765worker2 # [ 218.310566] nix-daemon[866]: accepted connection from pid 2044, user nix-grpc-daemon (trusted)6766worker2 # [ 218.329867] nix-daemon[866]: reaped child process 2047, status = succeeded6767lb # [ 218.343811] niks3-server[1130]: 2026/09/22 10:52:47 INFO Received uploads request method=POST path=/api/pending_closures6768worker2 # [ 218.356671] nix-grpc-daemon[1968]: time=2026-09-22T10:52:47.958Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6769worker2 # [ 218.358134] nix-grpc-daemon[1968]: time=2026-09-22T10:52:47.959Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)"6770lb # [ 218.382783] niks3-server[1130]: 2026/09/22 10:52:47 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6771lb # [ 218.400442] niks3-server[1130]: 2026/09/22 10:52:47 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst6772lb # [ 218.407952] niks3-server[1130]: 2026/09/22 10:52:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign6773worker2 # [ 218.413021] nix-grpc-daemon[1968]: time=2026-09-22T10:52:48.015Z level=INFO msg="Uploading 1 narinfos"6774lb # [ 218.416985] niks3-server[1130]: 2026/09/22 10:52:47 INFO Signed narinfos id=75 count=16775lb # [ 218.418958] niks3-server[1130]: 2026/09/22 10:52:47 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls6776lb # [ 218.427225] niks3-server[1130]: 2026/09/22 10:52:48 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete6777lb # [ 218.433490] niks3-server[1130]: 2026/09/22 10:52:48 INFO Completed upload id=756778lb # [ 218.435789] niks3-server[1130]: 2026/09/22 10:52:48 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo6779worker2 # [ 218.434185] nix-grpc-daemon[1968]: time=2026-09-22T10:52:48.037Z level=INFO msg="Upload complete. (300ms)"6780worker2 # [ 218.445263] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40662 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=16 outputs=16781worker1 # [ 218.441269] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:48Z level=debug event=want client=7 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=06782worker1 # [ 218.446996] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:48Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=16783worker2 # [ 218.461927] nix-daemon[866]: reaped child process 2026, status = succeeded6784worker2 # [ 218.496991] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:406626785worker2 # [ 218.512555] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:406626786worker2 # [ 218.526636] nix-grpc-daemon[1968]: time=2026-09-22T10:52:48.129Z level=INFO msg="All 2 paths already cached"6787client # copying 1 paths to 'lb:50051'...6788client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'...6789worker2 # [ 218.536947] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:406626790worker2 # [ 218.550692] nix-daemon[866]: accepted connection from pid 1908, user nix-grpc-daemon (trusted)6791worker2 # [ 218.760895] nix-daemon[866]: accepted connection from pid 2051, user nix-grpc-daemon (trusted)6792worker2 # [ 218.781323] nix-daemon[866]: reaped child process 2054, status = succeeded6793worker2 # [ 218.959481] nix-daemon[866]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)6794worker2 # [ 218.977389] nix-daemon[866]: reaped child process 2059, status = succeeded6795lb # [ 218.992560] niks3-server[1130]: 2026/09/22 10:52:48 INFO Received uploads request method=POST path=/api/pending_closures6796worker2 # [ 219.009745] nix-grpc-daemon[1968]: time=2026-09-22T10:52:48.612Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6797worker2 # [ 219.013559] nix-grpc-daemon[1968]: time=2026-09-22T10:52:48.612Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)"6798lb # [ 219.033254] niks3-server[1130]: 2026/09/22 10:52:48 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst6799worker2 # [ 219.051043] nix-grpc-daemon[1968]: time=2026-09-22T10:52:48.653Z level=INFO msg="Uploading 1 narinfos"6800lb # [ 219.045132] niks3-server[1130]: 2026/09/22 10:52:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign6801lb # [ 219.054963] niks3-server[1130]: 2026/09/22 10:52:48 INFO Signed narinfos id=76 count=16802lb # [ 219.059732] niks3-server[1130]: 2026/09/22 10:52:48 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls6803lb # [ 219.069332] niks3-server[1130]: 2026/09/22 10:52:48 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete6804lb # [ 219.073606] niks3-server[1130]: 2026/09/22 10:52:48 INFO Completed upload id=766805worker2 # [ 219.073571] nix-grpc-daemon[1968]: time=2026-09-22T10:52:48.676Z level=INFO msg="Upload complete. (506ms)"6806lb # [ 219.075690] niks3-server[1130]: 2026/09/22 10:52:48 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo6807worker2 # [ 219.080316] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40662 duration_s=0 paths=1 nar_bytes_in=7606808worker2 # [ 219.095503] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:406626809worker2 # [ 219.103048] nix-daemon[866]: accepted connection from pid 1908, user nix-grpc-daemon (trusted)6810worker2 # [ 219.108196] nix-daemon[866]: reaped child process 2049, status = succeeded6811client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051...6812client # farm-top> worker2-next: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6813worker2 # [ 219.121445] nix-daemon[866]: accepted connection from pid 1908, user nix-grpc-daemon (trusted)6814client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6815client # farm-top> LOG-farm-top-drain6816worker2 # [ 220.158349] nix-daemon[866]: reaped child process 2063, status = succeeded6817worker2 # [ 220.338243] nix-daemon[866]: accepted connection from pid 2079, user nix-grpc-daemon (trusted)6818worker2 # [ 220.355041] nix-daemon[866]: reaped child process 2082, status = succeeded6819lb # [ 220.366796] niks3-server[1130]: 2026/09/22 10:52:49 INFO Received uploads request method=POST path=/api/pending_closures6820worker2 # [ 220.380399] nix-grpc-daemon[1968]: time=2026-09-22T10:52:49.982Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6821worker2 # [ 220.381870] nix-grpc-daemon[1968]: time=2026-09-22T10:52:49.982Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)"6822lb # [ 220.407809] niks3-server[1130]: 2026/09/22 10:52:49 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6823lb # [ 220.418996] niks3-server[1130]: 2026/09/22 10:52:49 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst6824lb # [ 220.427951] niks3-server[1130]: 2026/09/22 10:52:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign6825worker2 # [ 220.433705] nix-grpc-daemon[1968]: time=2026-09-22T10:52:50.036Z level=INFO msg="Uploading 1 narinfos"6826lb # [ 220.436215] niks3-server[1130]: 2026/09/22 10:52:50 INFO Signed narinfos id=77 count=16827lb # [ 220.438775] niks3-server[1130]: 2026/09/22 10:52:50 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls6828lb # [ 220.452385] niks3-server[1130]: 2026/09/22 10:52:50 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete6829lb # [ 220.457445] niks3-server[1130]: 2026/09/22 10:52:50 INFO Completed upload id=776830worker2 # [ 220.457705] nix-grpc-daemon[1968]: time=2026-09-22T10:52:50.060Z level=INFO msg="Upload complete. (326ms)"6831lb # [ 220.460552] niks3-server[1130]: 2026/09/22 10:52:50 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo6832worker2 # [ 220.469100] nix-grpc-daemon[1908]: ts=2026-09-22T10:52:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40662 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=17 outputs=16833worker2 # [ 220.486491] nix-daemon[866]: reaped child process 2061, status = succeeded6834client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain >&2, in 7.00 seconds)6835client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain outPath6836client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain outPath, in 0.23 seconds)6837worker1: must succeed: rm /nix/.rw-store/fill6838worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.15 seconds)6839worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'6840worker1 # [ 221.498187] nix-grpc-daemon[1486]: ts=2026-09-22T10:52:51Z level=info event=healthy reason=min_free6841lb # [ 221.588827] envoy[927]: conn peer=[2001:db8:1::1]:45466 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9575 tx=4141 ms=67046842worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.12 seconds)6843(finished: subtest: low disk drains a worker and builds go to the other, in 9.76 seconds)6844(finished: run the VM test script, in 223.08 seconds)6845test script finished in 223.37s6846cleanup6847kill QemuMachine (pid 50)6848client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6849kill QemuMachine (pid 49)6850lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6851kill QemuMachine (pid 52)6852worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6853kill QemuMachine (pid 51)6854worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6855(finished: cleanup, in 1.15 seconds)6856additionally exposed symbols:6857 client, lb, worker1, worker2,6858 vlan1,6859 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh