vm-test-run-nix-grpc-farm
checks.aarch64-linux.farm
· build #471
· raw
1tribuchet: building on eliza2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker1 # Disk image does not exist, creating the virtualisation disk image...13worker1 # Formatting '/build/vm-state-worker1/tmp.FNy7z8jvAS', fmt=raw size=107374182414worker2 # Disk image does not exist, creating the virtualisation disk image...15worker1: QEMU running (pid 51)16worker2: QEMU running (pid 52)17client: QEMU running (pid 49)18lb # Disk image does not exist, creating the virtualisation disk image...19lb: QEMU running (pid 50)20worker2 # Formatting '/build/vm-state-worker2/tmp.V8lnM0Ixs9', fmt=raw size=107374182421lb # Formatting '/build/vm-state-lb/tmp.Ec0R3f5YD0', fmt=raw size=107374182422client # Disk image does not exist, creating the virtualisation disk image...23worker1 # mke2fs 1.47.4 (6-Mar-2025)24worker2 # mke2fs 1.47.4 (6-Mar-2025)25client # Formatting '/build/vm-state-client/tmp.4jN9OSFsna', fmt=raw size=107374182426worker1 # Discarding device blocks: 0/262144 done27worker2 # Discarding device blocks: 0/262144 done28client # mke2fs 1.47.4 (6-Mar-2025)29worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes30worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes31client # Discarding device blocks: 0/262144 done32worker1 # Filesystem UUID: d8132a6a-ae54-4b63-b1bd-4145e3baa66133worker2 # Filesystem UUID: 5eaeb1a5-07ac-4e59-a660-4d29c727ebf834client # Creating filesystem with 262144 4k blocks and 65536 inodes35worker1 # Superblock backups stored on blocks:36worker2 # Superblock backups stored on blocks:37client # Filesystem UUID: 9ef88811-bc5e-4f00-b803-60ab8875d70038worker1 # 32768, 98304, 163840, 22937639worker2 # 32768, 98304, 163840, 22937640client # Superblock backups stored on blocks:41worker1 # 42worker2 # 43client # 32768, 98304, 163840, 22937644worker1 # Allocating group tables: 0/8 done45worker2 # Allocating group tables: 0/8 done46client # 47worker1 # Writing inode tables: 0/8 done48worker2 # Writing inode tables: 0/8 done49client # Allocating group tables: 0/8 done50worker1 # Creating journal (8192 blocks): done51worker2 # Creating journal (8192 blocks): done52client # Writing inode tables: 0/8 done53worker1 # Writing superblocks and filesystem accounting information: 0/8 done54worker2 # Writing superblocks and filesystem accounting information: 0/8 done55client # Creating journal (8192 blocks): done56worker1 # 57worker2 # 58client # Writing superblocks and filesystem accounting information: 0/8 done59worker1 # Virtualisation disk image created.60worker2 # Virtualisation disk image created.61(finished: start all VMs, in 0.47 seconds)62lb # mke2fs 1.47.4 (6-Mar-2025)63client # 64lb # Discarding device blocks: 0/262144 done65client # Virtualisation disk image created.66lb # Creating filesystem with 262144 4k blocks and 65536 inodes67client: waiting for unit network-addresses-eth1.service68lb # Filesystem UUID: 52ff3af8-ca46-4cf7-9f9b-4fa114b7037a69client: waiting for the VM to finish booting70lb # Superblock backups stored on blocks:71lb # 32768, 98304, 163840, 22937672lb # 73lb # Allocating group tables: 0/8 done74lb # Writing inode tables: 0/8 done75lb # Creating journal (8192 blocks): done76lb # Writing superblocks and filesystem accounting information: 0/8 done77lb # 78lb # Virtualisation disk image created.79worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]80worker1 # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 202681worker1 # [ 0.000000] KASLR enabled82worker1 # [ 0.000000] random: crng init done83worker1 # [ 0.000000] Machine model: linux,dummy-virt84worker1 # [ 0.000000] efi: UEFI not found.85worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT86worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]87worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]88worker1 # [ 0.000000] Zone ranges:89worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]90worker1 # [ 0.000000] DMA32 empty91worker1 # [ 0.000000] Normal empty92worker1 # [ 0.000000] Device empty93worker1 # [ 0.000000] Movable zone start for each node94worker1 # [ 0.000000] Early memory node ranges95worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]96worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]97worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c40000098worker1 # [ 0.000000] psci: probing for conduit method from DT.99worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware.100worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs101worker1 # [ 0.000000] psci: Trusted OS migration not required102worker1 # [ 0.000000] psci: SMC Calling Convention v1.1103worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)104worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296105worker1 # [ 0.000000] Detected PIPT I-cache on CPU0106worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)107worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface108worker1 # [ 0.000000] CPU features: detected: Spectre-v4109worker1 # [ 0.000000] CPU features: detected: Spectre-BHB110worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38111worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23112worker1 # [ 0.000000] alternatives: applying boot alternatives113worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/85sl11dh0zky10jc8x3p4y9xadvl89kj-nixos-system-worker1-test/init regInfo=/nix/store/sf77vbcbghgk9hk2xbbk4q5n9r9ql4qy-closure-info/registration console=ttyAMA0,115200n8 console=tty0114worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/sf77vbcbghgk9hk2xbbk4q5n9r9ql4qy-closure-info/registration", will be passed to user space.115worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes116worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)117worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)118worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB119worker1 # [ 0.000000] software IO TLB: area num 1.120worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB121worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)122worker1 # [ 0.000000] Fallback order for Node 0: 0123worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216124worker1 # [ 0.000000] Policy zone: DMA125worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off126worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1127worker1 # [ 0.000000] allocated 3145728 bytes of page_ext128worker1 # [ 0.000000] ftrace: allocating 74885 entries in 294 pages129worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups130worker1 # [ 0.000000] rcu: Hierarchical RCU implementation.131worker1 # [ 0.000000] rcu: RCU event tracing is enabled.132worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.133worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled.134worker1 # [ 0.000000] Rude variant of Tasks RCU enabled.135worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled.136worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.137worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1138worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.139worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.140worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.141worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0142worker1 # [ 0.000000] GICv3: 256 SPIs implemented143worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented144worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq145worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI146worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0147worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000148worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]149worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1)150worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1)151worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000152worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000153worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.154worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns155worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).156worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns157worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns158worker1 # [ 0.000034] arm-pv: using stolen time PV159worker1 # [ 0.000641] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)160worker1 # [ 0.000794] Console: colour dummy device 80x25161worker1 # [ 0.000802] printk: legacy console [tty0] enabled162worker1 # [ 0.001018] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)163worker1 # [ 0.001025] pid_max: default: 32768 minimum: 301164worker1 # [ 0.001124] LSM: initializing lsm=capability,landlock,yama,bpf,ima165worker1 # [ 0.001298] landlock: Up and running.166worker1 # [ 0.001301] Yama: becoming mindful.167client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]168worker1 # [ 0.001940] LSM support for eBPF active169worker1 # [ 0.002112] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)170client # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026171client # [ 0.000000] KASLR enabled172worker1 # [ 0.002145] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)173client # [ 0.000000] random: crng init done174client # [ 0.000000] Machine model: linux,dummy-virt175worker1 # [ 0.003250] cacheinfo: Unable to detect cache hierarchy for CPU 0176client # [ 0.000000] efi: UEFI not found.177worker1 # [ 0.004031] rcu: Hierarchical SRCU implementation.178client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT179worker1 # [ 0.004035] rcu: Max phase no-delay instances is 1000.180worker1 # [ 0.005233] fsl-mc MSI: its@8080000 domain created181client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]182worker1 # [ 0.005323] EFI services will not be available.183client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]184worker1 # [ 0.005438] smp: Bringing up secondary CPUs ...185client # [ 0.000000] Zone ranges:186worker1 # [ 0.005446] smp: Brought up 1 node, 1 CPU187client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]188worker1 # [ 0.005449] SMP: Total of 1 processors activated.189client # [ 0.000000] DMA32 empty190client # [ 0.000000] Normal empty191worker1 # [ 0.005455] CPU: All CPU(s) started at EL1192client # [ 0.000000] Device empty193lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]194worker1 # [ 0.005467] CPU features: detected: Branch Target Identification195client # [ 0.000000] Movable zone start for each node196client # [ 0.000000] Early memory node ranges197worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]198worker1 # [ 0.005472] CPU features: detected: ARMv8.4 Translation Table Level199client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]200lb # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026201lb # [ 0.000000] KASLR enabled202worker1 # [ 0.005475] CPU features: detected: Instruction cache invalidation not required for I/D coherence203client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]204lb # [ 0.000000] random: crng init done205worker2 # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026206worker2 # [ 0.000000] KASLR enabled207lb # [ 0.000000] Machine model: linux,dummy-virt208client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000209worker1 # [ 0.005479] CPU features: detected: Data cache clean to the PoU not required for I/D coherence210worker2 # [ 0.000000] random: crng init done211lb # [ 0.000000] efi: UEFI not found.212client # [ 0.000000] psci: probing for conduit method from DT.213worker2 # [ 0.000000] Machine model: linux,dummy-virt214worker1 # [ 0.005482] CPU features: detected: Common not Private translations215client # [ 0.000000] psci: PSCIv1.3 detected in firmware.216worker2 # [ 0.000000] efi: UEFI not found.217lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT218worker1 # [ 0.005486] CPU features: detected: CRC32 instructions219client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs220lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]221worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT222worker1 # [ 0.005488] CPU features: detected: Data cache clean to Point of Deep Persistence223client # [ 0.000000] psci: Trusted OS migration not required224lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]225worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]226lb # [ 0.000000] Zone ranges:227client # [ 0.000000] psci: SMC Calling Convention v1.1228worker1 # [ 0.005492] CPU features: detected: Data cache clean to Point of Persistence229worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]230lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]231worker2 # [ 0.000000] Zone ranges:232client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)233lb # [ 0.000000] DMA32 empty234worker1 # [ 0.005495] CPU features: detected: Data independent timing control (DIT)235lb # [ 0.000000] Normal empty236worker1 # [ 0.005498] CPU features: detected: E0PD237worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]238client # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296239lb # [ 0.000000] Device empty240worker2 # [ 0.000000] DMA32 empty241worker1 # [ 0.005501] CPU features: detected: Enhanced Counter Virtualization242client # [ 0.000000] Detected PIPT I-cache on CPU0243worker2 # [ 0.000000] Normal empty244lb # [ 0.000000] Movable zone start for each node245worker2 # [ 0.000000] Device empty246lb # [ 0.000000] Early memory node ranges247worker1 # [ 0.005504] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)248worker2 # [ 0.000000] Movable zone start for each node249client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)250lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]251worker2 # [ 0.000000] Early memory node ranges252client # [ 0.000000] CPU features: detected: GICv3 CPU interface253worker1 # [ 0.005507] CPU features: detected: Enhanced Virtualization Traps254lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]255client # [ 0.000000] CPU features: detected: Spectre-v4256worker1 # [ 0.005510] CPU features: detected: Fine Grained Traps257worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]258lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000259client # [ 0.000000] CPU features: detected: Spectre-BHB260worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]261worker1 # [ 0.005515] CPU features: detected: Generic authentication (architected QARMA5 algorithm)262lb # [ 0.000000] psci: probing for conduit method from DT.263worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000264client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38265lb # [ 0.000000] psci: PSCIv1.3 detected in firmware.266worker1 # [ 0.005519] CPU features: detected: RCpc load-acquire (LDAPR)267worker2 # [ 0.000000] psci: probing for conduit method from DT.268client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23269lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs270worker1 # [ 0.005522] CPU features: detected: LSE atomic instructions271worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware.272client # [ 0.000000] alternatives: applying boot alternatives273lb # [ 0.000000] psci: Trusted OS migration not required274worker1 # [ 0.005525] CPU features: detected: Privileged Access Never275worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs276lb # [ 0.000000] psci: SMC Calling Convention v1.1277worker1 # [ 0.005528] CPU features: detected: PMUv3278worker2 # [ 0.000000] psci: Trusted OS migration not required279worker1 # [ 0.005531] CPU features: detected: RAS Extension Support280worker2 # [ 0.000000] psci: SMC Calling Convention v1.1281lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)282lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296283worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)284lb # [ 0.000000] Detected PIPT I-cache on CPU0285worker1 # [ 0.005534] CPU features: detected: RASv1p1 Extension Support286worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296287client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/llyf0bg5i97i0wydcrmwr522c4lqspsl-nixos-system-client-test/init regInfo=/nix/store/qv89ps7hlc2xvxz8pzqw4jziggm25i3g-closure-info/registration console=ttyAMA0,115200n8 console=tty0288worker1 # [ 0.005536] CPU features: detected: Random Number Generator289worker2 # [ 0.000000] Detected PIPT I-cache on CPU0290lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)291worker1 # [ 0.005539] CPU features: detected: Speculation barrier (SB)292worker1 # [ 0.005542] CPU features: detected: Stage-2 Force Write-Back293lb # [ 0.000000] CPU features: detected: GICv3 CPU interface294client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/qv89ps7hlc2xvxz8pzqw4jziggm25i3g-closure-info/registration", will be passed to user space.295worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)296worker1 # [ 0.005545] CPU features: detected: TLB range maintenance instructions297lb # [ 0.000000] CPU features: detected: Spectre-v4298client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes299worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface300lb # [ 0.000000] CPU features: detected: Spectre-BHB301worker1 # [ 0.005550] CPU features: detected: Speculative Store Bypassing Safe (SSBS)302worker2 # [ 0.000000] CPU features: detected: Spectre-v4303client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)304lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38305worker1 # [ 0.005591] alternatives: applying system-wide alternatives306worker2 # [ 0.000000] CPU features: detected: Spectre-BHB307lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23308client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)309worker1 # [ 0.008565] CPU features: detected: BBM Level 2 without TLB conflict abort310worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38311lb # [ 0.000000] alternatives: applying boot alternatives312client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB313worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23314client # [ 0.000000] software IO TLB: area num 1.315worker2 # [ 0.000000] alternatives: applying boot alternatives316worker1 # [ 0.008744] Memory: 1406556K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123772K reserved, 32768K cma-reserved)317client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB318worker1 # [ 0.009135] devtmpfs: initialized319client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)320worker1 # [ 0.010934] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)321client # [ 0.000000] Fallback order for Node 0: 0322worker1 # [ 0.010956] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).323lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/f4b0lk4znyrmdfpm6f1kvcry9mqdiq05-nixos-system-lb-test/init regInfo=/nix/store/lvhvpzdlnfy9xyrdr3lx689a20qywaqb-closure-info/registration console=ttyAMA0,115200n8 console=tty0324client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216325worker1 # [ 0.011143] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL326worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/wykyb0qy7gxax0kbx52lhrandvn5hhwl-nixos-system-worker2-test/init regInfo=/nix/store/vijscrzsxl5jgm579rc9xqvnsymkza5v-closure-info/registration console=ttyAMA0,115200n8 console=tty0327client # [ 0.000000] Policy zone: DMA328worker1 # [ 0.011148] 0 pages in range for non-PLT usage329lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/lvhvpzdlnfy9xyrdr3lx689a20qywaqb-closure-info/registration", will be passed to user space.330worker1 # [ 0.011149] 508288 pages in range for PLT usage331client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off332worker1 # [ 0.011260] pinctrl core: initialized pinctrl subsystem333lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes334client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1335worker1 # [ 0.012093] DMI not present or invalid.336worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/vijscrzsxl5jgm579rc9xqvnsymkza5v-closure-info/registration", will be passed to user space.337client # [ 0.000000] allocated 3145728 bytes of page_ext338lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)339worker1 # [ 0.015900] NET: Registered PF_NETLINK/PF_ROUTE protocol family340worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes341client # [ 0.000000] ftrace: allocating 74885 entries in 294 pages342worker1 # [ 0.018160] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations343client # [ 0.000000] ftrace: allocated 294 pages with 4 groups344lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)345worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)346client # [ 0.000000] rcu: Hierarchical RCU implementation.347lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB348worker1 # [ 0.018341] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations349worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)350client # [ 0.000000] rcu: RCU event tracing is enabled.351lb # [ 0.000000] software IO TLB: area num 1.352worker1 # [ 0.018542] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations353client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.354worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB355lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB356worker1 # [ 0.018564] audit: initializing netlink subsys (disabled)357worker2 # [ 0.000000] software IO TLB: area num 1.358client # [ 0.000000] Trampoline variant of Tasks RCU enabled.359lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)360client # [ 0.000000] Rude variant of Tasks RCU enabled.361worker1 # [ 0.019113] thermal_sys: Registered thermal governor 'fair_share'362worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB363lb # [ 0.000000] Fallback order for Node 0: 0364client # [ 0.000000] Tracing variant of Tasks RCU enabled.365worker1 # [ 0.019115] thermal_sys: Registered thermal governor 'bang_bang'366worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)367lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216368worker1 # [ 0.019119] thermal_sys: Registered thermal governor 'step_wise'369lb # [ 0.000000] Policy zone: DMA370client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.371worker2 # [ 0.000000] Fallback order for Node 0: 0372worker1 # [ 0.019124] thermal_sys: Registered thermal governor 'user_space'373lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off374client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1375worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216376worker2 # [ 0.000000] Policy zone: DMA377worker1 # [ 0.019127] thermal_sys: Registered thermal governor 'power_allocator'378lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1379client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.380worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off381lb # [ 0.000000] allocated 3145728 bytes of page_ext382worker1 # [ 0.019151] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1383worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1384lb # [ 0.000000] ftrace: allocating 74885 entries in 294 pages385client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.386worker1 # [ 0.019160] cpuidle: using governor ladder387lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups388worker2 # [ 0.000000] allocated 3145728 bytes of page_ext389worker1 # [ 0.019165] cpuidle: using governor menu390lb # [ 0.000000] rcu: Hierarchical RCU implementation.391client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.392worker2 # [ 0.000000] ftrace: allocating 74885 entries in 294 pages393worker1 # [ 0.019355] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.394lb # [ 0.000000] rcu: RCU event tracing is enabled.395worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups396client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0397worker1 # [ 0.019370] ASID allocator initialised with 65536 entries398worker2 # [ 0.000000] rcu: Hierarchical RCU implementation.399client # [ 0.000000] GICv3: 256 SPIs implemented400worker1 # [ 0.020605] Serial: AMBA PL011 UART driver401lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.402client # [ 0.000000] GICv3: 0 Extended SPIs implemented403worker2 # [ 0.000000] rcu: RCU event tracing is enabled.404lb # [ 0.000000] Trampoline variant of Tasks RCU enabled.405client # [ 0.000000] Root IRQ handler: gic_handle_irq406worker1 # [ 0.025882] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1407lb # [ 0.000000] Rude variant of Tasks RCU enabled.408worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.409client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI410worker1 # [ 0.026005] printk: console [ttyAMA0] enabled411lb # [ 0.000000] Tracing variant of Tasks RCU enabled.412worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled.413client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0414worker2 # [ 0.000000] Rude variant of Tasks RCU enabled.415lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.416client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000417worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled.418client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]419lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1420worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.421lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.422client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)423worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1424lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.425client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)426worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.427client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000428lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.429worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.430client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000431lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0432lb # [ 0.000000] GICv3: 256 SPIs implemented433worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.434client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.435lb # [ 0.000000] GICv3: 0 Extended SPIs implemented436worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0437lb # [ 0.000000] Root IRQ handler: gic_handle_irq438worker2 # [ 0.000000] GICv3: 256 SPIs implemented439client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns440lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI441worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented442lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0443client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).444worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq445worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI446lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000447worker1 # [ 0.148149] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages448lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]449client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns450worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0451worker1 # [ 0.148167] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page452worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000453client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns454lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)455worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]456client # [ 0.000027] arm-pv: using stolen time PV457worker1 # [ 0.148172] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages458worker1 # [ 0.148176] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page459lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)460worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1)461client # [ 0.000404] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)462lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000463worker1 # [ 0.148181] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages464client # [ 0.000535] Console: colour dummy device 80x25465worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1)466lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000467client # [ 0.000543] printk: legacy console [tty0] enabled468worker1 # [ 0.148185] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page469worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a00000470lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.471worker1 # [ 0.148189] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages472client # [ 0.000747] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)473worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a10000474worker1 # [ 0.148194] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page475client # [ 0.000753] pid_max: default: 32768 minimum: 301476lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns477worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.478client # [ 0.000821] LSM: initializing lsm=capability,landlock,yama,bpf,ima479lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).480client # [ 0.000953] landlock: Up and running.481client # [ 0.000956] Yama: becoming mindful.482worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns483client # [ 0.001410] LSM support for eBPF active484worker1 # [ 0.155309] fbcon: Taking over console485lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns486worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).487worker1 # [ 0.155323] ACPI: Interpreter disabled.488client # [ 0.001565] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)489lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns490worker1 # [ 0.157146] iommu: Default domain type: Translated491lb # [ 0.000027] arm-pv: using stolen time PV492worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns493client # [ 0.001597] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)494worker1 # [ 0.157155] iommu: DMA domain TLB invalidation policy: strict mode495client # [ 0.002613] cacheinfo: Unable to detect cache hierarchy for CPU 0496worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns497lb # [ 0.000368] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)498worker1 # [ 0.158784] SCSI subsystem initialized499worker2 # [ 0.000027] arm-pv: using stolen time PV500client # [ 0.003341] rcu: Hierarchical SRCU implementation.501lb # [ 0.000545] Console: colour dummy device 80x25502client # [ 0.003345] rcu: Max phase no-delay instances is 1000.503lb # [ 0.000553] printk: legacy console [tty0] enabled504worker2 # [ 0.000418] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)505client # [ 0.004525] fsl-mc MSI: its@8080000 domain created506worker2 # [ 0.000592] Console: colour dummy device 80x25507client # [ 0.004615] EFI services will not be available.508lb # [ 0.000751] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)509client # [ 0.004694] smp: Bringing up secondary CPUs ...510worker2 # [ 0.000600] printk: legacy console [tty0] enabled511lb # [ 0.000758] pid_max: default: 32768 minimum: 301512client # [ 0.004702] smp: Brought up 1 node, 1 CPU513lb # [ 0.000853] LSM: initializing lsm=capability,landlock,yama,bpf,ima514client # [ 0.004705] SMP: Total of 1 processors activated.515worker2 # [ 0.000786] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)516lb # [ 0.000978] landlock: Up and running.517client # [ 0.004710] CPU: All CPU(s) started at EL1518lb # [ 0.000980] Yama: becoming mindful.519worker2 # [ 0.000792] pid_max: default: 32768 minimum: 301520lb # [ 0.001410] LSM support for eBPF active521client # [ 0.004720] CPU features: detected: Branch Target Identification522worker2 # [ 0.000858] LSM: initializing lsm=capability,landlock,yama,bpf,ima523client # [ 0.004725] CPU features: detected: ARMv8.4 Translation Table Level524worker2 # [ 0.001007] landlock: Up and running.525lb # [ 0.001559] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)526worker2 # [ 0.001010] Yama: becoming mindful.527worker1 # [ 0.164367] usbcore: registered new interface driver usbfs528worker2 # [ 0.001474] LSM support for eBPF active529lb # [ 0.001597] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)530client # [ 0.004728] CPU features: detected: Instruction cache invalidation not required for I/D coherence531worker1 # [ 0.164399] usbcore: registered new interface driver hub532lb # [ 0.002668] cacheinfo: Unable to detect cache hierarchy for CPU 0533worker2 # [ 0.001619] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)534worker1 # [ 0.164416] usbcore: registered new device driver usb535client # [ 0.004732] CPU features: detected: Data cache clean to the PoU not required for I/D coherence536lb # [ 0.003362] rcu: Hierarchical SRCU implementation.537worker1 # [ 0.164679] pps_core: LinuxPPS API ver. 1 registered538worker2 # [ 0.001652] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)539client # [ 0.004736] CPU features: detected: Common not Private translations540lb # [ 0.003366] rcu: Max phase no-delay instances is 1000.541client # [ 0.004739] CPU features: detected: CRC32 instructions542worker2 # [ 0.002707] cacheinfo: Unable to detect cache hierarchy for CPU 0543lb # [ 0.004546] fsl-mc MSI: its@8080000 domain created544worker1 # [ 0.164684] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>545worker2 # [ 0.003401] rcu: Hierarchical SRCU implementation.546lb # [ 0.004644] EFI services will not be available.547worker1 # [ 0.164695] PTP clock support registered548client # [ 0.004742] CPU features: detected: Data cache clean to Point of Deep Persistence549lb # [ 0.004725] smp: Bringing up secondary CPUs ...550worker1 # [ 0.164742] EDAC MC: Ver: 3.0.0551worker2 # [ 0.003405] rcu: Max phase no-delay instances is 1000.552lb # [ 0.004733] smp: Brought up 1 node, 1 CPU553worker2 # [ 0.004633] fsl-mc MSI: its@8080000 domain created554client # [ 0.004745] CPU features: detected: Data cache clean to Point of Persistence555worker1 # [ 0.169183] scmi_core: SCMI protocol bus registered556lb # [ 0.004737] SMP: Total of 1 processors activated.557worker2 # [ 0.004724] EFI services will not be available.558client # [ 0.004749] CPU features: detected: Data independent timing control (DIT)559lb # [ 0.004745] CPU: All CPU(s) started at EL1560worker2 # [ 0.004834] smp: Bringing up secondary CPUs ...561client # [ 0.004752] CPU features: detected: E0PD562worker1 # [ 0.170118] FPGA manager framework563worker2 # [ 0.004842] smp: Brought up 1 node, 1 CPU564worker1 # [ 0.170988] vgaarb: loaded565lb # [ 0.004756] CPU features: detected: Branch Target Identification566client # [ 0.004754] CPU features: detected: Enhanced Counter Virtualization567worker2 # [ 0.004845] SMP: Total of 1 processors activated.568lb # [ 0.004761] CPU features: detected: ARMv8.4 Translation Table Level569worker2 # [ 0.004850] CPU: All CPU(s) started at EL1570worker1 # [ 0.171594] clocksource: Switched to clocksource arch_sys_counter571client # [ 0.004757] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)572worker2 # [ 0.004862] CPU features: detected: Branch Target Identification573lb # [ 0.004764] CPU features: detected: Instruction cache invalidation not required for I/D coherence574client # [ 0.004760] CPU features: detected: Enhanced Virtualization Traps575worker2 # [ 0.004867] CPU features: detected: ARMv8.4 Translation Table Level576client # [ 0.004763] CPU features: detected: Fine Grained Traps577lb # [ 0.004768] CPU features: detected: Data cache clean to the PoU not required for I/D coherence578client # [ 0.004766] CPU features: detected: Generic authentication (architected QARMA5 algorithm)579worker2 # [ 0.004870] CPU features: detected: Instruction cache invalidation not required for I/D coherence580lb # [ 0.004799] CPU features: detected: Common not Private translations581client # [ 0.004771] CPU features: detected: RCpc load-acquire (LDAPR)582lb # [ 0.004802] CPU features: detected: CRC32 instructions583worker2 # [ 0.004873] CPU features: detected: Data cache clean to the PoU not required for I/D coherence584worker1 # [ 0.175149] VFS: Disk quotas dquot_6.6.0585client # [ 0.004773] CPU features: detected: LSE atomic instructions586lb # [ 0.004805] CPU features: detected: Data cache clean to Point of Deep Persistence587worker2 # [ 0.004877] CPU features: detected: Common not Private translations588worker1 # [ 0.175189] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)589client # [ 0.004777] CPU features: detected: Privileged Access Never590lb # [ 0.004809] CPU features: detected: Data cache clean to Point of Persistence591worker2 # [ 0.004880] CPU features: detected: CRC32 instructions592client # [ 0.004779] CPU features: detected: PMUv3593client # [ 0.004782] CPU features: detected: RAS Extension Support594lb # [ 0.004812] CPU features: detected: Data independent timing control (DIT)595worker2 # [ 0.004883] CPU features: detected: Data cache clean to Point of Deep Persistence596lb # [ 0.004815] CPU features: detected: E0PD597client # [ 0.004785] CPU features: detected: RASv1p1 Extension Support598lb # [ 0.004817] CPU features: detected: Enhanced Counter Virtualization599client # [ 0.004787] CPU features: detected: Random Number Generator600worker1 # [ 0.178698] netfs: FS-Cache loaded601worker1 # [ 0.178815] pnp: PnP ACPI: disabled602lb # [ 0.004820] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)603lb # [ 0.004823] CPU features: detected: Enhanced Virtualization Traps604lb # [ 0.004826] CPU features: detected: Fine Grained Traps605lb # [ 0.004830] CPU features: detected: Generic authentication (architected QARMA5 algorithm)606lb # [ 0.004835] CPU features: detected: RCpc load-acquire (LDAPR)607lb # [ 0.004838] CPU features: detected: LSE atomic instructions608worker1 # [ 0.182613] NET: Registered PF_INET protocol family609lb # [ 0.004841] CPU features: detected: Privileged Access Never610lb # [ 0.004844] CPU features: detected: PMUv3611worker1 # [ 0.182880] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)612lb # [ 0.004846] CPU features: detected: RAS Extension Support613lb # [ 0.004849] CPU features: detected: RASv1p1 Extension Support614lb # [ 0.004852] CPU features: detected: Random Number Generator615worker2 # [ 0.004887] CPU features: detected: Data cache clean to Point of Persistence616lb # [ 0.004854] CPU features: detected: Speculation barrier (SB)617worker2 # [ 0.004890] CPU features: detected: Data independent timing control (DIT)618client # [ 0.004790] CPU features: detected: Speculation barrier (SB)619lb # [ 0.004857] CPU features: detected: Stage-2 Force Write-Back620worker2 # [ 0.004893] CPU features: detected: E0PD621client # [ 0.004792] CPU features: detected: Stage-2 Force Write-Back622worker2 # [ 0.004895] CPU features: detected: Enhanced Counter Virtualization623client # [ 0.004795] CPU features: detected: TLB range maintenance instructions624worker2 # [ 0.004898] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)625client # [ 0.004799] CPU features: detected: Speculative Store Bypassing Safe (SSBS)626worker2 # [ 0.004901] CPU features: detected: Enhanced Virtualization Traps627client # [ 0.004835] alternatives: applying system-wide alternatives628worker2 # [ 0.004904] CPU features: detected: Fine Grained Traps629client # [ 0.007693] CPU features: detected: BBM Level 2 without TLB conflict abort630worker2 # [ 0.004907] CPU features: detected: Generic authentication (architected QARMA5 algorithm)631worker2 # [ 0.004913] CPU features: detected: RCpc load-acquire (LDAPR)632client # [ 0.007882] Memory: 1406508K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123784K reserved, 32768K cma-reserved)633worker2 # [ 0.004915] CPU features: detected: LSE atomic instructions634client # [ 0.008268] devtmpfs: initialized635worker2 # [ 0.004918] CPU features: detected: Privileged Access Never636client # [ 0.010099] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)637worker2 # [ 0.004921] CPU features: detected: PMUv3638worker2 # [ 0.004923] CPU features: detected: RAS Extension Support639client # [ 0.010126] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).640worker2 # [ 0.004926] CPU features: detected: RASv1p1 Extension Support641client # [ 0.010302] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL642worker2 # [ 0.004929] CPU features: detected: Random Number Generator643client # [ 0.010306] 0 pages in range for non-PLT usage644lb # [ 0.004860] CPU features: detected: TLB range maintenance instructions645worker2 # [ 0.004931] CPU features: detected: Speculation barrier (SB)646client # [ 0.010307] 508288 pages in range for PLT usage647lb # [ 0.004864] CPU features: detected: Speculative Store Bypassing Safe (SSBS)648worker2 # [ 0.004934] CPU features: detected: Stage-2 Force Write-Back649client # [ 0.010420] pinctrl core: initialized pinctrl subsystem650lb # [ 0.004901] alternatives: applying system-wide alternatives651client # [ 0.011167] DMI not present or invalid.652worker2 # [ 0.004937] CPU features: detected: TLB range maintenance instructions653lb # [ 0.007776] CPU features: detected: BBM Level 2 without TLB conflict abort654client # [ 0.014316] NET: Registered PF_NETLINK/PF_ROUTE protocol family655worker2 # [ 0.004942] CPU features: detected: Speculative Store Bypassing Safe (SSBS)656client # [ 0.016801] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations657worker2 # [ 0.004977] alternatives: applying system-wide alternatives658lb # [ 0.008005] Memory: 1406536K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123764K reserved, 32768K cma-reserved)659client # [ 0.016985] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations660worker2 # [ 0.007916] CPU features: detected: BBM Level 2 without TLB conflict abort661lb # [ 0.008402] devtmpfs: initialized662client # [ 0.017180] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations663lb # [ 0.010218] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)664client # [ 0.017203] audit: initializing netlink subsys (disabled)665worker2 # [ 0.008164] Memory: 1406544K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123772K reserved, 32768K cma-reserved)666lb # [ 0.010239] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).667worker2 # [ 0.008559] devtmpfs: initialized668client # [ 0.017771] thermal_sys: Registered thermal governor 'fair_share'669lb # [ 0.010422] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL670worker2 # [ 0.010311] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)671client # [ 0.017773] thermal_sys: Registered thermal governor 'bang_bang'672lb # [ 0.010426] 0 pages in range for non-PLT usage673client # [ 0.017776] thermal_sys: Registered thermal governor 'step_wise'674lb # [ 0.010427] 508288 pages in range for PLT usage675worker2 # [ 0.010332] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).676client # [ 0.017781] thermal_sys: Registered thermal governor 'user_space'677lb # [ 0.010516] pinctrl core: initialized pinctrl subsystem678worker2 # [ 0.010519] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL679lb # [ 0.011240] DMI not present or invalid.680client # [ 0.017784] thermal_sys: Registered thermal governor 'power_allocator'681worker2 # [ 0.010524] 0 pages in range for non-PLT usage682lb # [ 0.014555] NET: Registered PF_NETLINK/PF_ROUTE protocol family683worker2 # [ 0.010524] 508288 pages in range for PLT usage684client # [ 0.017806] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1685lb # [ 0.016788] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations686worker2 # [ 0.010637] pinctrl core: initialized pinctrl subsystem687client # [ 0.017817] cpuidle: using governor ladder688worker2 # [ 0.011387] DMI not present or invalid.689client # [ 0.017822] cpuidle: using governor menu690lb # [ 0.016968] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations691worker2 # [ 0.014714] NET: Registered PF_NETLINK/PF_ROUTE protocol family692client # [ 0.018001] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.693lb # [ 0.017163] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations694worker2 # [ 0.017306] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations695client # [ 0.018016] ASID allocator initialised with 65536 entries696lb # [ 0.017184] audit: initializing netlink subsys (disabled)697client # [ 0.019185] Serial: AMBA PL011 UART driver698worker2 # [ 0.017480] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations699lb # [ 0.017772] thermal_sys: Registered thermal governor 'fair_share'700client # [ 0.024379] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1701worker2 # [ 0.017678] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations702lb # [ 0.017773] thermal_sys: Registered thermal governor 'bang_bang'703client # [ 0.024528] printk: console [ttyAMA0] enabled704worker2 # [ 0.017701] audit: initializing netlink subsys (disabled)705lb # [ 0.017777] thermal_sys: Registered thermal governor 'step_wise'706worker2 # [ 0.018251] thermal_sys: Registered thermal governor 'fair_share'707lb # [ 0.017782] thermal_sys: Registered thermal governor 'user_space'708client # [ 0.144367] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages709worker2 # [ 0.018253] thermal_sys: Registered thermal governor 'bang_bang'710lb # [ 0.017785] thermal_sys: Registered thermal governor 'power_allocator'711client # [ 0.144388] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page712worker2 # [ 0.018257] thermal_sys: Registered thermal governor 'step_wise'713lb # [ 0.017815] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1714client # [ 0.144393] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages715worker2 # [ 0.018262] thermal_sys: Registered thermal governor 'user_space'716lb # [ 0.017825] cpuidle: using governor ladder717client # [ 0.144398] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page718lb # [ 0.017830] cpuidle: using governor menu719worker2 # [ 0.018264] thermal_sys: Registered thermal governor 'power_allocator'720client # [ 0.144402] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages721lb # [ 0.018011] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.722worker2 # [ 0.018286] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1723lb # [ 0.018026] ASID allocator initialised with 65536 entries724client # [ 0.144406] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page725worker2 # [ 0.018296] cpuidle: using governor ladder726lb # [ 0.019220] Serial: AMBA PL011 UART driver727worker2 # [ 0.018301] cpuidle: using governor menu728client # [ 0.144411] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages729lb # [ 0.024404] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1730worker2 # [ 0.018489] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.731client # [ 0.144415] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page732lb # [ 0.024561] printk: console [ttyAMA0] enabled733worker2 # [ 0.018503] ASID allocator initialised with 65536 entries734worker2 # [ 0.019656] Serial: AMBA PL011 UART driver735lb # [ 0.143696] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages736worker2 # [ 0.024789] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1737client # [ 0.151750] fbcon: Taking over console738lb # [ 0.143712] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page739worker2 # [ 0.024938] printk: console [ttyAMA0] enabled740worker1 # [ 0.210869] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)741client # [ 0.151764] ACPI: Interpreter disabled.742lb # [ 0.143718] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages743worker1 # [ 0.210910] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)744lb # [ 0.143722] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page745worker2 # [ 0.145549] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages746worker1 # [ 0.210936] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)747worker2 # [ 0.145565] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page748lb # [ 0.143726] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages749worker1 # [ 0.211010] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)750lb # [ 0.143731] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page751worker2 # [ 0.145570] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages752worker1 # [ 0.211155] TCP: Hash tables configured (established 16384 bind 16384)753worker2 # [ 0.145574] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page754lb # [ 0.143735] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages755worker1 # [ 0.211232] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)756lb # [ 0.143739] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page757worker2 # [ 0.145578] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages758worker1 # [ 0.211267] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)759worker2 # [ 0.145582] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page760worker1 # [ 0.211303] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)761worker2 # [ 0.145587] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages762client # [ 0.159006] iommu: Default domain type: Translated763worker1 # [ 0.211389] NET: Registered PF_UNIX/PF_LOCAL protocol family764lb # [ 0.150942] fbcon: Taking over console765worker2 # [ 0.145591] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page766client # [ 0.159015] iommu: DMA domain TLB invalidation policy: strict mode767lb # [ 0.150955] ACPI: Interpreter disabled.768worker1 # [ 0.211417] NET: Registered PF_XDP protocol family769worker1 # [ 0.211434] PCI: CLS 0 bytes, default 64770client # [ 0.160621] SCSI subsystem initialized771worker1 # [ 0.211654] Trying to unpack rootfs image as initramfs...772worker2 # [ 0.152933] fbcon: Taking over console773client # [ 0.161307] usbcore: registered new interface driver usbfs774worker2 # [ 0.152947] ACPI: Interpreter disabled.775client # [ 0.161338] usbcore: registered new interface driver hub776client # [ 0.161354] usbcore: registered new device driver usb777client # [ 0.161619] pps_core: LinuxPPS API ver. 1 registered778client # [ 0.161625] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>779client # [ 0.161635] PTP clock support registered780client # [ 0.161685] EDAC MC: Ver: 3.0.0781client # [ 0.166267] scmi_core: SCMI protocol bus registered782lb # [ 0.159415] iommu: Default domain type: Translated783client # [ 0.167209] FPGA manager framework784client # [ 0.168154] vgaarb: loaded785lb # [ 0.159425] iommu: DMA domain TLB invalidation policy: strict mode786worker1 # [ 0.229025] kvm [1]: HYP mode not available787lb # [ 0.161103] SCSI subsystem initialized788client # [ 0.168795] clocksource: Switched to clocksource arch_sys_counter789worker2 # [ 0.161690] iommu: Default domain type: Translated790lb # [ 0.161792] usbcore: registered new interface driver usbfs791worker2 # [ 0.161699] iommu: DMA domain TLB invalidation policy: strict mode792lb # [ 0.161815] usbcore: registered new interface driver hub793lb # [ 0.161846] usbcore: registered new device driver usb794worker2 # [ 0.163434] SCSI subsystem initialized795client # [ 0.171727] VFS: Disk quotas dquot_6.6.0796lb # [ 0.162100] pps_core: LinuxPPS API ver. 1 registered797client # [ 0.171756] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)798worker2 # [ 0.164121] usbcore: registered new interface driver usbfs799lb # [ 0.162106] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>800lb # [ 0.162115] PTP clock support registered801worker2 # [ 0.164145] usbcore: registered new interface driver hub802lb # [ 0.162158] EDAC MC: Ver: 3.0.0803worker2 # [ 0.164182] usbcore: registered new device driver usb804worker2 # [ 0.164436] pps_core: LinuxPPS API ver. 1 registered805lb # [ 0.166732] scmi_core: SCMI protocol bus registered806client # [ 0.175372] netfs: FS-Cache loaded807client # [ 0.175487] pnp: PnP ACPI: disabled808lb # [ 0.167681] FPGA manager framework809worker2 # [ 0.164442] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>810lb # [ 0.168637] vgaarb: loaded811worker2 # [ 0.164452] PTP clock support registered812worker2 # [ 0.164500] EDAC MC: Ver: 3.0.0813worker2 # [ 0.169120] scmi_core: SCMI protocol bus registered814lb # [ 0.169253] clocksource: Switched to clocksource arch_sys_counter815lb # [ 0.169831] VFS: Disk quotas dquot_6.6.0816worker2 # [ 0.170079] FPGA manager framework817worker2 # [ 0.171013] vgaarb: loaded818lb # [ 0.169863] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)819lb # [ 0.172196] netfs: FS-Cache loaded820lb # [ 0.172306] pnp: PnP ACPI: disabled821worker2 # [ 0.171646] clocksource: Switched to clocksource arch_sys_counter822client # [ 0.179351] NET: Registered PF_INET protocol family823worker2 # [ 0.172239] VFS: Disk quotas dquot_6.6.0824client # [ 0.179606] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)825worker2 # [ 0.172270] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)826worker2 # [ 0.174695] netfs: FS-Cache loaded827worker2 # [ 0.174808] pnp: PnP ACPI: disabled828lb # [ 0.178084] NET: Registered PF_INET protocol family829lb # [ 0.178342] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)830worker2 # [ 0.180562] NET: Registered PF_INET protocol family831worker2 # [ 0.180839] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)832client # [ 0.208592] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)833client # [ 0.208632] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)834client # [ 0.208667] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)835client # [ 0.208740] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)836client # [ 0.212724] TCP: Hash tables configured (established 16384 bind 16384)837client # [ 0.213592] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)838lb # [ 0.207048] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)839client # [ 0.213633] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)840lb # [ 0.207092] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)841client # [ 0.213671] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)842client # [ 0.213773] NET: Registered PF_UNIX/PF_LOCAL protocol family843lb # [ 0.207121] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)844client # [ 0.213845] NET: Registered PF_XDP protocol family845lb # [ 0.207192] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)846client # [ 0.213863] PCI: CLS 0 bytes, default 64847lb # [ 0.207336] TCP: Hash tables configured (established 16384 bind 16384)848client # [ 0.214084] Trying to unpack rootfs image as initramfs...849lb # [ 0.207413] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)850lb # [ 0.207447] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)851lb # [ 0.207483] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)852lb # [ 0.207582] NET: Registered PF_UNIX/PF_LOCAL protocol family853worker2 # [ 0.210602] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)854lb # [ 0.207614] NET: Registered PF_XDP protocol family855worker2 # [ 0.210647] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)856lb # [ 0.207631] PCI: CLS 0 bytes, default 64857lb # [ 0.207848] Trying to unpack rootfs image as initramfs...858worker2 # [ 0.210675] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)859worker2 # [ 0.210751] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)860worker2 # [ 0.210893] TCP: Hash tables configured (established 16384 bind 16384)861client # [ 0.225798] kvm [1]: HYP mode not available862worker2 # [ 0.210975] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)863lb # [ 0.222883] kvm [1]: HYP mode not available864worker2 # [ 0.211012] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)865worker2 # [ 0.211050] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)866worker2 # [ 0.211181] NET: Registered PF_UNIX/PF_LOCAL protocol family867worker2 # [ 0.211243] NET: Registered PF_XDP protocol family868worker2 # [ 0.211260] PCI: CLS 0 bytes, default 64869worker2 # [ 0.211486] Trying to unpack rootfs image as initramfs...870worker2 # [ 0.229471] kvm [1]: HYP mode not available871worker1 # [ 0.313954] Initialise system trusted keyrings872worker1 # [ 0.319645] workingset: timestamp_bits=42 max_order=19 bucket_order=0873worker1 # [ 0.321075] squashfs: version 4.0 (2009/01/31) Phillip Lougher874worker1 # [ 0.321837] 9p: Installing v9fs 9p2000 file system support875worker1 # [ 0.342404] Key type asymmetric registered876worker1 # [ 0.342424] Asymmetric key parser 'x509' registered877worker1 # [ 0.342484] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)878worker1 # [ 0.347640] io scheduler mq-deadline registered879worker1 # [ 0.347655] io scheduler kyber registered880worker1 # [ 0.359736] pl061_gpio 9030000.pl061: PL061 GPIO chip registered881worker1 # [ 0.360340] ledtrig-cpu: registered to indicate activity on CPUs882worker1 # [ 0.360693] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:883worker1 # [ 0.360709] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000884worker1 # [ 0.360720] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000885worker1 # [ 0.360728] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000886worker1 # [ 0.360749] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits887worker1 # [ 0.360770] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]888worker1 # [ 0.360844] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00889worker1 # [ 0.360854] pci_bus 0000:00: root bus resource [bus 00-ff]890worker1 # [ 0.360859] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]891worker1 # [ 0.360865] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]892worker1 # [ 0.360870] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]893worker1 # [ 0.360941] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint894worker1 # [ 0.361369] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint895worker1 # [ 0.361550] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]896worker1 # [ 0.361566] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]897worker1 # [ 0.361595] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]898worker1 # [ 0.361610] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]899client # [ 0.317336] Initialise system trusted keyrings900worker1 # [ 0.362052] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint901worker1 # [ 0.362230] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]902client # [ 0.318074] workingset: timestamp_bits=42 max_order=19 bucket_order=0903worker1 # [ 0.362246] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]904client # [ 0.319374] squashfs: version 4.0 (2009/01/31) Phillip Lougher905worker1 # [ 0.362274] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]906worker1 # [ 0.362710] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint907client # [ 0.320137] 9p: Installing v9fs 9p2000 file system support908worker1 # [ 0.362888] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]909worker1 # [ 0.362903] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]910worker2 # [ 0.314433] Initialise system trusted keyrings911worker1 # [ 0.362932] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]912lb # [ 0.314964] Initialise system trusted keyrings913worker2 # [ 0.315165] workingset: timestamp_bits=42 max_order=19 bucket_order=0914worker1 # [ 0.363371] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint915lb # [ 0.315665] workingset: timestamp_bits=42 max_order=19 bucket_order=0916worker1 # [ 0.363548] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]917worker2 # [ 0.316511] squashfs: version 4.0 (2009/01/31) Phillip Lougher918lb # [ 0.316879] squashfs: version 4.0 (2009/01/31) Phillip Lougher919worker1 # [ 0.363563] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]920worker1 # [ 0.363615] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]921worker1 # [ 0.364074] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint922worker1 # [ 0.364250] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]923worker1 # [ 0.364266] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]924worker1 # [ 0.364295] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]925worker1 # [ 0.364746] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint926worker1 # [ 0.364923] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]927worker1 # [ 0.364939] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]928worker1 # [ 0.364967] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]929worker1 # [ 0.365440] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint930lb # [ 0.317677] 9p: Installing v9fs 9p2000 file system support931worker1 # [ 0.365617] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]932worker1 # [ 0.365632] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]933worker1 # [ 0.365661] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]934worker1 # [ 0.365676] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]935worker1 # [ 0.366115] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint936worker1 # [ 0.366297] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]937worker1 # [ 0.366326] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]938worker1 # [ 0.366763] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint939worker2 # [ 0.323700] 9p: Installing v9fs 9p2000 file system support940worker1 # [ 0.366944] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]941client # [ 0.340670] Key type asymmetric registered942worker1 # [ 0.366973] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]943client # [ 0.340688] Asymmetric key parser 'x509' registered944worker1 # [ 0.367355] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint945client # [ 0.340757] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)946worker1 # [ 0.367531] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]947lb # [ 0.337646] Key type asymmetric registered948lb # [ 0.337663] Asymmetric key parser 'x509' registered949lb # [ 0.337721] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)950client # [ 0.348849] io scheduler mq-deadline registered951client # [ 0.348864] io scheduler kyber registered952worker2 # [ 0.343592] Key type asymmetric registered953worker2 # [ 0.343609] Asymmetric key parser 'x509' registered954worker1 # [ 0.415811] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint955lb # [ 0.345307] io scheduler mq-deadline registered956worker2 # [ 0.344850] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)957worker1 # [ 0.416102] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]958lb # [ 0.345323] io scheduler kyber registered959worker2 # [ 0.345853] io scheduler mq-deadline registered960worker1 # [ 0.416120] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]961worker2 # [ 0.345864] io scheduler kyber registered962worker1 # [ 0.416149] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]963worker1 # [ 0.416614] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint964worker1 # [ 0.416794] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]965worker1 # [ 0.416810] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]966client # [ 0.357030] pl061_gpio 9030000.pl061: PL061 GPIO chip registered967worker1 # [ 0.416839] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]968worker1 # [ 0.417397] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned969worker1 # [ 0.417408] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned970client # [ 0.358321] ledtrig-cpu: registered to indicate activity on CPUs971worker1 # [ 0.417414] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned972client # [ 0.358670] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:973lb # [ 0.353402] pl061_gpio 9030000.pl061: PL061 GPIO chip registered974worker1 # [ 0.417457] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned975client # [ 0.358686] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000976worker1 # [ 0.417502] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned977client # [ 0.358711] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000978worker1 # [ 0.417547] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned979worker2 # [ 0.355791] pl061_gpio 9030000.pl061: PL061 GPIO chip registered980worker1 # [ 0.417592] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned981client # [ 0.358719] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000982lb # [ 0.354698] ledtrig-cpu: registered to indicate activity on CPUs983worker1 # [ 0.417636] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned984client # [ 0.358738] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits985lb # [ 0.355070] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:986worker1 # [ 0.417681] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned987client # [ 0.358762] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]988lb # [ 0.355086] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000989worker1 # [ 0.417727] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned990client # [ 0.358834] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00991lb # [ 0.355098] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000992client # [ 0.358843] pci_bus 0000:00: root bus resource [bus 00-ff]993worker1 # [ 0.417773] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned994lb # [ 0.355107] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000995client # [ 0.358849] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]996worker1 # [ 0.417818] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned997client # [ 0.358854] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]998lb # [ 0.355127] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits999worker1 # [ 0.417877] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1000client # [ 0.358859] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1001lb # [ 0.355155] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1002worker1 # [ 0.417922] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1003client # [ 0.358914] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1004worker2 # [ 0.357110] ledtrig-cpu: registered to indicate activity on CPUs1005lb # [ 0.355237] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001006worker1 # [ 0.417942] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1007lb # [ 0.355246] pci_bus 0000:00: root bus resource [bus 00-ff]1008worker2 # [ 0.357470] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1009client # [ 0.359349] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1010worker1 # [ 0.417963] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1011lb # [ 0.355252] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1012client # [ 0.359533] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1013worker2 # [ 0.357488] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001014worker1 # [ 0.417984] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1015lb # [ 0.355257] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1016client # [ 0.359549] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1017worker1 # [ 0.418007] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1018worker2 # [ 0.357501] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001019client # [ 0.359578] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1020lb # [ 0.355262] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1021worker1 # [ 0.418028] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1022client # [ 0.359594] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1023worker2 # [ 0.357510] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001024lb # [ 0.355316] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1025worker1 # [ 0.418049] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1026client # [ 0.360041] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1027lb # [ 0.355744] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1028worker2 # [ 0.357530] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1029worker1 # [ 0.418070] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1030client # [ 0.360222] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1031lb # [ 0.355930] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1032worker1 # [ 0.418092] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1033worker2 # [ 0.357552] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1034client # [ 0.360238] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1035lb # [ 0.355946] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1036worker1 # [ 0.418113] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1037client # [ 0.360267] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1038worker2 # [ 0.357624] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001039lb # [ 0.355976] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1040worker1 # [ 0.418134] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1041worker2 # [ 0.357633] pci_bus 0000:00: root bus resource [bus 00-ff]1042lb # [ 0.355992] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1043worker2 # [ 0.357639] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1044worker1 # [ 0.418156] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1045lb # [ 0.356444] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1046worker1 # [ 0.418177] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1047worker2 # [ 0.357644] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1048client # [ 0.379471] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1049lb # [ 0.356629] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1050worker1 # [ 0.418197] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1051client # [ 0.379656] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]1052worker2 # [ 0.357649] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1053lb # [ 0.356644] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1054worker1 # [ 0.418218] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1055client # [ 0.379672] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1056lb # [ 0.356673] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1057worker2 # [ 0.357707] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1058worker1 # [ 0.418238] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1059client # [ 0.379702] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1060lb # [ 0.357143] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1061worker1 # [ 0.418259] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1062worker2 # [ 0.358152] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1063client # [ 0.380159] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1064worker2 # [ 0.358339] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1065worker1 # [ 0.418279] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1066client # [ 0.380341] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1067worker2 # [ 0.358355] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1068worker1 # [ 0.418300] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1069client # [ 0.380356] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1070worker2 # [ 0.358385] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1071worker1 # [ 0.418321] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1072client # [ 0.380388] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1073worker2 # [ 0.358401] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1074worker1 # [ 0.418342] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1075worker1 # [ 0.418368] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1076worker2 # [ 0.358855] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1077worker1 # [ 0.418377] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1078worker2 # [ 0.359039] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1079worker1 # [ 0.418382] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1080worker2 # [ 0.359054] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1081worker1 # [ 0.419510] pci 0000:00:0a.0: enabling device (0000 -> 0002)1082worker2 # [ 0.359084] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1083worker2 # [ 0.359532] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1084worker2 # [ 0.359732] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]1085worker2 # [ 0.359748] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1086worker2 # [ 0.359777] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1087lb # [ 0.381363] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]1088lb # [ 0.381385] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1089worker2 # [ 0.360228] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1090worker2 # [ 0.360418] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1091lb # [ 0.381415] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1092worker2 # [ 0.360433] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1093lb # [ 0.381886] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1094worker2 # [ 0.360465] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1095lb # [ 0.382069] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1096lb # [ 0.382084] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1097worker2 # [ 0.360932] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1098worker2 # [ 0.361115] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1099worker1 # [ 0.463739] pci 0000:00:0a.0: quirk_usb_early_handoff+0x0/0xa60 took 43197 usecs1100lb # [ 0.382116] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1101worker2 # [ 0.361131] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1102lb # [ 0.382570] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1103worker2 # [ 0.361160] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1104lb # [ 0.382751] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1105lb # [ 0.382766] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1106worker2 # [ 0.361609] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1107worker2 # [ 0.361793] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1108lb # [ 0.382796] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1109client # [ 0.392898] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1110worker2 # [ 0.361808] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1111client # [ 0.393095] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1112lb # [ 0.383240] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1113worker2 # [ 0.361838] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1114client # [ 0.393111] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1115lb # [ 0.383422] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1116lb # [ 0.383437] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1117client # [ 0.393140] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1118worker2 # [ 0.362294] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1119lb # [ 0.383467] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1120worker2 # [ 0.362477] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1121client # [ 0.393609] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1122worker2 # [ 0.362492] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1123client # [ 0.393792] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1124lb # [ 0.383915] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1125worker2 # [ 0.362522] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1126client # [ 0.393807] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1127lb # [ 0.384097] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1128worker2 # [ 0.362538] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1129lb # [ 0.384113] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1130client # [ 0.393836] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1131lb # [ 0.384142] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1132worker2 # [ 0.363022] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1133client # [ 0.394326] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1134lb # [ 0.384157] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1135worker2 # [ 0.363211] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1136client # [ 0.394509] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1137client # [ 0.394525] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1138lb # [ 0.384644] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1139worker2 # [ 0.363240] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1140lb # [ 0.384925] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1141client # [ 0.394554] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1142client # [ 0.394570] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1143lb # [ 0.384959] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1144client # [ 0.395021] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1145client # [ 0.395207] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1146client # [ 0.395236] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1147client # [ 0.395686] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1148client # [ 0.395871] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1149client # [ 0.395900] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1150client # [ 0.396293] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1151client # [ 0.396474] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1152client # [ 0.396728] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1153client # [ 0.397028] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1154client # [ 0.397045] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1155client # [ 0.397074] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1156worker2 # [ 0.407727] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1157worker1 # [ 0.483772] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1158worker2 # [ 0.407925] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1159client # [ 0.397530] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1160worker1 # [ 0.485921] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1161worker2 # [ 0.407956] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1162client # [ 0.397712] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1163client # [ 0.397727] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1164worker2 # [ 0.408364] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1165client # [ 0.397757] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1166worker2 # [ 0.408552] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1167lb # [ 0.409473] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1168client # [ 0.398335] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1169worker2 # [ 0.408814] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1170lb # [ 0.409674] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1171client # [ 0.398346] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1172worker2 # [ 0.409062] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1173lb # [ 0.409704] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1174worker2 # [ 0.409078] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1175client # [ 0.398352] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1176lb # [ 0.410096] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1177worker2 # [ 0.409108] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1178client # [ 0.398396] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1179lb # [ 0.410277] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1180worker2 # [ 0.409575] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1181client # [ 0.398442] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1182lb # [ 0.410528] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1183worker2 # [ 0.409756] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1184lb # [ 0.410786] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1185client # [ 0.398488] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1186worker2 # [ 0.409772] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1187lb # [ 0.410803] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1188client # [ 0.398534] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1189worker2 # [ 0.409801] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1190lb # [ 0.410833] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1191worker2 # [ 0.410362] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1192client # [ 0.398579] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1193lb # [ 0.411288] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1194worker2 # [ 0.410373] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1195client # [ 0.398624] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1196lb # [ 0.411469] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1197worker2 # [ 0.410379] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1198lb # [ 0.411484] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1199worker1 # [ 0.496182] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1200client # [ 0.398669] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1201worker1 # [ 0.498460] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1202lb # [ 0.411513] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1203worker2 # [ 0.410423] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1204client # [ 0.398714] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1205lb # [ 0.412103] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1206worker2 # [ 0.410469] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1207client # [ 0.398760] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1208lb # [ 0.412114] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1209worker2 # [ 0.410515] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1210client # [ 0.398819] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1211lb # [ 0.412119] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1212client # [ 0.398864] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1213worker2 # [ 0.410561] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1214lb # [ 0.412164] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1215client # [ 0.398886] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1216worker2 # [ 0.410607] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1217lb # [ 0.412211] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1218client # [ 0.398907] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1219worker2 # [ 0.410655] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1220client # [ 0.398928] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1221lb # [ 0.412258] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1222worker2 # [ 0.410703] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1223client # [ 0.398952] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1224lb # [ 0.412305] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1225client # [ 0.398974] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1226worker2 # [ 0.410749] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1227lb # [ 0.412352] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1228client # [ 0.398995] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1229worker2 # [ 0.410795] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1230lb # [ 0.412400] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1231client # [ 0.399017] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1232worker2 # [ 0.410866] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1233lb # [ 0.412447] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1234client # [ 0.399038] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1235worker2 # [ 0.410912] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1236client # [ 0.399060] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1237lb # [ 0.412493] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1238worker2 # [ 0.410933] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1239client # [ 0.399082] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1240lb # [ 0.412540] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1241worker2 # [ 0.410954] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1242worker1 # [ 0.508615] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1243client # [ 0.399104] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1244worker2 # [ 0.410976] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1245lb # [ 0.412609] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1246worker1 # [ 0.510973] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1247client # [ 0.399125] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1248worker2 # [ 0.410999] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1249lb # [ 0.412655] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1250client # [ 0.399146] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1251lb # [ 0.412676] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1252worker2 # [ 0.411021] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1253client # [ 0.399167] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1254lb # [ 0.412697] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1255worker2 # [ 0.411042] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1256client # [ 0.399188] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1257lb # [ 0.412719] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1258worker2 # [ 0.411064] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1259client # [ 0.399210] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1260worker1 # [ 0.514586] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1261lb # [ 0.412743] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1262client # [ 0.399231] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1263worker2 # [ 0.411086] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1264client # [ 0.399252] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1265lb # [ 0.412764] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1266worker2 # [ 0.411107] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1267client # [ 0.399274] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1268lb # [ 0.412800] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1269worker2 # [ 0.411130] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1270client # [ 0.399295] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1271lb # [ 0.412822] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1272worker2 # [ 0.411152] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1273client # [ 0.399322] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1274worker2 # [ 0.411173] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1275lb # [ 0.412844] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1276client # [ 0.399331] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1277worker2 # [ 0.411194] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1278lb # [ 0.412866] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1279client # [ 0.399336] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1280worker2 # [ 0.411215] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1281lb # [ 0.412888] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1282client # [ 0.400485] pci 0000:00:0a.0: enabling device (0000 -> 0002)1283worker2 # [ 0.411236] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1284lb # [ 0.412910] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1285worker2 # [ 0.411257] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1286lb # [ 0.412931] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1287worker2 # [ 0.411279] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1288lb # [ 0.412952] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1289worker1 # [ 0.523047] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1290worker2 # [ 0.411301] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1291lb # [ 0.412974] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1292worker2 # [ 0.411322] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1293lb # [ 0.412995] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1294worker2 # [ 0.411344] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1295lb # [ 0.413016] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1296worker2 # [ 0.411370] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1297lb # [ 0.413038] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1298worker2 # [ 0.411379] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1299lb # [ 0.413059] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1300worker2 # [ 0.411384] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1301lb # [ 0.413081] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1302lb # [ 0.413102] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1303lb # [ 0.413129] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1304client # [ 0.460909] pci 0000:00:0a.0: quirk_usb_early_handoff+0x0/0xa60 took 59013 usecs1305lb # [ 0.413138] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1306lb # [ 0.413143] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1307lb # [ 0.414330] pci 0000:00:0a.0: enabling device (0000 -> 0002)1308worker1 # [ 0.529176] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1309worker2 # [ 0.456568] pci 0000:00:0a.0: enabling device (0000 -> 0002)1310worker1 # [ 0.530893] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1311worker1 # [ 0.541008] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1312client # [ 0.480921] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1313client # [ 0.482967] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1314worker1 # [ 0.553914] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1315lb # [ 0.479392] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1316worker2 # [ 0.481648] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1317worker1 # [ 0.556389] msm_serial: driver initialized1318worker1 # [ 0.556566] SuperH (H)SCI(F) driver initialized1319worker1 # [ 0.556619] STM32 USART driver initialized1320client # [ 0.492973] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1321client # [ 0.495354] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1322worker2 # [ 0.487813] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1323worker2 # [ 0.490046] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1324lb # [ 0.489459] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1325lb # [ 0.492992] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1326lb # [ 0.494974] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1327lb # [ 0.496966] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1328client # [ 0.504984] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1329client # [ 0.507958] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1330worker2 # [ 0.499922] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1331worker2 # [ 0.501915] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1332lb # [ 0.506637] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1333lb # [ 0.508711] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1334client # [ 0.519434] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1335worker1 # [ 0.588360] loop: module loaded1336worker1 # [ 0.588532] virtio_blk virtio5: 1/0/0 default/read/poll queues1337worker1 # [ 0.589259] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1338client # [ 0.522437] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1339client # [ 0.524399] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1340worker2 # [ 0.511800] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1341worker2 # [ 0.514752] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1342lb # [ 0.520287] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1343worker2 # [ 0.516831] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1344worker2 # [ 0.518862] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1345lb # [ 0.526936] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1346client # [ 0.534184] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1347client # [ 0.536769] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1348lb # [ 0.528726] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1349worker1 # [ 0.600199] megasas: 07.734.00.00-rc11350worker2 # [ 0.528926] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1351worker1 # [ 0.600903] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1352worker2 # [ 0.531462] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1353worker1 # [ 0.603190] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001354worker1 # [ 0.603214] Intel/Sharp Extended Query Table at 0x00311355worker1 # [ 0.604789] Using buffer write method1356worker1 # [ 0.604850] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1357worker1 # [ 0.606484] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001358worker1 # [ 0.606507] Intel/Sharp Extended Query Table at 0x00311359lb # [ 0.538763] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1360worker1 # [ 0.624231] Using buffer write method1361worker2 # [ 0.548937] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1362client # [ 0.557279] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1363worker1 # [ 0.624265] Concatenating MTD devices:1364worker1 # [ 0.624270] (0): "0.flash"1365worker1 # [ 0.624274] (1): "0.flash"1366worker1 # [ 0.624277] into device "0.flash"1367worker2 # [ 0.551490] msm_serial: driver initialized1368worker2 # [ 0.551628] SuperH (H)SCI(F) driver initialized1369client # [ 0.559809] msm_serial: driver initialized1370client # [ 0.559958] SuperH (H)SCI(F) driver initialized1371client # [ 0.560010] STM32 USART driver initialized1372lb # [ 0.553443] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1373lb # [ 0.556010] msm_serial: driver initialized1374lb # [ 0.556167] SuperH (H)SCI(F) driver initialized1375lb # [ 0.556221] STM32 USART driver initialized1376worker2 # [ 0.559740] STM32 USART driver initialized1377client # [ 0.586627] loop: module loaded1378client # [ 0.586801] virtio_blk virtio5: 1/0/0 default/read/poll queues1379client # [ 0.587505] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1380lb # [ 0.584015] loop: module loaded1381lb # [ 0.584182] virtio_blk virtio5: 1/0/0 default/read/poll queues1382lb # [ 0.584932] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1383worker2 # [ 0.587895] loop: module loaded1384worker2 # [ 0.588065] virtio_blk virtio5: 1/0/0 default/read/poll queues1385worker2 # [ 0.588797] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1386client # [ 0.601353] megasas: 07.734.00.00-rc11387client # [ 0.602037] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1388client # [ 0.604251] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001389client # [ 0.604275] Intel/Sharp Extended Query Table at 0x00311390worker2 # [ 0.596262] megasas: 07.734.00.00-rc11391client # [ 0.605819] Using buffer write method1392worker2 # [ 0.597083] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1393lb # [ 0.597844] megasas: 07.734.00.00-rc11394client # [ 0.605880] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1395lb # [ 0.598551] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1396worker2 # [ 0.599211] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001397worker2 # [ 0.599236] Intel/Sharp Extended Query Table at 0x00311398worker2 # [ 0.600844] Using buffer write method1399lb # [ 0.600697] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001400client # [ 0.607611] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001401lb # [ 0.600721] Intel/Sharp Extended Query Table at 0x00311402client # [ 0.607632] Intel/Sharp Extended Query Table at 0x00311403worker2 # [ 0.600912] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1404lb # [ 0.602281] Using buffer write method1405client # [ 0.616780] Using buffer write method1406client # [ 0.617261] Concatenating MTD devices:1407lb # [ 0.602339] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1408client # [ 0.617266] (0): "0.flash"1409client # [ 0.617270] (1): "0.flash"1410worker2 # [ 0.602580] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001411client # [ 0.617274] into device "0.flash"1412worker2 # [ 0.602610] Intel/Sharp Extended Query Table at 0x00311413lb # [ 0.603978] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001414lb # [ 0.604005] Intel/Sharp Extended Query Table at 0x00311415worker2 # [ 0.616150] Using buffer write method1416worker2 # [ 0.616174] Concatenating MTD devices:1417worker2 # [ 0.616178] (0): "0.flash"1418worker2 # [ 0.616182] (1): "0.flash"1419worker2 # [ 0.616186] into device "0.flash"1420lb # [ 0.622892] Using buffer write method1421lb # [ 0.622933] Concatenating MTD devices:1422lb # [ 0.622937] (0): "0.flash"1423lb # [ 0.622942] (1): "0.flash"1424lb # [ 0.622945] into device "0.flash"1425worker1 # [ 0.847955] Freeing initrd memory: 26148K1426worker1 # [ 0.853961] tun: Universal TUN/TAP device driver, 1.61427worker1 # [ 0.857626] thunder_xcv, ver 1.01428worker1 # [ 0.857669] thunder_bgx, ver 1.01429worker1 # [ 0.857691] nicpf, ver 1.01430worker1 # [ 0.858254] e1000: Intel(R) PRO/1000 Network Driver1431worker1 # [ 0.858261] e1000: Copyright (c) 1999-2006 Intel Corporation.1432worker1 # [ 0.858286] e1000e: Intel(R) PRO/1000 Network Driver1433worker1 # [ 0.858294] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1434worker1 # [ 0.858323] igb: Intel(R) Gigabit Ethernet Network Driver1435worker1 # [ 0.858328] igb: Copyright (c) 2007-2014 Intel Corporation.1436worker1 # [ 0.858353] igbvf: Intel(R) Gigabit Virtual Function Network Driver1437worker1 # [ 0.858358] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1438worker1 # [ 0.858500] sky2: driver version 1.301439worker1 # [ 0.867096] usbcore: registered new interface driver usb-storage1440worker1 # [ 0.867147] usbcore: registered new interface driver usbserial_generic1441worker1 # [ 0.867162] usbserial: USB Serial support registered for generic1442worker1 # [ 0.868324] ehci-pci 0000:00:0a.0: EHCI Host Controller1443worker1 # [ 0.868349] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11444worker1 # [ 0.868623] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001445worker1 # [ 0.873416] hv_vmbus: registering driver hyperv_keyboard1446worker1 # [ 0.874923] rtc-pl031 9010000.pl031: registered as rtc01447worker1 # [ 0.874948] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T12:32:53 UTC (1789993973)1448worker1 # [ 0.875253] i2c_dev: i2c /dev entries driver1449worker1 # [ 0.880302] sdhci: Secure Digital Host Controller Interface driver1450worker1 # [ 0.880312] sdhci: Copyright(c) Pierre Ossman1451worker1 # [ 0.880589] Synopsys Designware Multimedia Card Interface Driver1452worker1 # [ 0.880972] sdhci-pltfm: SDHCI platform and OF driver helper1453worker1 # [ 0.882687] hid: raw HID events driver (C) Jiri Kosina1454worker1 # [ 0.882942] usbcore: registered new interface driver usbhid1455worker1 # [ 0.882948] usbhid: USB HID core driver1456worker1 # [ 0.888133] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001457worker1 # [ 0.889161] hub 1-0:1.0: USB hub found1458worker1 # [ 0.889649] hub 1-0:1.0: 6 ports detected1459worker1 # [ 0.891045] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1460worker1 # [ 0.893762] drop_monitor: Initializing network drop monitor service1461worker1 # [ 0.893906] NET: Registered PF_INET6 protocol family1462worker1 # [ 0.895803] Segment Routing with IPv61463worker1 # [ 0.895822] In-situ OAM (IOAM) with IPv61464worker1 # [ 0.895847] NET: Registered PF_PACKET protocol family1465worker1 # [ 0.897477] 9pnet: Installing 9P2000 support1466worker1 # [ 0.899511] Key type dns_resolver registered1467client # [ 0.836504] Freeing initrd memory: 26160K1468worker1 # [ 0.906384] registered taskstats version 11469worker1 # [ 0.906537] Loading compiled-in X.509 certificates1470client # [ 0.842736] tun: Universal TUN/TAP device driver, 1.61471lb # [ 0.837699] Freeing initrd memory: 26140K1472worker1 # [ 0.915848] Demotion targets for Node 0: null1473worker1 # [ 0.915980] Key type .fscrypt registered1474client # [ 0.846356] thunder_xcv, ver 1.01475worker1 # [ 0.915985] Key type fscrypt-provisioning registered1476client # [ 0.846395] thunder_bgx, ver 1.01477client # [ 0.846416] nicpf, ver 1.01478worker1 # [ 0.916080] ima: No TPM chip found, activating TPM-bypass!1479client # [ 0.846949] e1000: Intel(R) PRO/1000 Network Driver1480worker1 # [ 0.916099] ima: Allocated hash algorithm: sha11481lb # [ 0.843619] tun: Universal TUN/TAP device driver, 1.61482worker1 # [ 0.916119] ima: No architecture policies found1483client # [ 0.846956] e1000: Copyright (c) 1999-2006 Intel Corporation.1484client # [ 0.846980] e1000e: Intel(R) PRO/1000 Network Driver1485worker1 # [ 0.920112] input: gpio-keys as /devices/platform/gpio-keys/input/input01486client # [ 0.846988] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1487client # [ 0.847017] igb: Intel(R) Gigabit Ethernet Network Driver1488client # [ 0.847022] igb: Copyright (c) 2007-2014 Intel Corporation.1489client # [ 0.847045] igbvf: Intel(R) Gigabit Virtual Function Network Driver1490worker2 # [ 0.847697] Freeing initrd memory: 26148K1491client # [ 0.847050] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1492client # [ 0.847179] sky2: driver version 1.301493lb # [ 0.847369] thunder_xcv, ver 1.01494lb # [ 0.847412] thunder_bgx, ver 1.01495client # [ 0.848728] usbcore: registered new interface driver usb-storage1496lb # [ 0.847433] nicpf, ver 1.01497client # [ 0.849685] ehci-pci 0000:00:0a.0: EHCI Host Controller1498lb # [ 0.847974] e1000: Intel(R) PRO/1000 Network Driver1499client # [ 0.849709] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11500lb # [ 0.847981] e1000: Copyright (c) 1999-2006 Intel Corporation.1501lb # [ 0.848003] e1000e: Intel(R) PRO/1000 Network Driver1502client # [ 0.850041] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001503lb # [ 0.848011] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1504lb # [ 0.848043] igb: Intel(R) Gigabit Ethernet Network Driver1505client # [ 0.860282] usbcore: registered new interface driver usbserial_generic1506lb # [ 0.848048] igb: Copyright (c) 2007-2014 Intel Corporation.1507client # [ 0.860301] usbserial: USB Serial support registered for generic1508worker2 # [ 0.853561] tun: Universal TUN/TAP device driver, 1.61509lb # [ 0.848071] igbvf: Intel(R) Gigabit Virtual Function Network Driver1510client # [ 0.862329] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001511lb # [ 0.848077] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1512client # [ 0.863335] hub 1-0:1.0: USB hub found1513lb # [ 0.848208] sky2: driver version 1.301514client # [ 0.863827] hub 1-0:1.0: 6 ports detected1515lb # [ 0.856935] usbcore: registered new interface driver usb-storage1516client # [ 0.864857] hv_vmbus: registering driver hyperv_keyboard1517lb # [ 0.857032] usbcore: registered new interface driver usbserial_generic1518lb # [ 0.857047] usbserial: USB Serial support registered for generic1519client # [ 0.866347] rtc-pl031 9010000.pl031: registered as rtc01520lb # [ 0.860108] ehci-pci 0000:00:0a.0: EHCI Host Controller1521client # [ 0.866373] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T12:32:53 UTC (1789993973)1522worker2 # [ 0.857193] thunder_xcv, ver 1.01523client # [ 0.866667] i2c_dev: i2c /dev entries driver1524worker2 # [ 0.857235] thunder_bgx, ver 1.01525lb # [ 0.860135] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11526worker2 # [ 0.857257] nicpf, ver 1.01527lb # [ 0.860397] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001528worker2 # [ 0.857782] e1000: Intel(R) PRO/1000 Network Driver1529worker2 # [ 0.857789] e1000: Copyright (c) 1999-2006 Intel Corporation.1530worker2 # [ 0.857813] e1000e: Intel(R) PRO/1000 Network Driver1531lb # [ 0.863288] hv_vmbus: registering driver hyperv_keyboard1532worker2 # [ 0.857821] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1533worker1 # [ 0.938042] clk: Disabling unused clocks1534worker1 # [ 0.938067] PM: genpd: Disabling unused power domains1535worker2 # [ 0.857850] igb: Intel(R) Gigabit Ethernet Network Driver1536client # [ 0.871631] sdhci: Secure Digital Host Controller Interface driver1537lb # [ 0.864834] rtc-pl031 9010000.pl031: registered as rtc01538client # [ 0.871642] sdhci: Copyright(c) Pierre Ossman1539worker2 # [ 0.857856] igb: Copyright (c) 2007-2014 Intel Corporation.1540client # [ 0.871901] Synopsys Designware Multimedia Card Interface Driver1541worker2 # [ 0.857878] igbvf: Intel(R) Gigabit Virtual Function Network Driver1542lb # [ 0.864861] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T12:32:53 UTC (1789993973)1543lb # [ 0.865168] i2c_dev: i2c /dev entries driver1544client # [ 0.872260] sdhci-pltfm: SDHCI platform and OF driver helper1545worker2 # [ 0.857884] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1546worker2 # [ 0.858012] sky2: driver version 1.301547worker1 # [ 0.942251] Freeing unused kernel memory: 4736K1548worker1 # [ 0.942444] Run /init as init process1549worker2 # [ 0.859516] usbcore: registered new interface driver usb-storage1550client # [ 0.876516] hid: raw HID events driver (C) Jiri Kosina1551worker2 # [ 0.859611] usbcore: registered new interface driver usbserial_generic1552client # [ 0.876761] usbcore: registered new interface driver usbhid1553worker2 # [ 0.859625] usbserial: USB Serial support registered for generic1554client # [ 0.876768] usbhid: USB HID core driver1555worker2 # [ 0.860313] ehci-pci 0000:00:0a.0: EHCI Host Controller1556worker2 # [ 0.860339] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11557worker2 # [ 0.860641] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001558worker2 # [ 0.872139] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001559lb # [ 0.870057] sdhci: Secure Digital Host Controller Interface driver1560client # [ 0.879673] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1561worker2 # [ 0.872527] hub 1-0:1.0: USB hub found1562lb # [ 0.870067] sdhci: Copyright(c) Pierre Ossman1563worker2 # [ 0.872548] hub 1-0:1.0: 6 ports detected1564lb # [ 0.870328] Synopsys Designware Multimedia Card Interface Driver1565client # [ 0.882229] drop_monitor: Initializing network drop monitor service1566lb # [ 0.870693] sdhci-pltfm: SDHCI platform and OF driver helper1567client # [ 0.882406] NET: Registered PF_INET6 protocol family1568lb # [ 0.872318] hid: raw HID events driver (C) Jiri Kosina1569worker2 # [ 0.875155] hv_vmbus: registering driver hyperv_keyboard1570client # [ 0.884313] Segment Routing with IPv61571lb # [ 0.872554] usbcore: registered new interface driver usbhid1572client # [ 0.884335] In-situ OAM (IOAM) with IPv61573lb # [ 0.872561] usbhid: USB HID core driver1574client # [ 0.884362] NET: Registered PF_PACKET protocol family1575worker2 # [ 0.876641] rtc-pl031 9010000.pl031: registered as rtc01576lb # [ 0.877703] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001577lb # [ 0.878032] hub 1-0:1.0: USB hub found1578worker2 # [ 0.876667] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T12:32:53 UTC (1789993973)1579lb # [ 0.878063] hub 1-0:1.0: 6 ports detected1580worker2 # [ 0.876956] i2c_dev: i2c /dev entries driver1581client # [ 0.886189] 9pnet: Installing 9P2000 support1582client # [ 0.888303] Key type dns_resolver registered1583lb # [ 0.880508] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1584worker1 # [ 0.957772] systemd[1]: Successfully made /usr/ read-only.1585lb # [ 0.883127] drop_monitor: Initializing network drop monitor service1586worker2 # [ 0.881896] sdhci: Secure Digital Host Controller Interface driver1587lb # [ 0.883264] NET: Registered PF_INET6 protocol family1588worker2 # [ 0.881906] sdhci: Copyright(c) Pierre Ossman1589lb # [ 0.885177] Segment Routing with IPv61590worker2 # [ 0.882166] Synopsys Designware Multimedia Card Interface Driver1591lb # [ 0.885196] In-situ OAM (IOAM) with IPv61592worker2 # [ 0.882519] sdhci-pltfm: SDHCI platform and OF driver helper1593lb # [ 0.885229] NET: Registered PF_PACKET protocol family1594worker2 # [ 0.886914] hid: raw HID events driver (C) Jiri Kosina1595worker2 # [ 0.887159] usbcore: registered new interface driver usbhid1596client # [ 0.894987] registered taskstats version 11597lb # [ 0.886863] 9pnet: Installing 9P2000 support1598worker2 # [ 0.887166] usbhid: USB HID core driver1599client # [ 0.895144] Loading compiled-in X.509 certificates1600lb # [ 0.888992] Key type dns_resolver registered1601worker2 # [ 0.889998] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1602worker2 # [ 0.891444] drop_monitor: Initializing network drop monitor service1603worker2 # [ 0.891565] NET: Registered PF_INET6 protocol family1604worker2 # [ 0.894575] Segment Routing with IPv61605worker2 # [ 0.894605] In-situ OAM (IOAM) with IPv61606worker2 # [ 0.894635] NET: Registered PF_PACKET protocol family1607client # [ 0.903559] Demotion targets for Node 0: null1608client # [ 0.903660] Key type .fscrypt registered1609lb # [ 0.895874] registered taskstats version 11610client # [ 0.903667] Key type fscrypt-provisioning registered1611lb # [ 0.896015] Loading compiled-in X.509 certificates1612client # [ 0.903754] ima: No TPM chip found, activating TPM-bypass!1613client # [ 0.903774] ima: Allocated hash algorithm: sha11614client # [ 0.903795] ima: No architecture policies found1615worker2 # [ 0.896505] 9pnet: Installing 9P2000 support1616client # [ 0.907895] input: gpio-keys as /devices/platform/gpio-keys/input/input01617worker2 # [ 0.898546] Key type dns_resolver registered1618lb # [ 0.904510] Demotion targets for Node 0: null1619lb # [ 0.904615] Key type .fscrypt registered1620lb # [ 0.904622] Key type fscrypt-provisioning registered1621lb # [ 0.904710] ima: No TPM chip found, activating TPM-bypass!1622lb # [ 0.904729] ima: Allocated hash algorithm: sha11623lb # [ 0.904748] ima: No architecture policies found1624worker2 # [ 0.905168] registered taskstats version 11625worker2 # [ 0.905319] Loading compiled-in X.509 certificates1626lb # [ 0.908949] input: gpio-keys as /devices/platform/gpio-keys/input/input01627worker2 # [ 0.913735] Demotion targets for Node 0: null1628worker2 # [ 0.913839] Key type .fscrypt registered1629worker2 # [ 0.913846] Key type fscrypt-provisioning registered1630worker2 # [ 0.913939] ima: No TPM chip found, activating TPM-bypass!1631worker2 # [ 0.913958] ima: Allocated hash algorithm: sha11632worker2 # [ 0.913978] ima: No architecture policies found1633worker2 # [ 0.918034] input: gpio-keys as /devices/platform/gpio-keys/input/input01634client # [ 0.925139] clk: Disabling unused clocks1635client # [ 0.925162] PM: genpd: Disabling unused power domains1636client # [ 0.929360] Freeing unused kernel memory: 4736K1637client # [ 0.929557] Run /init as init process1638lb # [ 0.926232] clk: Disabling unused clocks1639lb # [ 0.926260] PM: genpd: Disabling unused power domains1640lb # [ 0.930446] Freeing unused kernel memory: 4736K1641lb # [ 0.930652] Run /init as init process1642worker2 # [ 0.935889] clk: Disabling unused clocks1643worker2 # [ 0.935912] PM: genpd: Disabling unused power domains1644client # [ 0.946931] systemd[1]: Successfully made /usr/ read-only.1645worker2 # [ 0.940150] Freeing unused kernel memory: 4736K1646worker2 # [ 0.940343] Run /init as init process1647lb # [ 0.947448] systemd[1]: Successfully made /usr/ read-only.1648worker2 # [ 0.957817] systemd[1]: Successfully made /usr/ read-only.1649worker1 # [ 1.139663] usb 1-1: new high-speed USB device number 2 using ehci-pci1650client # [ 1.108865] usb 1-1: new high-speed USB device number 2 using ehci-pci1651worker2 # [ 1.119710] usb 1-1: new high-speed USB device number 2 using ehci-pci1652lb # [ 1.125333] usb 1-1: new high-speed USB device number 2 using ehci-pci1653worker1 # [ 1.292168] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11654worker1 # [ 1.298003] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1655worker1 # [ 1.310270] systemd[1]: Detected virtualization qemu.1656worker1 # [ 1.312403] systemd[1]: Detected architecture arm64.1657worker1 # [ 1.314294] systemd[1]: Running in initrd.1658worker1 # [ 1.316865] systemd[1]: Initializing machine ID from random generator.1659worker1 # [ 1.319709] systemd[1]: Hostname set to <worker1>.1660client # [ 1.263041] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11661worker2 # [ 1.271962] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11662lb # [ 1.277535] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11663client # [ 1.281688] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1664client # [ 1.293611] systemd[1]: Detected virtualization qemu.1665client # [ 1.295572] systemd[1]: Detected architecture arm64.1666client # [ 1.297515] systemd[1]: Running in initrd.1667client # [ 1.300020] systemd[1]: Initializing machine ID from random generator.1668lb # [ 1.283445] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1669client # [ 1.302926] systemd[1]: Hostname set to <client>.1670lb # [ 1.295594] systemd[1]: Detected virtualization qemu.1671lb # [ 1.297650] systemd[1]: Detected architecture arm64.1672lb # [ 1.299566] systemd[1]: Running in initrd.1673worker2 # [ 1.292612] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1674lb # [ 1.300514] systemd[1]: Initializing machine ID from random generator.1675lb # [ 1.300816] systemd[1]: Hostname set to <lb>.1676worker2 # [ 1.304794] systemd[1]: Detected virtualization qemu.1677worker2 # [ 1.306819] systemd[1]: Detected architecture arm64.1678worker2 # [ 1.308786] systemd[1]: Running in initrd.1679worker1 # [ 1.383838] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01680worker2 # [ 1.311283] systemd[1]: Initializing machine ID from random generator.1681worker2 # [ 1.314229] systemd[1]: Hostname set to <worker2>.1682client # [ 1.353065] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01683worker2 # [ 1.359909] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01684lb # [ 1.361547] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01685worker1 # [ 1.503642] usb 1-2: new high-speed USB device number 3 using ehci-pci1686client # [ 1.476836] usb 1-2: new high-speed USB device number 3 using ehci-pci1687lb # [ 1.481332] usb 1-2: new high-speed USB device number 3 using ehci-pci1688worker2 # [ 1.483724] usb 1-2: new high-speed USB device number 3 using ehci-pci1689worker1 # [ 1.615319] systemd[1]: bpf-restrict-fs: LSM BPF program attached1690worker1 # [ 1.662740] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21691worker1 # [ 1.667805] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01692client # [ 1.606478] systemd[1]: bpf-restrict-fs: LSM BPF program attached1693lb # [ 1.610437] systemd[1]: bpf-restrict-fs: LSM BPF program attached1694worker2 # [ 1.612653] systemd[1]: bpf-restrict-fs: LSM BPF program attached1695client # [ 1.641391] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21696client # [ 1.645974] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01697lb # [ 1.641713] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21698worker2 # [ 1.644206] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21699worker1 # [ 1.726460] systemd[1]: Queued start job for default target Initrd Default Target.1700lb # [ 1.649543] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01701worker2 # [ 1.649198] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01702worker1 # [ 1.738292] systemd[1]: Created slice Slice /system/modprobe.1703worker1 # [ 1.739666] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1704worker1 # [ 1.739712] systemd[1]: Expecting device /dev/disk/by-label/nixos...1705worker1 # [ 1.739742] systemd[1]: Reached target Path Units.1706worker1 # [ 1.739762] systemd[1]: Reached target Slice Units.1707worker1 # [ 1.739783] systemd[1]: Reached target Swaps.1708worker1 # [ 1.739806] systemd[1]: Reached target Timer Units.1709worker1 # [ 1.740024] systemd[1]: Listening on D-Bus System Message Bus Socket.1710worker1 # [ 1.740195] systemd[1]: Listening on Journal Socket (/dev/log).1711worker1 # [ 1.740365] systemd[1]: Listening on Journal Sockets.1712worker1 # [ 1.740512] systemd[1]: Listening on udev Control Socket.1713worker1 # [ 1.740653] systemd[1]: Listening on udev Kernel Socket.1714worker1 # [ 1.740678] systemd[1]: Reached target Socket Units.1715worker1 # [ 1.754537] systemd[1]: Starting Create List of Static Device Nodes...1716worker1 # [ 1.763803] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1717worker1 # [ 1.765123] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1718worker1 # [ 1.771786] systemd[1]: Mounting Kernel Configuration File System...1719worker1 # [ 1.782124] systemd[1]: Starting Journal Service...1720client # [ 1.723437] systemd[1]: Queued start job for default target Initrd Default Target.1721client # [ 1.735241] systemd[1]: Created slice Slice /system/modprobe.1722lb # [ 1.727303] systemd[1]: Queued start job for default target Initrd Default Target.1723worker1 # [ 1.802321] systemd[1]: Starting Load Kernel Modules...1724client # [ 1.736466] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1725worker1 # [ 1.802420] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1726client # [ 1.737945] systemd[1]: Expecting device /dev/disk/by-label/nixos...1727client # [ 1.739105] systemd[1]: Reached target Path Units.1728worker2 # [ 1.731162] systemd[1]: Queued start job for default target Initrd Default Target.1729client # [ 1.739956] systemd[1]: Reached target Slice Units.1730client # [ 1.740883] systemd[1]: Reached target Swaps.1731client # [ 1.741654] systemd[1]: Reached target Timer Units.1732client # [ 1.742688] systemd[1]: Listening on D-Bus System Message Bus Socket.1733lb # [ 1.735430] systemd[1]: Created slice Slice /system/modprobe.1734client # [ 1.743954] systemd[1]: Listening on Journal Socket (/dev/log).1735lb # [ 1.736521] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1736client # [ 1.745185] systemd[1]: Listening on Journal Sockets.1737lb # [ 1.737867] systemd[1]: Expecting device /dev/disk/by-label/nixos...1738client # [ 1.746219] systemd[1]: Listening on udev Control Socket.1739lb # [ 1.738930] systemd[1]: Reached target Path Units.1740client # [ 1.747289] systemd[1]: Listening on udev Kernel Socket.1741lb # [ 1.739746] systemd[1]: Reached target Slice Units.1742client # [ 1.748240] systemd[1]: Reached target Socket Units.1743lb # [ 1.740585] systemd[1]: Reached target Swaps.1744lb # [ 1.741377] systemd[1]: Reached target Timer Units.1745lb # [ 1.742384] systemd[1]: Listening on D-Bus System Message Bus Socket.1746worker2 # [ 1.742134] systemd[1]: Created slice Slice /system/modprobe.1747lb # [ 1.743617] systemd[1]: Listening on Journal Socket (/dev/log).1748worker2 # [ 1.743386] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1749client # [ 1.750773] systemd[1]: Starting Create List of Static Device Nodes...1750lb # [ 1.744750] systemd[1]: Listening on Journal Sockets.1751worker2 # [ 1.744878] systemd[1]: Expecting device /dev/disk/by-label/nixos...1752worker2 # [ 1.745986] systemd[1]: Reached target Path Units.1753lb # [ 1.745816] systemd[1]: Listening on udev Control Socket.1754worker2 # [ 1.746834] systemd[1]: Reached target Slice Units.1755lb # [ 1.745937] systemd[1]: Listening on udev Kernel Socket.1756worker2 # [ 1.747765] systemd[1]: Reached target Swaps.1757lb # [ 1.745960] systemd[1]: Reached target Socket Units.1758worker2 # [ 1.748575] systemd[1]: Reached target Timer Units.1759lb # [ 1.750059] systemd[1]: Starting Create List of Static Device Nodes...1760worker2 # [ 1.749714] systemd[1]: Listening on D-Bus System Message Bus Socket.1761worker2 # [ 1.750995] systemd[1]: Listening on Journal Socket (/dev/log).1762worker2 # [ 1.752205] systemd[1]: Listening on Journal Sockets.1763client # [ 1.760974] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1764worker2 # [ 1.752360] systemd[1]: Listening on udev Control Socket.1765worker2 # [ 1.752495] systemd[1]: Listening on udev Kernel Socket.1766worker1 # [ 1.831242] systemd-journald[73]: Collecting audit messages is disabled.1767client # [ 1.761062] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1768worker2 # [ 1.752522] systemd[1]: Reached target Socket Units.1769worker2 # [ 1.757687] systemd[1]: Starting Create List of Static Device Nodes...1770client # [ 1.765684] systemd[1]: Mounting Kernel Configuration File System...1771lb # [ 1.760301] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1772lb # [ 1.761598] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1773worker2 # [ 1.764740] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1774worker1 # [ 1.839773] systemd[1]: Starting Coldplug All udev Devices...1775worker2 # [ 1.765916] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1776client # [ 1.777448] systemd[1]: Starting Journal Service...1777worker1 # [ 1.843712] systemd[1]: Finished Create List of Static Device Nodes.1778worker1 # [ 1.844531] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1779worker1 # [ 1.844839] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1780worker2 # [ 1.775824] systemd[1]: Mounting Kernel Configuration File System...1781lb # [ 1.774740] systemd[1]: Mounting Kernel Configuration File System...1782worker2 # [ 1.785271] systemd[1]: Starting Journal Service...1783worker1 # [ 1.860407] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1784worker1 # [ 1.862183] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1785lb # [ 1.797492] systemd[1]: Starting Journal Service...1786client # [ 1.808565] systemd[1]: Starting Load Kernel Modules...1787lb # [ 1.801468] systemd[1]: Starting Load Kernel Modules...1788client # [ 1.808697] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1789lb # [ 1.802375] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1790worker1 # [ 1.871151] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01791worker1 # [ 1.871390] [drm] features: -virgl +edid -resource_blob -host_visible1792worker1 # [ 1.871400] [drm] features: -context_init1793lb # [ 1.806449] systemd[1]: Starting Coldplug All udev Devices...1794worker1 # [ 1.882347] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1795worker1 # [ 1.882897] systemd[1]: Mounted Kernel Configuration File System.1796lb # [ 1.817587] systemd[1]: Finished Create List of Static Device Nodes.1797lb # [ 1.819291] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1798worker1 # [ 1.894713] [drm] number of scanouts: 11799worker1 # [ 1.894741] [drm] number of cap sets: 01800client # [ 1.828681] systemd-journald[73]: Collecting audit messages is disabled.1801worker2 # [ 1.819816] systemd[1]: Starting Load Kernel Modules...1802worker2 # [ 1.819921] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1803client # [ 1.836988] systemd[1]: Starting Coldplug All udev Devices...1804worker1 # [ 1.896284] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1805worker1 # [ 1.896298] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01806lb # [ 1.833505] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1807worker2 # [ 1.833558] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1808worker2 # [ 1.835445] systemd[1]: Starting Coldplug All udev Devices...1809worker1 # [ 1.909905] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1810worker2 # [ 1.836562] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1811worker1 # [ 1.915961] systemd[1]: Starting Create Static Device Nodes in /dev...1812client # [ 1.849020] systemd[1]: Finished Create List of Static Device Nodes.1813client # [ 1.849915] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1814client # [ 1.850200] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1815worker2 # [ 1.845305] systemd[1]: Finished Create List of Static Device Nodes.1816worker2 # [ 1.846119] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1817worker2 # [ 1.846397] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1818worker2 # [ 1.846614] systemd[1]: Mounted Kernel Configuration File System.1819worker2 # [ 1.851641] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1820lb # [ 1.857443] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1821lb # [ 1.857975] systemd[1]: Mounted Kernel Configuration File System.1822worker1 # [ 1.928456] Console: switching to colour frame buffer device 160x501823worker1 # [ 1.935023] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1824client # [ 1.872994] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1825client # [ 1.874413] systemd[1]: Mounted Kernel Configuration File System.1826worker2 # [ 1.866320] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01827worker2 # [ 1.866555] [drm] features: -virgl +edid -resource_blob -host_visible1828worker2 # [ 1.866563] [drm] features: -context_init1829worker2 # [ 1.867248] [drm] number of scanouts: 11830worker2 # [ 1.867264] [drm] number of cap sets: 01831lb # [ 1.870530] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1832client # [ 1.880137] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1833lb # [ 1.874784] systemd-journald[73]: Collecting audit messages is disabled.1834lb # [ 1.885847] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1835worker2 # [ 1.879956] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1836worker2 # [ 1.879974] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01837client # [ 1.895957] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1838lb # [ 1.889357] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1839worker1 # [ 1.963760] systemd[1]: Finished Load Kernel Modules.1840lb # [ 1.891522] systemd[1]: Starting Create Static Device Nodes in /dev...1841worker1 # [ 1.966738] systemd[1]: Starting Apply Kernel Variables...1842client # [ 1.901489] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1843lb # [ 1.896988] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01844lb # [ 1.897222] [drm] features: -virgl +edid -resource_blob -host_visible1845lb # [ 1.897232] [drm] features: -context_init1846client # [ 1.908558] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01847client # [ 1.909891] systemd[1]: Starting Create Static Device Nodes in /dev...1848lb # [ 1.910575] [drm] number of scanouts: 11849worker1 # [ 1.983731] systemd[1]: Finished Create Static Device Nodes in /dev.1850lb # [ 1.910605] [drm] number of cap sets: 01851worker2 # [ 1.904003] Console: switching to colour frame buffer device 160x501852worker1 # [ 1.984117] systemd[1]: Reached target Preparation for Local File Systems.1853worker1 # [ 1.984151] systemd[1]: Reached target Local File Systems.1854client # [ 1.921198] [drm] features: -virgl +edid -resource_blob -host_visible1855worker1 # [ 1.988805] systemd[1]: Starting Rule-based Manager for Device Events and Files...1856client # [ 1.921214] [drm] features: -context_init1857client # [ 1.921947] [drm] number of scanouts: 11858client # [ 1.921966] [drm] number of cap sets: 01859lb # [ 1.913410] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1860lb # [ 1.913427] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01861worker2 # [ 1.911498] systemd-journald[73]: Collecting audit messages is disabled.1862client # [ 1.929185] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1863client # [ 1.929202] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01864worker2 # [ 1.928295] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1865worker2 # [ 1.930768] systemd[1]: Starting Create Static Device Nodes in /dev...1866worker2 # [ 1.935685] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1867worker1 # [ 2.015105] systemd[1]: Finished Apply Kernel Variables.1868worker2 # [ 1.948526] systemd[1]: Finished Load Kernel Modules.1869worker1 # [ 2.024850] systemd[1]: Started Journal Service.1870lb # [ 1.945715] Console: switching to colour frame buffer device 160x501871worker2 # [ 1.953131] systemd[1]: Starting Apply Kernel Variables...1872lb # [ 1.953989] systemd[1]: Finished Create Static Device Nodes in /dev.1873worker1 # [ 2.015210] systemd-modules-load[75]: Inserted module 'dm_mod'1874client # [ 1.957147] Console: switching to colour frame buffer device 160x501875lb # [ 1.954159] systemd[1]: Reached target Preparation for Local File Systems.1876lb # [ 1.954183] systemd[1]: Reached target Local File Systems.1877worker1 # [ 2.016567] systemd-modules-load[75]: Module 'virtio_balloon' is built in1878worker1 # [ 2.017633] systemd-modules-load[75]: Module 'virtio_console' is built in1879worker1 # [ 2.018664] systemd-modules-load[75]: Inserted module 'virtio_gpu'1880worker1 # [ 2.019636] systemd-modules-load[75]: Module 'virtio_rng' is built in1881client # [ 1.973474] systemd[1]: Finished Create Static Device Nodes in /dev.1882client # [ 1.973646] systemd[1]: Reached target Preparation for Local File Systems.1883client # [ 1.973670] systemd[1]: Reached target Local File Systems.1884lb # [ 1.965711] systemd[1]: Starting Rule-based Manager for Device Events and Files...1885client # [ 1.977545] systemd[1]: Starting Rule-based Manager for Device Events and Files...1886worker1 # [ 2.032378] systemd[1]: Starting Create System Files and Directories...1887worker2 # [ 1.972237] systemd[1]: Finished Create Static Device Nodes in /dev.1888worker2 # [ 1.972462] systemd[1]: Reached target Preparation for Local File Systems.1889worker2 # [ 1.972498] systemd[1]: Reached target Local File Systems.1890lb # [ 1.977879] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1891worker2 # [ 1.980015] systemd[1]: Starting Rule-based Manager for Device Events and Files...1892client # [ 1.989427] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1893lb # [ 1.994656] systemd[1]: Finished Load Kernel Modules.1894client # [ 2.009099] systemd[1]: Finished Load Kernel Modules.1895worker1 # [ 2.064602] systemd[1]: Finished Create System Files and Directories.1896client # [ 2.012097] systemd[1]: Starting Apply Kernel Variables...1897worker1 # [ 2.068700] systemd-udevd[81]: Using default interface naming scheme 'v261'.1898lb # [ 2.006525] systemd[1]: Starting Apply Kernel Variables...1899worker2 # [ 2.015959] systemd[1]: Finished Apply Kernel Variables.1900client # [ 2.031022] systemd-modules-load[75]: Inserted module 'dm_mod'1901worker1 # [ 2.095941] systemd[1]: Started Rule-based Manager for Device Events and Files.1902client # [ 2.032160] systemd-modules-load[75]: Module 'virtio_balloon' is built in1903client # [ 2.033238] systemd-modules-load[75]: Module 'virtio_console' is built in1904client # [ 2.034293] systemd-modules-load[75]: Inserted module 'virtio_gpu'1905client # [ 2.035254] systemd-modules-load[75]: Module 'virtio_rng' is built in1906client # [ 2.052989] systemd[1]: Started Journal Service.1907client # [ 2.053249] systemd[1]: Starting Create System Files and Directories...1908lb # [ 2.062039] systemd[1]: Finished Apply Kernel Variables.1909worker2 # [ 2.061152] systemd[1]: Started Journal Service.1910lb # [ 2.051080] systemd-modules-load[74]: Inserted module 'dm_mod'1911lb # [ 2.065460] systemd[1]: Started Journal Service.1912lb # [ 2.055249] systemd-modules-load[74]: Module 'virtio_balloon' is built in1913worker2 # [ 2.060247] systemd-modules-load[74]: Inserted module 'dm_mod'1914client # [ 2.068102] systemd[1]: Finished Apply Kernel Variables.1915lb # [ 2.061177] systemd-modules-load[74]: Module 'virtio_console' is built in1916lb # [ 2.062223] systemd-modules-load[74]: Inserted module 'virtio_gpu'1917worker2 # [ 2.064203] systemd-modules-load[74]: Module 'virtio_balloon' is built in1918lb # [ 2.063310] systemd-modules-load[74]: Module 'virtio_rng' is built in1919worker2 # [ 2.065330] systemd-modules-load[74]: Module 'virtio_console' is built in1920client # [ 2.072409] systemd-udevd[79]: Using default interface naming scheme 'v261'.1921worker2 # [ 2.066392] systemd-modules-load[74]: Inserted module 'virtio_gpu'1922worker2 # [ 2.067536] systemd-modules-load[74]: Module 'virtio_rng' is built in1923lb # [ 2.072627] systemd-udevd[79]: Using default interface naming scheme 'v261'.1924lb # [ 2.073721] systemd[1]: Starting Create System Files and Directories...1925worker1 # [ 2.152795] systemd[1]: Starting Virtual Console Setup...1926worker2 # [ 2.085422] systemd[1]: Starting Create System Files and Directories...1927worker2 # [ 2.087999] systemd-udevd[80]: Using default interface naming scheme 'v261'.1928worker2 # [ 2.098652] systemd[1]: Finished Create System Files and Directories.1929client # [ 2.108700] systemd[1]: Finished Create System Files and Directories.1930client # [ 2.109688] systemd[1]: Started Rule-based Manager for Device Events and Files.1931lb # [ 2.110828] systemd[1]: Started Rule-based Manager for Device Events and Files.1932lb # [ 2.113412] systemd[1]: Finished Create System Files and Directories.1933worker2 # [ 2.116750] systemd[1]: Started Rule-based Manager for Device Events and Files.1934worker1 # [ 2.200443] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1935worker1 # [ 2.203774] systemd[1]: Finished Virtual Console Setup.1936client # [ 2.160920] systemd[1]: Starting Virtual Console Setup...1937worker2 # [ 2.172131] systemd[1]: Starting Virtual Console Setup...1938lb # [ 2.172175] systemd[1]: Starting Virtual Console Setup...1939client # [ 2.220462] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1940client # [ 2.223658] systemd[1]: Finished Virtual Console Setup.1941worker2 # [ 2.220419] systemd-vconsole-setup[102]: Configuration of first virtual console was skipped, ignoring remaining ones.1942lb # [ 2.220531] systemd-vconsole-setup[102]: Configuration of first virtual console was skipped, ignoring remaining ones.1943worker2 # [ 2.223866] systemd[1]: Finished Virtual Console Setup.1944lb # [ 2.223823] systemd[1]: Finished Virtual Console Setup.1945worker1 # [ 2.814616] systemd[1]: Finished Coldplug All udev Devices.1946worker1 # [ 2.820734] systemd[1]: Reached target System Initialization.1947worker1 # [ 2.822116] systemd[1]: Reached target Basic System.1948worker2 # [ 2.808783] systemd[1]: Finished Coldplug All udev Devices.1949worker2 # [ 2.810587] systemd[1]: Reached target System Initialization.1950worker2 # [ 2.816988] systemd[1]: Reached target Basic System.1951client # [ 2.830379] systemd[1]: Finished Coldplug All udev Devices.1952client # [ 2.831300] systemd[1]: Reached target System Initialization.1953client # [ 2.832161] systemd[1]: Reached target Basic System.1954lb # [ 2.831927] systemd[1]: Finished Coldplug All udev Devices.1955lb # [ 2.836073] systemd[1]: Reached target System Initialization.1956lb # [ 2.837262] systemd[1]: Reached target Basic System.1957worker1 # [ 2.960445] (udev-worker)[107]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1958worker1 # [ 2.972566] (udev-worker)[92]: Network interface NamePolicy= disabled on kernel command line.1959worker1 # [ 2.993481] (udev-worker)[107]: Network interface NamePolicy= disabled on kernel command line.1960worker2 # [ 2.956385] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line.1961worker2 # [ 2.963194] (udev-worker)[100]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1962worker2 # [ 2.974182] (udev-worker)[100]: Network interface NamePolicy= disabled on kernel command line.1963lb # [ 2.979822] (udev-worker)[105]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1964client # [ 2.990193] (udev-worker)[99]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1965worker1 # [ 3.056627] systemd[1]: Found device /dev/disk/by-label/nixos.1966client # [ 2.994507] (udev-worker)[99]: Network interface NamePolicy= disabled on kernel command line.1967worker1 # [ 3.060511] systemd[1]: Reached target Initrd Root Device.1968worker1 # [ 3.063343] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1969client # [ 3.007876] (udev-worker)[106]: Network interface NamePolicy= disabled on kernel command line.1970lb # [ 3.016877] (udev-worker)[105]: Network interface NamePolicy= disabled on kernel command line.1971lb # [ 3.019848] (udev-worker)[95]: Network interface NamePolicy= disabled on kernel command line.1972worker1 # [ 3.110451] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks1973worker1 # [ 3.118523] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1974worker1 # [ 3.124304] systemd[1]: Mounting /sysroot...1975worker2 # [ 3.059977] systemd[1]: Found device /dev/disk/by-label/nixos.1976client # [ 3.072573] systemd[1]: Found device /dev/disk/by-label/nixos.1977worker2 # [ 3.066853] systemd[1]: Reached target Initrd Root Device.1978client # [ 3.074543] systemd[1]: Reached target Initrd Root Device.1979client # [ 3.077598] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1980worker2 # [ 3.073106] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1981lb # [ 3.081841] systemd[1]: Found device /dev/disk/by-label/nixos.1982worker1 # [ 3.172080] EXT4-fs (vda): mounted filesystem d8132a6a-ae54-4b63-b1bd-4145e3baa661 r/w with ordered data mode. Quota mode: none.1983lb # [ 3.087660] systemd[1]: Reached target Initrd Root Device.1984worker1 # [ 3.161850] systemd[1]: Mounted /sysroot.1985worker1 # [ 3.164892] systemd[1]: Reached target Initrd Root File System.1986lb # [ 3.092528] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1987worker1 # [ 3.168307] systemd[1]: Starting Mountpoints Configured in the Real Root...1988worker2 # [ 3.119664] systemd-fsck[111]: nixos: clean, 12/65536 files, 13019/262144 blocks1989worker1 # [ 3.195098] systemd-sysroot-fstab-check[120]: /sysroot should be mounted in the initrd, will request daemon-reload.1990client # [ 3.131122] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks1991worker2 # [ 3.128146] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1992client # [ 3.136283] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1993worker1 # [ 3.202766] systemd[1]: Reload requested from client PID 120 ('systemd-sysroot') (unit initrd-parse-etc.service)...1994worker2 # [ 3.132439] systemd[1]: Mounting /sysroot...1995worker1 # [ 3.204465] systemd[1]: Reloading...1996client # [ 3.149735] systemd[1]: Mounting /sysroot...1997lb # [ 3.143382] systemd-fsck[111]: nixos: clean, 12/65536 files, 13019/262144 blocks1998lb # [ 3.151001] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1999lb # [ 3.152775] systemd[1]: Mounting /sysroot...2000worker2 # [ 3.178527] EXT4-fs (vda): mounted filesystem 5eaeb1a5-07ac-4e59-a660-4d29c727ebf8 r/w with ordered data mode. Quota mode: none.
loading 1,571 hidden lines…
3572lb # [ 20.001561] envoy[902]: [2026-09-21 12:33:12.192][902][info][main] [source/server/server.cc:534] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id3573lb # [ 20.011786] envoy[902]: [2026-09-21 12:33:12.192][902][info][main] [source/server/server.cc:534] response trailer map: 144 bytes: grpc-message,grpc-status3574lb # [ 20.014335] envoy[902]: [2026-09-21 12:33:12.305][902][info][main] [source/server/server.cc:968] runtime: {}3575lb # [ 20.015826] envoy[902]: [2026-09-21 12:33:12.322][902][info][main] [source/server/server.cc:420] Raised file descriptor limits from 1024 to 524288.3576lb # [ 20.018339] envoy[902]: [2026-09-21 12:33:12.322][902][info][main] [source/server/server.cc:796] Starting admin HTTP server at 127.0.0.1:99013577lb # [ 20.020739] envoy[902]: [2026-09-21 12:33:12.322][902][info][admin] [source/server/admin/admin.cc:66] admin address: 127.0.0.1:99013578lb # [ 20.023106] envoy[902]: [2026-09-21 12:33:12.328][902][info][config] [source/server/configuration_impl.cc:181] loading tracing configuration3579lb # [ 20.025517] envoy[902]: [2026-09-21 12:33:12.328][902][info][config] [source/server/configuration_impl.cc:132] loading 0 static secret(s)3580lb # [ 20.027397] envoy[902]: [2026-09-21 12:33:12.328][902][info][config] [source/server/configuration_impl.cc:138] loading 2 cluster(s)3581lb # [ 20.029753] envoy[902]: [2026-09-21 12:33:12.530][902][info][config] [source/server/configuration_impl.cc:148] loading 1 listener(s)3582lb # [ 20.032060] envoy[902]: [2026-09-21 12:33:12.585][902][info][config] [source/server/configuration_impl.cc:164] loading stats configuration3583lb # [ 20.034197] envoy[902]: [2026-09-21 12:33:12.591][902][warning][main] [source/server/server.cc:1038] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.global_downstream_max_connections` resource monitor.3584lb # [ 20.040607] envoy[902]: [2026-09-21 12:33:12.591][902][info][main] [source/server/server.cc:1077] starting main dispatch loop3585worker2 # [ 20.054135] systemd[1]: scheduler-token.service: Deactivated successfully.3586worker2 # [ 20.064479] systemd[1]: Finished scheduler-token.service.3587worker2 # [ 20.067031] systemd[1]: scheduler-token.service: Consumed 54ms CPU time over 159ms wall clock time, 2.9M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.3588worker2 # [ 20.077533] systemd[1]: Starting Nix worker-protocol over gRPC...3589worker1 # [ 20.215470] nix-grpc-daemon[773]: ts=2026-09-21T12:33:12Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"3590worker1 # [ 20.227838] nix-grpc-daemon[773]: ts=2026-09-21T12:33:12Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc3591worker1 # [ 20.230532] nix-grpc-daemon[773]: ts=2026-09-21T12:33:12Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=13592worker1 # [ 20.269397] nix-grpc-daemon[773]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR3593worker1 # [ 20.271004] nix-grpc-daemon[773]: E0000 00:00:1789993992.909790 773 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.3594worker1 # [ 20.285943] nix-grpc-daemon[773]: E0000 00:00:1789993992.926403 773 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.3595worker1 # [ 20.289597] nix-grpc-daemon[773]: ts=2026-09-21T12:33:12Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket3596worker1 # [ 20.291317] systemd[1]: Started Nix worker-protocol over gRPC.3597worker1 # [ 20.328098] nix-grpc-daemon[773]: E0000 00:00:1789993992.967424 812 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.3598lb # [ 20.253122] envoy[902]: [2026-09-21 12:33:12.901][902][info][runtime] [source/common/runtime/runtime_impl.cc:567] RTDS has finished initialization3599lb # [ 20.258851] envoy[902]: [2026-09-21 12:33:12.906][902][info][upstream] [source/common/upstream/cluster_manager_impl.cc:250] cm init: all clusters initialized3600lb # [ 20.265647] envoy[902]: [2026-09-21 12:33:12.906][902][info][main] [source/server/server.cc:1057] all clusters initialized. initializing init manager3601worker1 # [ 20.346932] nix-grpc-daemon[773]: E0000 00:00:1789993992.987327 812 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.3602lb # [ 20.275330] envoy[902]: [2026-09-21 12:33:12.923][902][info][config] [source/common/listener_manager/listener_manager_impl.cc:1067] all dependencies initialized. starting workers3603worker1 # [ 20.352055] nix-grpc-daemon[773]: E0000 00:00:1789993992.990637 812 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.3604worker1 # [ 20.369195] nix-grpc-daemon[773]: E0000 00:00:1789993993.008946 812 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.3605worker1 # [ 20.384221] nix-grpc-daemon[773]: ts=2026-09-21T12:33:13Z level=info event=scheduler_connected addr=lb:500513606worker2 # [ 20.314261] nix-grpc-daemon[779]: ts=2026-09-21T12:33:12Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"3607worker1 # [ 20.387811] nix-grpc-daemon[773]: ts=2026-09-21T12:33:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3608worker2 # [ 20.325885] nix-grpc-daemon[779]: ts=2026-09-21T12:33:12Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc3609worker2 # [ 20.328341] nix-grpc-daemon[779]: ts=2026-09-21T12:33:12Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=13610worker2 # [ 20.362934] nix-grpc-daemon[779]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR3611worker2 # [ 20.364816] nix-grpc-daemon[779]: E0000 00:00:1789993992.997746 779 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.3612worker2 # [ 20.377462] nix-grpc-daemon[779]: E0000 00:00:1789993993.012173 779 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.3613worker2 # [ 20.381979] nix-grpc-daemon[779]: ts=2026-09-21T12:33:13Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket3614worker2 # [ 20.383727] systemd[1]: Started Nix worker-protocol over gRPC.3615worker2 # [ 20.417659] nix-grpc-daemon[779]: E0000 00:00:1789993993.051458 820 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.3616worker2 # [ 20.436086] nix-grpc-daemon[779]: E0000 00:00:1789993993.070814 820 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.3617worker2 # [ 20.440069] nix-grpc-daemon[779]: E0000 00:00:1789993993.073954 820 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.3618worker2 # [ 20.457915] nix-grpc-daemon[779]: E0000 00:00:1789993993.091935 820 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.3619worker2 # [ 20.472734] nix-grpc-daemon[779]: ts=2026-09-21T12:33:13Z level=info event=scheduler_connected addr=lb:500513620worker2 # [ 20.474513] nix-grpc-daemon[779]: ts=2026-09-21T12:33:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3621worker1 # [ 20.589142] nix-grpc-daemon[773]: ts=2026-09-21T12:33:13Z level=info event=scheduler_connected addr=lb:500513622worker1 # [ 20.594317] nix-grpc-daemon[773]: ts=2026-09-21T12:33:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3623worker2 # [ 20.676089] nix-grpc-daemon[779]: ts=2026-09-21T12:33:13Z level=info event=scheduler_connected addr=lb:500513624worker2 # [ 20.681131] nix-grpc-daemon[779]: ts=2026-09-21T12:33:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3625worker1 # [ 20.795107] nix-grpc-daemon[773]: ts=2026-09-21T12:33:13Z level=info event=scheduler_connected addr=lb:500513626worker1 # [ 20.800324] nix-grpc-daemon[773]: ts=2026-09-21T12:33:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3627worker2 # [ 20.882183] nix-grpc-daemon[779]: ts=2026-09-21T12:33:13Z level=info event=scheduler_connected addr=lb:500513628worker2 # [ 20.887079] nix-grpc-daemon[779]: ts=2026-09-21T12:33:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3629worker1 # [ 21.001161] nix-grpc-daemon[773]: ts=2026-09-21T12:33:13Z level=info event=scheduler_connected addr=lb:500513630worker1 # [ 21.006147] nix-grpc-daemon[773]: ts=2026-09-21T12:33:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3631worker2 # [ 21.088150] nix-grpc-daemon[779]: ts=2026-09-21T12:33:13Z level=info event=scheduler_connected addr=lb:500513632worker2 # [ 21.093198] nix-grpc-daemon[779]: ts=2026-09-21T12:33:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3633worker1 # [ 21.206967] nix-grpc-daemon[773]: ts=2026-09-21T12:33:13Z level=info event=scheduler_connected addr=lb:500513634worker1 # [ 21.212193] nix-grpc-daemon[773]: ts=2026-09-21T12:33:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3635worker2 # [ 21.294288] nix-grpc-daemon[779]: ts=2026-09-21T12:33:13Z level=info event=scheduler_connected addr=lb:500513636worker2 # [ 21.299458] nix-grpc-daemon[779]: ts=2026-09-21T12:33:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3637worker1 # [ 21.414779] nix-grpc-daemon[773]: ts=2026-09-21T12:33:14Z level=info event=scheduler_connected addr=lb:500513638worker1 # [ 21.418963] nix-grpc-daemon[773]: ts=2026-09-21T12:33:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3639worker2 # [ 21.502421] nix-grpc-daemon[779]: ts=2026-09-21T12:33:14Z level=info event=scheduler_connected addr=lb:500513640worker2 # [ 21.506520] nix-grpc-daemon[779]: ts=2026-09-21T12:33:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3641worker1 # [ 21.615162] nix-grpc-daemon[773]: ts=2026-09-21T12:33:14Z level=info event=scheduler_connected addr=lb:500513642worker1 # [ 21.620584] nix-grpc-daemon[773]: ts=2026-09-21T12:33:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3643worker2 # [ 21.702953] nix-grpc-daemon[779]: ts=2026-09-21T12:33:14Z level=info event=scheduler_connected addr=lb:500513644worker2 # [ 21.707841] nix-grpc-daemon[779]: ts=2026-09-21T12:33:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3645worker1 # [ 21.821326] nix-grpc-daemon[773]: ts=2026-09-21T12:33:14Z level=info event=scheduler_connected addr=lb:500513646worker1 # [ 21.826218] nix-grpc-daemon[773]: ts=2026-09-21T12:33:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3647worker2 # [ 21.909067] nix-grpc-daemon[779]: ts=2026-09-21T12:33:14Z level=info event=scheduler_connected addr=lb:500513648worker2 # [ 21.914089] nix-grpc-daemon[779]: ts=2026-09-21T12:33:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3649worker1 # [ 22.027051] nix-grpc-daemon[773]: ts=2026-09-21T12:33:14Z level=info event=scheduler_connected addr=lb:500513650worker1 # [ 22.032266] nix-grpc-daemon[773]: ts=2026-09-21T12:33:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3651worker2 # [ 22.115273] nix-grpc-daemon[779]: ts=2026-09-21T12:33:14Z level=info event=scheduler_connected addr=lb:500513652worker2 # [ 22.120415] nix-grpc-daemon[779]: ts=2026-09-21T12:33:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3653worker1 # [ 22.233212] nix-grpc-daemon[773]: ts=2026-09-21T12:33:14Z level=info event=scheduler_connected addr=lb:500513654worker1 # [ 22.238242] nix-grpc-daemon[773]: ts=2026-09-21T12:33:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3655worker2 # [ 22.321703] nix-grpc-daemon[779]: ts=2026-09-21T12:33:14Z level=info event=scheduler_connected addr=lb:500513656worker2 # [ 22.327143] nix-grpc-daemon[779]: ts=2026-09-21T12:33:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3657worker1 # [ 22.440685] nix-grpc-daemon[773]: ts=2026-09-21T12:33:15Z level=info event=scheduler_connected addr=lb:500513658worker1 # [ 22.444507] nix-grpc-daemon[773]: ts=2026-09-21T12:33:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3659worker2 # [ 22.529819] nix-grpc-daemon[779]: ts=2026-09-21T12:33:15Z level=info event=scheduler_connected addr=lb:500513660worker2 # [ 22.537932] nix-grpc-daemon[779]: ts=2026-09-21T12:33:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3661worker1 # [ 22.641202] nix-grpc-daemon[773]: ts=2026-09-21T12:33:15Z level=info event=scheduler_connected addr=lb:500513662worker1 # [ 22.646223] nix-grpc-daemon[773]: ts=2026-09-21T12:33:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3663worker2 # [ 22.730618] nix-grpc-daemon[779]: ts=2026-09-21T12:33:15Z level=info event=scheduler_connected addr=lb:500513664worker2 # [ 22.735796] nix-grpc-daemon[779]: ts=2026-09-21T12:33:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3665worker1 # [ 22.846901] nix-grpc-daemon[773]: ts=2026-09-21T12:33:15Z level=info event=scheduler_connected addr=lb:500513666worker1 # [ 22.850405] nix-grpc-daemon[773]: ts=2026-09-21T12:33:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3667worker2 # [ 22.936984] nix-grpc-daemon[779]: ts=2026-09-21T12:33:15Z level=info event=scheduler_connected addr=lb:500513668worker2 # [ 22.941937] nix-grpc-daemon[779]: ts=2026-09-21T12:33:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3669worker1 # [ 23.051317] nix-grpc-daemon[773]: ts=2026-09-21T12:33:15Z level=info event=scheduler_connected addr=lb:500513670worker1 # [ 23.056356] nix-grpc-daemon[773]: ts=2026-09-21T12:33:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3671worker2 # [ 23.143093] nix-grpc-daemon[779]: ts=2026-09-21T12:33:15Z level=info event=scheduler_connected addr=lb:500513672worker2 # [ 23.148266] nix-grpc-daemon[779]: ts=2026-09-21T12:33:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3673worker1 # [ 23.257083] nix-grpc-daemon[773]: ts=2026-09-21T12:33:15Z level=info event=scheduler_connected addr=lb:500513674worker1 # [ 23.262369] nix-grpc-daemon[773]: ts=2026-09-21T12:33:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3675worker2 # [ 23.349300] nix-grpc-daemon[779]: ts=2026-09-21T12:33:15Z level=info event=scheduler_connected addr=lb:500513676worker2 # [ 23.354450] nix-grpc-daemon[779]: ts=2026-09-21T12:33:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3677worker1 # [ 23.464976] nix-grpc-daemon[773]: ts=2026-09-21T12:33:16Z level=info event=scheduler_connected addr=lb:500513678worker1 # [ 23.469127] nix-grpc-daemon[773]: ts=2026-09-21T12:33:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3679worker2 # [ 23.557178] nix-grpc-daemon[779]: ts=2026-09-21T12:33:16Z level=info event=scheduler_connected addr=lb:500513680worker2 # [ 23.561388] nix-grpc-daemon[779]: ts=2026-09-21T12:33:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3681worker1 # [ 23.665345] nix-grpc-daemon[773]: ts=2026-09-21T12:33:16Z level=info event=scheduler_connected addr=lb:500513682worker1 # [ 23.670518] nix-grpc-daemon[773]: ts=2026-09-21T12:33:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3683worker2 # [ 23.757829] nix-grpc-daemon[779]: ts=2026-09-21T12:33:16Z level=info event=scheduler_connected addr=lb:500513684worker2 # [ 23.762835] nix-grpc-daemon[779]: ts=2026-09-21T12:33:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3685worker1 # [ 23.871289] nix-grpc-daemon[773]: ts=2026-09-21T12:33:16Z level=info event=scheduler_connected addr=lb:500513686worker1 # [ 23.876390] nix-grpc-daemon[773]: ts=2026-09-21T12:33:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3687worker2 # [ 23.964003] nix-grpc-daemon[779]: ts=2026-09-21T12:33:16Z level=info event=scheduler_connected addr=lb:500513688worker2 # [ 23.968970] nix-grpc-daemon[779]: ts=2026-09-21T12:33:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3689worker1 # [ 24.077236] nix-grpc-daemon[773]: ts=2026-09-21T12:33:16Z level=info event=scheduler_connected addr=lb:500513690worker1 # [ 24.082291] nix-grpc-daemon[773]: ts=2026-09-21T12:33:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3691worker2 # [ 24.169834] nix-grpc-daemon[779]: ts=2026-09-21T12:33:16Z level=info event=scheduler_connected addr=lb:500513692worker2 # [ 24.174846] nix-grpc-daemon[779]: ts=2026-09-21T12:33:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3693worker1 # [ 24.283088] nix-grpc-daemon[773]: ts=2026-09-21T12:33:16Z level=info event=scheduler_connected addr=lb:500513694worker1 # [ 24.288457] nix-grpc-daemon[773]: ts=2026-09-21T12:33:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3695worker2 # [ 24.376228] nix-grpc-daemon[779]: ts=2026-09-21T12:33:17Z level=info event=scheduler_connected addr=lb:500513696worker2 # [ 24.381485] nix-grpc-daemon[779]: ts=2026-09-21T12:33:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3697worker1 # [ 24.491443] nix-grpc-daemon[773]: ts=2026-09-21T12:33:17Z level=info event=scheduler_connected addr=lb:500513698worker1 # [ 24.495733] nix-grpc-daemon[773]: ts=2026-09-21T12:33:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3699worker2 # [ 24.584465] nix-grpc-daemon[779]: ts=2026-09-21T12:33:17Z level=info event=scheduler_connected addr=lb:500513700worker2 # [ 24.588650] nix-grpc-daemon[779]: ts=2026-09-21T12:33:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3701worker1 # [ 24.691885] nix-grpc-daemon[773]: ts=2026-09-21T12:33:17Z level=info event=scheduler_connected addr=lb:500513702worker1 # [ 24.697180] nix-grpc-daemon[773]: ts=2026-09-21T12:33:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3703worker2 # [ 24.785692] nix-grpc-daemon[779]: ts=2026-09-21T12:33:17Z level=info event=scheduler_connected addr=lb:500513704worker2 # [ 24.791159] nix-grpc-daemon[779]: ts=2026-09-21T12:33:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3705worker1 # [ 24.897990] nix-grpc-daemon[773]: ts=2026-09-21T12:33:17Z level=info event=scheduler_connected addr=lb:500513706worker1 # [ 24.903162] nix-grpc-daemon[773]: ts=2026-09-21T12:33:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3707worker2 # [ 24.992334] nix-grpc-daemon[779]: ts=2026-09-21T12:33:17Z level=info event=scheduler_connected addr=lb:500513708worker2 # [ 24.997583] nix-grpc-daemon[779]: ts=2026-09-21T12:33:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3709worker1 # [ 25.103990] nix-grpc-daemon[773]: ts=2026-09-21T12:33:17Z level=info event=scheduler_connected addr=lb:500513710worker1 # [ 25.109071] nix-grpc-daemon[773]: ts=2026-09-21T12:33:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3711worker2 # [ 25.198869] nix-grpc-daemon[779]: ts=2026-09-21T12:33:17Z level=info event=scheduler_connected addr=lb:500513712worker2 # [ 25.204344] nix-grpc-daemon[779]: ts=2026-09-21T12:33:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3713worker1 # [ 25.309926] nix-grpc-daemon[773]: ts=2026-09-21T12:33:17Z level=info event=scheduler_connected addr=lb:500513714worker1 # [ 25.315090] nix-grpc-daemon[773]: ts=2026-09-21T12:33:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3715worker2 # [ 25.405349] nix-grpc-daemon[779]: ts=2026-09-21T12:33:18Z level=info event=scheduler_connected addr=lb:500513716worker2 # [ 25.410339] nix-grpc-daemon[779]: ts=2026-09-21T12:33:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3717worker1 # [ 25.517234] nix-grpc-daemon[773]: ts=2026-09-21T12:33:18Z level=info event=scheduler_connected addr=lb:500513718worker1 # [ 25.521858] nix-grpc-daemon[773]: ts=2026-09-21T12:33:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3719worker2 # [ 25.613013] nix-grpc-daemon[779]: ts=2026-09-21T12:33:18Z level=info event=scheduler_connected addr=lb:500513720worker2 # [ 25.617662] nix-grpc-daemon[779]: ts=2026-09-21T12:33:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3721worker1 # [ 25.722720] nix-grpc-daemon[773]: ts=2026-09-21T12:33:18Z level=info event=scheduler_connected addr=lb:500513722worker1 # [ 25.728149] nix-grpc-daemon[773]: ts=2026-09-21T12:33:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3723worker2 # [ 25.818618] nix-grpc-daemon[779]: ts=2026-09-21T12:33:18Z level=info event=scheduler_connected addr=lb:500513724worker2 # [ 25.823499] nix-grpc-daemon[779]: ts=2026-09-21T12:33:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3725worker1 # [ 25.928999] nix-grpc-daemon[773]: ts=2026-09-21T12:33:18Z level=info event=scheduler_connected addr=lb:500513726worker1 # [ 25.934112] nix-grpc-daemon[773]: ts=2026-09-21T12:33:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3727worker2 # [ 26.024851] nix-grpc-daemon[779]: ts=2026-09-21T12:33:18Z level=info event=scheduler_connected addr=lb:500513728worker2 # [ 26.030058] nix-grpc-daemon[779]: ts=2026-09-21T12:33:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3729worker1 # [ 26.136457] nix-grpc-daemon[773]: ts=2026-09-21T12:33:18Z level=info event=scheduler_connected addr=lb:500513730worker1 # [ 26.144623] nix-grpc-daemon[773]: ts=2026-09-21T12:33:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3731worker2 # [ 26.231168] nix-grpc-daemon[779]: ts=2026-09-21T12:33:18Z level=info event=scheduler_connected addr=lb:500513732worker2 # [ 26.236285] nix-grpc-daemon[779]: ts=2026-09-21T12:33:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3733worker1 # [ 26.337433] nix-grpc-daemon[773]: ts=2026-09-21T12:33:18Z level=info event=scheduler_connected addr=lb:500513734worker1 # [ 26.343748] nix-grpc-daemon[773]: ts=2026-09-21T12:33:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3735worker2 # [ 26.437441] nix-grpc-daemon[779]: ts=2026-09-21T12:33:19Z level=info event=scheduler_connected addr=lb:500513736worker2 # [ 26.442601] nix-grpc-daemon[779]: ts=2026-09-21T12:33:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3737worker1 # [ 26.542664] nix-grpc-daemon[773]: ts=2026-09-21T12:33:19Z level=info event=scheduler_connected addr=lb:500513738worker1 # [ 26.546800] nix-grpc-daemon[773]: ts=2026-09-21T12:33:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3739worker2 # [ 26.645806] nix-grpc-daemon[779]: ts=2026-09-21T12:33:19Z level=info event=scheduler_connected addr=lb:500513740worker2 # [ 26.650032] nix-grpc-daemon[779]: ts=2026-09-21T12:33:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3741worker1 # [ 26.743258] nix-grpc-daemon[773]: ts=2026-09-21T12:33:19Z level=info event=scheduler_connected addr=lb:500513742worker1 # [ 26.748864] nix-grpc-daemon[773]: ts=2026-09-21T12:33:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3743worker2 # [ 26.847089] nix-grpc-daemon[779]: ts=2026-09-21T12:33:19Z level=info event=scheduler_connected addr=lb:500513744worker2 # [ 26.852614] nix-grpc-daemon[779]: ts=2026-09-21T12:33:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3745worker1 # [ 26.949766] nix-grpc-daemon[773]: ts=2026-09-21T12:33:19Z level=info event=scheduler_connected addr=lb:500513746worker1 # [ 26.955163] nix-grpc-daemon[773]: ts=2026-09-21T12:33:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3747worker2 # [ 27.053725] nix-grpc-daemon[779]: ts=2026-09-21T12:33:19Z level=info event=scheduler_connected addr=lb:500513748worker2 # [ 27.058698] nix-grpc-daemon[779]: ts=2026-09-21T12:33:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3749worker1 # [ 27.156539] nix-grpc-daemon[773]: ts=2026-09-21T12:33:19Z level=info event=scheduler_connected addr=lb:500513750worker1 # [ 27.161839] nix-grpc-daemon[773]: ts=2026-09-21T12:33:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3751worker2 # [ 27.259980] nix-grpc-daemon[779]: ts=2026-09-21T12:33:19Z level=info event=scheduler_connected addr=lb:500513752worker2 # [ 27.265478] nix-grpc-daemon[779]: ts=2026-09-21T12:33:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3753worker1 # [ 27.362526] nix-grpc-daemon[773]: ts=2026-09-21T12:33:20Z level=info event=scheduler_connected addr=lb:500513754worker1 # [ 27.368000] nix-grpc-daemon[773]: ts=2026-09-21T12:33:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3755worker2 # [ 27.466230] nix-grpc-daemon[779]: ts=2026-09-21T12:33:20Z level=info event=scheduler_connected addr=lb:500513756worker2 # [ 27.471676] nix-grpc-daemon[779]: ts=2026-09-21T12:33:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3757worker1 # [ 27.570787] nix-grpc-daemon[773]: ts=2026-09-21T12:33:20Z level=info event=scheduler_connected addr=lb:500513758worker1 # [ 27.574891] nix-grpc-daemon[773]: ts=2026-09-21T12:33:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3759worker2 # [ 27.675215] nix-grpc-daemon[779]: ts=2026-09-21T12:33:20Z level=info event=scheduler_connected addr=lb:500513760worker2 # [ 27.679618] nix-grpc-daemon[779]: ts=2026-09-21T12:33:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3761worker1 # [ 27.771092] nix-grpc-daemon[773]: ts=2026-09-21T12:33:20Z level=info event=scheduler_connected addr=lb:500513762worker1 # [ 27.776308] nix-grpc-daemon[773]: ts=2026-09-21T12:33:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3763worker2 # [ 27.876185] nix-grpc-daemon[779]: ts=2026-09-21T12:33:20Z level=info event=scheduler_connected addr=lb:500513764worker2 # [ 27.881274] nix-grpc-daemon[779]: ts=2026-09-21T12:33:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3765worker1 # [ 27.977061] nix-grpc-daemon[773]: ts=2026-09-21T12:33:20Z level=info event=scheduler_connected addr=lb:500513766worker1 # [ 27.981902] nix-grpc-daemon[773]: ts=2026-09-21T12:33:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3767worker2 # [ 28.082630] nix-grpc-daemon[779]: ts=2026-09-21T12:33:20Z level=info event=scheduler_connected addr=lb:500513768worker2 # [ 28.087940] nix-grpc-daemon[779]: ts=2026-09-21T12:33:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3769worker1 # [ 28.182758] nix-grpc-daemon[773]: ts=2026-09-21T12:33:20Z level=info event=scheduler_connected addr=lb:500513770worker1 # [ 28.187961] nix-grpc-daemon[773]: ts=2026-09-21T12:33:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3771worker2 # [ 28.289420] nix-grpc-daemon[779]: ts=2026-09-21T12:33:20Z level=info event=scheduler_connected addr=lb:500513772worker2 # [ 28.294912] nix-grpc-daemon[779]: ts=2026-09-21T12:33:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3773worker1 # [ 28.388927] nix-grpc-daemon[773]: ts=2026-09-21T12:33:21Z level=info event=scheduler_connected addr=lb:500513774worker1 # [ 28.394255] nix-grpc-daemon[773]: ts=2026-09-21T12:33:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3775worker2 # [ 28.496187] nix-grpc-daemon[779]: ts=2026-09-21T12:33:21Z level=info event=scheduler_connected addr=lb:500513776worker2 # [ 28.501496] nix-grpc-daemon[779]: ts=2026-09-21T12:33:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3777worker1 # [ 28.597204] nix-grpc-daemon[773]: ts=2026-09-21T12:33:21Z level=info event=scheduler_connected addr=lb:500513778worker1 # [ 28.601328] nix-grpc-daemon[773]: ts=2026-09-21T12:33:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3779worker2 # [ 28.704388] nix-grpc-daemon[779]: ts=2026-09-21T12:33:21Z level=info event=scheduler_connected addr=lb:500513780worker2 # [ 28.709563] nix-grpc-daemon[779]: ts=2026-09-21T12:33:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3781worker1 # [ 28.797839] nix-grpc-daemon[773]: ts=2026-09-21T12:33:21Z level=info event=scheduler_connected addr=lb:500513782worker1 # [ 28.803217] nix-grpc-daemon[773]: ts=2026-09-21T12:33:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3783worker2 # [ 28.910841] nix-grpc-daemon[779]: ts=2026-09-21T12:33:21Z level=info event=scheduler_connected addr=lb:500513784worker2 # [ 28.916244] nix-grpc-daemon[779]: ts=2026-09-21T12:33:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3785worker1 # [ 29.004389] nix-grpc-daemon[773]: ts=2026-09-21T12:33:21Z level=info event=scheduler_connected addr=lb:500513786worker1 # [ 29.009484] nix-grpc-daemon[773]: ts=2026-09-21T12:33:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3787worker2 # [ 29.117369] nix-grpc-daemon[779]: ts=2026-09-21T12:33:21Z level=info event=scheduler_connected addr=lb:500513788worker2 # [ 29.122352] nix-grpc-daemon[779]: ts=2026-09-21T12:33:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3789worker1 # [ 29.210556] nix-grpc-daemon[773]: ts=2026-09-21T12:33:21Z level=info event=scheduler_connected addr=lb:500513790worker1 # [ 29.219229] nix-grpc-daemon[773]: ts=2026-09-21T12:33:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3791worker2 # [ 29.324845] nix-grpc-daemon[779]: ts=2026-09-21T12:33:21Z level=info event=scheduler_connected addr=lb:500513792worker2 # [ 29.330052] nix-grpc-daemon[779]: ts=2026-09-21T12:33:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3793worker1 # [ 29.418503] nix-grpc-daemon[773]: ts=2026-09-21T12:33:22Z level=info event=scheduler_connected addr=lb:500513794worker1 # [ 29.427602] nix-grpc-daemon[773]: ts=2026-09-21T12:33:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3795worker2 # [ 29.531164] nix-grpc-daemon[779]: ts=2026-09-21T12:33:22Z level=info event=scheduler_connected addr=lb:500513796worker2 # [ 29.536604] nix-grpc-daemon[779]: ts=2026-09-21T12:33:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3797worker1 # [ 29.627418] nix-grpc-daemon[773]: ts=2026-09-21T12:33:22Z level=info event=scheduler_connected addr=lb:500513798worker1 # [ 29.631456] nix-grpc-daemon[773]: ts=2026-09-21T12:33:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3799worker2 # [ 29.739726] nix-grpc-daemon[779]: ts=2026-09-21T12:33:22Z level=info event=scheduler_connected addr=lb:500513800worker2 # [ 29.744192] nix-grpc-daemon[779]: ts=2026-09-21T12:33:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3801worker1 # [ 29.827941] nix-grpc-daemon[773]: ts=2026-09-21T12:33:22Z level=info event=scheduler_connected addr=lb:500513802worker1 # [ 29.833043] nix-grpc-daemon[773]: ts=2026-09-21T12:33:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3803worker2 # [ 29.940784] nix-grpc-daemon[779]: ts=2026-09-21T12:33:22Z level=info event=scheduler_connected addr=lb:500513804worker2 # [ 29.946173] nix-grpc-daemon[779]: ts=2026-09-21T12:33:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3805worker1 # [ 30.033733] nix-grpc-daemon[773]: ts=2026-09-21T12:33:22Z level=info event=scheduler_connected addr=lb:500513806worker1 # [ 30.038827] nix-grpc-daemon[773]: ts=2026-09-21T12:33:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3807worker2 # [ 30.147616] nix-grpc-daemon[779]: ts=2026-09-21T12:33:22Z level=info event=scheduler_connected addr=lb:500513808worker2 # [ 30.153237] nix-grpc-daemon[779]: ts=2026-09-21T12:33:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3809worker1 # [ 30.239972] nix-grpc-daemon[773]: ts=2026-09-21T12:33:22Z level=info event=scheduler_connected addr=lb:500513810worker1 # [ 30.246250] nix-grpc-daemon[773]: ts=2026-09-21T12:33:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3811worker2 # [ 30.354212] nix-grpc-daemon[779]: ts=2026-09-21T12:33:22Z level=info event=scheduler_connected addr=lb:500513812worker2 # [ 30.359445] nix-grpc-daemon[779]: ts=2026-09-21T12:33:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3813worker1 # [ 30.446291] nix-grpc-daemon[773]: ts=2026-09-21T12:33:23Z level=info event=scheduler_connected addr=lb:500513814worker1 # [ 30.451396] nix-grpc-daemon[773]: ts=2026-09-21T12:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3815worker2 # [ 30.560701] nix-grpc-daemon[779]: ts=2026-09-21T12:33:23Z level=info event=scheduler_connected addr=lb:500513816worker2 # [ 30.565855] nix-grpc-daemon[779]: ts=2026-09-21T12:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3817worker1 # [ 30.653989] nix-grpc-daemon[773]: ts=2026-09-21T12:33:23Z level=info event=scheduler_connected addr=lb:500513818worker1 # [ 30.658123] nix-grpc-daemon[773]: ts=2026-09-21T12:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3819worker2 # [ 30.768938] nix-grpc-daemon[779]: ts=2026-09-21T12:33:23Z level=info event=scheduler_connected addr=lb:500513820worker2 # [ 30.773023] nix-grpc-daemon[779]: ts=2026-09-21T12:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3821worker1 # [ 30.854373] nix-grpc-daemon[773]: ts=2026-09-21T12:33:23Z level=info event=scheduler_connected addr=lb:500513822worker1 # [ 30.859520] nix-grpc-daemon[773]: ts=2026-09-21T12:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3823worker2 # [ 30.969953] nix-grpc-daemon[779]: ts=2026-09-21T12:33:23Z level=info event=scheduler_connected addr=lb:500513824worker2 # [ 30.974846] nix-grpc-daemon[779]: ts=2026-09-21T12:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3825worker1 # [ 31.060324] nix-grpc-daemon[773]: ts=2026-09-21T12:33:23Z level=info event=scheduler_connected addr=lb:500513826worker1 # [ 31.065304] nix-grpc-daemon[773]: ts=2026-09-21T12:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3827worker2 # [ 31.175988] nix-grpc-daemon[779]: ts=2026-09-21T12:33:23Z level=info event=scheduler_connected addr=lb:500513828worker2 # [ 31.181100] nix-grpc-daemon[779]: ts=2026-09-21T12:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3829worker1 # [ 31.266126] nix-grpc-daemon[773]: ts=2026-09-21T12:33:23Z level=info event=scheduler_connected addr=lb:500513830worker1 # [ 31.271113] nix-grpc-daemon[773]: ts=2026-09-21T12:33:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3831worker2 # [ 31.382329] nix-grpc-daemon[779]: ts=2026-09-21T12:33:24Z level=info event=scheduler_connected addr=lb:500513832worker2 # [ 31.387544] nix-grpc-daemon[779]: ts=2026-09-21T12:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3833worker1 # [ 31.471720] nix-grpc-daemon[773]: ts=2026-09-21T12:33:24Z level=info event=scheduler_connected addr=lb:500513834worker1 # [ 31.477235] nix-grpc-daemon[773]: ts=2026-09-21T12:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3835worker2 # [ 31.588604] nix-grpc-daemon[779]: ts=2026-09-21T12:33:24Z level=info event=scheduler_connected addr=lb:500513836worker2 # [ 31.593531] nix-grpc-daemon[779]: ts=2026-09-21T12:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3837worker1 # [ 31.680289] nix-grpc-daemon[773]: ts=2026-09-21T12:33:24Z level=info event=scheduler_connected addr=lb:500513838worker1 # [ 31.684160] nix-grpc-daemon[773]: ts=2026-09-21T12:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3839worker2 # [ 31.796568] nix-grpc-daemon[779]: ts=2026-09-21T12:33:24Z level=info event=scheduler_connected addr=lb:500513840worker2 # [ 31.800651] nix-grpc-daemon[779]: ts=2026-09-21T12:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3841worker1 # [ 31.880557] nix-grpc-daemon[773]: ts=2026-09-21T12:33:24Z level=info event=scheduler_connected addr=lb:500513842worker1 # [ 31.885598] nix-grpc-daemon[773]: ts=2026-09-21T12:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3843worker2 # [ 31.997398] nix-grpc-daemon[779]: ts=2026-09-21T12:33:24Z level=info event=scheduler_connected addr=lb:500513844worker2 # [ 32.002444] nix-grpc-daemon[779]: ts=2026-09-21T12:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3845worker1 # [ 32.086437] nix-grpc-daemon[773]: ts=2026-09-21T12:33:24Z level=info event=scheduler_connected addr=lb:500513846worker1 # [ 32.091514] nix-grpc-daemon[773]: ts=2026-09-21T12:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3847worker2 # [ 32.203692] nix-grpc-daemon[779]: ts=2026-09-21T12:33:24Z level=info event=scheduler_connected addr=lb:500513848worker2 # [ 32.208845] nix-grpc-daemon[779]: ts=2026-09-21T12:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3849worker1 # [ 32.292322] nix-grpc-daemon[773]: ts=2026-09-21T12:33:24Z level=info event=scheduler_connected addr=lb:500513850worker1 # [ 32.297353] nix-grpc-daemon[773]: ts=2026-09-21T12:33:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3851worker2 # [ 32.409906] nix-grpc-daemon[779]: ts=2026-09-21T12:33:25Z level=info event=scheduler_connected addr=lb:500513852worker2 # [ 32.415040] nix-grpc-daemon[779]: ts=2026-09-21T12:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3853worker1 # [ 32.498063] nix-grpc-daemon[773]: ts=2026-09-21T12:33:25Z level=info event=scheduler_connected addr=lb:500513854worker1 # [ 32.503155] nix-grpc-daemon[773]: ts=2026-09-21T12:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3855worker2 # [ 32.615685] nix-grpc-daemon[779]: ts=2026-09-21T12:33:25Z level=info event=scheduler_connected addr=lb:500513856worker2 # [ 32.618692] nix-grpc-daemon[779]: ts=2026-09-21T12:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3857worker1 # [ 32.705713] nix-grpc-daemon[773]: ts=2026-09-21T12:33:25Z level=info event=scheduler_connected addr=lb:500513858worker1 # [ 32.709789] nix-grpc-daemon[773]: ts=2026-09-21T12:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3859worker2 # [ 32.821945] nix-grpc-daemon[779]: ts=2026-09-21T12:33:25Z level=info event=scheduler_connected addr=lb:500513860worker2 # [ 32.826182] nix-grpc-daemon[779]: ts=2026-09-21T12:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3861worker1 # [ 32.906207] nix-grpc-daemon[773]: ts=2026-09-21T12:33:25Z level=info event=scheduler_connected addr=lb:500513862worker1 # [ 32.910885] nix-grpc-daemon[773]: ts=2026-09-21T12:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3863lb # [ 32.940753] rustfs-bucket-start[1030]: mb s3://farm3864lb # [ 32.949712] systemd[1]: Finished rustfs-bucket.service.3865lb # [ 32.962136] systemd[1]: Starting niks3 server...3866worker2 # [ 33.023071] nix-grpc-daemon[779]: ts=2026-09-21T12:33:25Z level=info event=scheduler_connected addr=lb:500513867worker2 # [ 33.028449] nix-grpc-daemon[779]: ts=2026-09-21T12:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3868worker1 # [ 33.111744] nix-grpc-daemon[773]: ts=2026-09-21T12:33:25Z level=info event=scheduler_connected addr=lb:500513869worker1 # [ 33.117021] nix-grpc-daemon[773]: ts=2026-09-21T12:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3870lb # [ 33.226688] postgres[1050]: [1050] ERROR: relation "goose_db_version" does not exist at character 363871lb # [ 33.227969] postgres[1050]: [1050] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC3872worker2 # [ 33.229542] nix-grpc-daemon[779]: ts=2026-09-21T12:33:25Z level=info event=scheduler_connected addr=lb:500513873worker2 # [ 33.234740] nix-grpc-daemon[779]: ts=2026-09-21T12:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3874worker1 # [ 33.317706] nix-grpc-daemon[773]: ts=2026-09-21T12:33:25Z level=info event=scheduler_connected addr=lb:500513875worker1 # [ 33.322660] nix-grpc-daemon[773]: ts=2026-09-21T12:33:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3876lb # [ 33.259558] niks3-server[1040]: 2026/09/21 12:33:25 OK 20241026095416_initial_model.sql (19ms)3877lb # [ 33.262807] niks3-server[1040]: 2026/09/21 12:33:25 OK 20251210153512_drop_unused_gin_index.sql (1.81ms)3878lb # [ 33.268617] niks3-server[1040]: 2026/09/21 12:33:25 OK 20251218171726_add_pins.sql (5.76ms)3879lb # [ 33.273009] niks3-server[1040]: 2026/09/21 12:33:25 OK 20260628120000_add_object_size_and_stats.sql (4.36ms)3880lb # [ 33.278777] niks3-server[1040]: 2026/09/21 12:33:25 OK 20260905000000_add_claims.sql (5.73ms)3881lb # [ 33.283130] niks3-server[1040]: 2026/09/21 12:33:25 OK 20260920000000_drop_claims.sql (4.24ms)3882lb # [ 33.285092] niks3-server[1040]: 2026/09/21 12:33:25 goose: successfully migrated database to version: 202609200000003883lb # [ 33.289226] niks3-server[1040]: 2026/09/21 12:33:25 OK 1_commit_pending_closure.sql (6.07ms)3884lb # [ 33.291718] niks3-server[1040]: 2026/09/21 12:33:25 OK 2_object_stats_trigger.sql (2.37ms)3885lb # [ 33.293203] niks3-server[1040]: 2026/09/21 12:33:25 goose: up to current file version: 23886lb # [ 33.296939] niks3-server[1040]: 2026/09/21 12:33:25 INFO Loaded signing key name=farm-test-1 path=/nix/store/8lzz3wcf17v05vfdp1zcfaa9hipw3f70-key3887lb # [ 33.340417] niks3-server[1040]: 2026/09/21 12:33:25 INFO Read proxy enabled — serving cache objects from S33888lb # [ 33.341748] niks3-server[1040]: 2026/09/21 12:33:25 INFO Using socket-activated listener address=0.0.0.0:57513889lb # [ 33.345998] niks3-server[1040]: 2026/09/21 12:33:25 INFO systemd watchdog enabled interval=15s3890lb # [ 33.349811] systemd[1]: Started niks3 server.3891lb # [ 33.350494] niks3-server[1040]: 2026/09/21 12:33:25 INFO Starting HTTP server address=0.0.0.0:57513892lb # [ 33.352523] systemd[1]: Reached target Multi-User System.3893lb # [ 33.354920] systemd[1]: Startup finished in 921ms (kernel) + 5.697s (initrd) + 26.729s (userspace) = 33.349s.3894lb # [ 33.358676] niks3-server[1040]: 2026/09/21 12:33:26 INFO lead: acquired remote=192.168.1.3:505603895worker1 # [ 33.438030] nix-grpc-daemon[773]: ts=2026-09-21T12:33:26Z level=info event=scheduler_take_over3896worker2 # [ 33.435921] nix-grpc-daemon[779]: ts=2026-09-21T12:33:26Z level=info event=scheduler_connected addr=lb:500513897worker2 # [ 33.443406] nix-grpc-daemon[779]: ts=2026-09-21T12:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3898worker1 # [ 33.523432] nix-grpc-daemon[773]: ts=2026-09-21T12:33:26Z level=info event=scheduler_connected addr=lb:500513899worker1 # [ 33.528559] nix-grpc-daemon[773]: ts=2026-09-21T12:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3900worker2 # [ 33.638680] nix-grpc-daemon[779]: ts=2026-09-21T12:33:26Z level=info event=scheduler_connected addr=lb:500513901worker2 # [ 33.643935] nix-grpc-daemon[779]: ts=2026-09-21T12:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3902worker1 # [ 33.731186] nix-grpc-daemon[773]: ts=2026-09-21T12:33:26Z level=info event=scheduler_connected addr=lb:500513903worker1 # [ 33.735359] nix-grpc-daemon[773]: ts=2026-09-21T12:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3904worker2 # [ 33.847276] nix-grpc-daemon[779]: ts=2026-09-21T12:33:26Z level=info event=scheduler_connected addr=lb:500513905worker2 # [ 33.851599] nix-grpc-daemon[779]: ts=2026-09-21T12:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3906worker1 # [ 33.931622] nix-grpc-daemon[773]: ts=2026-09-21T12:33:26Z level=info event=scheduler_connected addr=lb:500513907worker1 # [ 33.936843] nix-grpc-daemon[773]: ts=2026-09-21T12:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3908worker2 # [ 34.048204] nix-grpc-daemon[779]: ts=2026-09-21T12:33:26Z level=info event=scheduler_connected addr=lb:500513909worker2 # [ 34.053393] nix-grpc-daemon[779]: ts=2026-09-21T12:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3910worker1 # [ 34.138067] nix-grpc-daemon[773]: ts=2026-09-21T12:33:26Z level=info event=scheduler_connected addr=lb:500513911worker1 # [ 34.143305] nix-grpc-daemon[773]: ts=2026-09-21T12:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3912worker2 # [ 34.254640] nix-grpc-daemon[779]: ts=2026-09-21T12:33:26Z level=info event=scheduler_connected addr=lb:500513913worker2 # [ 34.259703] nix-grpc-daemon[779]: ts=2026-09-21T12:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3914lb: (finished: waiting for unit niks3.service, in 20.39 seconds)3915lb: waiting for TCP port 5751 on localhost3916worker1 # [ 34.344207] nix-grpc-daemon[773]: ts=2026-09-21T12:33:26Z level=info event=scheduler_connected addr=lb:500513917worker1 # [ 34.350071] nix-grpc-daemon[773]: ts=2026-09-21T12:33:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3918lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded!3919lb: (finished: waiting for TCP port 5751 on localhost, in 0.09 seconds)3920lb: waiting for unit envoy.service3921lb: (finished: waiting for unit envoy.service, in 0.06 seconds)3922lb: waiting for TCP port 50051 on localhost3923lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded!3924lb # [ 34.456789] envoy[902]: conn peer=[::1]:57924 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=33925lb: (finished: waiting for TCP port 50051 on localhost, in 0.04 seconds)3926worker2 # [ 34.460600] nix-grpc-daemon[779]: ts=2026-09-21T12:33:27Z level=info event=scheduler_connected addr=lb:500513927worker1 # [ 34.552289] nix-grpc-daemon[773]: ts=2026-09-21T12:33:27Z level=info event=scheduler_connected addr=lb:500513928worker1 # [ 34.565728] nix-grpc-daemon[773]: ts=2026-09-21T12:33:27Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:421163929worker1 # [ 34.571835] nix-grpc-daemon[773]: ts=2026-09-21T12:33:27Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=03930worker1 # [ 34.577459] nix-grpc-daemon[773]: ts=2026-09-21T12:33:27Z level=info event=oidc_jwks provider=mock keys=13931worker1 # [ 34.584279] nix-grpc-daemon[773]: ts=2026-09-21T12:33:27Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:421163932worker1 # [ 34.590960] nix-grpc-daemon[773]: ts=2026-09-21T12:33:27Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=03933worker2 # [ 34.670876] systemd[1]: Starting scheduler-token.service...3934worker2 # [ 34.761291] systemd[1]: scheduler-token.service: Deactivated successfully.3935worker2 # [ 34.764560] systemd[1]: Finished scheduler-token.service.3936worker2 # [ 34.765674] systemd[1]: scheduler-token.service: Consumed 34ms CPU time over 93ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.3937lb: waiting until exactly no endpoints of aarch64-linux are unhealthy3938lb: must succeed: curl -sf localhost:9901/clusters3939lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)3940lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.10 seconds)3941subtest: one node schedules and both builders hold a WorkerSession on it3942lb: must succeed: curl -sf localhost:9901/clusters3943lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)3944worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics3945worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)3946lb: must succeed: curl -sf localhost:9901/clusters3947lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)3948lb: must succeed: curl -sf localhost:9901/clusters3949lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)3950lb: must succeed: curl -sf localhost:9901/clusters3951lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)3952lb: must succeed: curl -sf localhost:9901/clusters3953lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)3954worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'3955worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.06 seconds)3956worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics3957worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)3958lb: must succeed: curl -sf localhost:9901/clusters3959lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)3960lb: must succeed: curl -sf localhost:9901/clusters3961lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)3962worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3963worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)3964worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics3965worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)3966worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics3967worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)3968worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics3969worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)3970(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 0.86 seconds)3971subtest: balancer auth: client cert, bearer token, nothing, unknown CN3972worker2 # [ 36.069047] nix-grpc-daemon[779]: ts=2026-09-21T12:33:28Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:366383973worker2 # [ 36.086888] systemd[1]: Started Nix Daemon.3974worker2 # [ 36.382936] nix-daemon[847]: accepted connection from pid 779, user nix-grpc-daemon (trusted)3975worker2 # [ 36.407665] nix-grpc-daemon[779]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 23 ms (attempt 1/5)3976worker2 # [ 36.433384] nix-grpc-daemon[779]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 37 ms (attempt 2/5)3977worker2 # [ 36.474332] nix-grpc-daemon[779]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 80 ms (attempt 3/5)3978worker2 # [ 36.561194] nix-grpc-daemon[779]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 413 ms (attempt 4/5)3979worker2 # [ 36.976514] nix-grpc-daemon[779]: disabling binary cache 'https://cache.nixos.org' for 60 seconds3980worker2 # [ 36.982209] nix-grpc-daemon[779]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org3981worker2 # [ 37.000981] nix-grpc-daemon[779]: ts=2026-09-21T12:33:29Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:366383982worker2 # [ 37.015553] nix-grpc-daemon[779]: ts=2026-09-21T12:33:29Z level=debug event=path_info_miss path=llyf0bg5i97i0wydcrmwr522c4lqspsl-nixos-system-client-test error="error: path '/nix/store/llyf0bg5i97i0wydcrmwr522c4lqspsl-nixos-system-client-test' is required, but there is no substituter that can build it"3983worker2 # [ 37.028384] nix-grpc-daemon[779]: ts=2026-09-21T12:33:29Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:366383984worker2 # [ 37.038678] nix-grpc-daemon[779]: ts=2026-09-21T12:33:29Z level=debug event=path_info_miss path=llyf0bg5i97i0wydcrmwr522c4lqspsl-nixos-system-client-test error="error: path '/nix/store/llyf0bg5i97i0wydcrmwr522c4lqspsl-nixos-system-client-test' is required, but there is no substituter that can build it"3985client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt3986client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.10 seconds)3987client: must succeed: install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key3988client: (finished: must succeed: install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.05 seconds)3989worker1 # [ 37.569782] nix-grpc-daemon[773]: ts=2026-09-21T12:33:30Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:421243990worker1 # [ 37.588838] systemd[1]: Started Nix Daemon.3991worker1 # [ 37.846249] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)3992worker1 # [ 37.868937] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 46 ms (attempt 1/5)3993worker1 # [ 37.917540] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 44 ms (attempt 2/5)3994worker1 # [ 37.964507] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 67 ms (attempt 3/5)3995worker1 # [ 38.037836] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 310 ms (attempt 4/5)3996worker1 # [ 38.358466] nix-grpc-daemon[773]: disabling binary cache 'https://cache.nixos.org' for 60 seconds3997worker1 # [ 38.364855] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org3998worker1 # [ 38.383779] nix-grpc-daemon[773]: ts=2026-09-21T12:33:31Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:421243999worker1 # [ 38.399505] nix-grpc-daemon[773]: ts=2026-09-21T12:33:31Z level=debug event=path_info_miss path=llyf0bg5i97i0wydcrmwr522c4lqspsl-nixos-system-client-test error="error: path '/nix/store/llyf0bg5i97i0wydcrmwr522c4lqspsl-nixos-system-client-test' is required, but there is no substituter that can build it"4000worker2 # [ 38.345469] nix-grpc-daemon[779]: ts=2026-09-21T12:33:30Z level=info event=oidc_jwks provider=mock keys=14001worker2 # [ 38.352739] nix-grpc-daemon[779]: ts=2026-09-21T12:33:30Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:366384002worker2 # [ 38.361175] nix-grpc-daemon[779]: ts=2026-09-21T12:33:30Z level=debug event=path_info_miss path=llyf0bg5i97i0wydcrmwr522c4lqspsl-nixos-system-client-test error="error: path '/nix/store/llyf0bg5i97i0wydcrmwr522c4lqspsl-nixos-system-client-test' is required, but there is no substituter that can build it"4003client: must succeed: rm -r /var/lib/nix-grpc-store4004client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.05 seconds)4005worker2 # [ 38.698322] nix-grpc-daemon[779]: ts=2026-09-21T12:33:31Z level=info event=denied method=QueryMissing cn=- role=none4006worker2 # [ 38.961282] nix-grpc-daemon[779]: ts=2026-09-21T12:33:31Z level=info event=denied method=QueryMissing cn=stranger role=none4007(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.25 seconds)4008subtest: DAG build is scheduled across workers and lands in the cache4009client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&24010worker2 # [ 39.214345] nix-grpc-daemon[779]: ts=2026-09-21T12:33:31Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:366384011client # don't know how to build these paths:4012client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4013worker1 # [ 39.301332] nix-grpc-daemon[773]: ts=2026-09-21T12:33:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421244014worker1 # [ 39.334641] nix-grpc-daemon[773]: ts=2026-09-21T12:33:31Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=04015worker1 # [ 39.341863] nix-grpc-daemon[773]: ts=2026-09-21T12:33:31Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.3:50051 clients=14016worker1 # [ 39.380849] nix-grpc-daemon[773]: ts=2026-09-21T12:33:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244017worker1 # [ 39.402965] nix-grpc-daemon[773]: ts=2026-09-21T12:33:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421244018client # copying 1 paths to 'lb:50051'...4019worker1 # [ 39.420826] nix-grpc-daemon[773]: ts=2026-09-21T12:33:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421244020client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'...4021worker1 # [ 39.449726] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4022worker1 # [ 39.478999] nix-grpc-daemon[773]: ts=2026-09-21T12:33:32Z level=info event=niks3_push_started pid=8694023worker1 # [ 39.764811] nix-grpc-daemon[869]: time=2026-09-21T12:33:32.403Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""4024worker1 # [ 39.956581] nix-daemon[855]: accepted connection from pid 883, user nix-grpc-daemon (trusted)4025lb # [ 39.887488] envoy[902]: conn peer=[2001:db8:1::1]:41220 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=715 tx=365 ms=10154026worker1 # [ 39.975843] nix-daemon[855]: reaped child process 886, status = succeeded4027worker1 # [ 40.147458] nix-daemon[855]: accepted connection from pid 888, user nix-grpc-daemon (trusted)4028worker1 # [ 40.165242] nix-daemon[855]: reaped child process 891, status = succeeded4029lb # [ 39.892584] envoy[902]: conn peer=[2001:db8:1::1]:41226 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=365 ms=8924030lb # [ 39.892710] envoy[902]: conn peer=[2001:db8:1::1]:41236 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=124031lb # [ 39.892810] envoy[902]: conn peer=[2001:db8:1::1]:41244 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=124032lb # [ 40.104655] niks3-server[1040]: 2026/09/21 12:33:32 INFO Received uploads request method=POST path=/api/pending_closures4033worker1 # [ 40.213926] nix-grpc-daemon[869]: time=2026-09-21T12:33:32.854Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4034worker1 # [ 40.217560] nix-grpc-daemon[869]: time=2026-09-21T12:33:32.856Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)"4035lb # [ 40.187851] niks3-server[1040]: 2026/09/21 12:33:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4036lb # [ 40.200007] niks3-server[1040]: 2026/09/21 12:33:32 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst4037lb # [ 40.227800] niks3-server[1040]: 2026/09/21 12:33:32 INFO Signed narinfos id=1 count=14038worker1 # [ 40.304127] nix-grpc-daemon[869]: time=2026-09-21T12:33:32.944Z level=INFO msg="Uploading 1 narinfos"4039lb # [ 40.236579] niks3-server[1040]: 2026/09/21 12:33:32 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls4040lb # [ 40.251649] niks3-server[1040]: 2026/09/21 12:33:32 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4041lb # [ 40.259727] niks3-server[1040]: 2026/09/21 12:33:32 INFO Completed upload id=14042worker1 # [ 40.333046] nix-grpc-daemon[869]: time=2026-09-21T12:33:32.973Z level=INFO msg="Upload complete. (560ms)"4043lb # [ 40.271064] niks3-server[1040]: 2026/09/21 12:33:32 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo4044worker1 # [ 40.348594] nix-grpc-daemon[773]: ts=2026-09-21T12:33:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=0 paths=1 nar_bytes_in=5044045worker1 # [ 40.364456] nix-grpc-daemon[773]: ts=2026-09-21T12:33:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244046worker1 # [ 40.371231] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4047worker1 # [ 40.383171] nix-daemon[855]: reaped child process 867, status = succeeded4048client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051...4049client # farm-a> node-a: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4050worker1 # [ 40.391657] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4051client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4052client # farm-a> LOG-farm-a-t14053worker1 # [ 42.085827] nix-daemon[855]: reaped child process 895, status = succeeded4054worker1 # [ 42.278257] nix-daemon[855]: accepted connection from pid 911, user nix-grpc-daemon (trusted)4055worker1 # [ 42.298219] nix-daemon[855]: reaped child process 914, status = succeeded4056lb # [ 42.241976] niks3-server[1040]: 2026/09/21 12:33:34 INFO Received uploads request method=POST path=/api/pending_closures4057worker1 # [ 42.348589] nix-grpc-daemon[869]: time=2026-09-21T12:33:34.987Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4058worker1 # [ 42.350583] nix-grpc-daemon[869]: time=2026-09-21T12:33:34.988Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)"4059lb # [ 42.353353] niks3-server[1040]: 2026/09/21 12:33:35 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4060lb # [ 42.360348] niks3-server[1040]: 2026/09/21 12:33:35 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4061lb # [ 42.366720] niks3-server[1040]: 2026/09/21 12:33:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4062lb # [ 42.373238] niks3-server[1040]: 2026/09/21 12:33:35 INFO Signed narinfos id=2 count=14063worker1 # [ 42.445650] nix-grpc-daemon[869]: time=2026-09-21T12:33:35.084Z level=INFO msg="Uploading 1 narinfos"4064lb # [ 42.375264] niks3-server[1040]: 2026/09/21 12:33:35 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls4065lb # [ 42.387560] niks3-server[1040]: 2026/09/21 12:33:35 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4066lb # [ 42.392372] niks3-server[1040]: 2026/09/21 12:33:35 INFO Completed upload id=24067worker1 # [ 42.465131] nix-grpc-daemon[869]: time=2026-09-21T12:33:35.105Z level=INFO msg="Upload complete. (452ms)"4068lb # [ 42.395472] niks3-server[1040]: 2026/09/21 12:33:35 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo4069worker1 # [ 42.493070] nix-grpc-daemon[773]: ts=2026-09-21T12:33:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=2 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=14070worker1 # [ 42.512591] nix-grpc-daemon[773]: ts=2026-09-21T12:33:35Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=04071worker1 # [ 42.517504] nix-grpc-daemon[773]: ts=2026-09-21T12:33:35Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.3:50051 clients=14072worker1 # [ 42.521702] nix-daemon[855]: reaped child process 893, status = succeeded4073worker1 # [ 42.549200] nix-grpc-daemon[773]: ts=2026-09-21T12:33:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244074worker1 # [ 42.574795] nix-grpc-daemon[773]: ts=2026-09-21T12:33:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421244075worker1 # [ 42.599018] nix-grpc-daemon[869]: time=2026-09-21T12:33:35.239Z level=INFO msg="All 1 paths already cached"4076client # copying 1 paths to 'lb:50051'...4077client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'...4078worker1 # [ 42.613285] nix-grpc-daemon[773]: ts=2026-09-21T12:33:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421244079worker1 # [ 42.637499] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4080worker1 # [ 42.927644] nix-daemon[855]: accepted connection from pid 920, user nix-grpc-daemon (trusted)4081worker1 # [ 42.952465] nix-daemon[855]: reaped child process 924, status = succeeded4082worker1 # [ 43.150534] nix-daemon[855]: accepted connection from pid 926, user nix-grpc-daemon (trusted)4083worker1 # [ 43.169820] nix-daemon[855]: reaped child process 929, status = succeeded4084lb # [ 43.109544] niks3-server[1040]: 2026/09/21 12:33:35 INFO Received uploads request method=POST path=/api/pending_closures4085worker1 # [ 43.207350] nix-grpc-daemon[869]: time=2026-09-21T12:33:35.847Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4086worker1 # [ 43.209021] nix-grpc-daemon[869]: time=2026-09-21T12:33:35.847Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)"4087lb # [ 43.170773] niks3-server[1040]: 2026/09/21 12:33:35 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst4088lb # [ 43.174319] niks3-server[1040]: 2026/09/21 12:33:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4089worker1 # [ 43.250513] nix-grpc-daemon[869]: time=2026-09-21T12:33:35.890Z level=INFO msg="Uploading 1 narinfos"4090lb # [ 43.180345] niks3-server[1040]: 2026/09/21 12:33:35 INFO Signed narinfos id=3 count=14091lb # [ 43.183949] niks3-server[1040]: 2026/09/21 12:33:35 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls4092lb # [ 43.188645] niks3-server[1040]: 2026/09/21 12:33:35 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4093lb # [ 43.195133] niks3-server[1040]: 2026/09/21 12:33:35 INFO Completed upload id=34094worker1 # [ 43.268586] nix-grpc-daemon[869]: time=2026-09-21T12:33:35.909Z level=INFO msg="Upload complete. (609ms)"4095lb # [ 43.197506] niks3-server[1040]: 2026/09/21 12:33:35 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo4096worker1 # [ 43.280698] nix-grpc-daemon[773]: ts=2026-09-21T12:33:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=0 paths=1 nar_bytes_in=6244097worker1 # [ 43.292080] nix-grpc-daemon[773]: ts=2026-09-21T12:33:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244098worker1 # [ 43.304503] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4099worker1 # [ 43.309384] nix-daemon[855]: reaped child process 918, status = succeeded4100client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051...4101worker1 # [ 43.321364] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4102client # farm-b> node-a: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4103client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4104client # farm-b> LOG-farm-b-t14105worker1 # [ 44.147166] nix-daemon[855]: reaped child process 933, status = succeeded4106worker1 # [ 44.307668] nix-daemon[855]: accepted connection from pid 949, user nix-grpc-daemon (trusted)4107worker1 # [ 44.327404] nix-daemon[855]: reaped child process 952, status = succeeded4108lb # [ 44.266751] niks3-server[1040]: 2026/09/21 12:33:36 INFO Received uploads request method=POST path=/api/pending_closures4109worker1 # [ 44.351033] nix-grpc-daemon[869]: time=2026-09-21T12:33:36.990Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4110worker1 # [ 44.352656] nix-grpc-daemon[869]: time=2026-09-21T12:33:36.990Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)"4111lb # [ 44.311405] niks3-server[1040]: 2026/09/21 12:33:36 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4112lb # [ 44.323016] niks3-server[1040]: 2026/09/21 12:33:36 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst4113lb # [ 44.332921] niks3-server[1040]: 2026/09/21 12:33:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4114worker1 # [ 44.411531] nix-grpc-daemon[869]: time=2026-09-21T12:33:37.051Z level=INFO msg="Uploading 1 narinfos"4115lb # [ 44.340246] niks3-server[1040]: 2026/09/21 12:33:36 INFO Signed narinfos id=4 count=14116lb # [ 44.345918] niks3-server[1040]: 2026/09/21 12:33:36 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls4117lb # [ 44.354572] niks3-server[1040]: 2026/09/21 12:33:37 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4118lb # [ 44.361658] niks3-server[1040]: 2026/09/21 12:33:37 INFO Completed upload id=44119worker1 # [ 44.434137] nix-grpc-daemon[869]: time=2026-09-21T12:33:37.074Z level=INFO msg="Upload complete. (297ms)"4120lb # [ 44.364677] niks3-server[1040]: 2026/09/21 12:33:37 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo4121worker1 # [ 44.448445] nix-grpc-daemon[773]: ts=2026-09-21T12:33:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=14122worker1 # [ 44.466457] nix-grpc-daemon[773]: ts=2026-09-21T12:33:37Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=04123worker1 # [ 44.470798] nix-grpc-daemon[773]: ts=2026-09-21T12:33:37Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.3:50051 clients=14124worker1 # [ 44.475044] nix-daemon[855]: reaped child process 931, status = succeeded4125worker1 # [ 44.505773] nix-grpc-daemon[773]: ts=2026-09-21T12:33:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244126worker1 # [ 44.521430] nix-grpc-daemon[773]: ts=2026-09-21T12:33:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421244127worker1 # [ 44.537456] nix-grpc-daemon[869]: time=2026-09-21T12:33:37.177Z level=INFO msg="All 2 paths already cached"4128client # copying 1 paths to 'lb:50051'...4129worker1 # [ 44.547790] nix-grpc-daemon[773]: ts=2026-09-21T12:33:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421244130client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'...4131worker1 # [ 44.568334] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4132worker1 # [ 44.778763] nix-daemon[855]: accepted connection from pid 958, user nix-grpc-daemon (trusted)4133worker1 # [ 44.796760] nix-daemon[855]: reaped child process 961, status = succeeded4134worker1 # [ 44.972427] nix-daemon[855]: accepted connection from pid 963, user nix-grpc-daemon (trusted)4135worker1 # [ 44.989345] nix-daemon[855]: reaped child process 966, status = succeeded4136lb # [ 44.927927] niks3-server[1040]: 2026/09/21 12:33:37 INFO Received uploads request method=POST path=/api/pending_closures4137worker1 # [ 45.011832] nix-grpc-daemon[869]: time=2026-09-21T12:33:37.652Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4138worker1 # [ 45.013504] nix-grpc-daemon[869]: time=2026-09-21T12:33:37.652Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)"4139lb # [ 44.959621] niks3-server[1040]: 2026/09/21 12:33:37 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst4140lb # [ 44.973387] niks3-server[1040]: 2026/09/21 12:33:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4141worker1 # [ 45.053054] nix-grpc-daemon[869]: time=2026-09-21T12:33:37.693Z level=INFO msg="Uploading 1 narinfos"4142lb # [ 44.984058] niks3-server[1040]: 2026/09/21 12:33:37 INFO Signed narinfos id=5 count=14143lb # [ 44.986715] niks3-server[1040]: 2026/09/21 12:33:37 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls4144lb # [ 44.996201] niks3-server[1040]: 2026/09/21 12:33:37 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4145lb # [ 45.003931] niks3-server[1040]: 2026/09/21 12:33:37 INFO Completed upload id=54146worker1 # [ 45.076742] nix-grpc-daemon[869]: time=2026-09-21T12:33:37.717Z level=INFO msg="Upload complete. (487ms)"4147lb # [ 45.006456] niks3-server[1040]: 2026/09/21 12:33:37 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo4148worker1 # [ 45.086063] nix-grpc-daemon[773]: ts=2026-09-21T12:33:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=0 paths=1 nar_bytes_in=7524149worker1 # [ 45.098798] nix-grpc-daemon[773]: ts=2026-09-21T12:33:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244150worker1 # [ 45.105303] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4151worker1 # [ 45.110060] nix-daemon[855]: reaped child process 956, status = succeeded4152client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051...4153worker1 # [ 45.122664] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4154client # farm-top> node-a: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4155client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4156client # farm-top> LOG-farm-top-t14157worker1 # [ 46.364534] nix-daemon[855]: reaped child process 970, status = succeeded4158worker1 # [ 46.552192] nix-daemon[855]: accepted connection from pid 987, user nix-grpc-daemon (trusted)4159worker1 # [ 46.571241] nix-daemon[855]: reaped child process 990, status = succeeded4160lb # [ 46.510812] niks3-server[1040]: 2026/09/21 12:33:39 INFO Received uploads request method=POST path=/api/pending_closures4161worker1 # [ 46.594700] nix-grpc-daemon[869]: time=2026-09-21T12:33:39.234Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4162worker1 # [ 46.598072] nix-grpc-daemon[869]: time=2026-09-21T12:33:39.237Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)"4163lb # [ 46.551665] niks3-server[1040]: 2026/09/21 12:33:39 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4164lb # [ 46.561228] niks3-server[1040]: 2026/09/21 12:33:39 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst4165worker1 # [ 46.644867] nix-grpc-daemon[869]: time=2026-09-21T12:33:39.285Z level=INFO msg="Uploading 1 narinfos"4166lb # [ 46.572240] niks3-server[1040]: 2026/09/21 12:33:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4167lb # [ 46.579311] niks3-server[1040]: 2026/09/21 12:33:39 INFO Signed narinfos id=6 count=14168lb # [ 46.583600] niks3-server[1040]: 2026/09/21 12:33:39 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls4169lb # [ 46.588176] niks3-server[1040]: 2026/09/21 12:33:39 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4170lb # [ 46.596291] niks3-server[1040]: 2026/09/21 12:33:39 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo4171lb # [ 46.600771] niks3-server[1040]: 2026/09/21 12:33:39 INFO Completed upload id=64172worker1 # [ 46.673951] nix-grpc-daemon[869]: time=2026-09-21T12:33:39.314Z level=INFO msg="Upload complete. (341ms)"4173worker1 # [ 46.685185] nix-grpc-daemon[773]: ts=2026-09-21T12:33:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=14174worker1 # [ 46.701370] nix-daemon[855]: reaped child process 968, status = succeeded4175client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2, in 7.69 seconds)4176client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath4177client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath, in 0.24 seconds)4178client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4179client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.03 seconds)4180client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4181client # copying 3 paths...4182client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'...4183client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'...4184client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'...4185client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.30 seconds)4186(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 8.38 seconds)4187subtest: repeat is answered Cached by the scheduler without building4188worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4189worker1 # finding garbage collector roots...4190worker1 # deleting specified paths...4191worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4192worker1 # deleting unused links...4193worker1 # note: hard linking is currently saving 0.0 KiB4194worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.27 seconds)4195worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4196worker2 # finding garbage collector roots...4197worker2 # deleting specified paths...4198worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4199worker2 # deleting unused links...4200worker2 # note: hard linking is currently saving 0.0 KiB4201worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.30 seconds)4202client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&24203worker1 # [ 48.263185] nix-grpc-daemon[773]: ts=2026-09-21T12:33:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:421244204client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4205client # /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4206worker1 # [ 48.282675] nix-grpc-daemon[773]: ts=2026-09-21T12:33:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421244207worker1 # [ 48.293780] nix-grpc-daemon[869]: time=2026-09-21T12:33:40.933Z level=INFO msg="All 2 paths already cached"4208worker1 # [ 48.312788] nix-grpc-daemon[773]: ts=2026-09-21T12:33:40Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=14209client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2, in 0.36 seconds)4210client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath4211client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath, in 0.21 seconds)4212lb: must succeed: curl -sf localhost:9901/clusters4213lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)4214lb: must succeed: curl -sf localhost:9901/clusters4215lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4216worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4217worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4218(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.46 seconds)4219subtest: inputs already in the cache are not sent to the scheduler4220lb: must succeed: curl -sf localhost:9901/clusters4221lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4222lb: must succeed: curl -sf localhost:9901/clusters4223lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4224worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4225worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.07 seconds)4226client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b >&24227worker2 # [ 49.290045] nix-grpc-daemon[779]: ts=2026-09-21T12:33:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:366384228client # don't know how to build these paths:4229client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4230worker1 # [ 49.374847] nix-grpc-daemon[773]: ts=2026-09-21T12:33:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421244231worker1 # [ 49.387652] nix-grpc-daemon[869]: time=2026-09-21T12:33:42.027Z level=INFO msg="All 2 paths already cached"4232worker1 # [ 49.403784] nix-grpc-daemon[773]: ts=2026-09-21T12:33:42Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=04233worker1 # [ 49.410717] nix-grpc-daemon[773]: ts=2026-09-21T12:33:42Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.3:50051 clients=14234worker1 # [ 49.453186] nix-grpc-daemon[773]: ts=2026-09-21T12:33:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244235worker1 # [ 49.468943] nix-grpc-daemon[773]: ts=2026-09-21T12:33:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421244236worker1 # [ 49.483542] nix-grpc-daemon[869]: time=2026-09-21T12:33:42.124Z level=INFO msg="All 2 paths already cached"4237client # copying 1 paths to 'lb:50051'...4238worker1 # [ 49.493686] nix-grpc-daemon[773]: ts=2026-09-21T12:33:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421244239client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'...4240worker1 # [ 49.512122] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4241worker1 # [ 49.718882] nix-daemon[855]: accepted connection from pid 1019, user nix-grpc-daemon (trusted)4242worker1 # [ 49.737618] nix-daemon[855]: reaped child process 1022, status = succeeded4243worker1 # [ 49.911006] nix-daemon[855]: accepted connection from pid 1024, user nix-grpc-daemon (trusted)4244worker1 # [ 49.926957] nix-daemon[855]: reaped child process 1027, status = succeeded4245lb # [ 49.867017] niks3-server[1040]: 2026/09/21 12:33:42 INFO Received uploads request method=POST path=/api/pending_closures4246worker1 # [ 49.949508] nix-grpc-daemon[869]: time=2026-09-21T12:33:42.589Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4247worker1 # [ 49.950991] nix-grpc-daemon[869]: time=2026-09-21T12:33:42.589Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)"4248lb # [ 49.890695] envoy[902]: conn peer=[2001:db8:1::1]:41246 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9549 tx=4050 ms=74224249lb # [ 49.901425] envoy[902]: conn peer=[2001:db8:1::1]:48096 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=480 ms=714250worker1 # [ 49.987548] nix-grpc-daemon[869]: time=2026-09-21T12:33:42.627Z level=INFO msg="Uploading 1 narinfos"4251lb # [ 49.915143] niks3-server[1040]: 2026/09/21 12:33:42 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst4252lb # [ 49.927049] niks3-server[1040]: 2026/09/21 12:33:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4253lb # [ 49.933390] niks3-server[1040]: 2026/09/21 12:33:42 INFO Signed narinfos id=7 count=14254worker1 # [ 50.013636] nix-grpc-daemon[869]: time=2026-09-21T12:33:42.654Z level=INFO msg="Upload complete. (483ms)"4255lb # [ 49.941259] niks3-server[1040]: 2026/09/21 12:33:42 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls4256lb # [ 49.947121] niks3-server[1040]: 2026/09/21 12:33:42 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4257lb # [ 49.951849] niks3-server[1040]: 2026/09/21 12:33:42 INFO Completed upload id=74258worker1 # [ 50.024272] nix-grpc-daemon[773]: ts=2026-09-21T12:33:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=0 paths=1 nar_bytes_in=7524259lb # [ 49.954208] niks3-server[1040]: 2026/09/21 12:33:42 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo4260worker1 # [ 50.034657] nix-grpc-daemon[773]: ts=2026-09-21T12:33:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244261worker1 # [ 50.044512] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4262worker1 # [ 50.047364] nix-daemon[855]: reaped child process 1017, status = succeeded4263client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051...4264worker1 # [ 50.058872] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4265client # farm-top> node-a: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4266client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4267client # farm-top> LOG-farm-top-t1b4268worker1 # [ 51.103677] nix-daemon[855]: reaped child process 1031, status = succeeded4269worker1 # [ 51.285036] nix-daemon[855]: accepted connection from pid 1047, user nix-grpc-daemon (trusted)4270worker1 # [ 51.302086] nix-daemon[855]: reaped child process 1050, status = succeeded4271lb # [ 51.242727] niks3-server[1040]: 2026/09/21 12:33:43 INFO Received uploads request method=POST path=/api/pending_closures4272worker1 # [ 51.328504] nix-grpc-daemon[869]: time=2026-09-21T12:33:43.965Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4273worker1 # [ 51.330654] nix-grpc-daemon[869]: time=2026-09-21T12:33:43.966Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)"4274lb # [ 51.284785] niks3-server[1040]: 2026/09/21 12:33:43 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4275lb # [ 51.295349] niks3-server[1040]: 2026/09/21 12:33:43 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst4276lb # [ 51.307375] niks3-server[1040]: 2026/09/21 12:33:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4277worker1 # [ 51.386600] nix-grpc-daemon[869]: time=2026-09-21T12:33:44.026Z level=INFO msg="Uploading 1 narinfos"4278lb # [ 51.313394] niks3-server[1040]: 2026/09/21 12:33:43 INFO Signed narinfos id=8 count=14279lb # [ 51.318553] niks3-server[1040]: 2026/09/21 12:33:43 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls4280lb # [ 51.331662] niks3-server[1040]: 2026/09/21 12:33:43 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4281lb # [ 51.339016] niks3-server[1040]: 2026/09/21 12:33:43 INFO Completed upload id=84282worker1 # [ 51.411581] nix-grpc-daemon[869]: time=2026-09-21T12:33:44.052Z level=INFO msg="Upload complete. (339ms)"4283lb # [ 51.342115] niks3-server[1040]: 2026/09/21 12:33:43 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo4284worker1 # [ 51.425176] nix-grpc-daemon[773]: ts=2026-09-21T12:33:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=14285worker1 # [ 51.442739] nix-daemon[855]: reaped child process 1029, status = succeeded4286client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.37 seconds)4287client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b outPath4288client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.23 seconds)4289lb: must succeed: curl -sf localhost:9901/clusters4290lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4291lb: must succeed: curl -sf localhost:9901/clusters4292lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4293worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4294worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.07 seconds)4295(finished: subtest: inputs already in the cache are not sent to the scheduler, in 3.02 seconds)4296subtest: two clients wanting the same drv build it once4297client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup4298client # [ 51.925272] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup.4299client # Running as unit: dedup1.service; invocation ID: c621bc2bcb2b490997ffeeea48bcce974300client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup, in 0.09 seconds)4301client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup4302worker2 # [ 52.182280] nix-grpc-daemon[779]: ts=2026-09-21T12:33:44Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:366384303client # [ 52.196883] nix[880]: don't know how to build these paths:4304client # [ 52.198693] nix[880]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4305worker1 # [ 52.272173] nix-grpc-daemon[773]: ts=2026-09-21T12:33:44Z level=debug event=want client=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04306worker1 # [ 52.272628] nix-grpc-daemon[773]: ts=2026-09-21T12:33:44Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.4:50051 clients=14307worker2 # [ 52.257753] nix-grpc-daemon[779]: ts=2026-09-21T12:33:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366384308worker2 # [ 52.278544] nix-grpc-daemon[779]: ts=2026-09-21T12:33:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366384309client # [ 52.300177] nix[880]: copying 1 paths to 'lb:50051'...4310worker2 # [ 52.298417] nix-grpc-daemon[779]: ts=2026-09-21T12:33:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:366384311client # [ 52.307875] nix[880]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'...4312worker2 # [ 52.314519] nix-daemon[847]: accepted connection from pid 779, user nix-grpc-daemon (trusted)4313worker2 # [ 52.345696] nix-grpc-daemon[779]: ts=2026-09-21T12:33:44Z level=info event=niks3_push_started pid=8754314worker2 # [ 52.608755] nix-grpc-daemon[875]: time=2026-09-21T12:33:45.241Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""4315worker2 # [ 52.797286] nix-daemon[847]: accepted connection from pid 888, user nix-grpc-daemon (trusted)4316worker2 # [ 52.815833] nix-daemon[847]: reaped child process 891, status = succeeded4317worker2 # [ 52.988585] nix-daemon[847]: accepted connection from pid 893, user nix-grpc-daemon (trusted)4318worker2 # [ 53.004329] nix-daemon[847]: reaped child process 896, status = succeeded4319lb # [ 53.018050] niks3-server[1040]: 2026/09/21 12:33:45 INFO Received uploads request method=POST path=/api/pending_closures4320worker2 # [ 53.025553] nix-grpc-daemon[875]: time=2026-09-21T12:33:45.660Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4321worker2 # [ 53.028886] nix-grpc-daemon[875]: time=2026-09-21T12:33:45.662Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)"4322client # [ 53.039739] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup.4323lb # [ 53.051534] niks3-server[1040]: 2026/09/21 12:33:45 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst4324client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup, in 1.13 seconds)4325worker2 # [ 53.072222] nix-grpc-daemon[875]: time=2026-09-21T12:33:45.706Z level=INFO msg="Uploading 1 narinfos"4326lb # [ 53.064808] niks3-server[1040]: 2026/09/21 12:33:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4327lb # [ 53.076931] niks3-server[1040]: 2026/09/21 12:33:45 INFO Signed narinfos id=9 count=14328lb # [ 53.083266] niks3-server[1040]: 2026/09/21 12:33:45 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls4329lb # [ 53.094310] niks3-server[1040]: 2026/09/21 12:33:45 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4330lb # [ 53.103873] niks3-server[1040]: 2026/09/21 12:33:45 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo4331lb # [ 53.109019] niks3-server[1040]: 2026/09/21 12:33:45 INFO Completed upload id=94332worker2 # [ 53.110250] nix-grpc-daemon[875]: time=2026-09-21T12:33:45.745Z level=INFO msg="Upload complete. (493ms)"4333worker2 # [ 53.118880] nix-grpc-daemon[779]: ts=2026-09-21T12:33:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36638 duration_s=0 paths=1 nar_bytes_in=4324334worker2 # [ 53.131527] nix-grpc-daemon[779]: ts=2026-09-21T12:33:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366384335worker2 # [ 53.137898] nix-daemon[847]: accepted connection from pid 779, user nix-grpc-daemon (trusted)4336worker2 # [ 53.142804] nix-daemon[847]: reaped child process 873, status = succeeded4337client # Running as unit: dedup2.service; invocation ID: 18878402dd2f4d5da4424fbcf149accf4338client # [ 53.159237] nix[880]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4339worker2 # [ 53.153638] nix-daemon[847]: accepted connection from pid 779, user nix-grpc-daemon (trusted)4340client # [ 53.373933] nix[896]: don't know how to build these paths:4341client # [ 53.375608] nix[896]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4342worker1 # [ 53.436979] nix-grpc-daemon[773]: ts=2026-09-21T12:33:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:421244343worker1 # [ 53.451032] nix-grpc-daemon[773]: ts=2026-09-21T12:33:46Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04344client # [ 53.393066] nix[896]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4345worker2 # [ 53.385048] nix-grpc-daemon[779]: ts=2026-09-21T12:33:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366384346client: must succeed: sleep 24347client: (finished: must succeed: sleep 2, in 2.07 seconds)4348client: waiting for success: ! systemctl is-active dedup1 dedup24349worker2 # [ 58.006800] nix-daemon[847]: reaped child process 902, status = succeeded4350worker2 # [ 58.196067] nix-daemon[847]: accepted connection from pid 934, user nix-grpc-daemon (trusted)4351worker2 # [ 58.212219] nix-daemon[847]: reaped child process 937, status = succeeded4352lb # [ 58.225623] niks3-server[1040]: 2026/09/21 12:33:50 INFO Received uploads request method=POST path=/api/pending_closures4353worker2 # [ 58.238278] nix-grpc-daemon[875]: time=2026-09-21T12:33:50.871Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4354worker2 # [ 58.241114] nix-grpc-daemon[875]: time=2026-09-21T12:33:50.871Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)"4355lb # [ 58.269917] niks3-server[1040]: 2026/09/21 12:33:50 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4356lb # [ 58.284523] niks3-server[1040]: 2026/09/21 12:33:50 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4357lb # [ 58.296073] niks3-server[1040]: 2026/09/21 12:33:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4358lb # [ 58.305036] niks3-server[1040]: 2026/09/21 12:33:50 INFO Signed narinfos id=10 count=14359worker2 # [ 58.305860] nix-grpc-daemon[875]: time=2026-09-21T12:33:50.939Z level=INFO msg="Uploading 1 narinfos"4360lb # [ 58.308210] niks3-server[1040]: 2026/09/21 12:33:50 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls4361lb # [ 58.319501] niks3-server[1040]: 2026/09/21 12:33:50 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4362lb # [ 58.329155] niks3-server[1040]: 2026/09/21 12:33:50 INFO Completed upload id=104363worker2 # [ 58.330711] nix-grpc-daemon[875]: time=2026-09-21T12:33:50.965Z level=INFO msg="Upload complete. (349ms)"4364lb # [ 58.333621] niks3-server[1040]: 2026/09/21 12:33:50 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo4365worker2 # [ 58.345163] nix-grpc-daemon[779]: ts=2026-09-21T12:33:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36638 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14366worker2 # [ 58.360622] nix-grpc-daemon[779]: ts=2026-09-21T12:33:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36638 duration_s=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14367worker2 # [ 58.369664] nix-daemon[847]: reaped child process 898, status = succeeded4368client # [ 58.417220] systemd[1]: dedup1.service: Deactivated successfully.4369client # [ 58.421076] systemd[1]: dedup1.service: Consumed 174ms CPU time over 6.495s wall clock time, 39.3M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.4370client # [ 58.430649] systemd[1]: dedup2.service: Deactivated successfully.4371client # [ 58.435049] systemd[1]: dedup2.service: Consumed 205ms CPU time over 5.391s wall clock time, 35.2M memory peak, 6K incoming IP traffic, 5.7K outgoing IP traffic.4372client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.14 seconds)4373client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4374client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.07 seconds)4375worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4376worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4377worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4378worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4379worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4380worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4381worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4382worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4383(finished: subtest: two clients wanting the same drv build it once, in 7.44 seconds)4384subtest: build hook: nix-daemon with builders = grpc://lb4385client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag hook 2>/tmp/hook.log4386client # [ 59.512764] systemd[1]: Started Nix Daemon.4387client # [ 59.696211] nix-daemon[938]: accepted connection from pid 935, user root (trusted)4388lb # [ 59.892312] envoy[902]: conn peer=[2001:db8:1::1]:48112 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4202 tx=1825 ms=20774389lb # [ 59.897736] envoy[902]: conn peer=[2001:db8:1::1]:54570 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3009 tx=1470 ms=61784390lb # [ 59.907199] envoy[902]: conn peer=[2001:db8:1::1]:54584 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1443 tx=1082 ms=50094391worker1 # [ 59.993874] nix-grpc-daemon[773]: ts=2026-09-21T12:33:52Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:421244392worker2 # [ 59.926668] nix-grpc-daemon[779]: ts=2026-09-21T12:33:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366384393worker1 # [ 60.019074] nix-grpc-daemon[773]: ts=2026-09-21T12:33:52Z level=debug event=want client=6 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=04394worker1 # [ 60.026310] nix-grpc-daemon[773]: ts=2026-09-21T12:33:52Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.3:50051 clients=14395worker1 # [ 60.065925] nix-grpc-daemon[773]: ts=2026-09-21T12:33:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244396worker1 # [ 60.082715] nix-grpc-daemon[773]: ts=2026-09-21T12:33:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421244397worker1 # [ 60.098486] nix-grpc-daemon[773]: ts=2026-09-21T12:33:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421244398worker1 # [ 60.114639] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4399worker1 # [ 60.323701] nix-daemon[855]: accepted connection from pid 1090, user nix-grpc-daemon (trusted)4400worker1 # [ 60.344277] nix-daemon[855]: reaped child process 1093, status = succeeded4401worker1 # [ 60.520336] nix-daemon[855]: accepted connection from pid 1095, user nix-grpc-daemon (trusted)4402worker1 # [ 60.536862] nix-daemon[855]: reaped child process 1098, status = succeeded4403lb # [ 60.478817] niks3-server[1040]: 2026/09/21 12:33:53 INFO Received uploads request method=POST path=/api/pending_closures4404worker1 # [ 60.561659] nix-grpc-daemon[869]: time=2026-09-21T12:33:53.201Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4405worker1 # [ 60.563164] nix-grpc-daemon[869]: time=2026-09-21T12:33:53.201Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)"4406lb # [ 60.509147] niks3-server[1040]: 2026/09/21 12:33:53 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst4407lb # [ 60.525229] niks3-server[1040]: 2026/09/21 12:33:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4408worker1 # [ 60.604206] nix-grpc-daemon[869]: time=2026-09-21T12:33:53.244Z level=INFO msg="Uploading 1 narinfos"4409lb # [ 60.538024] niks3-server[1040]: 2026/09/21 12:33:53 INFO Signed narinfos id=11 count=14410lb # [ 60.548220] niks3-server[1040]: 2026/09/21 12:33:53 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls4411lb # [ 60.551898] niks3-server[1040]: 2026/09/21 12:33:53 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4412lb # [ 60.559014] niks3-server[1040]: 2026/09/21 12:33:53 INFO Completed upload id=114413worker1 # [ 60.631516] nix-grpc-daemon[869]: time=2026-09-21T12:33:53.272Z level=INFO msg="Upload complete. (499ms)"4414lb # [ 60.561763] niks3-server[1040]: 2026/09/21 12:33:53 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo4415worker1 # [ 60.640314] nix-grpc-daemon[773]: ts=2026-09-21T12:33:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=0 paths=1 nar_bytes_in=5124416worker1 # [ 60.654193] nix-grpc-daemon[773]: ts=2026-09-21T12:33:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244417worker1 # [ 60.661012] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4418worker1 # [ 60.666368] nix-daemon[855]: reaped child process 1088, status = succeeded4419worker1 # [ 60.678049] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4420worker1 # [ 61.528374] nix-daemon[855]: reaped child process 1102, status = succeeded4421worker1 # [ 61.701906] nix-daemon[855]: accepted connection from pid 1118, user nix-grpc-daemon (trusted)4422worker1 # [ 61.717605] nix-daemon[855]: reaped child process 1121, status = succeeded4423lb # [ 61.657205] niks3-server[1040]: 2026/09/21 12:33:54 INFO Received uploads request method=POST path=/api/pending_closures4424worker1 # [ 61.744394] nix-grpc-daemon[869]: time=2026-09-21T12:33:54.380Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4425worker1 # [ 61.745919] nix-grpc-daemon[869]: time=2026-09-21T12:33:54.380Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)"4426lb # [ 61.705693] niks3-server[1040]: 2026/09/21 12:33:54 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4427worker1 # [ 61.787155] nix-grpc-daemon[869]: time=2026-09-21T12:33:54.427Z level=INFO msg="Uploading 1 narinfos"4428lb # [ 61.719022] niks3-server[1040]: 2026/09/21 12:33:54 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv4429lb # [ 61.727862] niks3-server[1040]: 2026/09/21 12:33:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4430lb # [ 61.737408] niks3-server[1040]: 2026/09/21 12:33:54 INFO Signed narinfos id=12 count=14431lb # [ 61.740556] niks3-server[1040]: 2026/09/21 12:33:54 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls4432worker1 # [ 61.809754] nix-grpc-daemon[869]: time=2026-09-21T12:33:54.450Z level=INFO msg="Upload complete. (316ms)"4433lb # [ 61.745297] niks3-server[1040]: 2026/09/21 12:33:54 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4434lb # [ 61.749858] niks3-server[1040]: 2026/09/21 12:33:54 INFO Completed upload id=124435worker1 # [ 61.821208] nix-grpc-daemon[773]: ts=2026-09-21T12:33:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=6 outputs=14436lb # [ 61.752430] niks3-server[1040]: 2026/09/21 12:33:54 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo4437worker1 # [ 61.838277] nix-grpc-daemon[773]: ts=2026-09-21T12:33:54Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:421244438worker1 # [ 61.842311] nix-daemon[855]: reaped child process 1100, status = succeeded4439worker1 # [ 61.863558] nix-grpc-daemon[773]: ts=2026-09-21T12:33:54Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:421244440worker1 # [ 61.869816] nix-grpc-daemon[773]: ts=2026-09-21T12:33:54Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=0 paths=1 nar_bytes_out=1284441worker2 # [ 62.120464] nix-grpc-daemon[779]: ts=2026-09-21T12:33:54Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:366384442worker2 # [ 62.126866] nix-grpc-daemon[779]: ts=2026-09-21T12:33:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366384443worker1 # [ 62.224129] nix-grpc-daemon[773]: ts=2026-09-21T12:33:54Z level=debug event=want client=7 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=04444worker1 # [ 62.230817] nix-grpc-daemon[773]: ts=2026-09-21T12:33:54Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.3:50051 clients=14445worker1 # [ 62.239074] nix-grpc-daemon[773]: ts=2026-09-21T12:33:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244446worker1 # [ 62.249542] nix-grpc-daemon[773]: ts=2026-09-21T12:33:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421244447worker1 # [ 62.262882] nix-grpc-daemon[869]: time=2026-09-21T12:33:54.903Z level=INFO msg="All 1 paths already cached"4448worker1 # [ 62.271636] nix-grpc-daemon[773]: ts=2026-09-21T12:33:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421244449worker1 # [ 62.288570] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4450worker1 # [ 62.491354] nix-daemon[855]: accepted connection from pid 1127, user nix-grpc-daemon (trusted)4451worker1 # [ 62.508923] nix-daemon[855]: reaped child process 1130, status = succeeded4452worker1 # [ 62.677131] nix-daemon[855]: accepted connection from pid 1132, user nix-grpc-daemon (trusted)4453worker1 # [ 62.692650] nix-daemon[855]: reaped child process 1135, status = succeeded4454lb # [ 62.631003] niks3-server[1040]: 2026/09/21 12:33:55 INFO Received uploads request method=POST path=/api/pending_closures4455worker1 # [ 62.713865] nix-grpc-daemon[869]: time=2026-09-21T12:33:55.354Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4456worker1 # [ 62.715281] nix-grpc-daemon[869]: time=2026-09-21T12:33:55.354Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)"4457lb # [ 62.660682] niks3-server[1040]: 2026/09/21 12:33:55 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst4458lb # [ 62.672418] niks3-server[1040]: 2026/09/21 12:33:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4459worker1 # [ 62.753148] nix-grpc-daemon[869]: time=2026-09-21T12:33:55.393Z level=INFO msg="Uploading 1 narinfos"4460lb # [ 62.679852] niks3-server[1040]: 2026/09/21 12:33:55 INFO Signed narinfos id=13 count=14461lb # [ 62.690029] niks3-server[1040]: 2026/09/21 12:33:55 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls4462lb # [ 62.697896] niks3-server[1040]: 2026/09/21 12:33:55 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4463lb # [ 62.704157] niks3-server[1040]: 2026/09/21 12:33:55 INFO Completed upload id=134464worker1 # [ 62.777230] nix-grpc-daemon[869]: time=2026-09-21T12:33:55.417Z level=INFO msg="Upload complete. (472ms)"4465lb # [ 62.707243] niks3-server[1040]: 2026/09/21 12:33:55 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo4466worker1 # [ 62.784270] nix-grpc-daemon[773]: ts=2026-09-21T12:33:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=0 paths=1 nar_bytes_in=6244467worker1 # [ 62.797064] nix-grpc-daemon[773]: ts=2026-09-21T12:33:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244468worker1 # [ 62.803131] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4469worker1 # [ 62.807586] nix-daemon[855]: reaped child process 1125, status = succeeded4470worker1 # [ 62.820597] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4471worker1 # [ 63.239761] nix-daemon[855]: reaped child process 1139, status = succeeded4472worker1 # [ 63.396644] nix-daemon[855]: accepted connection from pid 1155, user nix-grpc-daemon (trusted)4473worker1 # [ 63.411563] nix-daemon[855]: reaped child process 1158, status = succeeded4474lb # [ 63.352458] niks3-server[1040]: 2026/09/21 12:33:56 INFO Received uploads request method=POST path=/api/pending_closures4475worker1 # [ 63.437453] nix-grpc-daemon[869]: time=2026-09-21T12:33:56.077Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4476worker1 # [ 63.438895] nix-grpc-daemon[869]: time=2026-09-21T12:33:56.077Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)"4477lb # [ 63.383572] niks3-server[1040]: 2026/09/21 12:33:56 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv4478lb # [ 63.397981] niks3-server[1040]: 2026/09/21 12:33:56 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst4479lb # [ 63.410159] niks3-server[1040]: 2026/09/21 12:33:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4480worker1 # [ 63.487934] nix-grpc-daemon[869]: time=2026-09-21T12:33:56.128Z level=INFO msg="Uploading 1 narinfos"4481lb # [ 63.419578] niks3-server[1040]: 2026/09/21 12:33:56 INFO Signed narinfos id=14 count=14482lb # [ 63.424119] niks3-server[1040]: 2026/09/21 12:33:56 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls4483worker1 # [ 63.503973] nix-grpc-daemon[869]: time=2026-09-21T12:33:56.144Z level=INFO msg="Upload complete. (275ms)"4484lb # [ 63.431345] niks3-server[1040]: 2026/09/21 12:33:56 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4485lb # [ 63.435821] niks3-server[1040]: 2026/09/21 12:33:56 INFO Completed upload id=144486lb # [ 63.438376] niks3-server[1040]: 2026/09/21 12:33:56 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo4487worker1 # [ 63.512728] nix-grpc-daemon[773]: ts=2026-09-21T12:33:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=0 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=7 outputs=14488worker2 # [ 63.448666] nix-grpc-daemon[779]: ts=2026-09-21T12:33:56Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:366384489worker1 # [ 63.525700] nix-daemon[855]: reaped child process 1137, status = succeeded4490worker2 # [ 63.464644] nix-grpc-daemon[779]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'...4491worker2 # [ 63.482564] nix-grpc-daemon[779]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'...4492worker2 # [ 63.523243] nix-grpc-daemon[779]: ts=2026-09-21T12:33:56Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:366384493worker2 # [ 63.529982] nix-grpc-daemon[779]: ts=2026-09-21T12:33:56Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36638 duration_s=0 paths=1 nar_bytes_out=1764494worker1 # [ 63.921044] nix-grpc-daemon[773]: ts=2026-09-21T12:33:56Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:421244495worker2 # [ 63.853245] nix-grpc-daemon[779]: ts=2026-09-21T12:33:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366384496worker2 # [ 63.864270] nix-grpc-daemon[875]: time=2026-09-21T12:33:56.498Z level=INFO msg="All 2 paths already cached"4497worker1 # [ 63.952848] nix-grpc-daemon[773]: ts=2026-09-21T12:33:56Z level=debug event=want client=8 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=04498worker1 # [ 63.960081] nix-grpc-daemon[773]: ts=2026-09-21T12:33:56Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.3:50051 clients=14499worker1 # [ 63.969878] nix-grpc-daemon[773]: ts=2026-09-21T12:33:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244500worker1 # [ 63.980891] nix-grpc-daemon[773]: ts=2026-09-21T12:33:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421244501worker1 # [ 63.994956] nix-grpc-daemon[869]: time=2026-09-21T12:33:56.634Z level=INFO msg="All 2 paths already cached"4502worker1 # [ 64.003604] nix-grpc-daemon[773]: ts=2026-09-21T12:33:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421244503worker1 # [ 64.018410] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4504worker1 # [ 64.219700] nix-daemon[855]: accepted connection from pid 1164, user nix-grpc-daemon (trusted)4505worker1 # [ 64.238073] nix-daemon[855]: reaped child process 1167, status = succeeded4506worker1 # [ 64.411431] nix-daemon[855]: accepted connection from pid 1169, user nix-grpc-daemon (trusted)4507worker1 # [ 64.428606] nix-daemon[855]: reaped child process 1172, status = succeeded4508lb # [ 64.368679] niks3-server[1040]: 2026/09/21 12:33:57 INFO Received uploads request method=POST path=/api/pending_closures4509worker1 # [ 64.452068] nix-grpc-daemon[869]: time=2026-09-21T12:33:57.092Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4510worker1 # [ 64.453880] nix-grpc-daemon[869]: time=2026-09-21T12:33:57.092Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)"4511lb # [ 64.397708] niks3-server[1040]: 2026/09/21 12:33:57 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst4512lb # [ 64.413981] niks3-server[1040]: 2026/09/21 12:33:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4513worker1 # [ 64.492488] nix-grpc-daemon[869]: time=2026-09-21T12:33:57.132Z level=INFO msg="Uploading 1 narinfos"4514lb # [ 64.426014] niks3-server[1040]: 2026/09/21 12:33:57 INFO Signed narinfos id=15 count=14515lb # [ 64.433038] niks3-server[1040]: 2026/09/21 12:33:57 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls4516lb # [ 64.444316] niks3-server[1040]: 2026/09/21 12:33:57 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4517lb # [ 64.449213] niks3-server[1040]: 2026/09/21 12:33:57 INFO Completed upload id=154518worker1 # [ 64.522039] nix-grpc-daemon[869]: time=2026-09-21T12:33:57.162Z level=INFO msg="Upload complete. (488ms)"4519lb # [ 64.451989] niks3-server[1040]: 2026/09/21 12:33:57 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo4520worker1 # [ 64.532975] nix-grpc-daemon[773]: ts=2026-09-21T12:33:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=0 paths=1 nar_bytes_in=7524521worker1 # [ 64.544346] nix-grpc-daemon[773]: ts=2026-09-21T12:33:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244522worker1 # [ 64.552162] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4523worker1 # [ 64.555516] nix-daemon[855]: reaped child process 1162, status = succeeded4524worker1 # [ 64.570075] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4525worker1 # [ 65.991900] nix-daemon[855]: reaped child process 1176, status = succeeded4526worker1 # [ 66.152591] nix-daemon[855]: accepted connection from pid 1192, user nix-grpc-daemon (trusted)4527worker1 # [ 66.168453] nix-daemon[855]: reaped child process 1195, status = succeeded4528lb # [ 66.111936] niks3-server[1040]: 2026/09/21 12:33:58 INFO Received uploads request method=POST path=/api/pending_closures4529worker1 # [ 66.196390] nix-grpc-daemon[869]: time=2026-09-21T12:33:58.834Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4530worker1 # [ 66.197847] nix-grpc-daemon[869]: time=2026-09-21T12:33:58.834Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)"4531lb # [ 66.147661] niks3-server[1040]: 2026/09/21 12:33:58 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst4532lb # [ 66.163600] niks3-server[1040]: 2026/09/21 12:33:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4533worker1 # [ 66.242873] nix-grpc-daemon[869]: time=2026-09-21T12:33:58.883Z level=INFO msg="Uploading 1 narinfos"4534lb # [ 66.176102] niks3-server[1040]: 2026/09/21 12:33:58 INFO Signed narinfos id=16 count=14535lb # [ 66.178558] niks3-server[1040]: 2026/09/21 12:33:58 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls4536lb # [ 66.185899] niks3-server[1040]: 2026/09/21 12:33:58 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv4537lb # [ 66.193836] niks3-server[1040]: 2026/09/21 12:33:58 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4538lb # [ 66.198236] niks3-server[1040]: 2026/09/21 12:33:58 INFO Completed upload id=164539worker1 # [ 66.271272] nix-grpc-daemon[869]: time=2026-09-21T12:33:58.911Z level=INFO msg="Upload complete. (291ms)"4540lb # [ 66.200688] niks3-server[1040]: 2026/09/21 12:33:58 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo4541worker1 # [ 66.281121] nix-grpc-daemon[773]: ts=2026-09-21T12:33:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=8 outputs=14542worker1 # [ 66.295917] nix-grpc-daemon[773]: ts=2026-09-21T12:33:58Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:421244543worker1 # [ 66.300418] nix-daemon[855]: reaped child process 1174, status = succeeded4544worker1 # [ 66.314216] nix-grpc-daemon[773]: ts=2026-09-21T12:33:58Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:421244545worker1 # [ 66.319110] nix-grpc-daemon[773]: ts=2026-09-21T12:33:58Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=0 paths=1 nar_bytes_out=2324546client # [ 66.381382] nix-daemon[938]: reaped child process 940, status = succeeded4547client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.11 seconds)4548client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top4549client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.05 seconds)4550client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4551client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds)4552client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4553client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.04 seconds)4554(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.24 seconds)4555subtest: an input only one worker has still reaches the builder4556client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag w1only4557client # this derivation will be built:4558client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv4559client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'...4560client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag w1only, in 0.44 seconds)4561client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure4562client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.20 seconds)4563worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4564worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.24 seconds)4565worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only4566worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.04 seconds)4567client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt a >&24568client # [ 67.668569] nix-daemon[938]: accepted connection from pid 1021, user root (trusted)4569client # this derivation will be built:4570client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4571client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'...4572worker1 # [ 67.950189] nix-grpc-daemon[773]: ts=2026-09-21T12:34:00Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:421244573worker2 # [ 67.883677] nix-grpc-daemon[779]: ts=2026-09-21T12:34:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366384574client # copying 1 paths...4575client # copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' to 'grpc://lb:50051'...4576worker2 # [ 67.899988] nix-grpc-daemon[779]: ts=2026-09-21T12:34:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:366384577worker2 # [ 67.915109] nix-daemon[847]: accepted connection from pid 779, user nix-grpc-daemon (trusted)4578worker2 # [ 68.124415] nix-daemon[847]: accepted connection from pid 955, user nix-grpc-daemon (trusted)4579worker2 # [ 68.143504] nix-daemon[847]: reaped child process 958, status = succeeded4580lb # [ 68.155017] niks3-server[1040]: 2026/09/21 12:34:00 INFO Received uploads request method=POST path=/api/pending_closures4581worker2 # [ 68.165892] nix-grpc-daemon[875]: time=2026-09-21T12:34:00.800Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4582worker2 # [ 68.167328] nix-grpc-daemon[875]: time=2026-09-21T12:34:00.800Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)"4583lb # [ 68.184705] niks3-server[1040]: 2026/09/21 12:34:00 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4584lb # [ 68.195771] niks3-server[1040]: 2026/09/21 12:34:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4585worker2 # [ 68.203781] nix-grpc-daemon[875]: time=2026-09-21T12:34:00.838Z level=INFO msg="Uploading 1 narinfos"4586lb # [ 68.207179] niks3-server[1040]: 2026/09/21 12:34:00 INFO Signed narinfos id=17 count=14587lb # [ 68.213100] niks3-server[1040]: 2026/09/21 12:34:00 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls4588lb # [ 68.221906] niks3-server[1040]: 2026/09/21 12:34:00 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4589lb # [ 68.226494] niks3-server[1040]: 2026/09/21 12:34:00 INFO Completed upload id=174590worker2 # [ 68.227919] nix-grpc-daemon[875]: time=2026-09-21T12:34:00.862Z level=INFO msg="Upload complete. (297ms)"4591lb # [ 68.229195] niks3-server[1040]: 2026/09/21 12:34:00 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo4592worker2 # [ 68.236278] nix-grpc-daemon[779]: ts=2026-09-21T12:34:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36638 duration_s=0 paths=1 nar_bytes_in=1204593worker2 # [ 68.248417] nix-daemon[847]: reaped child process 953, status = succeeded4594worker1 # [ 68.324333] nix-grpc-daemon[773]: ts=2026-09-21T12:34:00Z level=debug event=want client=9 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=04595worker1 # [ 68.331106] nix-grpc-daemon[773]: ts=2026-09-21T12:34:00Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=14596worker2 # [ 68.297513] nix-grpc-daemon[779]: ts=2026-09-21T12:34:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366384597worker2 # [ 68.313820] nix-grpc-daemon[779]: ts=2026-09-21T12:34:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366384598client # copying 2 paths to 'lb:50051'...4599client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'...4600worker2 # [ 68.333490] nix-grpc-daemon[779]: ts=2026-09-21T12:34:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:366384601client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'...4602worker2 # [ 68.348437] nix-daemon[847]: accepted connection from pid 779, user nix-grpc-daemon (trusted)4603worker2 # [ 68.557720] nix-daemon[847]: accepted connection from pid 962, user nix-grpc-daemon (trusted)4604worker2 # [ 68.574112] nix-daemon[847]: reaped child process 965, status = succeeded4605worker2 # [ 68.743357] nix-daemon[847]: accepted connection from pid 967, user nix-grpc-daemon (trusted)4606worker2 # [ 68.757989] nix-daemon[847]: reaped child process 970, status = succeeded4607lb # [ 68.772302] niks3-server[1040]: 2026/09/21 12:34:01 INFO Received uploads request method=POST path=/api/pending_closures4608lb # [ 68.781093] niks3-server[1040]: 2026/09/21 12:34:01 INFO Received uploads request method=POST path=/api/pending_closures4609worker2 # [ 68.790950] nix-grpc-daemon[875]: time=2026-09-21T12:34:01.425Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4610worker2 # [ 68.792514] nix-grpc-daemon[875]: time=2026-09-21T12:34:01.425Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)"4611worker2 # [ 68.794226] nix-grpc-daemon[875]: time=2026-09-21T12:34:01.425Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)"4612lb # [ 68.818376] niks3-server[1040]: 2026/09/21 12:34:01 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst4613lb # [ 68.834667] niks3-server[1040]: 2026/09/21 12:34:01 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst4614lb # [ 68.844929] niks3-server[1040]: 2026/09/21 12:34:01 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls4615worker2 # [ 68.846841] nix-grpc-daemon[875]: time=2026-09-21T12:34:01.480Z level=INFO msg="Uploading 3 narinfos"4616lb # [ 68.850401] niks3-server[1040]: 2026/09/21 12:34:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign4617lb # [ 68.856325] niks3-server[1040]: 2026/09/21 12:34:01 INFO Signed narinfos id=18 count=14618lb # [ 68.861964] niks3-server[1040]: 2026/09/21 12:34:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign4619lb # [ 68.868797] niks3-server[1040]: 2026/09/21 12:34:01 INFO Signed narinfos id=19 count=24620lb # [ 68.870613] niks3-server[1040]: 2026/09/21 12:34:01 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls4621lb # [ 68.880115] niks3-server[1040]: 2026/09/21 12:34:01 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4622lb # [ 68.882518] niks3-server[1040]: 2026/09/21 12:34:01 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo4623lb # [ 68.887780] niks3-server[1040]: 2026/09/21 12:34:01 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete4624lb # [ 68.893426] niks3-server[1040]: 2026/09/21 12:34:01 INFO Completed upload id=184625lb # [ 68.895630] niks3-server[1040]: 2026/09/21 12:34:01 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete4626lb # [ 68.898710] niks3-server[1040]: 2026/09/21 12:34:01 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4627lb # [ 68.902001] niks3-server[1040]: 2026/09/21 12:34:01 INFO Completed upload id=194628worker2 # [ 68.903690] nix-grpc-daemon[875]: time=2026-09-21T12:34:01.538Z level=INFO msg="Upload complete. (535ms)"4629worker2 # [ 68.917217] nix-grpc-daemon[779]: ts=2026-09-21T12:34:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36638 duration_s=0 paths=2 nar_bytes_in=9844630worker2 # [ 68.932543] nix-grpc-daemon[779]: ts=2026-09-21T12:34:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366384631worker2 # [ 68.943025] nix-daemon[847]: accepted connection from pid 779, user nix-grpc-daemon (trusted)4632worker2 # [ 68.951722] nix-daemon[847]: reaped child process 960, status = succeeded4633client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051...4634worker2 # [ 68.958320] nix-daemon[847]: accepted connection from pid 779, user nix-grpc-daemon (trusted)4635client # uses-input-w1onlya> worker2: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4636client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds4637worker2 # [ 69.584494] nix-daemon[847]: reaped child process 974, status = succeeded4638worker2 # [ 69.744917] nix-daemon[847]: accepted connection from pid 990, user nix-grpc-daemon (trusted)4639worker2 # [ 69.762555] nix-daemon[847]: reaped child process 993, status = succeeded4640lb # [ 69.773896] niks3-server[1040]: 2026/09/21 12:34:02 INFO Received uploads request method=POST path=/api/pending_closures4641worker2 # [ 69.784417] nix-grpc-daemon[875]: time=2026-09-21T12:34:02.418Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4642lb # [ 69.803598] niks3-server[1040]: 2026/09/21 12:34:02 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4643worker2 # [ 69.816953] nix-grpc-daemon[875]: time=2026-09-21T12:34:02.451Z level=INFO msg="Uploading 1 narinfos"4644lb # [ 69.817374] niks3-server[1040]: 2026/09/21 12:34:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign4645lb # [ 69.828147] niks3-server[1040]: 2026/09/21 12:34:02 INFO Signed narinfos id=20 count=14646lb # [ 69.831190] niks3-server[1040]: 2026/09/21 12:34:02 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls4647worker2 # [ 69.845986] nix-grpc-daemon[875]: time=2026-09-21T12:34:02.480Z level=INFO msg="Upload complete. (275ms)"4648lb # [ 69.843710] niks3-server[1040]: 2026/09/21 12:34:02 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete4649lb # [ 69.852519] niks3-server[1040]: 2026/09/21 12:34:02 INFO Completed upload id=204650lb # [ 69.856189] niks3-server[1040]: 2026/09/21 12:34:02 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo4651client # copying 1 paths...4652worker2 # [ 69.854005] nix-grpc-daemon[779]: ts=2026-09-21T12:34:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36638 duration_s=0 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=9 outputs=14653worker2 # [ 69.869532] nix-grpc-daemon[779]: ts=2026-09-21T12:34:02Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:366384654client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'...4655worker2 # [ 69.875227] nix-daemon[847]: reaped child process 972, status = succeeded4656lb # [ 69.892592] envoy[902]: conn peer=[2001:db8:1::1]:54598 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3400 tx=1741 ms=19424657lb # [ 69.898335] envoy[902]: conn peer=[2001:db8:1::1]:45632 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=333 ms=714658lb # [ 69.903981] envoy[902]: conn peer=[2001:db8:1::1]:45640 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3860 tx=2068 ms=14704659worker2 # [ 69.907318] nix-grpc-daemon[779]: ts=2026-09-21T12:34:02Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:366384660lb # [ 69.909163] envoy[902]: conn peer=[2001:db8:1::1]:45644 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=375 ms=714661worker2 # [ 69.912243] nix-grpc-daemon[779]: ts=2026-09-21T12:34:02Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36638 duration_s=0 paths=1 nar_bytes_out=1204662lb # [ 69.914060] envoy[902]: conn peer=[2001:db8:1::1]:45660 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=438 tx=406 ms=624663lb # [ 69.918405] envoy[902]: conn peer=[2001:db8:1::1]:45646 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4350 tx=2189 ms=24524664client # [ 70.047524] nix-daemon[938]: reaped child process 1024, status = succeeded4665client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.61 seconds)4666client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt b >&24667client # [ 70.268673] nix-daemon[938]: accepted connection from pid 1042, user root (trusted)4668client # this derivation will be built:4669client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4670worker2 # [ 70.475439] nix-grpc-daemon[779]: ts=2026-09-21T12:34:03Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:366384671client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'...4672worker2 # [ 70.482920] nix-grpc-daemon[779]: ts=2026-09-21T12:34:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366384673worker2 # [ 70.489725] nix-grpc-daemon[875]: time=2026-09-21T12:34:03.123Z level=INFO msg="All 1 paths already cached"4674worker1 # [ 70.578791] nix-grpc-daemon[773]: ts=2026-09-21T12:34:03Z level=debug event=want client=10 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=04675worker1 # [ 70.579181] nix-grpc-daemon[773]: ts=2026-09-21T12:34:03Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=14676worker1 # [ 70.579354] nix-grpc-daemon[773]: ts=2026-09-21T12:34:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244677worker1 # [ 70.605583] nix-grpc-daemon[773]: ts=2026-09-21T12:34:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421244678client # copying 1 paths to 'lb:50051'...4679client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'...4680worker1 # [ 70.624892] nix-grpc-daemon[773]: ts=2026-09-21T12:34:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:421244681worker1 # [ 70.639512] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4682worker1 # [ 70.660750] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 17 ms (attempt 1/5)4683worker1 # [ 70.683626] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 103 ms (attempt 2/5)4684worker1 # [ 70.794186] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 106 ms (attempt 3/5)4685worker1 # [ 70.907246] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 399 ms (attempt 4/5)4686worker1 # [ 71.314143] nix-grpc-daemon[773]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4687worker1 # [ 71.321656] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4688worker1 # [ 71.329019] nix-grpc-daemon[773]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'...4689worker1 # [ 71.555501] nix-daemon[855]: accepted connection from pid 1213, user nix-grpc-daemon (trusted)4690worker1 # [ 71.572877] nix-daemon[855]: reaped child process 1216, status = succeeded4691worker1 # [ 71.748841] nix-daemon[855]: accepted connection from pid 1218, user nix-grpc-daemon (trusted)4692worker1 # [ 71.764895] nix-daemon[855]: reaped child process 1221, status = succeeded4693lb # [ 71.704724] niks3-server[1040]: 2026/09/21 12:34:04 INFO Received uploads request method=POST path=/api/pending_closures4694worker1 # [ 71.788240] nix-grpc-daemon[869]: time=2026-09-21T12:34:04.428Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4695worker1 # [ 71.789694] nix-grpc-daemon[869]: time=2026-09-21T12:34:04.428Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)"4696lb # [ 71.733160] niks3-server[1040]: 2026/09/21 12:34:04 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst4697worker1 # [ 71.819870] nix-grpc-daemon[869]: time=2026-09-21T12:34:04.460Z level=INFO msg="Uploading 1 narinfos"4698lb # [ 71.746849] niks3-server[1040]: 2026/09/21 12:34:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign4699lb # [ 71.759455] niks3-server[1040]: 2026/09/21 12:34:04 INFO Signed narinfos id=21 count=14700lb # [ 71.766280] niks3-server[1040]: 2026/09/21 12:34:04 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls4701lb # [ 71.772074] niks3-server[1040]: 2026/09/21 12:34:04 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete4702lb # [ 71.780390] niks3-server[1040]: 2026/09/21 12:34:04 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo4703lb # [ 71.785604] niks3-server[1040]: 2026/09/21 12:34:04 INFO Completed upload id=214704worker1 # [ 71.858797] nix-grpc-daemon[869]: time=2026-09-21T12:34:04.499Z level=INFO msg="Upload complete. (505ms)"4705worker1 # [ 71.868686] nix-grpc-daemon[773]: ts=2026-09-21T12:34:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=1 paths=1 nar_bytes_in=5684706worker1 # [ 71.878939] nix-grpc-daemon[773]: ts=2026-09-21T12:34:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:421244707worker1 # [ 71.889462] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4708worker1 # [ 71.894511] nix-daemon[855]: reaped child process 1206, status = succeeded4709worker1 # [ 71.905312] nix-daemon[855]: accepted connection from pid 773, user nix-grpc-daemon (trusted)4710client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051...4711client # uses-input-w1onlyb> node-a: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4712client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds4713worker1 # [ 73.550778] nix-daemon[855]: reaped child process 1225, status = succeeded4714worker1 # [ 73.730166] nix-daemon[855]: accepted connection from pid 1241, user nix-grpc-daemon (trusted)4715worker1 # [ 73.746538] nix-daemon[855]: reaped child process 1244, status = succeeded4716lb # [ 73.685921] niks3-server[1040]: 2026/09/21 12:34:06 INFO Received uploads request method=POST path=/api/pending_closures4717worker1 # [ 73.771364] nix-grpc-daemon[869]: time=2026-09-21T12:34:06.410Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4718lb # [ 73.724140] niks3-server[1040]: 2026/09/21 12:34:06 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls4719lb # [ 73.733847] niks3-server[1040]: 2026/09/21 12:34:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign4720worker1 # [ 73.814081] nix-grpc-daemon[869]: time=2026-09-21T12:34:06.454Z level=INFO msg="Uploading 1 narinfos"4721lb # [ 73.743500] niks3-server[1040]: 2026/09/21 12:34:06 INFO Signed narinfos id=22 count=14722lb # [ 73.750177] niks3-server[1040]: 2026/09/21 12:34:06 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4723lb # [ 73.761818] niks3-server[1040]: 2026/09/21 12:34:06 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete4724worker1 # [ 73.842348] nix-grpc-daemon[869]: time=2026-09-21T12:34:06.482Z level=INFO msg="Upload complete. (325ms)"4725lb # [ 73.771728] niks3-server[1040]: 2026/09/21 12:34:06 INFO Completed upload id=224726client # copying 1 paths...4727lb # [ 73.780300] niks3-server[1040]: 2026/09/21 12:34:06 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo4728worker1 # [ 73.857068] nix-grpc-daemon[773]: ts=2026-09-21T12:34:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=10 outputs=14729worker1 # [ 73.873034] nix-grpc-daemon[773]: ts=2026-09-21T12:34:06Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:421244730client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'...4731worker1 # [ 73.878277] nix-daemon[855]: reaped child process 1223, status = succeeded4732worker1 # [ 73.899758] nix-grpc-daemon[773]: ts=2026-09-21T12:34:06Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:421244733worker1 # [ 73.905107] nix-grpc-daemon[773]: ts=2026-09-21T12:34:06Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:42124 duration_s=0 paths=1 nar_bytes_out=1204734client # [ 73.969649] nix-daemon[938]: reaped child process 1045, status = succeeded4735client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.92 seconds)4736client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt c >&24737client # [ 74.189756] nix-daemon[938]: accepted connection from pid 1064, user root (trusted)4738client # this derivation will be built:4739client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4740worker1 # [ 74.465672] nix-grpc-daemon[773]: ts=2026-09-21T12:34:07Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:421244741client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'...4742worker1 # [ 74.473117] nix-grpc-daemon[773]: ts=2026-09-21T12:34:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:421244743worker1 # [ 74.479734] nix-grpc-daemon[869]: time=2026-09-21T12:34:07.119Z level=INFO msg="All 1 paths already cached"4744worker1 # [ 74.492154] nix-grpc-daemon[773]: ts=2026-09-21T12:34:07Z level=debug event=want client=11 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=04745worker1 # [ 74.498619] nix-grpc-daemon[773]: ts=2026-09-21T12:34:07Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=14746worker2 # [ 74.431279] nix-grpc-daemon[779]: ts=2026-09-21T12:34:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366384747worker2 # [ 74.446957] nix-grpc-daemon[779]: ts=2026-09-21T12:34:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:366384748client # copying 1 paths to 'lb:50051'...4749worker2 # [ 74.462692] nix-grpc-daemon[875]: time=2026-09-21T12:34:07.095Z level=INFO msg="All 1 paths already cached"4750client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'...4751worker2 # [ 74.468368] nix-grpc-daemon[779]: ts=2026-09-21T12:34:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:366384752worker2 # [ 74.483402] nix-daemon[847]: accepted connection from pid 779, user nix-grpc-daemon (trusted)4753worker2 # [ 74.693939] nix-daemon[847]: accepted connection from pid 997, user nix-grpc-daemon (trusted)4754worker2 # [ 74.710261] nix-daemon[847]: reaped child process 1000, status = succeeded4755worker2 # [ 74.879356] nix-daemon[847]: accepted connection from pid 1002, user nix-grpc-daemon (trusted)4756worker2 # [ 74.895141] nix-daemon[847]: reaped child process 1005, status = succeeded4757lb # [ 74.906250] niks3-server[1040]: 2026/09/21 12:34:07 INFO Received uploads request method=POST path=/api/pending_closures4758worker2 # [ 74.916993] nix-grpc-daemon[875]: time=2026-09-21T12:34:07.551Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4759worker2 # [ 74.918445] nix-grpc-daemon[875]: time=2026-09-21T12:34:07.551Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)"4760lb # [ 74.932280] niks3-server[1040]: 2026/09/21 12:34:07 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst4761worker2 # [ 74.947556] nix-grpc-daemon[875]: time=2026-09-21T12:34:07.582Z level=INFO msg="Uploading 1 narinfos"4762lb # [ 74.945879] niks3-server[1040]: 2026/09/21 12:34:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign4763lb # [ 74.950714] niks3-server[1040]: 2026/09/21 12:34:07 INFO Signed narinfos id=23 count=14764lb # [ 74.956605] niks3-server[1040]: 2026/09/21 12:34:07 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls4765lb # [ 74.967450] niks3-server[1040]: 2026/09/21 12:34:07 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete4766worker2 # [ 74.972728] nix-grpc-daemon[875]: time=2026-09-21T12:34:07.607Z level=INFO msg="Upload complete. (471ms)"4767lb # [ 74.973471] niks3-server[1040]: 2026/09/21 12:34:07 INFO Completed upload id=234768lb # [ 74.977114] niks3-server[1040]: 2026/09/21 12:34:07 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo4769worker2 # [ 74.980121] nix-grpc-daemon[779]: ts=2026-09-21T12:34:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36638 duration_s=0 paths=1 nar_bytes_in=5684770worker2 # [ 74.991891] nix-grpc-daemon[779]: ts=2026-09-21T12:34:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:366384771worker2 # [ 74.998683] nix-daemon[847]: accepted connection from pid 779, user nix-grpc-daemon (trusted)4772worker2 # [ 75.003324] nix-daemon[847]: reaped child process 995, status = succeeded4773worker2 # [ 75.013880] nix-daemon[847]: accepted connection from pid 779, user nix-grpc-daemon (trusted)4774client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051...4775client # uses-input-w1onlyc> worker2: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4776client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds4777worker2 # [ 76.248002] nix-daemon[847]: reaped child process 1009, status = succeeded4778worker2 # [ 76.426249] nix-daemon[847]: accepted connection from pid 1025, user nix-grpc-daemon (trusted)4779worker2 # [ 76.446983] nix-daemon[847]: reaped child process 1028, status = succeeded4780lb # [ 76.462567] niks3-server[1040]: 2026/09/21 12:34:09 INFO Received uploads request method=POST path=/api/pending_closures4781worker2 # [ 76.473766] nix-grpc-daemon[875]: time=2026-09-21T12:34:09.108Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4782lb # [ 76.505076] niks3-server[1040]: 2026/09/21 12:34:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign4783worker2 # [ 76.515588] nix-grpc-daemon[875]: time=2026-09-21T12:34:09.150Z level=INFO msg="Uploading 1 narinfos"4784lb # [ 76.522659] niks3-server[1040]: 2026/09/21 12:34:09 INFO Signed narinfos id=24 count=14785lb # [ 76.531115] niks3-server[1040]: 2026/09/21 12:34:09 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls4786lb # [ 76.543383] niks3-server[1040]: 2026/09/21 12:34:09 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete4787lb # [ 76.549297] niks3-server[1040]: 2026/09/21 12:34:09 INFO Completed upload id=244788worker2 # [ 76.550101] nix-grpc-daemon[875]: time=2026-09-21T12:34:09.184Z level=INFO msg="Upload complete. (324ms)"4789lb # [ 76.552304] niks3-server[1040]: 2026/09/21 12:34:09 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4790lb # [ 76.558289] niks3-server[1040]: 2026/09/21 12:34:09 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo4791worker2 # [ 76.560497] nix-grpc-daemon[779]: ts=2026-09-21T12:34:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36638 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=11 outputs=14792worker2 # [ 76.576708] nix-daemon[847]: reaped child process 1007, status = succeeded4793client # copying 1 paths...4794worker1 # [ 76.685447] nix-grpc-daemon[773]: ts=2026-09-21T12:34:09Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:421244795worker1 # [ 76.703353] nix-grpc-daemon[773]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'...4796client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'...4797worker2 # [ 76.685747] nix-grpc-daemon[779]: ts=2026-09-21T12:34:09Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:366384798worker2 # [ 76.693642] nix-grpc-daemon[779]: ts=2026-09-21T12:34:09Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36638 duration_s=0 paths=1 nar_bytes_out=1204799client # [ 76.795239] nix-daemon[938]: reaped child process 1067, status = succeeded4800client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.82 seconds)4801(finished: subtest: an input only one worker has still reaches the builder, in 10.27 seconds)4802subtest: upload whose reference the worker lacks is completed from the cache4803client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag viacache4804client # this derivation will be built:4805client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv4806client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'...4807client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag viacache, in 0.46 seconds)4808client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A referrer --argstr tag viacache4809client # this derivation will be built:4810client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv4811client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'...4812client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A referrer --argstr tag viacache, in 0.42 seconds)4813client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure4814client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.20 seconds)4815worker1: must succeed: nix-store --import < /tmp/shared/ref.closure4816worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.26 seconds)4817worker2 # [ 78.209543] systemd[1]: Stopping Nix worker-protocol over gRPC...4818worker2 # [ 78.417148] nix-grpc-daemon[779]: ts=2026-09-21T12:34:11Z level=info event=signal_exit4819worker1 # [ 79.731782] nix-grpc-daemon[773]: ts=2026-09-21T12:34:12Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:421164820worker2 # [ 79.661959] nix-grpc-daemon[779]: ts=2026-09-21T12:34:12Z level=info event=scheduler_restarting4821worker1 # [ 79.736588] nix-grpc-daemon[773]: ts=2026-09-21T12:34:12Z level=info event=worker_gone addr=192.168.1.4:500514822worker2 # [ 79.689607] nix-grpc-daemon[779]: ts=2026-09-21T12:34:12Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4823worker2 # [ 79.713700] nix-daemon[847]: reaped child process 849, status = succeeded4824lb # [ 79.891414] envoy[902]: conn peer=[2001:db8:1::1]:45686 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=754825lb # [ 79.896322] envoy[902]: conn peer=[2001:db8:1::1]:45674 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4244 tx=1830 ms=20934826lb # [ 79.900920] envoy[902]: conn peer=[2001:db8:1::1]:45702 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3753 tx=1901 ms=34104827lb # [ 79.905504] envoy[902]: conn peer=[2001:db8:1::1]:38762 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=614828lb # [ 79.909966] envoy[902]: conn peer=[2001:db8:1::1]:38774 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3765 tx=2014 ms=23164829lb # [ 79.914617] envoy[902]: conn peer=[2001:db8:1::1]:38780 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=461 tx=328 ms=364830lb # [ 79.919449] envoy[902]: conn peer=[2001:db8:1::4]:60300 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=53447 tx=2679 ms=591984831worker2 # [ 80.537421] nix-grpc-daemon[779]: E0000 00:00:1789994053.172056 824 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true4832worker2 # [ 80.544978] nix-grpc-daemon[779]: E0000 00:00:1789994053.179740 824 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true4833worker2 # [ 81.349311] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4834worker2 # [ 81.355635] systemd[1]: Stopped Nix worker-protocol over gRPC.4835worker2 # [ 81.363100] systemd[1]: nix-grpc-daemon.service: Consumed 1.857s CPU time over 1min 1.270s wall clock time, 51.3M memory peak, 54.7K incoming IP traffic, 140.5K outgoing IP traffic.4836worker2 # [ 81.376517] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.4837worker2 # [ 81.378617] systemd[1]: Closed Nix worker-protocol over gRPC.4838worker2 # [ 81.382549] systemd[1]: nix-grpc-daemon.socket: Consumed 620us CPU time over 1min 10.233s wall clock time, 256K memory peak, 60.3K incoming IP traffic, 51.5K outgoing IP traffic.4839lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy4840lb: must succeed: curl -sf localhost:9901/clusters4841lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4842lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.08 seconds)4843client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&24844worker1 # [ 81.800313] nix-grpc-daemon[773]: ts=2026-09-21T12:34:14Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:421244845client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4846client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.29 seconds)4847worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4848worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.04 seconds)4849worker2 # [ 81.861545] systemd[1]: Listening on Nix worker-protocol over gRPC.4850worker1 # [ 81.807888] nix-grpc-daemon[773]: ts=2026-09-21T12:34:14Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:421244851worker1 # [ 82.010777] systemd[1]: Stopping Nix worker-protocol over gRPC...4852worker1 # [ 82.340785] nix-grpc-daemon[773]: ts=2026-09-21T12:34:14Z level=info event=signal_exit4853worker2 # [ 82.338503] systemd[1]: Starting scheduler-token.service...4854worker2 # [ 82.445996] systemd[1]: scheduler-token.service: Deactivated successfully.4855worker2 # [ 82.452456] systemd[1]: Finished scheduler-token.service.4856worker2 # [ 82.455238] systemd[1]: scheduler-token.service: Consumed 49ms CPU time over 110ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4857worker2 # [ 82.459406] systemd[1]: Starting Nix worker-protocol over gRPC...4858worker1 # [ 82.568427] nix-grpc-daemon[773]: ts=2026-09-21T12:34:15Z level=info event=scheduler_yield4859worker1 # [ 82.575035] nix-grpc-daemon[773]: ts=2026-09-21T12:34:15Z level=info event=scheduler_restarting addr=lb:500514860lb # [ 82.503375] niks3-server[1040]: 2026/09/21 12:34:15 INFO lead: released remote=192.168.1.3:505604861worker1 # [ 82.579768] nix-grpc-daemon[773]: ts=2026-09-21T12:34:15Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:421164862worker1 # [ 82.584212] nix-grpc-daemon[773]: ts=2026-09-21T12:34:15Z level=info event=worker_gone addr=192.168.1.3:500514863worker2 # [ 82.638321] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:15Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4864worker2 # [ 82.640929] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:15Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4865worker2 # [ 82.644983] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:15Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=14866worker2 # [ 82.652066] nix-grpc-daemon[1052]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR4867worker2 # [ 82.664642] nix-grpc-daemon[1052]: E0000 00:00:1789994055.286700 1052 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4868worker2 # [ 82.669486] nix-grpc-daemon[1052]: E0000 00:00:1789994055.290449 1052 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4869lb # [ 82.668589] niks3-server[1040]: 2026/09/21 12:34:15 INFO lead: acquired remote=192.168.1.4:580724870worker2 # [ 82.672372] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:15Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4871worker2 # [ 82.674413] systemd[1]: Started Nix worker-protocol over gRPC.4872worker2 # [ 82.704144] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:15Z level=info event=scheduler_take_over4873worker1 # [ 82.774566] nix-grpc-daemon[773]: ts=2026-09-21T12:34:15Z level=info event=scheduler_connected addr=lb:500514874worker2 # [ 82.705334] nix-grpc-daemon[1052]: E0000 00:00:1789994055.312276 1093 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4875worker2 # [ 82.707676] nix-grpc-daemon[1052]: E0000 00:00:1789994055.328588 1093 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4876worker1 # [ 82.779910] nix-grpc-daemon[773]: ts=2026-09-21T12:34:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4877worker2 # [ 82.715285] nix-grpc-daemon[1052]: E0000 00:00:1789994055.328742 1093 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4878worker2 # [ 82.718347] nix-grpc-daemon[1052]: E0000 00:00:1789994055.349482 1093 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4879worker2 # [ 82.727793] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:15Z level=info event=scheduler_connected addr=lb:500514880worker2 # [ 82.729604] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4881worker1 # [ 82.982433] nix-grpc-daemon[773]: ts=2026-09-21T12:34:15Z level=info event=scheduler_connected addr=lb:500514882worker1 # [ 82.986469] nix-grpc-daemon[773]: ts=2026-09-21T12:34:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4883worker2 # [ 82.931050] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:15Z level=info event=scheduler_connected addr=lb:500514884worker2 # [ 82.935864] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4885worker1 # [ 83.183002] nix-grpc-daemon[773]: ts=2026-09-21T12:34:15Z level=info event=scheduler_connected addr=lb:500514886worker1 # [ 83.188345] nix-grpc-daemon[773]: ts=2026-09-21T12:34:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4887worker2 # [ 83.136919] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:15Z level=info event=scheduler_connected addr=lb:500514888worker2 # [ 83.141824] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4889worker1 # [ 83.388895] nix-grpc-daemon[773]: ts=2026-09-21T12:34:16Z level=info event=scheduler_connected addr=lb:500514890worker1 # [ 83.393908] nix-grpc-daemon[773]: ts=2026-09-21T12:34:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4891worker2 # [ 83.342758] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:15Z level=info event=scheduler_connected addr=lb:500514892worker2 # [ 83.347842] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4893worker1 # [ 83.573809] nix-grpc-daemon[773]: ts=2026-09-21T12:34:16Z level=info event=scheduler_restarting4894worker2 # [ 83.548833] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:16Z level=info event=scheduler_connected addr=lb:500514895worker1 # [ 83.621303] nix-grpc-daemon[773]: ts=2026-09-21T12:34:16Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4896worker2 # [ 83.553585] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4897worker1 # [ 83.640371] nix-daemon[855]: reaped child process 857, status = succeeded4898worker2 # [ 83.755956] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:16Z level=info event=scheduler_connected addr=lb:500514899worker2 # [ 83.771732] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:16Z level=info event=oidc_jwks provider=mock keys=14900worker2 # [ 83.776221] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:16Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:609084901worker2 # [ 83.781295] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:16Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=04902worker1 # [ 84.448306] nix-grpc-daemon[773]: E0000 00:00:1789994057.087420 818 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true4903worker1 # [ 84.468112] nix-grpc-daemon[773]: E0000 00:00:1789994057.108437 818 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true4904worker1 # [ 85.337540] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4905worker1 # [ 85.343906] systemd[1]: Stopped Nix worker-protocol over gRPC.4906worker1 # [ 85.348997] systemd[1]: nix-grpc-daemon.service: Consumed 3.666s CPU time over 1min 5.386s wall clock time, 66.2M memory peak, 96.7K incoming IP traffic, 202.5K outgoing IP traffic.4907worker1 # [ 85.361908] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.4908worker1 # [ 85.363948] systemd[1]: Closed Nix worker-protocol over gRPC.4909worker1 # [ 85.366720] systemd[1]: nix-grpc-daemon.socket: Consumed 690us CPU time over 1min 13.544s wall clock time, 256K memory peak, 97.4K incoming IP traffic, 80.3K outgoing IP traffic.4910lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy4911lb: must succeed: curl -sf localhost:9901/clusters4912lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4913lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.09 seconds)4914client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&24915worker2 # [ 85.650691] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:609144916worker2 # [ 85.666132] nix-daemon[847]: accepted connection from pid 1052, user nix-grpc-daemon (trusted)4917worker2 # [ 85.698692] nix-grpc-daemon[1052]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 45 ms (attempt 1/5)4918worker2 # [ 85.751236] nix-grpc-daemon[1052]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 197 ms (attempt 2/5)4919worker2 # [ 85.948855] nix-grpc-daemon[1052]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 45 ms (attempt 3/5)4920worker2 # [ 85.993766] nix-grpc-daemon[1052]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 343 ms (attempt 4/5)4921worker2 # [ 86.344990] nix-grpc-daemon[1052]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4922worker2 # [ 86.350641] nix-grpc-daemon[1052]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4923client # copying 2 paths...4924client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'...4925worker2 # [ 86.375368] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:609144926client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'...4927worker2 # [ 86.393459] nix-daemon[847]: accepted connection from pid 1052, user nix-grpc-daemon (trusted)4928worker2 # [ 86.424313] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:19Z level=info event=niks3_push_started pid=11124929worker2 # [ 86.651445] nix-grpc-daemon[1112]: time=2026-09-21T12:34:19.285Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""4930worker2 # [ 86.830683] nix-daemon[847]: accepted connection from pid 1125, user nix-grpc-daemon (trusted)4931worker2 # [ 86.847707] nix-daemon[847]: reaped child process 1128, status = succeeded4932lb # [ 86.861830] niks3-server[1040]: 2026/09/21 12:34:19 INFO Received uploads request method=POST path=/api/pending_closures4933lb # [ 86.871897] niks3-server[1040]: 2026/09/21 12:34:19 INFO Received uploads request method=POST path=/api/pending_closures4934worker2 # [ 86.883571] nix-grpc-daemon[1112]: time=2026-09-21T12:34:19.516Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4935worker2 # [ 86.885973] nix-grpc-daemon[1112]: time=2026-09-21T12:34:19.516Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)"4936worker2 # [ 86.887940] nix-grpc-daemon[1112]: time=2026-09-21T12:34:19.516Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)"4937lb # [ 86.914550] niks3-server[1040]: 2026/09/21 12:34:19 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst4938lb # [ 86.934272] niks3-server[1040]: 2026/09/21 12:34:19 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst4939lb # [ 86.947151] niks3-server[1040]: 2026/09/21 12:34:19 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls4940worker2 # [ 86.956211] nix-grpc-daemon[1112]: time=2026-09-21T12:34:19.589Z level=INFO msg="Uploading 3 narinfos"4941lb # [ 86.956429] niks3-server[1040]: 2026/09/21 12:34:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign4942lb # [ 86.960534] niks3-server[1040]: 2026/09/21 12:34:19 INFO Signed narinfos id=25 count=14943lb # [ 86.962832] niks3-server[1040]: 2026/09/21 12:34:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign4944lb # [ 86.971143] niks3-server[1040]: 2026/09/21 12:34:19 INFO Signed narinfos id=26 count=24945lb # [ 86.976697] niks3-server[1040]: 2026/09/21 12:34:19 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls4946lb # [ 86.983391] niks3-server[1040]: 2026/09/21 12:34:19 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo4947lb # [ 86.993467] niks3-server[1040]: 2026/09/21 12:34:19 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo4948lb # [ 86.999333] niks3-server[1040]: 2026/09/21 12:34:19 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete4949lb # [ 87.006879] niks3-server[1040]: 2026/09/21 12:34:19 INFO Completed upload id=254950lb # [ 87.008565] niks3-server[1040]: 2026/09/21 12:34:19 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete4951lb # [ 87.012397] niks3-server[1040]: 2026/09/21 12:34:19 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo4952lb # [ 87.015362] niks3-server[1040]: 2026/09/21 12:34:19 INFO Completed upload id=264953worker2 # [ 87.017103] nix-grpc-daemon[1112]: time=2026-09-21T12:34:19.651Z level=INFO msg="Upload complete. (361ms)"4954worker2 # [ 87.029245] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60914 duration_s=0 paths=2 nar_bytes_in=3124955worker2 # [ 87.045562] nix-daemon[847]: reaped child process 1110, status = succeeded4956client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.68 seconds)4957worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache4958worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.04 seconds)4959worker1 # [ 87.258507] systemd[1]: Listening on Nix worker-protocol over gRPC.4960worker1 # [ 87.267349] systemd[1]: Starting Nix worker-protocol over gRPC...4961worker1 # [ 87.462240] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:20Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4962worker1 # [ 87.464903] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:20Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4963worker1 # [ 87.469034] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:20Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=14964worker1 # [ 87.476056] nix-grpc-daemon[1270]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR4965worker1 # [ 87.477466] nix-grpc-daemon[1270]: E0000 00:00:1789994060.116029 1270 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4966worker1 # [ 87.494429] nix-grpc-daemon[1270]: E0000 00:00:1789994060.122420 1270 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4967worker1 # [ 87.499973] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:20Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4968worker1 # [ 87.502654] systemd[1]: Started Nix worker-protocol over gRPC.4969lb: waiting until exactly no endpoints of aarch64-linux are unhealthy4970lb: must succeed: curl -sf localhost:9901/clusters4971worker1 # [ 87.528162] nix-grpc-daemon[1270]: E0000 00:00:1789994060.145303 1309 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4972worker1 # [ 87.530521] nix-grpc-daemon[1270]: E0000 00:00:1789994060.161158 1309 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4973worker1 # [ 87.540990] nix-grpc-daemon[1270]: E0000 00:00:1789994060.161313 1309 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4974worker1 # [ 87.543805] nix-grpc-daemon[1270]: E0000 00:00:1789994060.180617 1309 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4975worker1 # [ 87.559028] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:20Z level=info event=scheduler_connected addr=lb:500514976worker2 # [ 87.488876] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:20Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:609084977worker2 # [ 87.495839] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:20Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=04978lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)4979lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.10 seconds)4980(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 10.75 seconds)4981subtest: scheduler down: the other node takes the lock and keeps it4982lb: must succeed: curl -sf localhost:9901/clusters4983lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4984lb: must succeed: curl -sf localhost:9901/clusters4985lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4986lb: must succeed: curl -sf localhost:9901/clusters4987lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4988lb: must succeed: curl -sf localhost:9901/clusters4989lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4990worker2 # [ 87.800151] systemd[1]: Stopping Nix worker-protocol over gRPC...4991worker2 # [ 88.661958] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:21Z level=info event=signal_exit4992worker2 # [ 88.871960] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:21Z level=info event=scheduler_yield4993worker1 # [ 88.948235] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:21Z level=info event=scheduler_restarting addr=lb:500514994lb # [ 88.874503] niks3-server[1040]: 2026/09/21 12:34:21 INFO lead: released remote=192.168.1.4:580724995worker2 # [ 88.878895] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:21Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:609084996worker2 # [ 88.883723] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:21Z level=info event=worker_gone addr=192.168.1.3:500514997worker2 # [ 88.891560] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:21Z level=info event=scheduler_restarting addr=lb:500514998worker2 # [ 88.896135] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:21Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:609084999worker2 # [ 88.900984] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:21Z level=info event=worker_gone addr=192.168.1.4:500515000worker1 # [ 89.148305] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:21Z level=info event=scheduler_connected addr=lb:500515001worker1 # [ 89.153474] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5002worker2 # [ 89.091021] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:21Z level=info event=scheduler_connected addr=lb:500515003worker2 # [ 89.096239] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5004worker1 # [ 89.354109] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:21Z level=info event=scheduler_connected addr=lb:500515005worker1 # [ 89.359156] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5006worker2 # [ 89.297346] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:21Z level=info event=scheduler_connected addr=lb:500515007worker2 # [ 89.302424] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5008worker1 # [ 89.559846] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:22Z level=info event=scheduler_connected addr=lb:500515009worker1 # [ 89.564944] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5010worker2 # [ 89.503646] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:22Z level=info event=scheduler_connected addr=lb:500515011worker2 # [ 89.508861] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5012worker1 # [ 89.765653] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:22Z level=info event=scheduler_connected addr=lb:500515013worker1 # [ 89.770634] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5014worker2 # [ 89.711322] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:22Z level=info event=scheduler_connected addr=lb:500515015worker2 # [ 89.715592] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5016worker2 # [ 89.877779] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:22Z level=info event=scheduler_restarting5017lb # [ 89.891070] envoy[902]: conn peer=[2001:db8:1::1]:52814 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=235018lb # [ 89.891378] envoy[902]: conn peer=[2001:db8:1::3]:57942 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=11907 tx=3300 ms=632235019worker1 # [ 89.972492] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:22Z level=info event=scheduler_connected addr=lb:500515020lb # [ 89.891557] envoy[902]: conn peer=[2001:db8:1::1]:52820 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1007 tx=305 ms=13935021worker2 # [ 89.907923] nix-daemon[847]: reaped child process 1104, status = succeeded5022worker1 # [ 89.980225] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5023worker2 # [ 89.930995] nix-grpc-daemon[1052]: ts=2026-09-21T12:34:22Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5024worker1 # [ 90.181223] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:22Z level=info event=scheduler_connected addr=lb:500515025worker1 # [ 90.186220] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5026worker1 # [ 90.386991] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:23Z level=info event=scheduler_connected addr=lb:500515027worker1 # [ 90.392156] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5028worker1 # [ 90.592900] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:23Z level=info event=scheduler_connected addr=lb:500515029worker1 # [ 90.597887] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5030worker2 # [ 90.700744] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5031worker2 # [ 90.703661] systemd[1]: Stopped Nix worker-protocol over gRPC.5032worker2 # [ 90.707461] systemd[1]: nix-grpc-daemon.service: Consumed 494ms CPU time over 8.237s wall clock time, 33.6M memory peak, 20.1K incoming IP traffic, 30.4K outgoing IP traffic.5033worker2 # [ 90.719753] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5034worker2 # [ 90.725832] systemd[1]: Closed Nix worker-protocol over gRPC.5035worker1 # [ 90.798614] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:23Z level=info event=scheduler_connected addr=lb:500515036worker2 # [ 90.729002] systemd[1]: nix-grpc-daemon.socket: Consumed 695us CPU time over 8.865s wall clock time, 256K memory peak, 20.1K incoming IP traffic, 15K outgoing IP traffic.5037worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5038worker1 # [ 90.803744] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5039worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.09 seconds)5040worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5041worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5042worker1 # [ 91.006391] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:23Z level=info event=scheduler_connected addr=lb:500515043worker1 # [ 91.010599] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5044worker1 # [ 91.206847] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:23Z level=info event=scheduler_connected addr=lb:500515045worker1 # [ 91.211799] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5046worker1 # [ 91.412633] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:24Z level=info event=scheduler_connected addr=lb:500515047worker1 # [ 91.417631] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5048worker1 # [ 91.618357] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:24Z level=info event=scheduler_connected addr=lb:500515049worker1 # [ 91.623477] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5050worker1 # [ 91.824246] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:24Z level=info event=scheduler_connected addr=lb:500515051worker1 # [ 91.829218] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5052worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5053worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5054worker1 # [ 92.030401] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:24Z level=info event=scheduler_connected addr=lb:500515055worker1 # [ 92.033475] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5056worker1 # [ 92.234404] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:24Z level=info event=scheduler_connected addr=lb:500515057worker1 # [ 92.239593] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5058worker1 # [ 92.440419] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:25Z level=info event=scheduler_connected addr=lb:500515059worker1 # [ 92.445230] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5060worker1 # [ 92.500965] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:25Z level=info event=scheduler_take_over5061lb # [ 92.428760] niks3-server[1040]: 2026/09/21 12:34:25 INFO lead: acquired remote=192.168.1.3:490005062worker1 # [ 92.645844] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:25Z level=info event=scheduler_connected addr=lb:500515063worker1 # [ 92.655004] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:25Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:419725064worker1 # [ 92.660531] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:25Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05065worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5066worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5067client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby >&25068worker1 # [ 93.365258] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:419745069worker1 # [ 93.380433] nix-daemon[855]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5070client # don't know how to build these paths:5071client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5072worker1 # [ 93.405304] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419745073worker1 # [ 93.426305] nix-grpc-daemon[1270]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 99 ms (attempt 1/5)5074worker1 # [ 93.532836] nix-grpc-daemon[1270]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 111 ms (attempt 2/5)5075worker1 # [ 93.651490] nix-grpc-daemon[1270]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 202 ms (attempt 3/5)5076worker1 # [ 93.861156] nix-grpc-daemon[1270]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 619 ms (attempt 4/5)5077worker1 # [ 94.492857] nix-grpc-daemon[1270]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5078worker1 # [ 94.497226] nix-grpc-daemon[1270]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5079worker1 # [ 94.522856] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:27Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=05080worker1 # [ 94.530297] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:27Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.3:50051 clients=15081worker1 # [ 94.572185] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419745082worker1 # [ 94.588126] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419745083client # copying 1 paths to 'lb:50051'...5084worker1 # [ 94.602476] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:419745085client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'...5086worker1 # [ 94.621161] nix-daemon[855]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5087worker1 # [ 94.649309] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:27Z level=info event=niks3_push_started pid=13475088worker1 # [ 94.865236] nix-grpc-daemon[1347]: time=2026-09-21T12:34:27.504Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""5089worker1 # [ 95.040531] nix-daemon[855]: accepted connection from pid 1360, user nix-grpc-daemon (trusted)5090worker1 # [ 95.057832] nix-daemon[855]: reaped child process 1363, status = succeeded5091worker1 # [ 95.231734] nix-daemon[855]: accepted connection from pid 1365, user nix-grpc-daemon (trusted)5092worker1 # [ 95.250720] nix-daemon[855]: reaped child process 1368, status = succeeded5093lb # [ 95.190399] niks3-server[1040]: 2026/09/21 12:34:27 INFO Received uploads request method=POST path=/api/pending_closures5094worker1 # [ 95.275821] nix-grpc-daemon[1347]: time=2026-09-21T12:34:27.915Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5095worker1 # [ 95.277595] nix-grpc-daemon[1347]: time=2026-09-21T12:34:27.915Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)"5096lb # [ 95.230953] niks3-server[1040]: 2026/09/21 12:34:27 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst5097worker1 # [ 95.316234] nix-grpc-daemon[1347]: time=2026-09-21T12:34:27.956Z level=INFO msg="Uploading 1 narinfos"5098lb # [ 95.246828] niks3-server[1040]: 2026/09/21 12:34:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5099lb # [ 95.256463] niks3-server[1040]: 2026/09/21 12:34:27 INFO Signed narinfos id=27 count=15100lb # [ 95.258817] niks3-server[1040]: 2026/09/21 12:34:27 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls5101lb # [ 95.268224] niks3-server[1040]: 2026/09/21 12:34:27 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5102lb # [ 95.276880] niks3-server[1040]: 2026/09/21 12:34:27 INFO Completed upload id=275103worker1 # [ 95.350000] nix-grpc-daemon[1347]: time=2026-09-21T12:34:27.990Z level=INFO msg="Upload complete. (481ms)"5104worker1 # [ 95.356413] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41974 duration_s=0 paths=1 nar_bytes_in=5205105worker1 # [ 95.370136] nix-grpc-daemon[1347]: time=2026-09-21T12:34:28.004Z level=WARN msg="Failed to register uploaded object" key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo error="server returned 404: object is not pending upload\n"5106worker1 # [ 95.376491] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419745107worker1 # [ 95.384099] nix-daemon[855]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5108worker1 # [ 95.386902] nix-daemon[855]: reaped child process 1345, status = succeeded5109client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051...5110client # farm-a> node-a: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5111worker1 # [ 95.399224] nix-daemon[855]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5112client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5113client # farm-a> LOG-farm-a-on-standby5114worker1 # [ 96.844067] nix-daemon[855]: reaped child process 1372, status = succeeded5115worker1 # [ 97.030898] nix-daemon[855]: accepted connection from pid 1388, user nix-grpc-daemon (trusted)5116worker1 # [ 97.048218] nix-daemon[855]: reaped child process 1391, status = succeeded5117lb # [ 96.987405] niks3-server[1040]: 2026/09/21 12:34:29 INFO Received uploads request method=POST path=/api/pending_closures5118worker1 # [ 97.076419] nix-grpc-daemon[1347]: time=2026-09-21T12:34:29.715Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5119worker1 # [ 97.077960] nix-grpc-daemon[1347]: time=2026-09-21T12:34:29.715Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)"5120lb # [ 97.046972] niks3-server[1040]: 2026/09/21 12:34:29 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5121lb # [ 97.055141] niks3-server[1040]: 2026/09/21 12:34:29 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5122worker1 # [ 97.135143] nix-grpc-daemon[1347]: time=2026-09-21T12:34:29.774Z level=INFO msg="Uploading 1 narinfos"5123lb # [ 97.065347] niks3-server[1040]: 2026/09/21 12:34:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5124lb # [ 97.072293] niks3-server[1040]: 2026/09/21 12:34:29 INFO Signed narinfos id=28 count=15125lb # [ 97.074595] niks3-server[1040]: 2026/09/21 12:34:29 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls5126lb # [ 97.081364] niks3-server[1040]: 2026/09/21 12:34:29 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5127lb # [ 97.087721] niks3-server[1040]: 2026/09/21 12:34:29 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo5128lb # [ 97.092878] niks3-server[1040]: 2026/09/21 12:34:29 INFO Completed upload id=285129worker1 # [ 97.166060] nix-grpc-daemon[1347]: time=2026-09-21T12:34:29.806Z level=INFO msg="Upload complete. (355ms)"5130worker1 # [ 97.187279] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41974 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=15131worker1 # [ 97.202250] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:29Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=05132worker1 # [ 97.206631] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:29Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.3:50051 clients=15133worker1 # [ 97.211260] nix-daemon[855]: reaped child process 1370, status = succeeded5134worker1 # [ 97.233262] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419745135worker1 # [ 97.248668] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419745136worker1 # [ 97.263000] nix-grpc-daemon[1347]: time=2026-09-21T12:34:29.903Z level=INFO msg="All 1 paths already cached"5137client # copying 1 paths to 'lb:50051'...5138client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'...5139worker1 # [ 97.271786] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:419745140worker1 # [ 97.290559] nix-daemon[855]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5141worker1 # [ 97.491816] nix-daemon[855]: accepted connection from pid 1397, user nix-grpc-daemon (trusted)5142worker1 # [ 97.509550] nix-daemon[855]: reaped child process 1400, status = succeeded5143worker1 # [ 97.684108] nix-daemon[855]: accepted connection from pid 1402, user nix-grpc-daemon (trusted)5144worker1 # [ 97.700714] nix-daemon[855]: reaped child process 1405, status = succeeded5145lb # [ 97.638824] niks3-server[1040]: 2026/09/21 12:34:30 INFO Received uploads request method=POST path=/api/pending_closures5146worker1 # [ 97.723507] nix-grpc-daemon[1347]: time=2026-09-21T12:34:30.363Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5147worker1 # [ 97.725109] nix-grpc-daemon[1347]: time=2026-09-21T12:34:30.363Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)"5148lb # [ 97.669493] niks3-server[1040]: 2026/09/21 12:34:30 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst5149lb # [ 97.682607] niks3-server[1040]: 2026/09/21 12:34:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5150worker1 # [ 97.762253] nix-grpc-daemon[1347]: time=2026-09-21T12:34:30.402Z level=INFO msg="Uploading 1 narinfos"5151lb # [ 97.695704] niks3-server[1040]: 2026/09/21 12:34:30 INFO Signed narinfos id=29 count=15152lb # [ 97.700952] niks3-server[1040]: 2026/09/21 12:34:30 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls5153worker1 # [ 97.782307] nix-grpc-daemon[1347]: time=2026-09-21T12:34:30.422Z level=INFO msg="Upload complete. (475ms)"5154lb # [ 97.710346] niks3-server[1040]: 2026/09/21 12:34:30 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5155lb # [ 97.716326] niks3-server[1040]: 2026/09/21 12:34:30 INFO Completed upload id=295156worker1 # [ 97.788242] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41974 duration_s=0 paths=1 nar_bytes_in=6405157lb # [ 97.719364] niks3-server[1040]: 2026/09/21 12:34:30 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo5158worker1 # [ 97.800308] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419745159worker1 # [ 97.812190] nix-daemon[855]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5160worker1 # [ 97.814870] nix-daemon[855]: reaped child process 1395, status = succeeded5161client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051...5162worker1 # [ 97.825807] nix-daemon[855]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5163client # farm-b> node-a: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5164client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5165client # farm-b> LOG-farm-b-on-standby5166worker1 # [ 98.870701] nix-daemon[855]: reaped child process 1409, status = succeeded5167worker1 # [ 99.049437] nix-daemon[855]: accepted connection from pid 1425, user nix-grpc-daemon (trusted)5168worker1 # [ 99.068231] nix-daemon[855]: reaped child process 1428, status = succeeded5169lb # [ 99.007545] niks3-server[1040]: 2026/09/21 12:34:31 INFO Received uploads request method=POST path=/api/pending_closures5170worker1 # [ 99.091697] nix-grpc-daemon[1347]: time=2026-09-21T12:34:31.731Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5171worker1 # [ 99.094003] nix-grpc-daemon[1347]: time=2026-09-21T12:34:31.733Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)"5172lb # [ 99.045371] niks3-server[1040]: 2026/09/21 12:34:31 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst5173lb # [ 99.058351] niks3-server[1040]: 2026/09/21 12:34:31 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5174lb # [ 99.068290] niks3-server[1040]: 2026/09/21 12:34:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5175worker1 # [ 99.146974] nix-grpc-daemon[1347]: time=2026-09-21T12:34:31.787Z level=INFO msg="Uploading 1 narinfos"5176lb # [ 99.076811] niks3-server[1040]: 2026/09/21 12:34:31 INFO Signed narinfos id=30 count=15177lb # [ 99.079511] niks3-server[1040]: 2026/09/21 12:34:31 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls5178lb # [ 99.090965] niks3-server[1040]: 2026/09/21 12:34:31 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5179lb # [ 99.098588] niks3-server[1040]: 2026/09/21 12:34:31 INFO Completed upload id=305180worker1 # [ 99.171665] nix-grpc-daemon[1347]: time=2026-09-21T12:34:31.812Z level=INFO msg="Upload complete. (333ms)"5181lb # [ 99.101913] niks3-server[1040]: 2026/09/21 12:34:31 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo5182worker1 # [ 99.185996] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41974 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=15183worker1 # [ 99.205798] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:31Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=05184worker1 # [ 99.210342] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:31Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.3:50051 clients=15185worker1 # [ 99.215079] nix-daemon[855]: reaped child process 1407, status = succeeded5186worker1 # [ 99.245449] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419745187worker1 # [ 99.261220] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419745188worker1 # [ 99.275848] nix-grpc-daemon[1347]: time=2026-09-21T12:34:31.916Z level=INFO msg="All 2 paths already cached"5189client # copying 1 paths to 'lb:50051'...5190client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'...5191worker1 # [ 99.284598] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:419745192worker1 # [ 99.297935] nix-daemon[855]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5193worker1 # [ 99.494238] nix-daemon[855]: accepted connection from pid 1434, user nix-grpc-daemon (trusted)5194worker1 # [ 99.511133] nix-daemon[855]: reaped child process 1437, status = succeeded5195worker1 # [ 99.683873] nix-daemon[855]: accepted connection from pid 1439, user nix-grpc-daemon (trusted)5196worker1 # [ 99.700300] nix-daemon[855]: reaped child process 1442, status = succeeded5197lb # [ 99.638747] niks3-server[1040]: 2026/09/21 12:34:32 INFO Received uploads request method=POST path=/api/pending_closures5198worker1 # [ 99.721380] nix-grpc-daemon[1347]: time=2026-09-21T12:34:32.361Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5199worker1 # [ 99.722836] nix-grpc-daemon[1347]: time=2026-09-21T12:34:32.361Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)"5200lb # [ 99.668147] niks3-server[1040]: 2026/09/21 12:34:32 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst5201worker1 # [ 99.751594] nix-grpc-daemon[1347]: time=2026-09-21T12:34:32.391Z level=INFO msg="Uploading 1 narinfos"5202lb # [ 99.682768] niks3-server[1040]: 2026/09/21 12:34:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5203lb # [ 99.693567] niks3-server[1040]: 2026/09/21 12:34:32 INFO Signed narinfos id=31 count=15204lb # [ 99.701752] niks3-server[1040]: 2026/09/21 12:34:32 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls5205lb # [ 99.705384] niks3-server[1040]: 2026/09/21 12:34:32 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5206worker1 # [ 99.782409] nix-grpc-daemon[1347]: time=2026-09-21T12:34:32.422Z level=INFO msg="Upload complete. (468ms)"5207lb # [ 99.710078] niks3-server[1040]: 2026/09/21 12:34:32 INFO Completed upload id=315208lb # [ 99.713578] niks3-server[1040]: 2026/09/21 12:34:32 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo5209worker1 # [ 99.788209] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41974 duration_s=0 paths=1 nar_bytes_in=7685210worker1 # [ 99.801448] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419745211worker1 # [ 99.807155] nix-daemon[855]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5212worker1 # [ 99.811968] nix-daemon[855]: reaped child process 1432, status = succeeded5213client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051...5214client # farm-top> node-a: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5215worker1 # [ 99.824755] nix-daemon[855]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5216lb # [ 99.892605] envoy[902]: conn peer=[2001:db8:1::4]:42940 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=7963 tx=535 ms=71945217client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5218client # farm-top> LOG-farm-top-on-standby5219worker1 # [ 101.071605] nix-daemon[855]: reaped child process 1446, status = succeeded5220worker1 # [ 101.250477] nix-daemon[855]: accepted connection from pid 1462, user nix-grpc-daemon (trusted)5221worker1 # [ 101.268270] nix-daemon[855]: reaped child process 1465, status = succeeded5222lb # [ 101.211570] niks3-server[1040]: 2026/09/21 12:34:33 INFO Received uploads request method=POST path=/api/pending_closures5223worker1 # [ 101.296422] nix-grpc-daemon[1347]: time=2026-09-21T12:34:33.935Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5224worker1 # [ 101.298004] nix-grpc-daemon[1347]: time=2026-09-21T12:34:33.935Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)"5225lb # [ 101.247544] niks3-server[1040]: 2026/09/21 12:34:33 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5226lb # [ 101.258935] niks3-server[1040]: 2026/09/21 12:34:33 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst5227lb # [ 101.270068] niks3-server[1040]: 2026/09/21 12:34:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5228worker1 # [ 101.348114] nix-grpc-daemon[1347]: time=2026-09-21T12:34:33.988Z level=INFO msg="Uploading 1 narinfos"5229lb # [ 101.277981] niks3-server[1040]: 2026/09/21 12:34:33 INFO Signed narinfos id=32 count=15230lb # [ 101.282894] niks3-server[1040]: 2026/09/21 12:34:33 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls5231lb # [ 101.290410] niks3-server[1040]: 2026/09/21 12:34:33 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5232lb # [ 101.299271] niks3-server[1040]: 2026/09/21 12:34:33 INFO Completed upload id=325233worker1 # [ 101.372050] nix-grpc-daemon[1347]: time=2026-09-21T12:34:34.012Z level=INFO msg="Upload complete. (336ms)"5234lb # [ 101.302037] niks3-server[1040]: 2026/09/21 12:34:33 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo5235worker1 # [ 101.385060] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41974 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=15236worker1 # [ 101.400263] nix-daemon[855]: reaped child process 1444, status = succeeded5237client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby >&2, in 8.33 seconds)5238client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby outPath5239client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby outPath, in 0.23 seconds)5240worker2 # [ 101.678473] systemd[1]: Listening on Nix worker-protocol over gRPC.5241worker2 # [ 101.687391] systemd[1]: Starting scheduler-token.service...5242worker2 # [ 101.777397] systemd[1]: scheduler-token.service: Deactivated successfully.5243worker2 # [ 101.780611] systemd[1]: Finished scheduler-token.service.5244worker2 # [ 101.781803] systemd[1]: scheduler-token.service: Consumed 37ms CPU time over 93ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5245worker2 # [ 101.790271] systemd[1]: Starting Nix worker-protocol over gRPC...5246worker2 # [ 101.966613] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:34Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5247worker2 # [ 101.969285] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:34Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5248worker2 # [ 101.973337] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:34Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15249worker2 # [ 101.980055] nix-grpc-daemon[1152]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5250worker2 # [ 101.981520] nix-grpc-daemon[1152]: E0000 00:00:1789994074.614650 1152 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5251worker2 # [ 101.994797] nix-grpc-daemon[1152]: E0000 00:00:1789994074.620981 1152 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5252worker2 # [ 102.003740] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:34Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5253lb: must succeed: curl -sf localhost:9901/clusters5254worker2 # [ 102.007625] systemd[1]: Started Nix worker-protocol over gRPC.5255worker2 # [ 102.012107] nix-grpc-daemon[1152]: E0000 00:00:1789994074.645215 1194 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5256worker2 # [ 102.040052] nix-grpc-daemon[1152]: E0000 00:00:1789994074.662745 1194 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5257worker2 # [ 102.045702] nix-grpc-daemon[1152]: E0000 00:00:1789994074.662915 1194 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5258worker2 # [ 102.048605] nix-grpc-daemon[1152]: E0000 00:00:1789994074.679809 1194 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5259worker2 # [ 102.057824] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:34Z level=info event=scheduler_connected addr=lb:500515260worker1 # [ 102.140205] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:34Z level=info event=oidc_jwks provider=mock keys=15261worker1 # [ 102.140611] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:34Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:419725262worker1 # [ 102.150182] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:34Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05263lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5264worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5265worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5266lb: must succeed: curl -sf localhost:9901/clusters5267lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5268lb: must succeed: curl -sf localhost:9901/clusters5269lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5270worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5271worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5272(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 14.83 seconds)5273subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5274client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs5275client # [ 102.452357] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs.5276client # Running as unit: rs.service; invocation ID: 0be367d4dcd24a51bb5287cfee95147f5277client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs, in 0.09 seconds)5278worker2 # [ 102.715316] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:553525279worker2 # [ 102.730538] nix-daemon[847]: accepted connection from pid 1152, user nix-grpc-daemon (trusted)5280client # [ 102.754993] nix[1168]: don't know how to build these paths:5281client # [ 102.758017] nix[1168]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5282worker1 # [ 102.834231] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:35Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=05283worker1 # [ 102.834539] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:35Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.3:50051 clients=15284worker1 # [ 102.878120] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419745285worker1 # [ 102.893735] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419745286client # [ 102.843821] nix[1168]: copying 1 paths to 'lb:50051'...5287client # [ 102.846474] nix[1168]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'...5288worker1 # [ 102.908384] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:419745289worker1 # [ 102.925247] nix-daemon[855]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5290worker1 # [ 103.135902] nix-daemon[855]: accepted connection from pid 1478, user nix-grpc-daemon (trusted)5291worker1 # [ 103.156132] nix-daemon[855]: reaped child process 1481, status = succeeded5292worker1 # [ 103.329297] nix-daemon[855]: accepted connection from pid 1483, user nix-grpc-daemon (trusted)5293worker1 # [ 103.346357] nix-daemon[855]: reaped child process 1486, status = succeeded5294lb # [ 103.285710] niks3-server[1040]: 2026/09/21 12:34:35 INFO Received uploads request method=POST path=/api/pending_closures5295worker1 # [ 103.368384] nix-grpc-daemon[1347]: time=2026-09-21T12:34:36.008Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5296worker1 # [ 103.370245] nix-grpc-daemon[1347]: time=2026-09-21T12:34:36.008Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)"5297lb # [ 103.314461] niks3-server[1040]: 2026/09/21 12:34:35 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst5298worker1 # [ 103.400924] nix-grpc-daemon[1347]: time=2026-09-21T12:34:36.041Z level=INFO msg="Uploading 1 narinfos"5299lb # [ 103.331271] niks3-server[1040]: 2026/09/21 12:34:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5300lb # [ 103.340938] niks3-server[1040]: 2026/09/21 12:34:35 INFO Signed narinfos id=33 count=15301lb # [ 103.345978] niks3-server[1040]: 2026/09/21 12:34:35 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls5302lb # [ 103.356491] niks3-server[1040]: 2026/09/21 12:34:35 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5303lb # [ 103.361695] niks3-server[1040]: 2026/09/21 12:34:36 INFO Completed upload id=335304worker1 # [ 103.434773] nix-grpc-daemon[1347]: time=2026-09-21T12:34:36.075Z level=INFO msg="Upload complete. (492ms)"5305lb # [ 103.364705] niks3-server[1040]: 2026/09/21 12:34:36 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo5306worker1 # [ 103.444226] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41974 duration_s=0 paths=1 nar_bytes_in=4165307worker1 # [ 103.454166] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419745308worker1 # [ 103.461155] nix-daemon[855]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5309worker1 # [ 103.466091] nix-daemon[855]: reaped child process 1476, status = succeeded5310client # [ 103.411324] nix[1168]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051...5311worker1 # [ 103.479791] nix-daemon[855]: accepted connection from pid 1270, user nix-grpc-daemon (trusted)5312lb: must succeed: curl -sf localhost:9901/clusters5313lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5314lb: must succeed: curl -sf localhost:9901/clusters5315lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5316lb: must succeed: curl -sf localhost:9901/clusters5317lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5318lb: must succeed: curl -sf localhost:9901/clusters5319lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5320worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5321worker2 # [ 105.394017] systemd-journald[268]: Received client request to rotate journal, rotating.5322worker2 # [ 105.415426] systemd-journald[268]: Vacuuming done, freed 0B of archived journals from /var/log/journal/a88252f7f1ab4e67aed68b68d54c340b.5323worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.10 seconds)5324worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service5325worker1 # [ 105.552889] systemd[1]: Stopping Nix worker-protocol over gRPC...5326worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.05 seconds)5327client: must succeed: sleep 5.05328worker1 # [ 106.495114] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:39Z level=info event=draining5329lb # [ 109.891637] envoy[902]: conn peer=[2001:db8:1::1]:44684 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9666 tx=4155 ms=80365330client: (finished: must succeed: sleep 5.0, in 5.06 seconds)5331worker1 # [ 110.909123] nix-daemon[855]: reaped child process 1490, status = succeeded5332worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5333worker1 # [ 111.075109] nix-daemon[855]: accepted connection from pid 1531, user nix-grpc-daemon (trusted)5334worker1 # [ 111.096233] nix-daemon[855]: reaped child process 1534, status = succeeded5335lb # [ 111.035687] niks3-server[1040]: 2026/09/21 12:34:43 INFO Received uploads request method=POST path=/api/pending_closures5336worker1 # [ 111.122488] nix-grpc-daemon[1347]: time=2026-09-21T12:34:43.760Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5337worker1 # [ 111.124176] nix-grpc-daemon[1347]: time=2026-09-21T12:34:43.760Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)"5338lb # [ 111.077408] niks3-server[1040]: 2026/09/21 12:34:43 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5339worker1 # [ 111.166395] nix-grpc-daemon[1347]: time=2026-09-21T12:34:43.806Z level=INFO msg="Uploading 1 narinfos"5340lb # [ 111.094094] niks3-server[1040]: 2026/09/21 12:34:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5341lb # [ 111.104882] niks3-server[1040]: 2026/09/21 12:34:43 INFO Signed narinfos id=34 count=15342lb # [ 111.107438] niks3-server[1040]: 2026/09/21 12:34:43 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5343lb # [ 111.115970] niks3-server[1040]: 2026/09/21 12:34:43 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls5344lb # [ 111.123881] niks3-server[1040]: 2026/09/21 12:34:43 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5345lb # [ 111.129571] niks3-server[1040]: 2026/09/21 12:34:43 INFO Completed upload id=345346worker1 # [ 111.202595] nix-grpc-daemon[1347]: time=2026-09-21T12:34:43.843Z level=INFO msg="Upload complete. (308ms)"5347lb # [ 111.132204] niks3-server[1040]: 2026/09/21 12:34:43 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo5348worker1 # [ 111.214426] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41974 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=15349worker1 # [ 111.230231] nix-daemon[855]: reaped child process 1488, status = succeeded5350client # [ 111.191010] systemd[1]: rs.service: Deactivated successfully.5351client # [ 111.195324] systemd[1]: rs.service: Consumed 187ms CPU time over 8.742s wall clock time, 41.2M memory peak, 7K incoming IP traffic, 8.4K outgoing IP traffic.5352worker1 # [ 111.495558] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:44Z level=info event=signal_exit5353worker1 # [ 111.523230] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:44Z level=info event=scheduler_yield5354lb # [ 111.452212] niks3-server[1040]: 2026/09/21 12:34:44 INFO lead: released remote=192.168.1.3:490005355worker1 # [ 111.530793] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:44Z level=info event=scheduler_restarting addr=lb:500515356worker2 # [ 111.461166] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:44Z level=info event=scheduler_restarting addr=lb:500515357worker1 # [ 111.536484] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:44Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:419725358worker1 # [ 111.542129] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:44Z level=info event=worker_gone addr=192.168.1.4:500515359worker1 # [ 111.547590] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:44Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:419725360worker1 # [ 111.552326] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:44Z level=info event=worker_gone addr=192.168.1.3:500515361worker1 # [ 111.731794] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:44Z level=info event=scheduler_connected addr=lb:500515362worker2 # [ 111.661129] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:44Z level=info event=scheduler_connected addr=lb:500515363worker1 # [ 111.737179] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5364worker2 # [ 111.666442] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5365worker1 # [ 111.937885] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:44Z level=info event=scheduler_connected addr=lb:500515366worker2 # [ 111.867400] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:44Z level=info event=scheduler_connected addr=lb:500515367worker1 # [ 111.943127] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5368worker2 # [ 111.872664] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5369worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 1.13 seconds)5370client: waiting for success: ! systemctl is-active rs5371lb # [ 112.006114] niks3-server[1040]: 2026/09/21 12:34:44 INFO lead: acquired remote=192.168.1.4:444885372worker2 # [ 112.011731] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:44Z level=info event=scheduler_take_over5373client: (finished: waiting for success: ! systemctl is-active rs, in 0.07 seconds)5374client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5375worker1 # [ 112.143928] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:44Z level=info event=scheduler_connected addr=lb:500515376worker2 # [ 112.073839] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:44Z level=info event=scheduler_connected addr=lb:500515377worker1 # [ 112.149285] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5378worker2 # [ 112.079127] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5379client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds)5380client: must fail: journalctl -u rs | grep -q 'reconnecting'5381client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.04 seconds)5382lb: must succeed: curl -sf localhost:9901/clusters5383lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5384worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5385worker1 # [ 112.351901] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:44Z level=info event=scheduler_connected addr=lb:500515386worker2 # [ 112.281960] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:44Z level=info event=scheduler_connected addr=lb:500515387worker1 # [ 112.356317] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5388worker2 # [ 112.287418] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5389worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.10 seconds)5390worker1 # [ 112.529051] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:45Z level=info event=scheduler_restarting5391worker1 # [ 112.543474] nix-grpc-daemon[1270]: E0000 00:00:1789994085.183821 1315 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5392worker2 # [ 112.484935] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:45Z level=info event=scheduler_connected addr=lb:500515393worker2 # [ 112.489972] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5394worker1 # [ 112.576681] nix-grpc-daemon[1270]: E0000 00:00:1789994085.201342 1315 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true5395worker1 # [ 112.595955] nix-grpc-daemon[1270]: ts=2026-09-21T12:34:45Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5396worker1 # [ 112.600557] nix-daemon[855]: reaped child process 1337, status = succeeded5397worker2 # [ 112.691095] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:45Z level=info event=scheduler_connected addr=lb:500515398worker2 # [ 112.696707] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5399worker2 # [ 112.897824] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:45Z level=info event=scheduler_connected addr=lb:500515400worker2 # [ 112.903269] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5401worker2 # [ 113.104182] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:45Z level=info event=scheduler_connected addr=lb:500515402worker2 # [ 113.120148] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:45Z level=info event=oidc_jwks provider=mock keys=15403worker2 # [ 113.125252] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:45Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:549045404worker2 # [ 113.130605] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:45Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05405lb: must succeed: curl -sf localhost:9901/clusters5406lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5407worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5408worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5409worker1 # [ 114.288733] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5410worker1 # [ 114.291758] systemd[1]: Stopped Nix worker-protocol over gRPC.5411worker1 # [ 114.298845] systemd[1]: nix-grpc-daemon.service: Consumed 1.813s CPU time over 27.019s wall clock time, 37.2M memory peak, 49.9K incoming IP traffic, 93.9K outgoing IP traffic.5412worker1 # [ 114.314984] systemd[1]: Starting Nix worker-protocol over gRPC...5413worker1 # [ 114.509396] nix-grpc-daemon[1541]: ts=2026-09-21T12:34:47Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5414worker1 # [ 114.513006] nix-grpc-daemon[1541]: ts=2026-09-21T12:34:47Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5415worker1 # [ 114.515390] nix-grpc-daemon[1541]: ts=2026-09-21T12:34:47Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15416worker1 # [ 114.523946] nix-grpc-daemon[1541]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5417worker1 # [ 114.529307] nix-grpc-daemon[1541]: E0000 00:00:1789994087.162357 1541 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5418worker1 # [ 114.532921] nix-grpc-daemon[1541]: E0000 00:00:1789994087.164371 1541 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5419worker1 # [ 114.535892] nix-grpc-daemon[1541]: ts=2026-09-21T12:34:47Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5420worker1 # [ 114.539121] systemd[1]: Started Nix worker-protocol over gRPC.5421worker1 # [ 114.544066] nix-grpc-daemon[1541]: E0000 00:00:1789994087.183635 1580 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5422lb: must succeed: curl -sf localhost:9901/clusters5423worker1 # [ 114.572048] nix-grpc-daemon[1541]: E0000 00:00:1789994087.201334 1580 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5424worker1 # [ 114.574428] nix-grpc-daemon[1541]: E0000 00:00:1789994087.201510 1580 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5425worker1 # [ 114.581283] nix-grpc-daemon[1541]: E0000 00:00:1789994087.220868 1580 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5426worker1 # [ 114.593863] nix-grpc-daemon[1541]: ts=2026-09-21T12:34:47Z level=info event=scheduler_connected addr=lb:500515427worker2 # [ 114.525846] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:47Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:549045428worker2 # [ 114.526173] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:47Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05429lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5430worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5431worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5432(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.26 seconds)5433subtest: scheduler restart mid-build: build finishes, no double build5434client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid5435client # [ 114.715760] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid.5436client # Running as unit: mid.service; invocation ID: b8096baa586b4e9db4172765c09430a55437client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid, in 0.09 seconds)5438worker2 # [ 114.969809] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:47Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:553525439client # [ 114.985344] nix[1204]: don't know how to build these paths:5440client # [ 114.986361] nix[1204]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5441worker2 # [ 114.989945] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:47Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05442worker2 # [ 114.997692] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:47Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=15443worker2 # [ 115.037071] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:553525444worker2 # [ 115.053900] nix-grpc-daemon[1152]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 10 ms (attempt 1/5)5445worker2 # [ 115.071529] nix-grpc-daemon[1152]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 106 ms (attempt 2/5)5446worker2 # [ 115.185561] nix-grpc-daemon[1152]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 54 ms (attempt 3/5)5447worker2 # [ 115.247169] nix-grpc-daemon[1152]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 150 ms (attempt 4/5)5448worker2 # [ 115.405095] nix-grpc-daemon[1152]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5449worker2 # [ 115.410835] nix-grpc-daemon[1152]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5450worker2 # [ 115.429176] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:553525451client # [ 115.451050] nix[1204]: copying 1 paths to 'lb:50051'...5452worker2 # [ 115.443698] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:553525453client # [ 115.453599] nix[1204]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'...5454worker2 # [ 115.459555] nix-daemon[847]: accepted connection from pid 1152, user nix-grpc-daemon (trusted)5455worker2 # [ 115.487186] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:48Z level=info event=niks3_push_started pid=12715456worker2 # [ 115.709874] nix-grpc-daemon[1271]: time=2026-09-21T12:34:48.344Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""5457worker2 # [ 115.880098] nix-daemon[847]: accepted connection from pid 1284, user nix-grpc-daemon (trusted)5458worker2 # [ 115.894706] nix-daemon[847]: reaped child process 1287, status = succeeded5459worker2 # [ 116.090926] nix-daemon[847]: accepted connection from pid 1289, user nix-grpc-daemon (trusted)5460worker2 # [ 116.107129] nix-daemon[847]: reaped child process 1296, status = succeeded5461lb # [ 116.118560] niks3-server[1040]: 2026/09/21 12:34:48 INFO Received uploads request method=POST path=/api/pending_closures5462worker2 # [ 116.130583] nix-grpc-daemon[1271]: time=2026-09-21T12:34:48.764Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5463worker2 # [ 116.132259] nix-grpc-daemon[1271]: time=2026-09-21T12:34:48.764Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)"5464lb # [ 116.153604] niks3-server[1040]: 2026/09/21 12:34:48 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst5465worker2 # [ 116.169956] nix-grpc-daemon[1271]: time=2026-09-21T12:34:48.804Z level=INFO msg="Uploading 1 narinfos"5466lb # [ 116.167994] niks3-server[1040]: 2026/09/21 12:34:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5467lb # [ 116.178795] niks3-server[1040]: 2026/09/21 12:34:48 INFO Signed narinfos id=35 count=15468lb # [ 116.187468] niks3-server[1040]: 2026/09/21 12:34:48 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls5469lb # [ 116.197054] niks3-server[1040]: 2026/09/21 12:34:48 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5470lb # [ 116.208940] niks3-server[1040]: 2026/09/21 12:34:48 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo5471lb # [ 116.235622] niks3-server[1040]: 2026/09/21 12:34:48 INFO Completed upload id=355472worker2 # [ 116.237315] nix-grpc-daemon[1271]: time=2026-09-21T12:34:48.872Z level=INFO msg="Upload complete. (523ms)"5473worker2 # [ 116.250102] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55352 duration_s=0 paths=1 nar_bytes_in=4245474worker2 # [ 116.267003] nix-grpc-daemon[1152]: ts=2026-09-21T12:34:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:553525475worker2 # [ 116.277693] nix-daemon[847]: accepted connection from pid 1152, user nix-grpc-daemon (trusted)5476worker2 # [ 116.281807] nix-daemon[847]: reaped child process 1269, status = succeeded5477client # [ 116.301242] nix[1204]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5478worker2 # [ 116.296188] nix-daemon[847]: accepted connection from pid 1152, user nix-grpc-daemon (trusted)5479lb: must succeed: curl -sf localhost:9901/clusters5480lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5481lb: must succeed: curl -sf localhost:9901/clusters5482lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5483worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5484worker2 # [ 117.532004] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1152 (nix-grpc-daemon) on client request.5485lb # [ 117.532967] niks3-server[1040]: 2026/09/21 12:34:50 INFO lead: released remote=192.168.1.4:444885486worker2 # [ 117.534778] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5487client # [ 117.556742] nix[1204]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again5488worker1 # [ 117.620773] nix-grpc-daemon[1541]: ts=2026-09-21T12:34:50Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"5489worker2 # [ 117.558900] systemd[1]: Starting scheduler-token.service...5490client # [ 117.557161] nix[1204]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5491worker2 # [ 117.569530] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5492worker2 # [ 117.571014] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5493worker2 # [ 117.577481] systemd[1]: nix-grpc-daemon.service: Consumed 578ms CPU time over 15.777s wall clock time, 34.6M memory peak, 16.8K incoming IP traffic, 33K outgoing IP traffic.5494worker2 # [ 117.603133] nix-daemon[847]: reaped child process 1211, status = succeeded5495worker2 # [ 117.605046] nix-daemon[847]: reaped child process 1298, status = succeeded5496worker2 # [ 117.612413] nix-daemon[1300]: unexpected Nix daemon error: error: interrupted by the user5497worker2 # [ 117.618751] nix-daemon[847]: reaped child process 1300, status = failed with exit code 15498worker2 # [ 117.660133] systemd[1]: scheduler-token.service: Deactivated successfully.5499worker2 # [ 117.661286] systemd[1]: Finished scheduler-token.service.5500worker2 # [ 117.662259] systemd[1]: scheduler-token.service: Consumed 24ms CPU time over 98ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5501worker2 # [ 117.664512] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.5502worker2 # [ 117.667221] systemd[1]: Starting scheduler-token.service...5503worker2 # [ 117.737606] systemd[1]: scheduler-token.service: Deactivated successfully.5504worker2 # [ 117.740154] systemd[1]: Finished scheduler-token.service.5505worker2 # [ 117.740936] systemd[1]: scheduler-token.service: Consumed 22ms CPU time over 70ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5506worker2 # [ 117.745691] systemd[1]: Starting Nix worker-protocol over gRPC...5507worker1 # [ 117.821005] nix-grpc-daemon[1541]: ts=2026-09-21T12:34:50Z level=info event=scheduler_connected addr=lb:500515508worker2 # [ 117.909970] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:50Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5509worker2 # [ 117.912539] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:50Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5510worker2 # [ 117.916550] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:50Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15511worker2 # [ 117.924077] nix-grpc-daemon[1340]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5512worker2 # [ 117.925535] nix-grpc-daemon[1340]: E0000 00:00:1789994090.557726 1340 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5513worker2 # [ 117.937273] nix-grpc-daemon[1340]: E0000 00:00:1789994090.563973 1340 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5514worker2 # [ 117.942705] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:50Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5515lb # [ 117.940189] niks3-server[1040]: 2026/09/21 12:34:50 INFO lead: acquired remote=192.168.1.4:411305516worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.47 seconds)5517worker2 # [ 117.947050] systemd[1]: Started Nix worker-protocol over gRPC.5518client: must succeed: sleep 5.05519worker2 # [ 117.948337] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:50Z level=info event=scheduler_take_over5520worker2 # [ 117.949582] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:50Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:549145521worker2 # [ 117.951236] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:50Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05522worker2 # [ 117.956127] nix-grpc-daemon[1340]: E0000 00:00:1789994090.588885 1379 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5523worker2 # [ 117.984312] nix-grpc-daemon[1340]: E0000 00:00:1789994090.606351 1379 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5524worker2 # [ 117.989792] nix-grpc-daemon[1340]: E0000 00:00:1789994090.606494 1379 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5525worker2 # [ 117.992709] nix-grpc-daemon[1340]: E0000 00:00:1789994090.623983 1379 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5526worker2 # [ 118.002055] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:50Z level=info event=scheduler_connected addr=lb:500515527worker2 # [ 118.007198] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:50Z level=info event=oidc_jwks provider=mock keys=15528worker2 # [ 118.008722] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:50Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:549145529worker2 # [ 118.010426] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:50Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05530worker2 # [ 118.055959] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:50Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05531worker2 # [ 118.058344] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:50Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.3:50051 clients=15532worker1 # [ 118.187265] nix-grpc-daemon[1541]: ts=2026-09-21T12:34:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427585533worker1 # [ 118.201218] nix-daemon[855]: accepted connection from pid 1541, user nix-grpc-daemon (trusted)5534worker1 # [ 118.225356] nix-grpc-daemon[1541]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 60 ms (attempt 1/5)5535worker1 # [ 118.292452] nix-grpc-daemon[1541]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 195 ms (attempt 2/5)5536worker1 # [ 118.497134] nix-grpc-daemon[1541]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 32 ms (attempt 3/5)5537worker1 # [ 118.529423] nix-grpc-daemon[1541]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 476 ms (attempt 4/5)5538worker1 # [ 119.014624] nix-grpc-daemon[1541]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5539worker1 # [ 119.019981] nix-grpc-daemon[1541]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5540worker1 # [ 119.034795] nix-grpc-daemon[1541]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' from 'http://lb:5751'...5541worker1 # [ 119.056250] nix-daemon[855]: accepted connection from pid 1541, user nix-grpc-daemon (trusted)5542client # [ 119.012448] nix[1204]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5543worker1 # [ 119.081011] nix-daemon[855]: accepted connection from pid 1541, user nix-grpc-daemon (trusted)5544lb # [ 119.889320] envoy[902]: conn peer=[2001:db8:1::1]:53570 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=1405 ms=84425545lb # [ 119.894258] envoy[902]: conn peer=[2001:db8:1::3]:45906 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=4282 tx=1297 ms=250205546lb # [ 119.899214] envoy[902]: conn peer=[2001:db8:1::4]:45912 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=7245 tx=590 ms=154975547client: (finished: must succeed: sleep 5.0, in 5.06 seconds)5548client: waiting for success: ! systemctl is-active mid5549worker1 # [ 123.505277] nix-grpc-daemon[1541]: ts=2026-09-21T12:34:56Z level=info event=niks3_push_started pid=16435550worker1 # [ 123.551563] nix-daemon[855]: reaped child process 1618, status = succeeded5551worker1 # [ 123.751243] nix-grpc-daemon[1643]: time=2026-09-21T12:34:56.391Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""5552worker1 # [ 123.925164] nix-daemon[855]: accepted connection from pid 1656, user nix-grpc-daemon (trusted)5553worker1 # [ 123.945006] nix-daemon[855]: reaped child process 1659, status = succeeded5554lb # [ 123.887262] niks3-server[1040]: 2026/09/21 12:34:56 INFO Received uploads request method=POST path=/api/pending_closures5555worker1 # [ 123.968069] nix-grpc-daemon[1643]: time=2026-09-21T12:34:56.606Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5556worker1 # [ 123.969590] nix-grpc-daemon[1643]: time=2026-09-21T12:34:56.606Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)"5557lb # [ 123.920511] niks3-server[1040]: 2026/09/21 12:34:56 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst5558lb # [ 123.927978] niks3-server[1040]: 2026/09/21 12:34:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5559worker1 # [ 124.001931] nix-grpc-daemon[1643]: time=2026-09-21T12:34:56.641Z level=INFO msg="Uploading 1 narinfos"5560lb # [ 123.930284] niks3-server[1040]: 2026/09/21 12:34:56 INFO Signed narinfos id=36 count=15561lb # [ 123.939704] niks3-server[1040]: 2026/09/21 12:34:56 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5562lb # [ 123.949295] niks3-server[1040]: 2026/09/21 12:34:56 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5563worker1 # [ 124.028727] nix-grpc-daemon[1643]: time=2026-09-21T12:34:56.669Z level=INFO msg="Upload complete. (273ms)"5564lb # [ 123.955966] niks3-server[1040]: 2026/09/21 12:34:56 INFO Completed upload id=365565worker1 # [ 124.036108] nix-grpc-daemon[1541]: ts=2026-09-21T12:34:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42758 duration_s=5 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=15566worker1 # [ 124.044344] nix-grpc-daemon[1643]: time=2026-09-21T12:34:56.680Z level=WARN msg="Failed to register uploaded object" key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo error="server returned 404: object is not pending upload\n"5567worker1 # [ 124.049424] nix-grpc-daemon[1643]: time=2026-09-21T12:34:56.680Z level=WARN msg="Failed to register uploaded object" key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls error="server returned 404: object is not pending upload\n"5568worker1 # [ 124.053301] nix-daemon[855]: reaped child process 1616, status = succeeded5569client # [ 124.021497] systemd[1]: mid.service: Deactivated successfully.5570client # [ 124.026171] systemd[1]: mid.service: Consumed 191ms CPU time over 9.309s wall clock time, 39.2M memory peak, 8.7K incoming IP traffic, 10.3K outgoing IP traffic.5571client: (finished: waiting for success: ! systemctl is-active mid, in 1.14 seconds)5572client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }5573client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.06 seconds)5574(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 9.87 seconds)5575subtest: interrupting the client stops the build on the worker5576client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr5577client # [ 124.557443] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr.5578client # Running as unit: intr.service; invocation ID: f0a578b7cbb848669411cfa6473c37465579client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr, in 0.06 seconds)5580worker2 # [ 124.801239] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:57Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:372585581worker2 # [ 124.816412] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5582client # [ 124.841328] nix[1240]: don't know how to build these paths:5583client # [ 124.844276] nix[1240]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5584worker2 # [ 124.848834] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:57Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=05585worker2 # [ 124.856647] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:57Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.4:50051 clients=15586worker2 # [ 124.894024] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372585587worker2 # [ 124.911548] nix-grpc-daemon[1340]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 72 ms (attempt 1/5)5588worker2 # [ 124.991589] nix-grpc-daemon[1340]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 5 ms (attempt 2/5)5589worker2 # [ 125.004256] nix-grpc-daemon[1340]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 100 ms (attempt 3/5)5590worker2 # [ 125.112063] nix-grpc-daemon[1340]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 469 ms (attempt 4/5)5591worker2 # [ 125.581589] nix-grpc-daemon[1340]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5592worker2 # [ 125.587420] nix-grpc-daemon[1340]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5593worker2 # [ 125.605551] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:372585594client # [ 125.627950] nix[1240]: copying 1 paths to 'lb:50051'...5595worker2 # [ 125.620814] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:372585596client # [ 125.628449] nix[1240]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'...5597worker2 # [ 125.635173] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5598worker2 # [ 125.661841] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:58Z level=info event=niks3_push_started pid=14165599worker2 # [ 125.886492] nix-grpc-daemon[1416]: time=2026-09-21T12:34:58.520Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""5600worker2 # [ 126.072584] nix-daemon[847]: accepted connection from pid 1433, user nix-grpc-daemon (trusted)5601worker2 # [ 126.088855] nix-daemon[847]: reaped child process 1436, status = succeeded5602worker2 # [ 126.259611] nix-daemon[847]: accepted connection from pid 1438, user nix-grpc-daemon (trusted)5603worker2 # [ 126.274397] nix-daemon[847]: reaped child process 1441, status = succeeded5604lb # [ 126.289496] niks3-server[1040]: 2026/09/21 12:34:58 INFO Received uploads request method=POST path=/api/pending_closures5605worker2 # [ 126.306784] nix-grpc-daemon[1416]: time=2026-09-21T12:34:58.940Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5606worker2 # [ 126.308686] nix-grpc-daemon[1416]: time=2026-09-21T12:34:58.940Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)"5607worker2 # [ 126.350094] nix-grpc-daemon[1416]: time=2026-09-21T12:34:58.984Z level=INFO msg="Uploading 1 narinfos"5608lb # [ 126.341149] niks3-server[1040]: 2026/09/21 12:34:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5609lb # [ 126.356322] niks3-server[1040]: 2026/09/21 12:34:58 INFO Signed narinfos id=37 count=15610lb # [ 126.358958] niks3-server[1040]: 2026/09/21 12:34:59 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst5611lb # [ 126.379036] niks3-server[1040]: 2026/09/21 12:34:59 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5612lb # [ 126.383431] niks3-server[1040]: 2026/09/21 12:34:59 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls5613lb # [ 126.387560] niks3-server[1040]: 2026/09/21 12:34:59 INFO Completed upload id=375614worker2 # [ 126.389285] nix-grpc-daemon[1416]: time=2026-09-21T12:34:59.024Z level=INFO msg="Upload complete. (498ms)"5615lb # [ 126.389843] niks3-server[1040]: 2026/09/21 12:34:59 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo5616worker2 # [ 126.404595] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37258 duration_s=0 paths=1 nar_bytes_in=4245617worker2 # [ 126.416504] nix-grpc-daemon[1340]: ts=2026-09-21T12:34:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372585618worker2 # [ 126.422858] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5619worker2 # [ 126.427788] nix-daemon[847]: reaped child process 1414, status = succeeded5620worker2 # [ 126.439197] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5621client # [ 126.445490] nix[1240]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051...5622client: must succeed: systemctl kill -s INT intr5623worker2 # [ 128.291800] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:00Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5624client # [ 128.298265] systemd[1]: intr.service: Sent signal SIGINT to main process 1240 (nix) on client request.5625client # [ 128.306554] nix[1240]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5626client # [ 128.316185] nix[1240]: error: interrupted by the user5627client: (finished: must succeed: systemctl kill -s INT intr, in 0.08 seconds)5628client # [ 128.351046] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE5629client # [ 128.355485] systemd[1]: intr.service: Failed with result 'exit-code'.5630client # [ 128.358363] systemd[1]: intr.service: Consumed 161ms CPU time over 3.796s wall clock time, 41.4M memory peak, 6.4K incoming IP traffic, 8.3K outgoing IP traffic.5631worker2 # [ 128.480404] nix-daemon[1445]: unexpected Nix daemon error: error: interrupted by the user5632worker2 # [ 128.485295] nix-daemon[847]: reaped child process 1443, status = succeeded5633worker2 # [ 128.490238] nix-daemon[847]: reaped child process 1445, status = failed with exit code 15634(finished: subtest: interrupting the client stops the build on the worker, in 5.13 seconds)5635subtest: a deploy mid-build drains: the build finishes, the new generation takes over5636client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw5637client # [ 129.711703] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw.5638client # Running as unit: sw.service; invocation ID: ac7195386d9245b3bb7a239d7b45dd865639client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw, in 0.09 seconds)5640lb # [ 129.892530] envoy[902]: conn peer=[2001:db8:1::1]:38782 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3940 tx=1851 ms=90165641client # [ 129.968787] nix[1260]: don't know how to build these paths:5642client # [ 129.969722] nix[1260]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv5643worker2 # [ 129.961079] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:02Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:372585644worker2 # [ 129.975118] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:02Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=05645worker2 # [ 129.981587] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:02Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.3:50051 clients=15646worker1 # [ 130.094584] nix-grpc-daemon[1541]: ts=2026-09-21T12:35:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427585647worker1 # [ 130.111575] nix-grpc-daemon[1541]: ts=2026-09-21T12:35:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:427585648client # [ 130.060249] nix[1260]: copying 1 paths to 'lb:50051'...5649worker1 # [ 130.126223] nix-grpc-daemon[1541]: ts=2026-09-21T12:35:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:427585650client # [ 130.064145] nix[1260]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'...5651worker1 # [ 130.143037] nix-daemon[855]: accepted connection from pid 1541, user nix-grpc-daemon (trusted)5652worker1 # [ 130.351452] nix-daemon[855]: accepted connection from pid 1691, user nix-grpc-daemon (trusted)5653worker1 # [ 130.371255] nix-daemon[855]: reaped child process 1694, status = succeeded5654worker1 # [ 130.544931] nix-daemon[855]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)5655worker1 # [ 130.561283] nix-daemon[855]: reaped child process 1699, status = succeeded5656lb # [ 129.897591] envoy[902]: conn peer=[2001:db8:1::1]:50756 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3000 tx=829 ms=35325657lb # [ 130.502703] niks3-server[1040]: 2026/09/21 12:35:03 INFO Received uploads request method=POST path=/api/pending_closures5658worker1 # [ 130.590487] nix-grpc-daemon[1643]: time=2026-09-21T12:35:03.230Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5659worker1 # [ 130.591975] nix-grpc-daemon[1643]: time=2026-09-21T12:35:03.230Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)"5660lb # [ 130.534819] niks3-server[1040]: 2026/09/21 12:35:03 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst5661worker1 # [ 130.620751] nix-grpc-daemon[1643]: time=2026-09-21T12:35:03.261Z level=INFO msg="Uploading 1 narinfos"5662lb # [ 130.547712] niks3-server[1040]: 2026/09/21 12:35:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign5663lb # [ 130.560326] niks3-server[1040]: 2026/09/21 12:35:03 INFO Signed narinfos id=38 count=15664lb # [ 130.567211] niks3-server[1040]: 2026/09/21 12:35:03 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls5665lb # [ 130.580082] niks3-server[1040]: 2026/09/21 12:35:03 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete5666worker1 # [ 130.659514] nix-grpc-daemon[1643]: time=2026-09-21T12:35:03.300Z level=INFO msg="Upload complete. (499ms)"5667lb # [ 130.587251] niks3-server[1040]: 2026/09/21 12:35:03 INFO Completed upload id=385668lb # [ 130.590319] niks3-server[1040]: 2026/09/21 12:35:03 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo5669worker1 # [ 130.668246] nix-grpc-daemon[1541]: ts=2026-09-21T12:35:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42758 duration_s=0 paths=1 nar_bytes_in=4165670worker1 # [ 130.682269] nix-grpc-daemon[1541]: ts=2026-09-21T12:35:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:427585671worker1 # [ 130.689336] nix-daemon[855]: accepted connection from pid 1541, user nix-grpc-daemon (trusted)5672worker1 # [ 130.694707] nix-daemon[855]: reaped child process 1689, status = succeeded5673client # [ 130.639879] nix[1260]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051...5674worker1 # [ 130.708535] nix-daemon[855]: accepted connection from pid 1541, user nix-grpc-daemon (trusted)5675worker1: must succeed: systemctl reload nix-grpc-daemon.service5676worker1 # [ 132.478752] systemd[1]: Reloading Nix worker-protocol over gRPC...5677worker1 # [ 132.535225] systemd[1]: Reloaded Nix worker-protocol over gRPC.5678worker1 # [ 132.538958] nix-grpc-daemon[1541]: ts=2026-09-21T12:35:05Z level=info event=draining5679worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.11 seconds)5680worker1: must succeed: pgrep -f 'read -t [9]0 x'5681worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.06 seconds)5682client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain >&25683client # don't know how to build these paths:5684client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5685worker2 # [ 132.793391] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:372585686worker2 # [ 132.801218] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:372585687worker2 # [ 132.822720] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:05Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=05688worker2 # [ 132.829664] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:05Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.4:50051 clients=15689worker2 # [ 132.869336] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372585690worker2 # [ 132.890790] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:372585691client # copying 1 paths to 'lb:50051'...5692worker2 # [ 132.906669] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:372585693client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'...5694worker2 # [ 132.932566] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5695worker2 # [ 133.145425] nix-daemon[847]: accepted connection from pid 1500, user nix-grpc-daemon (trusted)5696worker2 # [ 133.163849] nix-daemon[847]: reaped child process 1503, status = succeeded5697worker2 # [ 133.335201] nix-daemon[847]: accepted connection from pid 1505, user nix-grpc-daemon (trusted)5698worker2 # [ 133.350990] nix-daemon[847]: reaped child process 1508, status = succeeded5699lb # [ 133.362311] niks3-server[1040]: 2026/09/21 12:35:06 INFO Received uploads request method=POST path=/api/pending_closures5700worker2 # [ 133.374111] nix-grpc-daemon[1416]: time=2026-09-21T12:35:06.008Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5701worker2 # [ 133.375558] nix-grpc-daemon[1416]: time=2026-09-21T12:35:06.008Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)"5702lb # [ 133.394893] niks3-server[1040]: 2026/09/21 12:35:06 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst5703worker2 # [ 133.408713] nix-grpc-daemon[1416]: time=2026-09-21T12:35:06.043Z level=INFO msg="Uploading 1 narinfos"5704lb # [ 133.409457] niks3-server[1040]: 2026/09/21 12:35:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign5705lb # [ 133.418297] niks3-server[1040]: 2026/09/21 12:35:06 INFO Signed narinfos id=39 count=15706lb # [ 133.428444] niks3-server[1040]: 2026/09/21 12:35:06 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls5707lb # [ 133.431652] niks3-server[1040]: 2026/09/21 12:35:06 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete5708lb # [ 133.438666] niks3-server[1040]: 2026/09/21 12:35:06 INFO Completed upload id=395709worker2 # [ 133.440115] nix-grpc-daemon[1416]: time=2026-09-21T12:35:06.074Z level=INFO msg="Upload complete. (489ms)"5710lb # [ 133.440948] niks3-server[1040]: 2026/09/21 12:35:06 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo5711worker2 # [ 133.448213] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37258 duration_s=0 paths=1 nar_bytes_in=5285712worker2 # [ 133.458244] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372585713worker2 # [ 133.467525] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5714worker2 # [ 133.470272] nix-daemon[847]: reaped child process 1498, status = succeeded5715client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051...5716client # farm-a> worker2: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5717worker2 # [ 133.482900] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5718client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5719client # farm-a> LOG-farm-a-during-drain5720worker2 # [ 134.928508] nix-daemon[847]: reaped child process 1512, status = succeeded5721worker2 # [ 135.102427] nix-daemon[847]: accepted connection from pid 1528, user nix-grpc-daemon (trusted)5722worker2 # [ 135.120293] nix-daemon[847]: reaped child process 1531, status = succeeded5723lb # [ 135.132613] niks3-server[1040]: 2026/09/21 12:35:07 INFO Received uploads request method=POST path=/api/pending_closures5724worker2 # [ 135.147715] nix-grpc-daemon[1416]: time=2026-09-21T12:35:07.779Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5725worker2 # [ 135.149901] nix-grpc-daemon[1416]: time=2026-09-21T12:35:07.779Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)"5726lb # [ 135.174860] niks3-server[1040]: 2026/09/21 12:35:07 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5727lb # [ 135.185193] niks3-server[1040]: 2026/09/21 12:35:07 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst5728worker2 # [ 135.198625] nix-grpc-daemon[1416]: time=2026-09-21T12:35:07.832Z level=INFO msg="Uploading 1 narinfos"5729lb # [ 135.199602] niks3-server[1040]: 2026/09/21 12:35:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign5730lb # [ 135.208547] niks3-server[1040]: 2026/09/21 12:35:07 INFO Signed narinfos id=40 count=15731lb # [ 135.210975] niks3-server[1040]: 2026/09/21 12:35:07 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls5732lb # [ 135.219780] niks3-server[1040]: 2026/09/21 12:35:07 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete5733lb # [ 135.224575] niks3-server[1040]: 2026/09/21 12:35:07 INFO Completed upload id=405734worker2 # [ 135.226155] nix-grpc-daemon[1416]: time=2026-09-21T12:35:07.860Z level=INFO msg="Upload complete. (332ms)"5735lb # [ 135.227177] niks3-server[1040]: 2026/09/21 12:35:07 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo5736worker2 # [ 135.242522] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37258 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=15737worker2 # [ 135.257862] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:07Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=05738worker2 # [ 135.261607] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:07Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.4:50051 clients=15739worker2 # [ 135.265458] nix-daemon[847]: reaped child process 1510, status = succeeded5740worker2 # [ 135.285422] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372585741worker2 # [ 135.299336] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:372585742worker2 # [ 135.313520] nix-grpc-daemon[1416]: time=2026-09-21T12:35:07.948Z level=INFO msg="All 1 paths already cached"5743client # copying 1 paths to 'lb:50051'...5744worker2 # [ 135.324340] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:372585745client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'...5746worker2 # [ 135.347780] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5747worker2 # [ 135.547309] nix-daemon[847]: accepted connection from pid 1537, user nix-grpc-daemon (trusted)5748worker2 # [ 135.563598] nix-daemon[847]: reaped child process 1540, status = succeeded5749worker2 # [ 135.737030] nix-daemon[847]: accepted connection from pid 1542, user nix-grpc-daemon (trusted)5750worker2 # [ 135.753605] nix-daemon[847]: reaped child process 1545, status = succeeded5751lb # [ 135.763110] niks3-server[1040]: 2026/09/21 12:35:08 INFO Received uploads request method=POST path=/api/pending_closures5752worker2 # [ 135.774957] nix-grpc-daemon[1416]: time=2026-09-21T12:35:08.409Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5753worker2 # [ 135.776522] nix-grpc-daemon[1416]: time=2026-09-21T12:35:08.409Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)"5754lb # [ 135.792299] niks3-server[1040]: 2026/09/21 12:35:08 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst5755lb # [ 135.804849] niks3-server[1040]: 2026/09/21 12:35:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign5756worker2 # [ 135.813140] nix-grpc-daemon[1416]: time=2026-09-21T12:35:08.447Z level=INFO msg="Uploading 1 narinfos"5757lb # [ 135.815856] niks3-server[1040]: 2026/09/21 12:35:08 INFO Signed narinfos id=41 count=15758lb # [ 135.822781] niks3-server[1040]: 2026/09/21 12:35:08 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls5759lb # [ 135.832357] niks3-server[1040]: 2026/09/21 12:35:08 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete5760lb # [ 135.838920] niks3-server[1040]: 2026/09/21 12:35:08 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo5761lb # [ 135.844490] niks3-server[1040]: 2026/09/21 12:35:08 INFO Completed upload id=415762worker2 # [ 135.846028] nix-grpc-daemon[1416]: time=2026-09-21T12:35:08.480Z level=INFO msg="Upload complete. (481ms)"5763worker2 # [ 135.854148] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37258 duration_s=0 paths=1 nar_bytes_in=6405764worker2 # [ 135.867844] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372585765worker2 # [ 135.877096] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5766worker2 # [ 135.879919] nix-daemon[847]: reaped child process 1535, status = succeeded5767client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051...5768client # farm-b> worker2: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv5769worker2 # [ 135.893281] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5770client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5771client # farm-b> LOG-farm-b-during-drain5772worker2 # [ 137.528520] nix-daemon[847]: reaped child process 1549, status = succeeded5773worker2 # [ 137.702174] nix-daemon[847]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)5774worker2 # [ 137.718084] nix-daemon[847]: reaped child process 1568, status = succeeded5775lb # [ 137.729077] niks3-server[1040]: 2026/09/21 12:35:10 INFO Received uploads request method=POST path=/api/pending_closures5776worker2 # [ 137.740205] nix-grpc-daemon[1416]: time=2026-09-21T12:35:10.374Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5777worker2 # [ 137.741675] nix-grpc-daemon[1416]: time=2026-09-21T12:35:10.374Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)"5778lb # [ 137.766570] niks3-server[1040]: 2026/09/21 12:35:10 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv5779lb # [ 137.777945] niks3-server[1040]: 2026/09/21 12:35:10 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst5780lb # [ 137.785582] niks3-server[1040]: 2026/09/21 12:35:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign5781lb # [ 137.795661] niks3-server[1040]: 2026/09/21 12:35:10 INFO Signed narinfos id=42 count=15782worker2 # [ 137.800734] nix-grpc-daemon[1416]: time=2026-09-21T12:35:10.435Z level=INFO msg="Uploading 1 narinfos"5783lb # [ 137.808088] niks3-server[1040]: 2026/09/21 12:35:10 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls5784lb # [ 137.816636] niks3-server[1040]: 2026/09/21 12:35:10 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete5785lb # [ 137.825481] niks3-server[1040]: 2026/09/21 12:35:10 INFO Completed upload id=425786worker2 # [ 137.827015] nix-grpc-daemon[1416]: time=2026-09-21T12:35:10.461Z level=INFO msg="Upload complete. (323ms)"5787lb # [ 137.828217] niks3-server[1040]: 2026/09/21 12:35:10 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo5788worker2 # [ 137.845058] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37258 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=15789worker2 # [ 137.860386] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:10Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=05790worker2 # [ 137.865395] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:10Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.4:50051 clients=15791worker2 # [ 137.869491] nix-daemon[847]: reaped child process 1547, status = succeeded5792worker2 # [ 137.889536] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372585793worker2 # [ 137.903965] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:372585794worker2 # [ 137.917170] nix-grpc-daemon[1416]: time=2026-09-21T12:35:10.551Z level=INFO msg="All 2 paths already cached"5795client # copying 1 paths to 'lb:50051'...5796client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'...5797worker2 # [ 137.927004] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:372585798worker2 # [ 137.943053] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5799worker2 # [ 138.153373] nix-daemon[847]: accepted connection from pid 1574, user nix-grpc-daemon (trusted)5800worker2 # [ 138.170339] nix-daemon[847]: reaped child process 1577, status = succeeded5801worker2 # [ 138.343885] nix-daemon[847]: accepted connection from pid 1579, user nix-grpc-daemon (trusted)5802worker2 # [ 138.359105] nix-daemon[847]: reaped child process 1582, status = succeeded5803lb # [ 138.372217] niks3-server[1040]: 2026/09/21 12:35:11 INFO Received uploads request method=POST path=/api/pending_closures5804worker2 # [ 138.382501] nix-grpc-daemon[1416]: time=2026-09-21T12:35:11.017Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5805worker2 # [ 138.383979] nix-grpc-daemon[1416]: time=2026-09-21T12:35:11.017Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)"5806lb # [ 138.399354] niks3-server[1040]: 2026/09/21 12:35:11 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst5807lb # [ 138.412619] niks3-server[1040]: 2026/09/21 12:35:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign5808worker2 # [ 138.419677] nix-grpc-daemon[1416]: time=2026-09-21T12:35:11.054Z level=INFO msg="Uploading 1 narinfos"5809lb # [ 138.422856] niks3-server[1040]: 2026/09/21 12:35:11 INFO Signed narinfos id=43 count=15810lb # [ 138.429459] niks3-server[1040]: 2026/09/21 12:35:11 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls5811lb # [ 138.436713] niks3-server[1040]: 2026/09/21 12:35:11 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete5812worker2 # [ 138.444626] nix-grpc-daemon[1416]: time=2026-09-21T12:35:11.079Z level=INFO msg="Upload complete. (480ms)"5813lb # [ 138.444079] niks3-server[1040]: 2026/09/21 12:35:11 INFO Completed upload id=435814lb # [ 138.446807] niks3-server[1040]: 2026/09/21 12:35:11 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo5815worker2 # [ 138.452173] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37258 duration_s=0 paths=1 nar_bytes_in=7685816worker2 # [ 138.466975] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372585817worker2 # [ 138.470368] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5818worker2 # [ 138.475717] nix-daemon[847]: reaped child process 1572, status = succeeded5819client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051...5820worker2 # [ 138.487140] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)5821client # farm-top> worker2: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5822client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5823client # farm-top> LOG-farm-top-during-drain5824worker2 # [ 139.917091] nix-daemon[847]: reaped child process 1586, status = succeeded5825worker2 # [ 140.082343] nix-daemon[847]: accepted connection from pid 1602, user nix-grpc-daemon (trusted)5826worker2 # [ 140.100040] nix-daemon[847]: reaped child process 1605, status = succeeded5827lb # [ 140.117711] niks3-server[1040]: 2026/09/21 12:35:12 INFO Received uploads request method=POST path=/api/pending_closures5828worker2 # [ 140.128403] nix-grpc-daemon[1416]: time=2026-09-21T12:35:12.761Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5829worker2 # [ 140.129890] nix-grpc-daemon[1416]: time=2026-09-21T12:35:12.761Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)"5830lb # [ 140.155446] niks3-server[1040]: 2026/09/21 12:35:12 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5831worker2 # [ 140.169209] nix-grpc-daemon[1416]: time=2026-09-21T12:35:12.803Z level=INFO msg="Uploading 1 narinfos"5832lb # [ 140.164752] niks3-server[1040]: 2026/09/21 12:35:12 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst5833lb # [ 140.179734] niks3-server[1040]: 2026/09/21 12:35:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign5834lb # [ 140.189545] niks3-server[1040]: 2026/09/21 12:35:12 INFO Signed narinfos id=44 count=15835lb # [ 140.192661] niks3-server[1040]: 2026/09/21 12:35:12 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls5836worker2 # [ 140.198396] nix-grpc-daemon[1416]: time=2026-09-21T12:35:12.833Z level=INFO msg="Upload complete. (301ms)"5837lb # [ 140.197545] niks3-server[1040]: 2026/09/21 12:35:12 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete5838lb # [ 140.203594] niks3-server[1040]: 2026/09/21 12:35:12 INFO Completed upload id=445839lb # [ 140.208486] niks3-server[1040]: 2026/09/21 12:35:12 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo5840worker2 # [ 140.209477] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37258 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=15841worker2 # [ 140.225638] nix-daemon[847]: reaped child process 1584, status = succeeded5842client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain >&2, in 7.73 seconds)5843client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain outPath5844client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain outPath, in 0.23 seconds)5845worker1: must succeed: pkill -f 'read -t [9]0 x'5846worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.06 seconds)5847client: waiting for success: ! systemctl is-active sw5848worker1 # [ 140.753386] nix-daemon[855]: reaped child process 1703, status = succeeded5849worker1 # [ 140.929872] nix-daemon[855]: accepted connection from pid 1744, user nix-grpc-daemon (trusted)5850worker1 # [ 140.947923] nix-daemon[855]: reaped child process 1747, status = succeeded5851lb # [ 140.886300] niks3-server[1040]: 2026/09/21 12:35:13 INFO Received uploads request method=POST path=/api/pending_closures5852worker1 # [ 140.969728] nix-grpc-daemon[1643]: time=2026-09-21T12:35:13.609Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5853worker1 # [ 140.971349] nix-grpc-daemon[1643]: time=2026-09-21T12:35:13.609Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)"5854lb # [ 140.917576] niks3-server[1040]: 2026/09/21 12:35:13 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst5855lb # [ 140.933906] niks3-server[1040]: 2026/09/21 12:35:13 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv5856worker1 # [ 141.016083] nix-grpc-daemon[1643]: time=2026-09-21T12:35:13.655Z level=INFO msg="Uploading 1 narinfos"5857lb # [ 140.945908] niks3-server[1040]: 2026/09/21 12:35:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign5858lb # [ 140.952585] niks3-server[1040]: 2026/09/21 12:35:13 INFO Signed narinfos id=45 count=15859lb # [ 140.954632] niks3-server[1040]: 2026/09/21 12:35:13 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls5860lb # [ 140.960487] niks3-server[1040]: 2026/09/21 12:35:13 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete5861lb # [ 140.966823] niks3-server[1040]: 2026/09/21 12:35:13 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo5862lb # [ 140.970617] niks3-server[1040]: 2026/09/21 12:35:13 INFO Completed upload id=455863worker1 # [ 141.042970] nix-grpc-daemon[1643]: time=2026-09-21T12:35:13.683Z level=INFO msg="Upload complete. (316ms)"5864worker1 # [ 141.053064] nix-grpc-daemon[1541]: ts=2026-09-21T12:35:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42758 duration_s=10 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=15865worker1 # [ 141.070344] nix-daemon[855]: reaped child process 1701, status = succeeded5866client # [ 141.029826] systemd[1]: sw.service: Deactivated successfully.5867client # [ 141.034402] systemd[1]: sw.service: Consumed 177ms CPU time over 11.321s wall clock time, 41.2M memory peak, 7K incoming IP traffic, 8.4K outgoing IP traffic.5868worker1 # [ 141.539882] nix-grpc-daemon[1541]: ts=2026-09-21T12:35:14Z level=info event=signal_exit5869client: (finished: waiting for success: ! systemctl is-active sw, in 1.12 seconds)5870client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }5871client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.06 seconds)5872worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service5873worker2 # [ 142.697060] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:15Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:549145874worker2 # [ 142.701992] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:15Z level=info event=worker_gone addr=192.168.1.3:500515875worker1 # [ 142.768002] nix-grpc-daemon[1541]: ts=2026-09-21T12:35:15Z level=info event=scheduler_restarting5876worker1 # [ 142.793633] nix-grpc-daemon[1541]: ts=2026-09-21T12:35:15Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5877worker1 # [ 142.818517] nix-daemon[855]: reaped child process 1609, status = succeeded5878worker1 # [ 143.344715] nix-grpc-daemon[1541]: E0000 00:00:1789994115.985063 1586 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true5879worker1 # [ 143.353708] nix-grpc-daemon[1541]: E0000 00:00:1789994115.994098 1586 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5880worker1 # [ 144.585249] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5881worker1 # [ 144.591106] systemd[1]: nix-grpc-daemon.service: Consumed 890ms CPU time over 30.268s wall clock time, 33M memory peak, 21K incoming IP traffic, 27.2K outgoing IP traffic.5882worker1 # [ 144.609214] systemd[1]: Starting Nix worker-protocol over gRPC...5883worker1 # [ 144.810373] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:17Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5884worker1 # [ 144.813015] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:17Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5885worker1 # [ 144.817044] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:17Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15886worker1 # [ 144.824062] nix-grpc-daemon[1761]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5887worker1 # [ 144.833699] nix-grpc-daemon[1761]: E0000 00:00:1789994117.464113 1761 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5888worker1 # [ 144.837824] nix-grpc-daemon[1761]: E0000 00:00:1789994117.467720 1761 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5889worker1 # [ 144.841423] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:17Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5890worker1 # [ 144.843402] systemd[1]: Started Nix worker-protocol over gRPC.5891worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 3.05 seconds)5892lb: waiting until exactly no endpoints of aarch64-linux are unhealthy5893lb: must succeed: curl -sf localhost:9901/clusters5894worker1 # [ 144.868135] nix-grpc-daemon[1761]: E0000 00:00:1789994117.485577 1803 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5895worker1 # [ 144.870481] nix-grpc-daemon[1761]: E0000 00:00:1789994117.502602 1803 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5896worker1 # [ 144.882407] nix-grpc-daemon[1761]: E0000 00:00:1789994117.502749 1803 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5897worker1 # [ 144.885383] nix-grpc-daemon[1761]: E0000 00:00:1789994117.522071 1803 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5898worker1 # [ 144.899448] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:17Z level=info event=scheduler_connected addr=lb:500515899worker2 # [ 144.831959] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:17Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:549145900worker2 # [ 144.832744] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:17Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05901lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5902lb: must succeed: curl -sf localhost:9901/clusters5903lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5904lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.18 seconds)5905lb: must succeed: curl -sf localhost:9901/clusters5906lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5907worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5908worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5909(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 16.47 seconds)5910subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does5911client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&15912worker2 # [ 146.391393] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:372585913worker2 # [ 146.397963] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:372585914worker2 # [ 146.437392] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:19Z level=debug event=want client=5 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=05915worker2 # [ 146.448327] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:19Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=15916worker2 # [ 146.454175] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:19Z level=debug event=want client=5 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=05917worker2 # [ 146.459329] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:19Z level=debug event=want client=5 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=05918worker1 # [ 146.563137] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349945919worker1 # [ 146.578805] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)5920worker1 # [ 146.605303] nix-grpc-daemon[1761]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 85 ms (attempt 1/5)5921worker1 # [ 146.697587] nix-grpc-daemon[1761]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 166 ms (attempt 2/5)5922worker1 # [ 146.871273] nix-grpc-daemon[1761]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 260 ms (attempt 3/5)5923worker1 # [ 147.131654] nix-grpc-daemon[1761]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 731 ms (attempt 4/5)5924worker1 # [ 147.871684] nix-grpc-daemon[1761]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5925worker1 # [ 147.877221] nix-grpc-daemon[1761]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5926worker1 # [ 147.898488] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349945927worker1 # [ 147.914278] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:349945928worker1 # [ 147.929726] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)5929worker1 # [ 147.956862] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:20Z level=info event=niks3_push_started pid=18215930worker1 # [ 148.180180] nix-grpc-daemon[1821]: time=2026-09-21T12:35:20.820Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""5931worker1 # [ 148.359628] nix-daemon[855]: accepted connection from pid 1834, user nix-grpc-daemon (trusted)5932worker1 # [ 148.376311] nix-daemon[855]: reaped child process 1837, status = succeeded5933worker1 # [ 148.557505] nix-daemon[855]: accepted connection from pid 1839, user nix-grpc-daemon (trusted)5934worker1 # [ 148.574232] nix-daemon[855]: reaped child process 1842, status = succeeded5935lb # [ 148.513930] niks3-server[1040]: 2026/09/21 12:35:21 INFO Received uploads request method=POST path=/api/pending_closures5936worker1 # [ 148.599491] nix-grpc-daemon[1821]: time=2026-09-21T12:35:21.238Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5937worker1 # [ 148.601360] nix-grpc-daemon[1821]: time=2026-09-21T12:35:21.238Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)"5938lb # [ 148.550977] niks3-server[1040]: 2026/09/21 12:35:21 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst5939lb # [ 148.564081] niks3-server[1040]: 2026/09/21 12:35:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign5940worker1 # [ 148.643065] nix-grpc-daemon[1821]: time=2026-09-21T12:35:21.283Z level=INFO msg="Uploading 1 narinfos"5941lb # [ 148.575488] niks3-server[1040]: 2026/09/21 12:35:21 INFO Signed narinfos id=46 count=15942lb # [ 148.582548] niks3-server[1040]: 2026/09/21 12:35:21 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls5943lb # [ 148.595726] niks3-server[1040]: 2026/09/21 12:35:21 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete5944worker1 # [ 148.673984] nix-grpc-daemon[1821]: time=2026-09-21T12:35:21.314Z level=INFO msg="Upload complete. (489ms)"5945lb # [ 148.601790] niks3-server[1040]: 2026/09/21 12:35:21 INFO Completed upload id=465946lb # [ 148.605663] niks3-server[1040]: 2026/09/21 12:35:21 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo5947worker1 # [ 148.680210] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=0 paths=1 nar_bytes_in=5125948worker1 # [ 148.688901] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349945949worker1 # [ 148.701086] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)5950worker1 # [ 148.703999] nix-daemon[855]: reaped child process 1819, status = succeeded5951worker1 # [ 148.713932] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)5952lb # [ 149.890458] envoy[902]: conn peer=[2001:db8:1::1]:33756 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9681 tx=4100 ms=74395953lb # [ 149.895532] envoy[902]: conn peer=[2001:db8:1::1]:50770 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2978 tx=1429 ms=110355954lb # [ 149.900553] envoy[902]: conn peer=[2001:db8:1::3]:54344 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=986 tx=425 ms=281875955worker1 # [ 150.360674] nix-daemon[855]: reaped child process 1846, status = succeeded5956worker1 # [ 150.541835] nix-daemon[855]: accepted connection from pid 1862, user nix-grpc-daemon (trusted)5957worker1 # [ 150.559798] nix-daemon[855]: reaped child process 1865, status = succeeded5958lb # [ 150.502748] niks3-server[1040]: 2026/09/21 12:35:23 INFO Received uploads request method=POST path=/api/pending_closures5959worker1 # [ 150.588402] nix-grpc-daemon[1821]: time=2026-09-21T12:35:23.226Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5960worker1 # [ 150.589930] nix-grpc-daemon[1821]: time=2026-09-21T12:35:23.226Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)"5961lb # [ 150.534348] niks3-server[1040]: 2026/09/21 12:35:23 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv5962lb # [ 150.550293] niks3-server[1040]: 2026/09/21 12:35:23 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst5963lb # [ 150.564555] niks3-server[1040]: 2026/09/21 12:35:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign5964worker1 # [ 150.643031] nix-grpc-daemon[1821]: time=2026-09-21T12:35:23.282Z level=INFO msg="Uploading 1 narinfos"5965lb # [ 150.574404] niks3-server[1040]: 2026/09/21 12:35:23 INFO Signed narinfos id=47 count=15966lb # [ 150.579885] niks3-server[1040]: 2026/09/21 12:35:23 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls5967lb # [ 150.587061] niks3-server[1040]: 2026/09/21 12:35:23 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete5968lb # [ 150.594977] niks3-server[1040]: 2026/09/21 12:35:23 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo5969lb # [ 150.599545] niks3-server[1040]: 2026/09/21 12:35:23 INFO Completed upload id=475970worker1 # [ 150.672371] nix-grpc-daemon[1821]: time=2026-09-21T12:35:23.312Z level=INFO msg="Upload complete. (345ms)"5971worker2 # [ 150.612393] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:23Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=15972worker1 # [ 150.683743] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=7 outputs=15973worker2 # [ 150.621427] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:23Z level=debug event=want client=5 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=05974worker1 # [ 150.701632] nix-daemon[855]: reaped child process 1844, status = succeeded5975worker1 # [ 150.729423] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349945976worker1 # [ 150.745147] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349945977worker1 # [ 150.759552] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:349945978worker1 # [ 150.773263] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)5979worker1 # [ 150.972791] nix-daemon[855]: accepted connection from pid 1869, user nix-grpc-daemon (trusted)5980worker1 # [ 150.992135] nix-daemon[855]: reaped child process 1872, status = succeeded5981worker1 # [ 151.167207] nix-daemon[855]: accepted connection from pid 1874, user nix-grpc-daemon (trusted)5982worker1 # [ 151.183485] nix-daemon[855]: reaped child process 1877, status = succeeded5983lb # [ 151.122491] niks3-server[1040]: 2026/09/21 12:35:23 INFO Received uploads request method=POST path=/api/pending_closures5984worker1 # [ 151.204139] nix-grpc-daemon[1821]: time=2026-09-21T12:35:23.844Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5985worker1 # [ 151.206193] nix-grpc-daemon[1821]: time=2026-09-21T12:35:23.844Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)"5986lb # [ 151.149086] niks3-server[1040]: 2026/09/21 12:35:23 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst5987lb # [ 151.164191] niks3-server[1040]: 2026/09/21 12:35:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign5988worker1 # [ 151.243399] nix-grpc-daemon[1821]: time=2026-09-21T12:35:23.883Z level=INFO msg="Uploading 1 narinfos"5989lb # [ 151.174694] niks3-server[1040]: 2026/09/21 12:35:23 INFO Signed narinfos id=48 count=15990lb # [ 151.181999] niks3-server[1040]: 2026/09/21 12:35:23 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls5991worker1 # [ 151.264180] nix-grpc-daemon[1821]: time=2026-09-21T12:35:23.904Z level=INFO msg="Upload complete. (475ms)"5992lb # [ 151.192313] niks3-server[1040]: 2026/09/21 12:35:23 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete5993lb # [ 151.197800] niks3-server[1040]: 2026/09/21 12:35:23 INFO Completed upload id=485994worker1 # [ 151.272188] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=0 paths=1 nar_bytes_in=5125995lb # [ 151.201040] niks3-server[1040]: 2026/09/21 12:35:23 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo5996worker1 # [ 151.281570] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349945997worker1 # [ 151.289264] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)5998worker1 # [ 151.291764] nix-daemon[855]: reaped child process 1867, status = succeeded5999worker1 # [ 151.303340] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6000worker1 # [ 152.744804] nix-daemon[855]: reaped child process 1881, status = succeeded6001worker1 # [ 152.908369] nix-daemon[855]: accepted connection from pid 1897, user nix-grpc-daemon (trusted)6002worker1 # [ 152.923602] nix-daemon[855]: reaped child process 1900, status = succeeded6003lb # [ 152.866757] niks3-server[1040]: 2026/09/21 12:35:25 INFO Received uploads request method=POST path=/api/pending_closures6004worker1 # [ 152.949732] nix-grpc-daemon[1821]: time=2026-09-21T12:35:25.589Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6005worker1 # [ 152.951415] nix-grpc-daemon[1821]: time=2026-09-21T12:35:25.589Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)"6006lb # [ 152.900524] niks3-server[1040]: 2026/09/21 12:35:25 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6007lb # [ 152.914600] niks3-server[1040]: 2026/09/21 12:35:25 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv6008lb # [ 152.923489] niks3-server[1040]: 2026/09/21 12:35:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6009worker1 # [ 153.002187] nix-grpc-daemon[1821]: time=2026-09-21T12:35:25.642Z level=INFO msg="Uploading 1 narinfos"6010lb # [ 152.934461] niks3-server[1040]: 2026/09/21 12:35:25 INFO Signed narinfos id=49 count=16011lb # [ 152.941861] niks3-server[1040]: 2026/09/21 12:35:25 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls6012lb # [ 152.947884] niks3-server[1040]: 2026/09/21 12:35:25 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6013lb # [ 152.955299] niks3-server[1040]: 2026/09/21 12:35:25 INFO Completed upload id=496014worker1 # [ 153.027963] nix-grpc-daemon[1821]: time=2026-09-21T12:35:25.668Z level=INFO msg="Upload complete. (314ms)"6015lb # [ 152.958207] niks3-server[1040]: 2026/09/21 12:35:25 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo6016worker2 # [ 152.966169] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:25Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=16017worker1 # [ 153.041006] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=8 outputs=16018worker2 # [ 152.979629] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:25Z level=debug event=want client=5 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=06019worker1 # [ 153.059212] nix-daemon[855]: reaped child process 1879, status = succeeded6020worker1 # [ 153.093626] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349946021worker1 # [ 153.108752] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349946022worker1 # [ 153.123183] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:349946023worker1 # [ 153.137167] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6024worker1 # [ 153.333795] nix-daemon[855]: accepted connection from pid 1904, user nix-grpc-daemon (trusted)6025worker1 # [ 153.351069] nix-daemon[855]: reaped child process 1907, status = succeeded6026worker1 # [ 153.526416] nix-daemon[855]: accepted connection from pid 1909, user nix-grpc-daemon (trusted)6027worker1 # [ 153.542141] nix-daemon[855]: reaped child process 1912, status = succeeded6028lb # [ 153.480830] niks3-server[1040]: 2026/09/21 12:35:26 INFO Received uploads request method=POST path=/api/pending_closures6029worker1 # [ 153.567984] nix-grpc-daemon[1821]: time=2026-09-21T12:35:26.208Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6030worker1 # [ 153.569559] nix-grpc-daemon[1821]: time=2026-09-21T12:35:26.208Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)"6031worker1 # [ 153.598501] nix-grpc-daemon[1821]: time=2026-09-21T12:35:26.238Z level=INFO msg="Uploading 1 narinfos"6032lb # [ 153.512677] niks3-server[1040]: 2026/09/21 12:35:26 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst6033lb # [ 153.528110] niks3-server[1040]: 2026/09/21 12:35:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6034lb # [ 153.542909] niks3-server[1040]: 2026/09/21 12:35:26 INFO Signed narinfos id=50 count=16035lb # [ 153.549321] niks3-server[1040]: 2026/09/21 12:35:26 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls6036lb # [ 153.562420] niks3-server[1040]: 2026/09/21 12:35:26 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6037lb # [ 153.568740] niks3-server[1040]: 2026/09/21 12:35:26 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo6038lb # [ 153.572796] niks3-server[1040]: 2026/09/21 12:35:26 INFO Completed upload id=506039worker1 # [ 153.645679] nix-grpc-daemon[1821]: time=2026-09-21T12:35:26.286Z level=INFO msg="Upload complete. (493ms)"6040worker1 # [ 153.656295] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=0 paths=1 nar_bytes_in=5126041worker1 # [ 153.670681] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349946042worker1 # [ 153.677627] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6043worker1 # [ 153.682911] nix-daemon[855]: reaped child process 1902, status = succeeded6044worker1 # [ 153.695476] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6045worker1 # [ 154.749335] nix-daemon[855]: reaped child process 1916, status = succeeded6046worker1 # [ 154.935816] nix-daemon[855]: accepted connection from pid 1932, user nix-grpc-daemon (trusted)6047worker1 # [ 154.954162] nix-daemon[855]: reaped child process 1935, status = succeeded6048lb # [ 154.894779] niks3-server[1040]: 2026/09/21 12:35:27 INFO Received uploads request method=POST path=/api/pending_closures6049worker1 # [ 154.978692] nix-grpc-daemon[1821]: time=2026-09-21T12:35:27.617Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6050worker1 # [ 154.980264] nix-grpc-daemon[1821]: time=2026-09-21T12:35:27.617Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)"6051lb # [ 154.929130] niks3-server[1040]: 2026/09/21 12:35:27 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv6052lb # [ 154.940390] niks3-server[1040]: 2026/09/21 12:35:27 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6053worker1 # [ 155.026783] nix-grpc-daemon[1821]: time=2026-09-21T12:35:27.667Z level=INFO msg="Uploading 1 narinfos"6054lb # [ 154.953715] niks3-server[1040]: 2026/09/21 12:35:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6055lb # [ 154.960630] niks3-server[1040]: 2026/09/21 12:35:27 INFO Signed narinfos id=51 count=16056lb # [ 154.966457] niks3-server[1040]: 2026/09/21 12:35:27 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls6057lb # [ 154.977147] niks3-server[1040]: 2026/09/21 12:35:27 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6058lb # [ 154.983723] niks3-server[1040]: 2026/09/21 12:35:27 INFO Completed upload id=516059worker1 # [ 155.056843] nix-grpc-daemon[1821]: time=2026-09-21T12:35:27.697Z level=INFO msg="Upload complete. (345ms)"6060lb # [ 154.986885] niks3-server[1040]: 2026/09/21 12:35:27 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo6061worker2 # [ 154.995071] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:27Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=16062worker1 # [ 155.069092] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=9 outputs=16063worker2 # [ 155.008141] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:27Z level=debug event=want client=5 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=06064worker1 # [ 155.087240] nix-daemon[855]: reaped child process 1914, status = succeeded6065worker1 # [ 155.121530] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349946066worker1 # [ 155.136637] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349946067worker1 # [ 155.150809] nix-grpc-daemon[1821]: time=2026-09-21T12:35:27.791Z level=INFO msg="All 1 paths already cached"6068worker1 # [ 155.160088] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:349946069worker1 # [ 155.173532] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6070worker1 # [ 155.370926] nix-daemon[855]: accepted connection from pid 1939, user nix-grpc-daemon (trusted)6071worker1 # [ 155.389207] nix-daemon[855]: reaped child process 1942, status = succeeded6072worker1 # [ 155.562944] nix-daemon[855]: accepted connection from pid 1944, user nix-grpc-daemon (trusted)6073worker1 # [ 155.580111] nix-daemon[855]: reaped child process 1947, status = succeeded6074lb # [ 155.521331] niks3-server[1040]: 2026/09/21 12:35:28 INFO Received uploads request method=POST path=/api/pending_closures6075worker1 # [ 155.603576] nix-grpc-daemon[1821]: time=2026-09-21T12:35:28.243Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6076worker1 # [ 155.605170] nix-grpc-daemon[1821]: time=2026-09-21T12:35:28.243Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)"6077lb # [ 155.552695] niks3-server[1040]: 2026/09/21 12:35:28 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst6078worker1 # [ 155.634303] nix-grpc-daemon[1821]: time=2026-09-21T12:35:28.274Z level=INFO msg="Uploading 1 narinfos"6079lb # [ 155.566362] niks3-server[1040]: 2026/09/21 12:35:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6080lb # [ 155.577389] niks3-server[1040]: 2026/09/21 12:35:28 INFO Signed narinfos id=52 count=16081lb # [ 155.587329] niks3-server[1040]: 2026/09/21 12:35:28 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls6082lb # [ 155.592823] niks3-server[1040]: 2026/09/21 12:35:28 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6083worker1 # [ 155.665347] nix-grpc-daemon[1821]: time=2026-09-21T12:35:28.305Z level=INFO msg="Upload complete. (475ms)"6084lb # [ 155.598047] niks3-server[1040]: 2026/09/21 12:35:28 INFO Completed upload id=526085worker1 # [ 155.672345] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=0 paths=1 nar_bytes_in=6246086lb # [ 155.601468] niks3-server[1040]: 2026/09/21 12:35:28 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo6087worker1 # [ 155.680643] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349946088worker1 # [ 155.691586] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6089worker1 # [ 155.693825] nix-daemon[855]: reaped child process 1937, status = succeeded6090worker1 # [ 155.704650] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6091worker1 # [ 156.549146] nix-daemon[855]: reaped child process 1951, status = succeeded6092worker1 # [ 156.731122] nix-daemon[855]: accepted connection from pid 1967, user nix-grpc-daemon (trusted)6093worker1 # [ 156.747967] nix-daemon[855]: reaped child process 1970, status = succeeded6094lb # [ 156.690953] niks3-server[1040]: 2026/09/21 12:35:29 INFO Received uploads request method=POST path=/api/pending_closures6095worker1 # [ 156.776397] nix-grpc-daemon[1821]: time=2026-09-21T12:35:29.414Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6096worker1 # [ 156.777874] nix-grpc-daemon[1821]: time=2026-09-21T12:35:29.414Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)"6097lb # [ 156.732794] niks3-server[1040]: 2026/09/21 12:35:29 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv6098lb # [ 156.744350] niks3-server[1040]: 2026/09/21 12:35:29 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst6099worker1 # [ 156.828201] nix-grpc-daemon[1821]: time=2026-09-21T12:35:29.468Z level=INFO msg="Uploading 1 narinfos"6100lb # [ 156.756490] niks3-server[1040]: 2026/09/21 12:35:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6101lb # [ 156.764213] niks3-server[1040]: 2026/09/21 12:35:29 INFO Signed narinfos id=53 count=16102lb # [ 156.766554] niks3-server[1040]: 2026/09/21 12:35:29 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls6103lb # [ 156.774050] niks3-server[1040]: 2026/09/21 12:35:29 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6104lb # [ 156.780763] niks3-server[1040]: 2026/09/21 12:35:29 INFO Completed upload id=536105worker1 # [ 156.853876] nix-grpc-daemon[1821]: time=2026-09-21T12:35:29.494Z level=INFO msg="Upload complete. (338ms)"6106lb # [ 156.783543] niks3-server[1040]: 2026/09/21 12:35:29 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo6107worker2 # [ 156.795886] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:29Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=16108worker1 # [ 156.866709] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=10 outputs=16109worker2 # [ 156.803495] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:29Z level=debug event=want client=5 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=06110worker1 # [ 156.888630] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349946111worker1 # [ 156.892571] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349946112worker1 # [ 156.897609] nix-daemon[855]: reaped child process 1949, status = succeeded6113worker1 # [ 156.903913] nix-grpc-daemon[1821]: time=2026-09-21T12:35:29.544Z level=INFO msg="All 1 paths already cached"6114worker1 # [ 156.910939] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:349946115worker1 # [ 156.921381] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6116worker1 # [ 157.117362] nix-daemon[855]: accepted connection from pid 1974, user nix-grpc-daemon (trusted)6117worker1 # [ 157.135441] nix-daemon[855]: reaped child process 1977, status = succeeded6118worker1 # [ 157.314363] nix-daemon[855]: accepted connection from pid 1979, user nix-grpc-daemon (trusted)6119worker1 # [ 157.332597] nix-daemon[855]: reaped child process 1982, status = succeeded6120lb # [ 157.272103] niks3-server[1040]: 2026/09/21 12:35:29 INFO Received uploads request method=POST path=/api/pending_closures6121worker1 # [ 157.353727] nix-grpc-daemon[1821]: time=2026-09-21T12:35:29.994Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6122worker1 # [ 157.355879] nix-grpc-daemon[1821]: time=2026-09-21T12:35:29.996Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)"6123lb # [ 157.302779] niks3-server[1040]: 2026/09/21 12:35:29 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst6124worker1 # [ 157.386717] nix-grpc-daemon[1821]: time=2026-09-21T12:35:30.026Z level=INFO msg="Uploading 1 narinfos"6125lb # [ 157.319234] niks3-server[1040]: 2026/09/21 12:35:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6126lb # [ 157.329429] niks3-server[1040]: 2026/09/21 12:35:29 INFO Signed narinfos id=54 count=16127lb # [ 157.333452] niks3-server[1040]: 2026/09/21 12:35:29 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls6128lb # [ 157.342231] niks3-server[1040]: 2026/09/21 12:35:29 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6129lb # [ 157.347818] niks3-server[1040]: 2026/09/21 12:35:29 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo6130lb # [ 157.352669] niks3-server[1040]: 2026/09/21 12:35:30 INFO Completed upload id=546131worker1 # [ 157.425878] nix-grpc-daemon[1821]: time=2026-09-21T12:35:30.066Z level=INFO msg="Upload complete. (491ms)"6132worker1 # [ 157.434904] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=0 paths=1 nar_bytes_in=6246133worker1 # [ 157.444998] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349946134worker1 # [ 157.458476] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6135worker1 # [ 157.465089] nix-daemon[855]: reaped child process 1972, status = succeeded6136worker1 # [ 157.479081] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6137worker1 # [ 158.724710] nix-daemon[855]: reaped child process 1986, status = succeeded6138worker1 # [ 158.913194] nix-daemon[855]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)6139worker1 # [ 158.932377] nix-daemon[855]: reaped child process 2005, status = succeeded6140lb # [ 158.871488] niks3-server[1040]: 2026/09/21 12:35:31 INFO Received uploads request method=POST path=/api/pending_closures6141worker1 # [ 158.956451] nix-grpc-daemon[1821]: time=2026-09-21T12:35:31.596Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6142worker1 # [ 158.961390] nix-grpc-daemon[1821]: time=2026-09-21T12:35:31.598Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)"6143lb # [ 158.915699] niks3-server[1040]: 2026/09/21 12:35:31 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst6144lb # [ 158.928239] niks3-server[1040]: 2026/09/21 12:35:31 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv6145lb # [ 158.935777] niks3-server[1040]: 2026/09/21 12:35:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6146worker1 # [ 159.014196] nix-grpc-daemon[1821]: time=2026-09-21T12:35:31.654Z level=INFO msg="Uploading 1 narinfos"6147lb # [ 158.945196] niks3-server[1040]: 2026/09/21 12:35:31 INFO Signed narinfos id=55 count=16148lb # [ 158.951222] niks3-server[1040]: 2026/09/21 12:35:31 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls6149lb # [ 158.959655] niks3-server[1040]: 2026/09/21 12:35:31 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6150lb # [ 158.967815] niks3-server[1040]: 2026/09/21 12:35:31 INFO Completed upload id=556151worker1 # [ 159.040976] nix-grpc-daemon[1821]: time=2026-09-21T12:35:31.681Z level=INFO msg="Upload complete. (346ms)"6152lb # [ 158.970938] niks3-server[1040]: 2026/09/21 12:35:31 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo6153worker2 # [ 158.982454] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:31Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=16154worker1 # [ 159.052713] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=11 outputs=16155worker2 # [ 158.991972] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:31Z level=debug event=want client=5 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=06156worker1 # [ 159.075336] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349946157worker1 # [ 159.081195] nix-daemon[855]: reaped child process 1984, status = succeeded6158worker1 # [ 159.089129] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349946159worker1 # [ 159.101910] nix-grpc-daemon[1821]: time=2026-09-21T12:35:31.742Z level=INFO msg="All 1 paths already cached"6160worker1 # [ 159.110496] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:349946161worker1 # [ 159.124522] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6162worker1 # [ 159.313583] nix-daemon[855]: accepted connection from pid 2009, user nix-grpc-daemon (trusted)6163worker1 # [ 159.331922] nix-daemon[855]: reaped child process 2012, status = succeeded6164worker1 # [ 159.505120] nix-daemon[855]: accepted connection from pid 2014, user nix-grpc-daemon (trusted)6165worker1 # [ 159.522870] nix-daemon[855]: reaped child process 2017, status = succeeded6166lb # [ 159.461012] niks3-server[1040]: 2026/09/21 12:35:32 INFO Received uploads request method=POST path=/api/pending_closures6167worker1 # [ 159.545164] nix-grpc-daemon[1821]: time=2026-09-21T12:35:32.185Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6168worker1 # [ 159.546648] nix-grpc-daemon[1821]: time=2026-09-21T12:35:32.185Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)"6169lb # [ 159.489359] niks3-server[1040]: 2026/09/21 12:35:32 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst6170worker1 # [ 159.576415] nix-grpc-daemon[1821]: time=2026-09-21T12:35:32.216Z level=INFO msg="Uploading 1 narinfos"6171lb # [ 159.507002] niks3-server[1040]: 2026/09/21 12:35:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6172lb # [ 159.515598] niks3-server[1040]: 2026/09/21 12:35:32 INFO Signed narinfos id=56 count=16173lb # [ 159.524298] niks3-server[1040]: 2026/09/21 12:35:32 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls6174lb # [ 159.527509] niks3-server[1040]: 2026/09/21 12:35:32 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6175lb # [ 159.533152] niks3-server[1040]: 2026/09/21 12:35:32 INFO Completed upload id=566176worker1 # [ 159.606173] nix-grpc-daemon[1821]: time=2026-09-21T12:35:32.246Z level=INFO msg="Upload complete. (467ms)"6177lb # [ 159.535932] niks3-server[1040]: 2026/09/21 12:35:32 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo6178worker1 # [ 159.612259] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=0 paths=1 nar_bytes_in=6246179worker1 # [ 159.624974] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349946180worker1 # [ 159.632525] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6181worker1 # [ 159.637372] nix-daemon[855]: reaped child process 2007, status = succeeded6182worker1 # [ 159.650875] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6183worker1 # [ 160.698400] nix-daemon[855]: reaped child process 2021, status = succeeded6184worker1 # [ 160.881177] nix-daemon[855]: accepted connection from pid 2037, user nix-grpc-daemon (trusted)6185worker1 # [ 160.897980] nix-daemon[855]: reaped child process 2040, status = succeeded6186lb # [ 160.837767] niks3-server[1040]: 2026/09/21 12:35:33 INFO Received uploads request method=POST path=/api/pending_closures6187worker1 # [ 160.927880] nix-grpc-daemon[1821]: time=2026-09-21T12:35:33.565Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6188worker1 # [ 160.929614] nix-grpc-daemon[1821]: time=2026-09-21T12:35:33.565Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)"6189lb # [ 160.873485] niks3-server[1040]: 2026/09/21 12:35:33 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv6190lb # [ 160.887149] niks3-server[1040]: 2026/09/21 12:35:33 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst6191lb # [ 160.899760] niks3-server[1040]: 2026/09/21 12:35:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6192worker1 # [ 160.978105] nix-grpc-daemon[1821]: time=2026-09-21T12:35:33.618Z level=INFO msg="Uploading 1 narinfos"6193lb # [ 160.908765] niks3-server[1040]: 2026/09/21 12:35:33 INFO Signed narinfos id=57 count=16194lb # [ 160.910917] niks3-server[1040]: 2026/09/21 12:35:33 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls6195lb # [ 160.920230] niks3-server[1040]: 2026/09/21 12:35:33 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6196lb # [ 160.924867] niks3-server[1040]: 2026/09/21 12:35:33 INFO Completed upload id=576197worker1 # [ 160.997976] nix-grpc-daemon[1821]: time=2026-09-21T12:35:33.638Z level=INFO msg="Upload complete. (336ms)"6198lb # [ 160.927177] niks3-server[1040]: 2026/09/21 12:35:33 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo6199worker2 # [ 160.938724] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:33Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=16200worker1 # [ 161.009673] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=12 outputs=16201worker2 # [ 160.946445] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:33Z level=debug event=want client=5 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=06202worker1 # [ 161.028741] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349946203worker1 # [ 161.036241] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349946204worker1 # [ 161.041719] nix-daemon[855]: reaped child process 2019, status = succeeded6205worker1 # [ 161.044863] nix-grpc-daemon[1821]: time=2026-09-21T12:35:33.685Z level=INFO msg="All 2 paths already cached"6206worker1 # [ 161.052163] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:349946207worker1 # [ 161.062804] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6208worker1 # [ 161.250162] nix-daemon[855]: accepted connection from pid 2044, user nix-grpc-daemon (trusted)6209worker1 # [ 161.266600] nix-daemon[855]: reaped child process 2047, status = succeeded6210worker1 # [ 161.441617] nix-daemon[855]: accepted connection from pid 2049, user nix-grpc-daemon (trusted)6211worker1 # [ 161.458163] nix-daemon[855]: reaped child process 2052, status = succeeded6212lb # [ 161.396417] niks3-server[1040]: 2026/09/21 12:35:34 INFO Received uploads request method=POST path=/api/pending_closures6213worker1 # [ 161.478552] nix-grpc-daemon[1821]: time=2026-09-21T12:35:34.118Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6214worker1 # [ 161.480336] nix-grpc-daemon[1821]: time=2026-09-21T12:35:34.118Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)"6215lb # [ 161.423714] niks3-server[1040]: 2026/09/21 12:35:34 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst6216worker1 # [ 161.507247] nix-grpc-daemon[1821]: time=2026-09-21T12:35:34.147Z level=INFO msg="Uploading 1 narinfos"6217lb # [ 161.439260] niks3-server[1040]: 2026/09/21 12:35:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6218lb # [ 161.449120] niks3-server[1040]: 2026/09/21 12:35:34 INFO Signed narinfos id=58 count=16219lb # [ 161.457166] niks3-server[1040]: 2026/09/21 12:35:34 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls6220lb # [ 161.461374] niks3-server[1040]: 2026/09/21 12:35:34 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6221worker1 # [ 161.538505] nix-grpc-daemon[1821]: time=2026-09-21T12:35:34.179Z level=INFO msg="Upload complete. (463ms)"6222lb # [ 161.466073] niks3-server[1040]: 2026/09/21 12:35:34 INFO Completed upload id=586223lb # [ 161.469867] niks3-server[1040]: 2026/09/21 12:35:34 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo6224worker1 # [ 161.544209] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=0 paths=1 nar_bytes_in=7526225worker1 # [ 161.552542] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349946226worker1 # [ 161.564162] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6227worker1 # [ 161.566990] nix-daemon[855]: reaped child process 2042, status = succeeded6228worker1 # [ 161.577324] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6229worker1 # [ 162.817800] nix-daemon[855]: reaped child process 2056, status = succeeded6230worker1 # [ 162.986199] nix-daemon[855]: accepted connection from pid 2072, user nix-grpc-daemon (trusted)6231worker1 # [ 163.003253] nix-daemon[855]: reaped child process 2075, status = succeeded6232lb # [ 162.942946] niks3-server[1040]: 2026/09/21 12:35:35 INFO Received uploads request method=POST path=/api/pending_closures6233worker1 # [ 163.028409] nix-grpc-daemon[1821]: time=2026-09-21T12:35:35.665Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6234worker1 # [ 163.029901] nix-grpc-daemon[1821]: time=2026-09-21T12:35:35.665Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)"6235lb # [ 162.988334] niks3-server[1040]: 2026/09/21 12:35:35 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst6236lb # [ 162.996719] niks3-server[1040]: 2026/09/21 12:35:35 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv6237worker1 # [ 163.078983] nix-grpc-daemon[1821]: time=2026-09-21T12:35:35.719Z level=INFO msg="Uploading 1 narinfos"6238lb # [ 163.006078] niks3-server[1040]: 2026/09/21 12:35:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6239lb # [ 163.015457] niks3-server[1040]: 2026/09/21 12:35:35 INFO Signed narinfos id=59 count=16240lb # [ 163.020505] niks3-server[1040]: 2026/09/21 12:35:35 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls6241lb # [ 163.029140] niks3-server[1040]: 2026/09/21 12:35:35 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6242lb # [ 163.034572] niks3-server[1040]: 2026/09/21 12:35:35 INFO Completed upload id=596243worker1 # [ 163.107753] nix-grpc-daemon[1821]: time=2026-09-21T12:35:35.748Z level=INFO msg="Upload complete. (318ms)"6244lb # [ 163.037911] niks3-server[1040]: 2026/09/21 12:35:35 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo6245worker2 # [ 163.046410] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:35Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=16246worker1 # [ 163.118962] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=13 outputs=16247worker1 # [ 163.134371] nix-daemon[855]: reaped child process 2054, status = succeeded6248worker1 # [ 163.161279] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349946249worker1 # [ 163.175294] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349946250worker1 # [ 163.189331] nix-grpc-daemon[1821]: time=2026-09-21T12:35:35.829Z level=INFO msg="All 2 paths already cached"6251worker1 # [ 163.197742] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:349946252worker1 # [ 163.211562] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6253worker1 # [ 163.412141] nix-daemon[855]: accepted connection from pid 2079, user nix-grpc-daemon (trusted)6254worker1 # [ 163.431703] nix-daemon[855]: reaped child process 2082, status = succeeded6255worker1 # [ 163.602849] nix-daemon[855]: accepted connection from pid 2084, user nix-grpc-daemon (trusted)6256worker1 # [ 163.621473] nix-daemon[855]: reaped child process 2087, status = succeeded6257lb # [ 163.562791] niks3-server[1040]: 2026/09/21 12:35:36 INFO Received uploads request method=POST path=/api/pending_closures6258worker1 # [ 163.644100] nix-grpc-daemon[1821]: time=2026-09-21T12:35:36.284Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6259worker1 # [ 163.645623] nix-grpc-daemon[1821]: time=2026-09-21T12:35:36.284Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)"6260lb # [ 163.592746] niks3-server[1040]: 2026/09/21 12:35:36 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst6261worker1 # [ 163.677370] nix-grpc-daemon[1821]: time=2026-09-21T12:35:36.317Z level=INFO msg="Uploading 1 narinfos"6262lb # [ 163.608662] niks3-server[1040]: 2026/09/21 12:35:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6263lb # [ 163.619116] niks3-server[1040]: 2026/09/21 12:35:36 INFO Signed narinfos id=60 count=16264lb # [ 163.623048] niks3-server[1040]: 2026/09/21 12:35:36 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls6265lb # [ 163.631031] niks3-server[1040]: 2026/09/21 12:35:36 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6266worker1 # [ 163.708395] nix-grpc-daemon[1821]: time=2026-09-21T12:35:36.348Z level=INFO msg="Upload complete. (481ms)"6267lb # [ 163.636047] niks3-server[1040]: 2026/09/21 12:35:36 INFO Completed upload id=606268lb # [ 163.639001] niks3-server[1040]: 2026/09/21 12:35:36 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo6269worker1 # [ 163.716149] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=0 paths=1 nar_bytes_in=7526270worker1 # [ 163.725827] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349946271worker1 # [ 163.735413] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6272worker1 # [ 163.740232] nix-daemon[855]: reaped child process 2077, status = succeeded6273worker1 # [ 163.753219] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6274worker1 # [ 164.591823] nix-daemon[855]: reaped child process 2091, status = succeeded6275worker1 # [ 164.756524] nix-daemon[855]: accepted connection from pid 2107, user nix-grpc-daemon (trusted)6276worker1 # [ 164.773445] nix-daemon[855]: reaped child process 2110, status = succeeded6277lb # [ 164.714965] niks3-server[1040]: 2026/09/21 12:35:37 INFO Received uploads request method=POST path=/api/pending_closures6278worker1 # [ 164.799180] nix-grpc-daemon[1821]: time=2026-09-21T12:35:37.437Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6279worker1 # [ 164.800956] nix-grpc-daemon[1821]: time=2026-09-21T12:35:37.437Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)"6280lb # [ 164.756183] niks3-server[1040]: 2026/09/21 12:35:37 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst6281lb # [ 164.766277] niks3-server[1040]: 2026/09/21 12:35:37 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv6282worker1 # [ 164.845979] nix-grpc-daemon[1821]: time=2026-09-21T12:35:37.486Z level=INFO msg="Uploading 1 narinfos"6283lb # [ 164.775720] niks3-server[1040]: 2026/09/21 12:35:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6284lb # [ 164.784371] niks3-server[1040]: 2026/09/21 12:35:37 INFO Signed narinfos id=61 count=16285lb # [ 164.787062] niks3-server[1040]: 2026/09/21 12:35:37 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls6286lb # [ 164.795502] niks3-server[1040]: 2026/09/21 12:35:37 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6287lb # [ 164.802613] niks3-server[1040]: 2026/09/21 12:35:37 INFO Completed upload id=616288worker1 # [ 164.875773] nix-grpc-daemon[1821]: time=2026-09-21T12:35:37.516Z level=INFO msg="Upload complete. (313ms)"6289lb # [ 164.805634] niks3-server[1040]: 2026/09/21 12:35:37 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo6290worker2 # [ 164.814162] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:37Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=16291worker1 # [ 164.889199] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=14 outputs=16292worker1 # [ 164.906224] nix-daemon[855]: reaped child process 2089, status = succeeded6293worker1 # [ 164.929291] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349946294worker1 # [ 164.944950] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349946295worker1 # [ 164.959383] nix-grpc-daemon[1821]: time=2026-09-21T12:35:37.599Z level=INFO msg="All 2 paths already cached"6296worker1 # [ 164.968127] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:349946297worker1 # [ 164.983147] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6298worker1 # [ 165.180094] nix-daemon[855]: accepted connection from pid 2114, user nix-grpc-daemon (trusted)6299worker1 # [ 165.199397] nix-daemon[855]: reaped child process 2117, status = succeeded6300worker1 # [ 165.374770] nix-daemon[855]: accepted connection from pid 2119, user nix-grpc-daemon (trusted)6301worker1 # [ 165.391980] nix-daemon[855]: reaped child process 2122, status = succeeded6302lb # [ 165.330767] niks3-server[1040]: 2026/09/21 12:35:37 INFO Received uploads request method=POST path=/api/pending_closures6303worker1 # [ 165.412076] nix-grpc-daemon[1821]: time=2026-09-21T12:35:38.052Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6304worker1 # [ 165.413562] nix-grpc-daemon[1821]: time=2026-09-21T12:35:38.052Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)"6305lb # [ 165.362727] niks3-server[1040]: 2026/09/21 12:35:38 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst6306lb # [ 165.375477] niks3-server[1040]: 2026/09/21 12:35:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6307worker1 # [ 165.455848] nix-grpc-daemon[1821]: time=2026-09-21T12:35:38.096Z level=INFO msg="Uploading 1 narinfos"6308lb # [ 165.382990] niks3-server[1040]: 2026/09/21 12:35:38 INFO Signed narinfos id=62 count=16309lb # [ 165.390451] niks3-server[1040]: 2026/09/21 12:35:38 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls6310lb # [ 165.401230] niks3-server[1040]: 2026/09/21 12:35:38 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6311lb # [ 165.406636] niks3-server[1040]: 2026/09/21 12:35:38 INFO Completed upload id=626312worker1 # [ 165.479719] nix-grpc-daemon[1821]: time=2026-09-21T12:35:38.120Z level=INFO msg="Upload complete. (479ms)"6313lb # [ 165.409473] niks3-server[1040]: 2026/09/21 12:35:38 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo6314worker1 # [ 165.488279] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=0 paths=1 nar_bytes_in=7526315worker1 # [ 165.502481] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:349946316worker1 # [ 165.509109] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6317worker1 # [ 165.514209] nix-daemon[855]: reaped child process 2112, status = succeeded6318worker1 # [ 165.527975] nix-daemon[855]: accepted connection from pid 1761, user nix-grpc-daemon (trusted)6319worker1 # [ 166.967540] nix-daemon[855]: reaped child process 2126, status = succeeded6320worker1 # [ 167.149823] nix-daemon[855]: accepted connection from pid 2142, user nix-grpc-daemon (trusted)6321worker1 # [ 167.166467] nix-daemon[855]: reaped child process 2145, status = succeeded6322lb # [ 167.105925] niks3-server[1040]: 2026/09/21 12:35:39 INFO Received uploads request method=POST path=/api/pending_closures6323worker1 # [ 167.192287] nix-grpc-daemon[1821]: time=2026-09-21T12:35:39.828Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6324worker1 # [ 167.195228] nix-grpc-daemon[1821]: time=2026-09-21T12:35:39.828Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)"6325lb # [ 167.150266] niks3-server[1040]: 2026/09/21 12:35:39 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst6326worker1 # [ 167.232729] nix-grpc-daemon[1821]: time=2026-09-21T12:35:39.872Z level=INFO msg="Uploading 1 narinfos"6327lb # [ 167.159721] niks3-server[1040]: 2026/09/21 12:35:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6328lb # [ 167.170722] niks3-server[1040]: 2026/09/21 12:35:39 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv6329lb # [ 167.178686] niks3-server[1040]: 2026/09/21 12:35:39 INFO Signed narinfos id=63 count=16330lb # [ 167.185972] niks3-server[1040]: 2026/09/21 12:35:39 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls6331worker1 # [ 167.263626] nix-grpc-daemon[1821]: time=2026-09-21T12:35:39.904Z level=INFO msg="Upload complete. (325ms)"6332lb # [ 167.191153] niks3-server[1040]: 2026/09/21 12:35:39 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6333lb # [ 167.198225] niks3-server[1040]: 2026/09/21 12:35:39 INFO Completed upload id=636334lb # [ 167.201603] niks3-server[1040]: 2026/09/21 12:35:39 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo6335worker1 # [ 167.277193] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34994 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=15 outputs=16336worker1 # [ 167.294646] nix-daemon[855]: reaped child process 2124, status = succeeded6337client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 21.18 seconds)6338worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6339worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6340client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features '["gpu"]'6341client # [ 167.445364] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6342client # Running as unit: gpu.service; invocation ID: 212918432bcb4481896406accb9ffb646343client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds)6344lb: must succeed: curl -sf localhost:9901/clusters6345lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6346lb: must succeed: curl -sf localhost:9901/clusters6347lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6348worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6349worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)6350client # [ 167.693602] nix[1340]: don't know how to build these paths:6351client # [ 167.695247] nix[1340]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv6352worker2 # [ 167.685794] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:372586353worker1 # [ 167.761685] nix-grpc-daemon[1761]: ts=2026-09-21T12:35:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:349946354client # [ 167.722527] nix[1340]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, waiting up to 30s for a worker6355worker2 # [ 167.714674] nix-grpc-daemon[1340]: ts=2026-09-21T12:35:40Z level=debug event=want client=6 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=06356lb: must succeed: curl -sf localhost:9901/clusters6357lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6358lb: must succeed: curl -sf localhost:9901/clusters6359lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6360worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6361worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6362client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6363lb # [ 169.892418] envoy[902]: conn peer=[2001:db8:1::1]:60878 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27161 tx=11443 ms=208356364client # [ 197.725252] nix[1340]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}6365client # [ 197.737712] nix[1340]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed6366client # [ 197.777045] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6367client # [ 197.784696] systemd[1]: gpu.service: Failed with result 'exit-code'.6368client # [ 197.787962] systemd[1]: gpu.service: Consumed 172ms CPU time over 30.337s wall clock time, 37.5M memory peak, 5.1K incoming IP traffic, 5K outgoing IP traffic.6369client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.26 seconds)6370client: must succeed: journalctl -u gpu -o cat6371client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)6372lb: must succeed: curl -sf localhost:9901/clusters6373lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6374lb: must succeed: curl -sf localhost:9901/clusters6375lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6376worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6377worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6378(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 52.27 seconds)6379subtest: low disk drains a worker and builds go to the other6380worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6381worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.10 seconds)6382worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6383worker1 # [ 199.858654] nix-grpc-daemon[1761]: ts=2026-09-21T12:36:12Z level=info event=unhealthy reason=min_free6384lb # [ 199.890505] envoy[902]: conn peer=[2001:db8:1::1]:50008 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=897 tx=499 ms=300546385worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.13 seconds)6386client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain >&26387client # don't know how to build these paths:6388client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6389worker1 # [ 200.944849] nix-grpc-daemon[1761]: ts=2026-09-21T12:36:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:349946390worker2 # [ 200.878915] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:372586391worker2 # [ 200.905080] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:13Z level=debug event=want client=7 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=06392worker2 # [ 200.911655] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:13Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=16393worker2 # [ 200.948883] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372586394worker2 # [ 200.963741] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:372586395client # copying 1 paths to 'lb:50051'...6396client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'...6397worker2 # [ 200.980253] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:372586398worker2 # [ 200.995998] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)6399worker2 # [ 201.211396] nix-daemon[847]: accepted connection from pid 1639, user nix-grpc-daemon (trusted)6400worker2 # [ 201.229126] nix-daemon[847]: reaped child process 1642, status = succeeded6401worker2 # [ 201.405666] nix-daemon[847]: accepted connection from pid 1644, user nix-grpc-daemon (trusted)6402worker2 # [ 201.421793] nix-daemon[847]: reaped child process 1647, status = succeeded6403lb # [ 201.434021] niks3-server[1040]: 2026/09/21 12:36:14 INFO Received uploads request method=POST path=/api/pending_closures6404worker2 # [ 201.445167] nix-grpc-daemon[1416]: time=2026-09-21T12:36:14.079Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6405worker2 # [ 201.446626] nix-grpc-daemon[1416]: time=2026-09-21T12:36:14.079Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)"6406lb # [ 201.471606] niks3-server[1040]: 2026/09/21 12:36:14 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst6407worker2 # [ 201.484851] nix-grpc-daemon[1416]: time=2026-09-21T12:36:14.119Z level=INFO msg="Uploading 1 narinfos"6408lb # [ 201.484921] niks3-server[1040]: 2026/09/21 12:36:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6409lb # [ 201.495477] niks3-server[1040]: 2026/09/21 12:36:14 INFO Signed narinfos id=64 count=16410lb # [ 201.503277] niks3-server[1040]: 2026/09/21 12:36:14 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls6411worker2 # [ 201.509869] nix-grpc-daemon[1416]: time=2026-09-21T12:36:14.144Z level=INFO msg="Upload complete. (496ms)"6412lb # [ 201.509220] niks3-server[1040]: 2026/09/21 12:36:14 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6413lb # [ 201.514874] niks3-server[1040]: 2026/09/21 12:36:14 INFO Completed upload id=646414lb # [ 201.518327] niks3-server[1040]: 2026/09/21 12:36:14 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo6415worker2 # [ 201.520122] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37258 duration_s=0 paths=1 nar_bytes_in=5126416worker2 # [ 201.530125] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372586417worker2 # [ 201.536840] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)6418worker2 # [ 201.541366] nix-daemon[847]: reaped child process 1637, status = succeeded6419client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051...6420client # farm-a> worker2: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6421worker2 # [ 201.553306] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)6422client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6423client # farm-a> LOG-farm-a-drain6424worker2 # [ 203.184200] nix-daemon[847]: reaped child process 1651, status = succeeded6425worker2 # [ 203.351753] nix-daemon[847]: accepted connection from pid 1667, user nix-grpc-daemon (trusted)6426worker2 # [ 203.369721] nix-daemon[847]: reaped child process 1670, status = succeeded6427lb # [ 203.381176] niks3-server[1040]: 2026/09/21 12:36:16 INFO Received uploads request method=POST path=/api/pending_closures6428worker2 # [ 203.392376] nix-grpc-daemon[1416]: time=2026-09-21T12:36:16.026Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6429worker2 # [ 203.393854] nix-grpc-daemon[1416]: time=2026-09-21T12:36:16.026Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)"6430lb # [ 203.414944] niks3-server[1040]: 2026/09/21 12:36:16 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6431lb # [ 203.431636] niks3-server[1040]: 2026/09/21 12:36:16 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6432lb # [ 203.440080] niks3-server[1040]: 2026/09/21 12:36:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6433worker2 # [ 203.447651] nix-grpc-daemon[1416]: time=2026-09-21T12:36:16.082Z level=INFO msg="Uploading 1 narinfos"6434lb # [ 203.448951] niks3-server[1040]: 2026/09/21 12:36:16 INFO Signed narinfos id=65 count=16435lb # [ 203.451811] niks3-server[1040]: 2026/09/21 12:36:16 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls6436lb # [ 203.459497] niks3-server[1040]: 2026/09/21 12:36:16 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6437lb # [ 203.466401] niks3-server[1040]: 2026/09/21 12:36:16 INFO Completed upload id=656438worker2 # [ 203.467954] nix-grpc-daemon[1416]: time=2026-09-21T12:36:16.102Z level=INFO msg="Upload complete. (304ms)"6439lb # [ 203.468783] niks3-server[1040]: 2026/09/21 12:36:16 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo6440worker2 # [ 203.484851] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37258 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=16 outputs=16441worker2 # [ 203.498770] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:16Z level=debug event=want client=7 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=06442worker2 # [ 203.502526] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:16Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=16443worker2 # [ 203.506179] nix-daemon[847]: reaped child process 1649, status = succeeded6444worker2 # [ 203.525305] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372586445worker2 # [ 203.539943] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:372586446worker2 # [ 203.553009] nix-grpc-daemon[1416]: time=2026-09-21T12:36:16.187Z level=INFO msg="All 1 paths already cached"6447client # copying 1 paths to 'lb:50051'...6448client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'...6449worker2 # [ 203.562592] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:372586450worker2 # [ 203.576114] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)6451worker2 # [ 203.772290] nix-daemon[847]: accepted connection from pid 1676, user nix-grpc-daemon (trusted)6452worker2 # [ 203.787197] nix-daemon[847]: reaped child process 1679, status = succeeded6453worker2 # [ 203.964040] nix-daemon[847]: accepted connection from pid 1681, user nix-grpc-daemon (trusted)6454worker2 # [ 203.979035] nix-daemon[847]: reaped child process 1684, status = succeeded6455lb # [ 203.993606] niks3-server[1040]: 2026/09/21 12:36:16 INFO Received uploads request method=POST path=/api/pending_closures6456worker2 # [ 204.003751] nix-grpc-daemon[1416]: time=2026-09-21T12:36:16.638Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6457worker2 # [ 204.005423] nix-grpc-daemon[1416]: time=2026-09-21T12:36:16.638Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)"6458lb # [ 204.023788] niks3-server[1040]: 2026/09/21 12:36:16 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst6459worker2 # [ 204.041038] nix-grpc-daemon[1416]: time=2026-09-21T12:36:16.675Z level=INFO msg="Uploading 1 narinfos"6460lb # [ 204.033420] niks3-server[1040]: 2026/09/21 12:36:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign6461lb # [ 204.045016] niks3-server[1040]: 2026/09/21 12:36:16 INFO Signed narinfos id=66 count=16462lb # [ 204.047866] niks3-server[1040]: 2026/09/21 12:36:16 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls6463lb # [ 204.056073] niks3-server[1040]: 2026/09/21 12:36:16 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete6464lb # [ 204.063965] niks3-server[1040]: 2026/09/21 12:36:16 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo6465lb # [ 204.067953] niks3-server[1040]: 2026/09/21 12:36:16 INFO Completed upload id=666466worker2 # [ 204.069448] nix-grpc-daemon[1416]: time=2026-09-21T12:36:16.704Z level=INFO msg="Upload complete. (478ms)"6467worker2 # [ 204.077073] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37258 duration_s=0 paths=1 nar_bytes_in=6326468worker2 # [ 204.090674] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372586469worker2 # [ 204.099844] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)6470worker2 # [ 204.102277] nix-daemon[847]: reaped child process 1674, status = succeeded6471client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051...6472worker2 # [ 204.115851] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)6473client # farm-b> worker2: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6474client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6475client # farm-b> LOG-farm-b-drain6476worker2 # [ 205.564766] nix-daemon[847]: reaped child process 1688, status = succeeded6477worker2 # [ 205.742012] nix-daemon[847]: accepted connection from pid 1704, user nix-grpc-daemon (trusted)6478worker2 # [ 205.759317] nix-daemon[847]: reaped child process 1707, status = succeeded6479lb # [ 205.770564] niks3-server[1040]: 2026/09/21 12:36:18 INFO Received uploads request method=POST path=/api/pending_closures6480worker2 # [ 205.784467] nix-grpc-daemon[1416]: time=2026-09-21T12:36:18.416Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6481worker2 # [ 205.786275] nix-grpc-daemon[1416]: time=2026-09-21T12:36:18.416Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)"6482lb # [ 205.814168] niks3-server[1040]: 2026/09/21 12:36:18 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst6483lb # [ 205.827510] niks3-server[1040]: 2026/09/21 12:36:18 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6484lb # [ 205.836295] niks3-server[1040]: 2026/09/21 12:36:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign6485worker2 # [ 205.844381] nix-grpc-daemon[1416]: time=2026-09-21T12:36:18.478Z level=INFO msg="Uploading 1 narinfos"6486lb # [ 205.842450] niks3-server[1040]: 2026/09/21 12:36:18 INFO Signed narinfos id=67 count=16487lb # [ 205.847444] niks3-server[1040]: 2026/09/21 12:36:18 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls6488lb # [ 205.860489] niks3-server[1040]: 2026/09/21 12:36:18 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete6489lb # [ 205.866012] niks3-server[1040]: 2026/09/21 12:36:18 INFO Completed upload id=676490worker2 # [ 205.867589] nix-grpc-daemon[1416]: time=2026-09-21T12:36:18.502Z level=INFO msg="Upload complete. (339ms)"6491lb # [ 205.869102] niks3-server[1040]: 2026/09/21 12:36:18 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo6492worker2 # [ 205.881139] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37258 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=17 outputs=16493worker2 # [ 205.898602] nix-daemon[847]: reaped child process 1686, status = succeeded6494worker2 # [ 205.927389] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:18Z level=debug event=want client=7 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=06495worker2 # [ 205.934101] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:18Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=16496worker2 # [ 205.974739] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372586497worker2 # [ 205.992159] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:372586498worker2 # [ 206.006464] nix-grpc-daemon[1416]: time=2026-09-21T12:36:18.641Z level=INFO msg="All 2 paths already cached"6499client # copying 1 paths to 'lb:50051'...6500client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'...6501worker2 # [ 206.015950] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:372586502worker2 # [ 206.030372] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)6503worker2 # [ 206.236486] nix-daemon[847]: accepted connection from pid 1713, user nix-grpc-daemon (trusted)6504worker2 # [ 206.255985] nix-daemon[847]: reaped child process 1716, status = succeeded6505worker2 # [ 206.426457] nix-daemon[847]: accepted connection from pid 1718, user nix-grpc-daemon (trusted)6506worker2 # [ 206.445145] nix-daemon[847]: reaped child process 1721, status = succeeded6507lb # [ 206.455328] niks3-server[1040]: 2026/09/21 12:36:19 INFO Received uploads request method=POST path=/api/pending_closures6508worker2 # [ 206.465872] nix-grpc-daemon[1416]: time=2026-09-21T12:36:19.100Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6509worker2 # [ 206.467359] nix-grpc-daemon[1416]: time=2026-09-21T12:36:19.100Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)"6510lb # [ 206.480842] niks3-server[1040]: 2026/09/21 12:36:19 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst6511worker2 # [ 206.494567] nix-grpc-daemon[1416]: time=2026-09-21T12:36:19.129Z level=INFO msg="Uploading 1 narinfos"6512lb # [ 206.497212] niks3-server[1040]: 2026/09/21 12:36:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign6513lb # [ 206.507128] niks3-server[1040]: 2026/09/21 12:36:19 INFO Signed narinfos id=68 count=16514lb # [ 206.512245] niks3-server[1040]: 2026/09/21 12:36:19 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls6515lb # [ 206.519947] niks3-server[1040]: 2026/09/21 12:36:19 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete6516lb # [ 206.524216] niks3-server[1040]: 2026/09/21 12:36:19 INFO Completed upload id=686517worker2 # [ 206.525508] nix-grpc-daemon[1416]: time=2026-09-21T12:36:19.160Z level=INFO msg="Upload complete. (477ms)"6518lb # [ 206.526995] niks3-server[1040]: 2026/09/21 12:36:19 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo6519worker2 # [ 206.532123] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37258 duration_s=0 paths=1 nar_bytes_in=7606520worker2 # [ 206.544195] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:372586521worker2 # [ 206.550382] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)6522worker2 # [ 206.554897] nix-daemon[847]: reaped child process 1711, status = succeeded6523client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051...6524client # farm-top> worker2: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6525worker2 # [ 206.568584] nix-daemon[847]: accepted connection from pid 1340, user nix-grpc-daemon (trusted)6526client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6527worker2 # [ 207.591281] nix-daemon[847]: reaped child process 1725, status = succeeded6528client # farm-top> LOG-farm-top-drain6529worker2 # [ 207.744702] nix-daemon[847]: accepted connection from pid 1741, user nix-grpc-daemon (trusted)6530worker2 # [ 207.762334] nix-daemon[847]: reaped child process 1744, status = succeeded6531lb # [ 207.778682] niks3-server[1040]: 2026/09/21 12:36:20 INFO Received uploads request method=POST path=/api/pending_closures6532worker2 # [ 207.789678] nix-grpc-daemon[1416]: time=2026-09-21T12:36:20.423Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6533worker2 # [ 207.791174] nix-grpc-daemon[1416]: time=2026-09-21T12:36:20.423Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)"6534lb # [ 207.819579] niks3-server[1040]: 2026/09/21 12:36:20 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst6535worker2 # [ 207.832038] nix-grpc-daemon[1416]: time=2026-09-21T12:36:20.466Z level=INFO msg="Uploading 1 narinfos"6536lb # [ 207.834366] niks3-server[1040]: 2026/09/21 12:36:20 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6537lb # [ 207.842510] niks3-server[1040]: 2026/09/21 12:36:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign6538lb # [ 207.848284] niks3-server[1040]: 2026/09/21 12:36:20 INFO Signed narinfos id=69 count=16539lb # [ 207.850421] niks3-server[1040]: 2026/09/21 12:36:20 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls6540worker2 # [ 207.856137] nix-grpc-daemon[1416]: time=2026-09-21T12:36:20.490Z level=INFO msg="Upload complete. (278ms)"6541lb # [ 207.855354] niks3-server[1040]: 2026/09/21 12:36:20 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete6542lb # [ 207.860523] niks3-server[1040]: 2026/09/21 12:36:20 INFO Completed upload id=696543lb # [ 207.863081] niks3-server[1040]: 2026/09/21 12:36:20 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo6544worker2 # [ 207.865481] nix-grpc-daemon[1340]: ts=2026-09-21T12:36:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37258 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=18 outputs=16545worker2 # [ 207.878801] nix-daemon[847]: reaped child process 1723, status = succeeded6546client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain >&2, in 7.31 seconds)6547client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain outPath6548client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain outPath, in 0.23 seconds)6549worker1: must succeed: rm /nix/.rw-store/fill6550worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.15 seconds)6551worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'6552worker1 # [ 208.867355] nix-grpc-daemon[1761]: ts=2026-09-21T12:36:21Z level=info event=healthy reason=min_free6553worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.12 seconds)6554(finished: subtest: low disk drains a worker and builds go to the other, in 11.16 seconds)6555(finished: run the VM test script, in 210.41 seconds)6556test script finished in 210.71s6557cleanup6558kill QemuMachine (pid 49)6559client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6560lb # [ 209.891639] envoy[902]: conn peer=[2001:db8:1::1]:50500 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9519 tx=4091 ms=70116561kill QemuMachine (pid 50)6562lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6563kill QemuMachine (pid 51)6564worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6565kill QemuMachine (pid 52)6566worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6567(finished: cleanup, in 1.07 seconds)6568additionally exposed symbols:6569 client, lb, worker1, worker2,6570 vlan1,6571 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh