vm-test-run-nix-grpc-farm
checks.aarch64-linux.farm
· build #483
· raw
1tribuchet: building on eliza2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker2 # Disk image does not exist, creating the virtualisation disk image...13worker2 # Formatting '/build/vm-state-worker2/tmp.RX4m3DgMQP', fmt=raw size=107374182414worker2: QEMU running (pid 52)15lb # Disk image does not exist, creating the virtualisation disk image...16lb: QEMU running (pid 50)17worker1 # Disk image does not exist, creating the virtualisation disk image...18lb # Formatting '/build/vm-state-lb/tmp.BMGh4oM2p7', fmt=raw size=107374182419client: QEMU running (pid 49)20client # Disk image does not exist, creating the virtualisation disk image...21worker1: QEMU running (pid 51)22worker2 # mke2fs 1.47.4 (6-Mar-2025)23client # Formatting '/build/vm-state-client/tmp.2GGwe3iwTG', fmt=raw size=107374182424worker1 # Formatting '/build/vm-state-worker1/tmp.vcNYQyNEGo', fmt=raw size=107374182425lb # mke2fs 1.47.4 (6-Mar-2025)26client # mke2fs 1.47.4 (6-Mar-2025)27worker2 # Discarding device blocks: 0/262144 done28(finished: start all VMs, in 0.52 seconds)29worker1 # mke2fs 1.47.4 (6-Mar-2025)30lb # Discarding device blocks: 0/262144 done31client: waiting for unit network-addresses-eth1.service32worker1 # Discarding device blocks: 0/262144 done33lb # Creating filesystem with 262144 4k blocks and 65536 inodes34client: waiting for the VM to finish booting35worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes36lb # Filesystem UUID: c88aeaea-09cc-4382-9997-984451cb02f337worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes38client # Discarding device blocks: 0/262144 done39worker1 # Filesystem UUID: bfe28ab3-f4bb-4f60-b1ac-da25a5b20f1e40lb # Superblock backups stored on blocks:41worker2 # Filesystem UUID: af75b1bc-0c30-4a07-b9d8-a5d001cce35c42client # Creating filesystem with 262144 4k blocks and 65536 inodes43worker1 # Superblock backups stored on blocks:44lb # 32768, 98304, 163840, 22937645worker2 # Superblock backups stored on blocks:46client # Filesystem UUID: ccf360dd-28b6-4d05-9a1a-872ed67b91f147worker1 # 32768, 98304, 163840, 22937648lb # 49worker2 # 32768, 98304, 163840, 22937650client # Superblock backups stored on blocks:51worker1 # 52lb # Allocating group tables: 0/8 done53worker2 # 54client # 32768, 98304, 163840, 22937655worker1 # Allocating group tables: 0/8 done56lb # Writing inode tables: 0/8 done57worker2 # Allocating group tables: 0/8 done58client # 59worker1 # Writing inode tables: 0/8 done60lb # Creating journal (8192 blocks): done61worker2 # Writing inode tables: 0/8 done62client # Allocating group tables: 0/8 done63worker1 # Creating journal (8192 blocks): done64lb # Writing superblocks and filesystem accounting information: 0/8 done65worker2 # Creating journal (8192 blocks): done66client # Writing inode tables: 0/8 done67worker1 # Writing superblocks and filesystem accounting information: 0/8 done68lb # 69worker2 # Writing superblocks and filesystem accounting information: 0/8 done70client # Creating journal (8192 blocks): done71worker1 # 72lb # Virtualisation disk image created.73worker2 # 74client # Writing superblocks and filesystem accounting information: 0/8 done75worker2 # Virtualisation disk image created.76client # 77worker1 # Virtualisation disk image created.78client # Virtualisation disk image created.79worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]80worker2 # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 202681worker2 # [ 0.000000] KASLR enabled82worker2 # [ 0.000000] random: crng init done83worker2 # [ 0.000000] Machine model: linux,dummy-virt84worker2 # [ 0.000000] efi: UEFI not found.85worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT86worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]87worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]88worker2 # [ 0.000000] Zone ranges:89worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]90worker2 # [ 0.000000] DMA32 empty91worker2 # [ 0.000000] Normal empty92worker2 # [ 0.000000] Device empty93worker2 # [ 0.000000] Movable zone start for each node94worker2 # [ 0.000000] Early memory node ranges95worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]96worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]97worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c40000098worker2 # [ 0.000000] psci: probing for conduit method from DT.99worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware.100worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs101worker2 # [ 0.000000] psci: Trusted OS migration not required102worker2 # [ 0.000000] psci: SMC Calling Convention v1.1103worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)104worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296105worker2 # [ 0.000000] Detected PIPT I-cache on CPU0106worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)107worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface108worker2 # [ 0.000000] CPU features: detected: Spectre-v4109worker2 # [ 0.000000] CPU features: detected: Spectre-BHB110worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38111worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23112worker2 # [ 0.000000] alternatives: applying boot alternatives113worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/vb88dwa6ixfld6155zc5rrhk4lp3ydsz-nixos-system-worker2-test/init regInfo=/nix/store/jlkx2iki7vwjabpbg95pq08zjmcv5z23-closure-info/registration console=ttyAMA0,115200n8 console=tty0114worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/jlkx2iki7vwjabpbg95pq08zjmcv5z23-closure-info/registration", will be passed to user space.115worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes116worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)117worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)118worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB119worker2 # [ 0.000000] software IO TLB: area num 1.120worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB121worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)122worker2 # [ 0.000000] Fallback order for Node 0: 0123worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216124worker2 # [ 0.000000] Policy zone: DMA125worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off126worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1127worker2 # [ 0.000000] allocated 3145728 bytes of page_ext128worker2 # [ 0.000000] ftrace: allocating 74885 entries in 294 pages129worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups130worker2 # [ 0.000000] rcu: Hierarchical RCU implementation.131worker2 # [ 0.000000] rcu: RCU event tracing is enabled.132worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.133worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled.134worker2 # [ 0.000000] Rude variant of Tasks RCU enabled.135worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled.136worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.137worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1138worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.139worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.140worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.141worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0142worker2 # [ 0.000000] GICv3: 256 SPIs implemented143worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented144worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq145worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI146worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0147worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000148worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]149worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)150worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)151worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000152worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000153worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.154worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns155worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).156worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns157worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns158worker2 # [ 0.000033] arm-pv: using stolen time PV159worker2 # [ 0.000662] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)160worker2 # [ 0.000815] Console: colour dummy device 80x25161worker2 # [ 0.000823] printk: legacy console [tty0] enabled162worker2 # [ 0.001012] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)163worker2 # [ 0.001019] pid_max: default: 32768 minimum: 301164worker2 # [ 0.001088] LSM: initializing lsm=capability,landlock,yama,bpf,ima165worker2 # [ 0.001290] landlock: Up and running.166worker2 # [ 0.001293] Yama: becoming mindful.167worker2 # [ 0.001911] LSM support for eBPF active168worker2 # [ 0.002046] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)169worker2 # [ 0.002078] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)170worker2 # [ 0.003202] cacheinfo: Unable to detect cache hierarchy for CPU 0171worker2 # [ 0.003916] rcu: Hierarchical SRCU implementation.172worker2 # [ 0.003919] rcu: Max phase no-delay instances is 1000.173worker2 # [ 0.005109] fsl-mc MSI: its@8080000 domain created174worker2 # [ 0.005198] EFI services will not be available.175worker2 # [ 0.005276] smp: Bringing up secondary CPUs ...176worker2 # [ 0.005284] smp: Brought up 1 node, 1 CPU177worker2 # [ 0.005287] SMP: Total of 1 processors activated.178worker2 # [ 0.005292] CPU: All CPU(s) started at EL1179worker2 # [ 0.005304] CPU features: detected: Branch Target Identification180worker2 # [ 0.005309] CPU features: detected: ARMv8.4 Translation Table Level181worker2 # [ 0.005312] CPU features: detected: Instruction cache invalidation not required for I/D coherence182worker2 # [ 0.005315] CPU features: detected: Data cache clean to the PoU not required for I/D coherence183worker2 # [ 0.005319] CPU features: detected: Common not Private translations184worker2 # [ 0.005322] CPU features: detected: CRC32 instructions185worker2 # [ 0.005325] CPU features: detected: Data cache clean to Point of Deep Persistence186worker2 # [ 0.005328] CPU features: detected: Data cache clean to Point of Persistence187worker2 # [ 0.005332] CPU features: detected: Data independent timing control (DIT)188worker2 # [ 0.005335] CPU features: detected: E0PD189worker2 # [ 0.005337] CPU features: detected: Enhanced Counter Virtualization190worker2 # [ 0.005340] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)191worker2 # [ 0.005343] CPU features: detected: Enhanced Virtualization Traps192worker2 # [ 0.005346] CPU features: detected: Fine Grained Traps193worker2 # [ 0.005350] CPU features: detected: Generic authentication (architected QARMA5 algorithm)194worker2 # [ 0.005356] CPU features: detected: RCpc load-acquire (LDAPR)195worker2 # [ 0.005359] CPU features: detected: LSE atomic instructions196worker2 # [ 0.005362] CPU features: detected: Privileged Access Never197worker2 # [ 0.005364] CPU features: detected: PMUv3198worker2 # [ 0.005367] CPU features: detected: RAS Extension Support199worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]200worker2 # [ 0.005370] CPU features: detected: RASv1p1 Extension Support201worker2 # [ 0.005372] CPU features: detected: Random Number Generator202client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]203worker2 # [ 0.005375] CPU features: detected: Speculation barrier (SB)204worker1 # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026205worker1 # [ 0.000000] KASLR enabled206worker2 # [ 0.005378] CPU features: detected: Stage-2 Force Write-Back207worker1 # [ 0.000000] random: crng init done208client # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026209client # [ 0.000000] KASLR enabled210worker1 # [ 0.000000] Machine model: linux,dummy-virt211worker2 # [ 0.005381] CPU features: detected: TLB range maintenance instructions212client # [ 0.000000] random: crng init done213worker1 # [ 0.000000] efi: UEFI not found.214client # [ 0.000000] Machine model: linux,dummy-virt215worker2 # [ 0.005385] CPU features: detected: Speculative Store Bypassing Safe (SSBS)216worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT217client # [ 0.000000] efi: UEFI not found.218worker2 # [ 0.005424] alternatives: applying system-wide alternatives219worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]220client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT221worker2 # [ 0.008359] CPU features: detected: BBM Level 2 without TLB conflict abort222worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]223worker1 # [ 0.000000] Zone ranges:224client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]225client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]226worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]227client # [ 0.000000] Zone ranges:228worker2 # [ 0.008567] Memory: 1406516K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123772K reserved, 32768K cma-reserved)229worker1 # [ 0.000000] DMA32 empty230worker1 # [ 0.000000] Normal empty231worker2 # [ 0.008953] devtmpfs: initialized232client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]233worker1 # [ 0.000000] Device empty234client # [ 0.000000] DMA32 empty235worker2 # [ 0.010776] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)236worker1 # [ 0.000000] Movable zone start for each node237client # [ 0.000000] Normal empty238worker1 # [ 0.000000] Early memory node ranges239client # [ 0.000000] Device empty240worker2 # [ 0.010798] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).241client # [ 0.000000] Movable zone start for each node242worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]243client # [ 0.000000] Early memory node ranges244worker2 # [ 0.010987] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL245worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]246client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]247worker2 # [ 0.010991] 0 pages in range for non-PLT usage248worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000249worker2 # [ 0.010992] 508288 pages in range for PLT usage250client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]251worker1 # [ 0.000000] psci: probing for conduit method from DT.252worker2 # [ 0.011127] pinctrl core: initialized pinctrl subsystem253client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000254worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware.255worker2 # [ 0.011883] DMI not present or invalid.256client # [ 0.000000] psci: probing for conduit method from DT.257worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs258worker2 # [ 0.015594] NET: Registered PF_NETLINK/PF_ROUTE protocol family259client # [ 0.000000] psci: PSCIv1.3 detected in firmware.260worker1 # [ 0.000000] psci: Trusted OS migration not required261client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs262worker2 # [ 0.017818] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations263worker1 # [ 0.000000] psci: SMC Calling Convention v1.1264client # [ 0.000000] psci: Trusted OS migration not required265worker2 # [ 0.018014] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations266worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)267client # [ 0.000000] psci: SMC Calling Convention v1.1268worker2 # [ 0.018213] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations269worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296270client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)271lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]272worker1 # [ 0.000000] Detected PIPT I-cache on CPU0273worker2 # [ 0.018234] audit: initializing netlink subsys (disabled)274client # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296275worker2 # [ 0.018775] thermal_sys: Registered thermal governor 'fair_share'276worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)277client # [ 0.000000] Detected PIPT I-cache on CPU0278lb # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026279worker2 # [ 0.018777] thermal_sys: Registered thermal governor 'bang_bang'280lb # [ 0.000000] KASLR enabled281lb # [ 0.000000] random: crng init done282client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)283worker2 # [ 0.018781] thermal_sys: Registered thermal governor 'step_wise'284worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface285lb # [ 0.000000] Machine model: linux,dummy-virt286client # [ 0.000000] CPU features: detected: GICv3 CPU interface287worker1 # [ 0.000000] CPU features: detected: Spectre-v4288lb # [ 0.000000] efi: UEFI not found.289worker2 # [ 0.018786] thermal_sys: Registered thermal governor 'user_space'290client # [ 0.000000] CPU features: detected: Spectre-v4291worker1 # [ 0.000000] CPU features: detected: Spectre-BHB292worker2 # [ 0.018789] thermal_sys: Registered thermal governor 'power_allocator'293client # [ 0.000000] CPU features: detected: Spectre-BHB294lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT295worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38296client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38297lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]298worker2 # [ 0.018815] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1299worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23300worker2 # [ 0.018823] cpuidle: using governor ladder301client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23302lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]303worker1 # [ 0.000000] alternatives: applying boot alternatives304worker2 # [ 0.018829] cpuidle: using governor menu305lb # [ 0.000000] Zone ranges:306client # [ 0.000000] alternatives: applying boot alternatives307lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]308worker2 # [ 0.019049] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.309lb # [ 0.000000] DMA32 empty310worker2 # [ 0.019066] ASID allocator initialised with 65536 entries311lb # [ 0.000000] Normal empty312lb # [ 0.000000] Device empty313worker2 # [ 0.020315] Serial: AMBA PL011 UART driver314lb # [ 0.000000] Movable zone start for each node315lb # [ 0.000000] Early memory node ranges316worker2 # [ 0.025639] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1317worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/4aqy0fhz5rhki6gmcbxm00zx4bjdyvaq-nixos-system-worker1-test/init regInfo=/nix/store/0zp4svlw8ivqwyzb7f2wg5kd4innxfww-closure-info/registration console=ttyAMA0,115200n8 console=tty0318lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]319worker2 # [ 0.025764] printk: console [ttyAMA0] enabled320client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/23mddq5b0nkdsfx6621y4k5jikrlcj3j-nixos-system-client-test/init regInfo=/nix/store/413ndf5drg73bc291i547g77mdshn7d4-closure-info/registration console=ttyAMA0,115200n8 console=tty0321lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]322lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000323worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/0zp4svlw8ivqwyzb7f2wg5kd4innxfww-closure-info/registration", will be passed to user space.324lb # [ 0.000000] psci: probing for conduit method from DT.325worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes326lb # [ 0.000000] psci: PSCIv1.3 detected in firmware.327client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/413ndf5drg73bc291i547g77mdshn7d4-closure-info/registration", will be passed to user space.328lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs329worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)330client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes331lb # [ 0.000000] psci: Trusted OS migration not required332worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)333lb # [ 0.000000] psci: SMC Calling Convention v1.1334client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)335worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB336lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)337client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)338worker1 # [ 0.000000] software IO TLB: area num 1.339lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296340client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB341worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB342client # [ 0.000000] software IO TLB: area num 1.343lb # [ 0.000000] Detected PIPT I-cache on CPU0344worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)345client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB346lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)347worker1 # [ 0.000000] Fallback order for Node 0: 0348lb # [ 0.000000] CPU features: detected: GICv3 CPU interface349client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)350worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216351lb # [ 0.000000] CPU features: detected: Spectre-v4352worker1 # [ 0.000000] Policy zone: DMA353client # [ 0.000000] Fallback order for Node 0: 0354lb # [ 0.000000] CPU features: detected: Spectre-BHB355worker2 # [ 0.148933] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages356worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off357client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216358lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38359client # [ 0.000000] Policy zone: DMA360worker2 # [ 0.148951] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page361worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1362lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23363client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off364worker1 # [ 0.000000] allocated 3145728 bytes of page_ext365worker2 # [ 0.148956] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages366lb # [ 0.000000] alternatives: applying boot alternatives367client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1368worker1 # [ 0.000000] ftrace: allocating 74885 entries in 294 pages369worker2 # [ 0.148960] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page370client # [ 0.000000] allocated 3145728 bytes of page_ext371worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups372worker2 # [ 0.148964] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages373client # [ 0.000000] ftrace: allocating 74885 entries in 294 pages374worker1 # [ 0.000000] rcu: Hierarchical RCU implementation.375worker2 # [ 0.148969] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page376worker1 # [ 0.000000] rcu: RCU event tracing is enabled.377client # [ 0.000000] ftrace: allocated 294 pages with 4 groups378client # [ 0.000000] rcu: Hierarchical RCU implementation.379worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.380worker2 # [ 0.148973] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages381client # [ 0.000000] rcu: RCU event tracing is enabled.382worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled.383worker2 # [ 0.148977] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page384lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/a9n40kyqx09p2mx9kz6ck1ddzmmfmfqv-nixos-system-lb-test/init regInfo=/nix/store/kxfc7nc3yvm950aydaljrdz4dxr491hv-closure-info/registration console=ttyAMA0,115200n8 console=tty0385worker1 # [ 0.000000] Rude variant of Tasks RCU enabled.386client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.387worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled.388client # [ 0.000000] Trampoline variant of Tasks RCU enabled.389client # [ 0.000000] Rude variant of Tasks RCU enabled.390worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.391lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/kxfc7nc3yvm950aydaljrdz4dxr491hv-closure-info/registration", will be passed to user space.392client # [ 0.000000] Tracing variant of Tasks RCU enabled.393worker2 # [ 0.156124] fbcon: Taking over console394worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1395lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes396worker2 # [ 0.156139] ACPI: Interpreter disabled.397client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.398worker2 # [ 0.157970] iommu: Default domain type: Translated399worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.400lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)401client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1402worker2 # [ 0.157979] iommu: DMA domain TLB invalidation policy: strict mode403lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)404worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.405client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.406worker2 # [ 0.159613] SCSI subsystem initialized407worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.408client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.409lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB410worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0411lb # [ 0.000000] software IO TLB: area num 1.412worker1 # [ 0.000000] GICv3: 256 SPIs implemented413client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.414lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB415worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented416client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0417worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq418client # [ 0.000000] GICv3: 256 SPIs implemented419lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)420worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI421client # [ 0.000000] GICv3: 0 Extended SPIs implemented422lb # [ 0.000000] Fallback order for Node 0: 0423worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0424client # [ 0.000000] Root IRQ handler: gic_handle_irq425lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216426client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI427worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000428lb # [ 0.000000] Policy zone: DMA429client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0430worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]431lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off432client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000433worker2 # [ 0.165116] usbcore: registered new interface driver usbfs434lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1435worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)436client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]437worker2 # [ 0.165151] usbcore: registered new interface driver hub438lb # [ 0.000000] allocated 3145728 bytes of page_ext439worker2 # [ 0.165177] usbcore: registered new device driver usb440lb # [ 0.000000] ftrace: allocating 74885 entries in 294 pages441worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)442client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)443worker2 # [ 0.165445] pps_core: LinuxPPS API ver. 1 registered444lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups445worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000446lb # [ 0.000000] rcu: Hierarchical RCU implementation.447client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)448worker2 # [ 0.165450] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>449worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000450lb # [ 0.000000] rcu: RCU event tracing is enabled.451client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000452worker2 # [ 0.165468] PTP clock support registered453worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.454worker2 # [ 0.165517] EDAC MC: Ver: 3.0.0455lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.456client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000457lb # [ 0.000000] Trampoline variant of Tasks RCU enabled.458worker2 # [ 0.169928] scmi_core: SCMI protocol bus registered459worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns460lb # [ 0.000000] Rude variant of Tasks RCU enabled.461client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.462worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).463lb # [ 0.000000] Tracing variant of Tasks RCU enabled.464worker2 # [ 0.170846] FPGA manager framework465worker2 # [ 0.171728] vgaarb: loaded466client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns467lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.468worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns469client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).470worker2 # [ 0.172345] clocksource: Switched to clocksource arch_sys_counter471lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1472worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns473lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.474worker1 # [ 0.000029] arm-pv: using stolen time PV475client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns476client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns477lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.478worker1 # [ 0.000448] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)479client # [ 0.000030] arm-pv: using stolen time PV480worker1 # [ 0.000601] Console: colour dummy device 80x25481lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.482worker1 # [ 0.000608] printk: legacy console [tty0] enabled483worker2 # [ 0.175965] VFS: Disk quotas dquot_6.6.0484client # [ 0.000465] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)485lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0486client # [ 0.000634] Console: colour dummy device 80x25487lb # [ 0.000000] GICv3: 256 SPIs implemented488worker2 # [ 0.175999] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)489worker1 # [ 0.000801] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)490client # [ 0.000642] printk: legacy console [tty0] enabled491lb # [ 0.000000] GICv3: 0 Extended SPIs implemented492worker1 # [ 0.000807] pid_max: default: 32768 minimum: 301493lb # [ 0.000000] Root IRQ handler: gic_handle_irq494worker1 # [ 0.000884] LSM: initializing lsm=capability,landlock,yama,bpf,ima495lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI496client # [ 0.000833] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)497worker1 # [ 0.001042] landlock: Up and running.498lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0499client # [ 0.000840] pid_max: default: 32768 minimum: 301500worker1 # [ 0.001045] Yama: becoming mindful.501worker1 # [ 0.001530] LSM support for eBPF active502worker2 # [ 0.179526] netfs: FS-Cache loaded503lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000504client # [ 0.000909] LSM: initializing lsm=capability,landlock,yama,bpf,ima505worker2 # [ 0.179644] pnp: PnP ACPI: disabled506lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]507client # [ 0.001085] landlock: Up and running.508worker1 # [ 0.001676] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)509client # [ 0.001087] Yama: becoming mindful.510client # [ 0.001592] LSM support for eBPF active511lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)512worker1 # [ 0.001709] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)513client # [ 0.001755] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)514worker1 # [ 0.002868] cacheinfo: Unable to detect cache hierarchy for CPU 0515lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)516worker1 # [ 0.003561] rcu: Hierarchical SRCU implementation.517client # [ 0.001792] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)518lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000519worker1 # [ 0.003565] rcu: Max phase no-delay instances is 1000.520client # [ 0.002890] cacheinfo: Unable to detect cache hierarchy for CPU 0521worker1 # [ 0.004783] fsl-mc MSI: its@8080000 domain created522lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000523client # [ 0.003633] rcu: Hierarchical SRCU implementation.524worker1 # [ 0.004873] EFI services will not be available.525worker2 # [ 0.183612] NET: Registered PF_INET protocol family526client # [ 0.003637] rcu: Max phase no-delay instances is 1000.527worker1 # [ 0.004952] smp: Bringing up secondary CPUs ...528lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.529worker2 # [ 0.183887] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)530worker1 # [ 0.004960] smp: Brought up 1 node, 1 CPU531client # [ 0.004830] fsl-mc MSI: its@8080000 domain created532worker1 # [ 0.004963] SMP: Total of 1 processors activated.533client # [ 0.004920] EFI services will not be available.534lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns535worker1 # [ 0.004968] CPU: All CPU(s) started at EL1536client # [ 0.005040] smp: Bringing up secondary CPUs ...537lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).538client # [ 0.005049] smp: Brought up 1 node, 1 CPU539worker1 # [ 0.004979] CPU features: detected: Branch Target Identification540client # [ 0.005052] SMP: Total of 1 processors activated.541worker1 # [ 0.004984] CPU features: detected: ARMv8.4 Translation Table Level542lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns543client # [ 0.005058] CPU: All CPU(s) started at EL1544client # [ 0.005069] CPU features: detected: Branch Target Identification545lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns546worker1 # [ 0.004987] CPU features: detected: Instruction cache invalidation not required for I/D coherence547lb # [ 0.000028] arm-pv: using stolen time PV548client # [ 0.005074] CPU features: detected: ARMv8.4 Translation Table Level549worker1 # [ 0.004991] CPU features: detected: Data cache clean to the PoU not required for I/D coherence550lb # [ 0.000370] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)551client # [ 0.005077] CPU features: detected: Instruction cache invalidation not required for I/D coherence552worker1 # [ 0.004994] CPU features: detected: Common not Private translations553lb # [ 0.000538] Console: colour dummy device 80x25554worker1 # [ 0.004997] CPU features: detected: CRC32 instructions555client # [ 0.005081] CPU features: detected: Data cache clean to the PoU not required for I/D coherence556lb # [ 0.000545] printk: legacy console [tty0] enabled557worker1 # [ 0.005000] CPU features: detected: Data cache clean to Point of Deep Persistence558client # [ 0.005084] CPU features: detected: Common not Private translations559lb # [ 0.000739] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)560client # [ 0.005087] CPU features: detected: CRC32 instructions561worker1 # [ 0.005004] CPU features: detected: Data cache clean to Point of Persistence562lb # [ 0.000746] pid_max: default: 32768 minimum: 301563worker1 # [ 0.005007] CPU features: detected: Data independent timing control (DIT)564client # [ 0.005090] CPU features: detected: Data cache clean to Point of Deep Persistence565lb # [ 0.000861] LSM: initializing lsm=capability,landlock,yama,bpf,ima566worker1 # [ 0.005010] CPU features: detected: E0PD567lb # [ 0.000977] landlock: Up and running.568client # [ 0.005094] CPU features: detected: Data cache clean to Point of Persistence569worker1 # [ 0.005012] CPU features: detected: Enhanced Counter Virtualization570lb # [ 0.000979] Yama: becoming mindful.571lb # [ 0.001389] LSM support for eBPF active572client # [ 0.005097] CPU features: detected: Data independent timing control (DIT)573worker1 # [ 0.005015] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)574client # [ 0.005100] CPU features: detected: E0PD575lb # [ 0.001518] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)576worker1 # [ 0.005019] CPU features: detected: Enhanced Virtualization Traps577client # [ 0.005102] CPU features: detected: Enhanced Counter Virtualization578lb # [ 0.001552] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)579worker1 # [ 0.005022] CPU features: detected: Fine Grained Traps580client # [ 0.005105] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)581lb # [ 0.002615] cacheinfo: Unable to detect cache hierarchy for CPU 0582worker1 # [ 0.005025] CPU features: detected: Generic authentication (architected QARMA5 algorithm)583client # [ 0.005109] CPU features: detected: Enhanced Virtualization Traps584lb # [ 0.003394] rcu: Hierarchical SRCU implementation.585worker1 # [ 0.005031] CPU features: detected: RCpc load-acquire (LDAPR)586client # [ 0.005111] CPU features: detected: Fine Grained Traps587lb # [ 0.003397] rcu: Max phase no-delay instances is 1000.588worker1 # [ 0.005033] CPU features: detected: LSE atomic instructions589lb # [ 0.004573] fsl-mc MSI: its@8080000 domain created590client # [ 0.005115] CPU features: detected: Generic authentication (architected QARMA5 algorithm)591lb # [ 0.004669] EFI services will not be available.592worker1 # [ 0.005037] CPU features: detected: Privileged Access Never593client # [ 0.005121] CPU features: detected: RCpc load-acquire (LDAPR)594worker1 # [ 0.005039] CPU features: detected: PMUv3595lb # [ 0.004756] smp: Bringing up secondary CPUs ...596lb # [ 0.004764] smp: Brought up 1 node, 1 CPU597client # [ 0.005123] CPU features: detected: LSE atomic instructions598worker1 # [ 0.005042] CPU features: detected: RAS Extension Support599lb # [ 0.004768] SMP: Total of 1 processors activated.600client # [ 0.005126] CPU features: detected: Privileged Access Never601worker1 # [ 0.005044] CPU features: detected: RASv1p1 Extension Support602lb # [ 0.004773] CPU: All CPU(s) started at EL1603client # [ 0.005129] CPU features: detected: PMUv3604worker1 # [ 0.005047] CPU features: detected: Random Number Generator605lb # [ 0.004784] CPU features: detected: Branch Target Identification606client # [ 0.005132] CPU features: detected: RAS Extension Support607worker1 # [ 0.005050] CPU features: detected: Speculation barrier (SB)608lb # [ 0.004789] CPU features: detected: ARMv8.4 Translation Table Level609worker1 # [ 0.005053] CPU features: detected: Stage-2 Force Write-Back610lb # [ 0.004792] CPU features: detected: Instruction cache invalidation not required for I/D coherence611lb # [ 0.004796] CPU features: detected: Data cache clean to the PoU not required for I/D coherence612lb # [ 0.004799] CPU features: detected: Common not Private translations613lb # [ 0.004802] CPU features: detected: CRC32 instructions614lb # [ 0.004805] CPU features: detected: Data cache clean to Point of Deep Persistence615lb # [ 0.004809] CPU features: detected: Data cache clean to Point of Persistence616lb # [ 0.004812] CPU features: detected: Data independent timing control (DIT)617lb # [ 0.004815] CPU features: detected: E0PD618lb # [ 0.004818] CPU features: detected: Enhanced Counter Virtualization619lb # [ 0.004821] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)620client # [ 0.005134] CPU features: detected: RASv1p1 Extension Support621lb # [ 0.004824] CPU features: detected: Enhanced Virtualization Traps622client # [ 0.005137] CPU features: detected: Random Number Generator623worker1 # [ 0.005056] CPU features: detected: TLB range maintenance instructions624lb # [ 0.004827] CPU features: detected: Fine Grained Traps625client # [ 0.005140] CPU features: detected: Speculation barrier (SB)626worker1 # [ 0.005060] CPU features: detected: Speculative Store Bypassing Safe (SSBS)627client # [ 0.005143] CPU features: detected: Stage-2 Force Write-Back628lb # [ 0.004830] CPU features: detected: Generic authentication (architected QARMA5 algorithm)629worker1 # [ 0.005097] alternatives: applying system-wide alternatives630client # [ 0.005146] CPU features: detected: TLB range maintenance instructions631lb # [ 0.004835] CPU features: detected: RCpc load-acquire (LDAPR)632worker1 # [ 0.008004] CPU features: detected: BBM Level 2 without TLB conflict abort633client # [ 0.005150] CPU features: detected: Speculative Store Bypassing Safe (SSBS)634lb # [ 0.004838] CPU features: detected: LSE atomic instructions635client # [ 0.005188] alternatives: applying system-wide alternatives636lb # [ 0.004841] CPU features: detected: Privileged Access Never637worker1 # [ 0.008229] Memory: 1406536K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123772K reserved, 32768K cma-reserved)638lb # [ 0.004843] CPU features: detected: PMUv3639client # [ 0.008161] CPU features: detected: BBM Level 2 without TLB conflict abort640worker1 # [ 0.008591] devtmpfs: initialized641lb # [ 0.004846] CPU features: detected: RAS Extension Support642worker1 # [ 0.010380] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)643lb # [ 0.004849] CPU features: detected: RASv1p1 Extension Support644client # [ 0.008326] Memory: 1406512K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123784K reserved, 32768K cma-reserved)645lb # [ 0.004852] CPU features: detected: Random Number Generator646worker1 # [ 0.010403] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).647client # [ 0.008728] devtmpfs: initialized648lb # [ 0.004854] CPU features: detected: Speculation barrier (SB)649worker1 # [ 0.010586] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL650client # [ 0.010541] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)651worker1 # [ 0.010591] 0 pages in range for non-PLT usage652lb # [ 0.004857] CPU features: detected: Stage-2 Force Write-Back653worker1 # [ 0.010592] 508288 pages in range for PLT usage654lb # [ 0.004860] CPU features: detected: TLB range maintenance instructions655client # [ 0.010563] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).656worker1 # [ 0.010724] pinctrl core: initialized pinctrl subsystem657worker2 # [ 0.212712] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)658client # [ 0.010749] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL659worker1 # [ 0.011453] DMI not present or invalid.660lb # [ 0.004865] CPU features: detected: Speculative Store Bypassing Safe (SSBS)661client # [ 0.010754] 0 pages in range for non-PLT usage662worker2 # [ 0.212753] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)663lb # [ 0.004901] alternatives: applying system-wide alternatives664worker1 # [ 0.014624] NET: Registered PF_NETLINK/PF_ROUTE protocol family665client # [ 0.010755] 508288 pages in range for PLT usage666worker1 # [ 0.016824] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations667worker2 # [ 0.212778] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)668lb # [ 0.007787] CPU features: detected: BBM Level 2 without TLB conflict abort669client # [ 0.010863] pinctrl core: initialized pinctrl subsystem670client # [ 0.011685] DMI not present or invalid.671worker2 # [ 0.212847] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)672worker1 # [ 0.017001] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations673client # [ 0.014898] NET: Registered PF_NETLINK/PF_ROUTE protocol family674worker2 # [ 0.212991] TCP: Hash tables configured (established 16384 bind 16384)675lb # [ 0.007960] Memory: 1406544K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123764K reserved, 32768K cma-reserved)676worker1 # [ 0.017199] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations677lb # [ 0.008366] devtmpfs: initialized678client # [ 0.017101] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations679worker1 # [ 0.017221] audit: initializing netlink subsys (disabled)680worker2 # [ 0.213069] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)681lb # [ 0.010210] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)682client # [ 0.017280] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations683worker1 # [ 0.017797] thermal_sys: Registered thermal governor 'fair_share'684worker2 # [ 0.213105] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)685worker1 # [ 0.017799] thermal_sys: Registered thermal governor 'bang_bang'686lb # [ 0.010233] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).687client # [ 0.017478] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations688worker2 # [ 0.213139] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)689worker1 # [ 0.017803] thermal_sys: Registered thermal governor 'step_wise'690client # [ 0.017500] audit: initializing netlink subsys (disabled)691lb # [ 0.010413] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL692worker2 # [ 0.213224] NET: Registered PF_UNIX/PF_LOCAL protocol family693worker1 # [ 0.017809] thermal_sys: Registered thermal governor 'user_space'694lb # [ 0.010418] 0 pages in range for non-PLT usage695worker2 # [ 0.213263] NET: Registered PF_XDP protocol family696client # [ 0.018067] thermal_sys: Registered thermal governor 'fair_share'697lb # [ 0.010419] 508288 pages in range for PLT usage698worker2 # [ 0.213279] PCI: CLS 0 bytes, default 64699worker1 # [ 0.017811] thermal_sys: Registered thermal governor 'power_allocator'700client # [ 0.018069] thermal_sys: Registered thermal governor 'bang_bang'701lb # [ 0.010509] pinctrl core: initialized pinctrl subsystem702worker2 # [ 0.213509] Trying to unpack rootfs image as initramfs...703worker1 # [ 0.017842] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1704client # [ 0.018073] thermal_sys: Registered thermal governor 'step_wise'705worker1 # [ 0.017850] cpuidle: using governor ladder706client # [ 0.018079] thermal_sys: Registered thermal governor 'user_space'707worker1 # [ 0.017856] cpuidle: using governor menu708client # [ 0.018081] thermal_sys: Registered thermal governor 'power_allocator'709worker1 # [ 0.018046] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.710worker1 # [ 0.018061] ASID allocator initialised with 65536 entries711client # [ 0.018106] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1712worker1 # [ 0.019203] Serial: AMBA PL011 UART driver713client # [ 0.018114] cpuidle: using governor ladder714client # [ 0.018120] cpuidle: using governor menu715worker1 # [ 0.024313] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1716client # [ 0.018300] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.717worker1 # [ 0.024467] printk: console [ttyAMA0] enabled718client # [ 0.018316] ASID allocator initialised with 65536 entries719client # [ 0.019485] Serial: AMBA PL011 UART driver720worker1 # [ 0.143856] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages721client # [ 0.024647] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1722worker1 # [ 0.143874] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page723client # [ 0.024803] printk: console [ttyAMA0] enabled724worker2 # [ 0.229600] kvm [1]: HYP mode not available725worker1 # [ 0.143879] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages726lb # [ 0.011267] DMI not present or invalid.727worker1 # [ 0.143883] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page728client # [ 0.145764] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages729lb # [ 0.014410] NET: Registered PF_NETLINK/PF_ROUTE protocol family730worker1 # [ 0.143888] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages731client # [ 0.145784] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page732lb # [ 0.016596] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations733worker1 # [ 0.143892] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page734client # [ 0.145789] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages735lb # [ 0.016785] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations736worker1 # [ 0.143896] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages737client # [ 0.145793] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page738worker1 # [ 0.143900] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page739lb # [ 0.016981] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations740client # [ 0.145797] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages741lb # [ 0.017003] audit: initializing netlink subsys (disabled)742client # [ 0.145802] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page743lb # [ 0.017504] thermal_sys: Registered thermal governor 'fair_share'744client # [ 0.145806] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages745lb # [ 0.017506] thermal_sys: Registered thermal governor 'bang_bang'746worker1 # [ 0.151087] fbcon: Taking over console747client # [ 0.145810] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page748worker1 # [ 0.151102] ACPI: Interpreter disabled.749lb # [ 0.017510] thermal_sys: Registered thermal governor 'step_wise'750worker1 # [ 0.152836] iommu: Default domain type: Translated751lb # [ 0.017515] thermal_sys: Registered thermal governor 'user_space'752worker1 # [ 0.152845] iommu: DMA domain TLB invalidation policy: strict mode753lb # [ 0.017517] thermal_sys: Registered thermal governor 'power_allocator'754worker1 # [ 0.154492] SCSI subsystem initialized755client # [ 0.153261] fbcon: Taking over console756lb # [ 0.017540] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1757client # [ 0.153275] ACPI: Interpreter disabled.758lb # [ 0.017550] cpuidle: using governor ladder759worker1 # [ 0.155119] usbcore: registered new interface driver usbfs760client # [ 0.155131] iommu: Default domain type: Translated761lb # [ 0.017555] cpuidle: using governor menu762worker1 # [ 0.155149] usbcore: registered new interface driver hub763client # [ 0.155139] iommu: DMA domain TLB invalidation policy: strict mode764lb # [ 0.017739] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.765worker1 # [ 0.155166] usbcore: registered new device driver usb766lb # [ 0.017754] ASID allocator initialised with 65536 entries767client # [ 0.156835] SCSI subsystem initialized768worker1 # [ 0.155427] pps_core: LinuxPPS API ver. 1 registered769lb # [ 0.018921] Serial: AMBA PL011 UART driver770worker1 # [ 0.155433] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>771lb # [ 0.024089] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1772worker1 # [ 0.155449] PTP clock support registered773lb # [ 0.024251] printk: console [ttyAMA0] enabled774worker1 # [ 0.155496] EDAC MC: Ver: 3.0.0775lb # [ 0.144579] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages776lb # [ 0.144596] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page777lb # [ 0.144601] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages778lb # [ 0.144606] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page779lb # [ 0.144610] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages780client # [ 0.163274] usbcore: registered new interface driver usbfs781lb # [ 0.144614] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page782client # [ 0.163305] usbcore: registered new interface driver hub783lb # [ 0.144619] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages784client # [ 0.163323] usbcore: registered new device driver usb785lb # [ 0.144623] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page786client # [ 0.163607] pps_core: LinuxPPS API ver. 1 registered787worker1 # [ 0.166793] scmi_core: SCMI protocol bus registered788client # [ 0.163613] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>789client # [ 0.163623] PTP clock support registered790worker1 # [ 0.167771] FPGA manager framework791lb # [ 0.152091] fbcon: Taking over console792client # [ 0.163685] EDAC MC: Ver: 3.0.0793worker1 # [ 0.168762] vgaarb: loaded794lb # [ 0.152106] ACPI: Interpreter disabled.795client # [ 0.168413] scmi_core: SCMI protocol bus registered796lb # [ 0.154003] iommu: Default domain type: Translated797worker1 # [ 0.169385] clocksource: Switched to clocksource arch_sys_counter798client # [ 0.169392] FPGA manager framework799lb # [ 0.154013] iommu: DMA domain TLB invalidation policy: strict mode800worker1 # [ 0.169968] VFS: Disk quotas dquot_6.6.0801client # [ 0.170431] vgaarb: loaded802lb # [ 0.155682] SCSI subsystem initialized803worker1 # [ 0.170001] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)804client # [ 0.171082] clocksource: Switched to clocksource arch_sys_counter805worker1 # [ 0.172362] netfs: FS-Cache loaded806worker1 # [ 0.172479] pnp: PnP ACPI: disabled807client # [ 0.173797] VFS: Disk quotas dquot_6.6.0808client # [ 0.173825] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)809lb # [ 0.160962] usbcore: registered new interface driver usbfs810lb # [ 0.160994] usbcore: registered new interface driver hub811client # [ 0.177428] netfs: FS-Cache loaded812client # [ 0.177550] pnp: PnP ACPI: disabled813lb # [ 0.161019] usbcore: registered new device driver usb814lb # [ 0.161302] pps_core: LinuxPPS API ver. 1 registered815lb # [ 0.161308] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>816lb # [ 0.161319] PTP clock support registered817lb # [ 0.161373] EDAC MC: Ver: 3.0.0818worker1 # [ 0.178318] NET: Registered PF_INET protocol family819lb # [ 0.166008] scmi_core: SCMI protocol bus registered820worker1 # [ 0.178585] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)821lb # [ 0.166987] FPGA manager framework822lb # [ 0.167888] vgaarb: loaded823client # [ 0.181398] NET: Registered PF_INET protocol family824lb # [ 0.168546] clocksource: Switched to clocksource arch_sys_counter825client # [ 0.181660] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)826lb # [ 0.172107] VFS: Disk quotas dquot_6.6.0827lb # [ 0.172134] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)828lb # [ 0.175794] netfs: FS-Cache loaded829lb # [ 0.175927] pnp: PnP ACPI: disabled830lb # [ 0.179800] NET: Registered PF_INET protocol family831lb # [ 0.180050] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)832client # [ 0.210711] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)833client # [ 0.210758] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)834worker1 # [ 0.207575] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)835client # [ 0.210785] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)836worker1 # [ 0.207617] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)837client # [ 0.210859] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)838worker1 # [ 0.207644] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)839client # [ 0.211003] TCP: Hash tables configured (established 16384 bind 16384)840worker1 # [ 0.207715] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)841worker1 # [ 0.207860] TCP: Hash tables configured (established 16384 bind 16384)842worker1 # [ 0.207941] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)843worker1 # [ 0.207974] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)844worker1 # [ 0.208010] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)845worker1 # [ 0.208095] NET: Registered PF_UNIX/PF_LOCAL protocol family846worker1 # [ 0.208148] NET: Registered PF_XDP protocol family847worker1 # [ 0.208166] PCI: CLS 0 bytes, default 64848worker1 # [ 0.208409] Trying to unpack rootfs image as initramfs...849worker2 # [ 0.313307] Initialise system trusted keyrings850client # [ 0.215969] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)851client # [ 0.216010] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)852client # [ 0.216046] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)853client # [ 0.216172] NET: Registered PF_UNIX/PF_LOCAL protocol family854client # [ 0.216235] NET: Registered PF_XDP protocol family855client # [ 0.216255] PCI: CLS 0 bytes, default 64856client # [ 0.216485] Trying to unpack rootfs image as initramfs...857worker1 # [ 0.222995] kvm [1]: HYP mode not available858worker2 # [ 0.320404] workingset: timestamp_bits=42 max_order=19 bucket_order=0859worker2 # [ 0.321784] squashfs: version 4.0 (2009/01/31) Phillip Lougher860lb # [ 0.209272] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)861worker2 # [ 0.322558] 9p: Installing v9fs 9p2000 file system support862lb # [ 0.209315] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)863lb # [ 0.209341] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)864lb # [ 0.209416] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)865client # [ 0.228136] kvm [1]: HYP mode not available866lb # [ 0.209559] TCP: Hash tables configured (established 16384 bind 16384)867lb # [ 0.209638] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)868lb # [ 0.209672] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)869lb # [ 0.209707] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)870lb # [ 0.209825] NET: Registered PF_UNIX/PF_LOCAL protocol family871lb # [ 0.209847] NET: Registered PF_XDP protocol family872lb # [ 0.209865] PCI: CLS 0 bytes, default 64873lb # [ 0.210092] Trying to unpack rootfs image as initramfs...874lb # [ 0.226188] kvm [1]: HYP mode not available875worker2 # [ 0.343168] Key type asymmetric registered876worker2 # [ 0.343190] Asymmetric key parser 'x509' registered877worker2 # [ 0.343256] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)878worker2 # [ 0.348391] io scheduler mq-deadline registered879worker2 # [ 0.348406] io scheduler kyber registered880worker2 # [ 0.360500] pl061_gpio 9030000.pl061: PL061 GPIO chip registered881worker2 # [ 0.361135] ledtrig-cpu: registered to indicate activity on CPUs882worker2 # [ 0.361524] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:883worker2 # [ 0.361542] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000884worker2 # [ 0.361554] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000885worker2 # [ 0.361562] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000886worker2 # [ 0.361589] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits887worker2 # [ 0.361613] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]888worker2 # [ 0.361688] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00889worker2 # [ 0.361698] pci_bus 0000:00: root bus resource [bus 00-ff]890worker2 # [ 0.361704] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]891worker2 # [ 0.361710] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]892worker2 # [ 0.361715] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]893worker2 # [ 0.361834] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint894worker2 # [ 0.362282] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint895worker2 # [ 0.362469] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]896worker2 # [ 0.362485] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]897worker2 # [ 0.362515] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]898worker2 # [ 0.362531] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]899worker2 # [ 0.362984] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint900worker2 # [ 0.363181] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]901worker2 # [ 0.363198] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]902worker2 # [ 0.363228] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]903worker2 # [ 0.363678] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint904worker2 # [ 0.363861] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]905worker2 # [ 0.363877] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]906worker2 # [ 0.363907] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]907worker2 # [ 0.364386] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint908worker2 # [ 0.364568] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]909worker2 # [ 0.364584] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]910worker2 # [ 0.364617] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]911worker2 # [ 0.365073] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint912worker2 # [ 0.365256] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]913worker2 # [ 0.365272] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]914worker2 # [ 0.365301] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]915worker2 # [ 0.365757] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint916worker2 # [ 0.365939] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]917worker2 # [ 0.365955] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]918worker2 # [ 0.365985] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]919worker2 # [ 0.366434] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint920worker2 # [ 0.366616] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]921worker2 # [ 0.366632] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]922worker2 # [ 0.366662] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]923worker2 # [ 0.366678] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]924worker2 # [ 0.367174] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint925worker1 # [ 0.313938] Initialise system trusted keyrings926worker2 # [ 0.367361] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]927worker1 # [ 0.314668] workingset: timestamp_bits=42 max_order=19 bucket_order=0928worker2 # [ 0.367391] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]929worker1 # [ 0.315922] squashfs: version 4.0 (2009/01/31) Phillip Lougher930worker2 # [ 0.367848] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint931worker2 # [ 0.368034] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]932worker1 # [ 0.316709] 9p: Installing v9fs 9p2000 file system support933worker2 # [ 0.368064] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]934client # [ 0.319662] Initialise system trusted keyrings935client # [ 0.320406] workingset: timestamp_bits=42 max_order=19 bucket_order=0936client # [ 0.321725] squashfs: version 4.0 (2009/01/31) Phillip Lougher937client # [ 0.327121] 9p: Installing v9fs 9p2000 file system support938worker2 # [ 0.416522] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint939worker2 # [ 0.416715] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]940lb # [ 0.317081] Initialise system trusted keyrings941worker2 # [ 0.416965] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint942lb # [ 0.317830] workingset: timestamp_bits=42 max_order=19 bucket_order=0943worker2 # [ 0.417262] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]944lb # [ 0.319108] squashfs: version 4.0 (2009/01/31) Phillip Lougher945worker2 # [ 0.417279] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]946worker2 # [ 0.417309] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]947lb # [ 0.319871] 9p: Installing v9fs 9p2000 file system support948worker2 # [ 0.417785] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint949worker1 # [ 0.337278] Key type asymmetric registered950worker2 # [ 0.417968] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]951worker1 # [ 0.337301] Asymmetric key parser 'x509' registered952worker2 # [ 0.417984] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]953worker1 # [ 0.337368] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)954worker2 # [ 0.418014] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]955worker2 # [ 0.418594] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned956worker2 # [ 0.418605] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned957worker2 # [ 0.418611] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned958worker2 # [ 0.418657] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned959worker2 # [ 0.418703] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned960worker2 # [ 0.418750] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned961worker1 # [ 0.345439] io scheduler mq-deadline registered962worker2 # [ 0.418797] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned963worker1 # [ 0.345460] io scheduler kyber registered964worker2 # [ 0.418844] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned965worker2 # [ 0.418891] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned966client # [ 0.347870] Key type asymmetric registered967worker2 # [ 0.418939] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned968client # [ 0.347890] Asymmetric key parser 'x509' registered969worker2 # [ 0.418986] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned970client # [ 0.347958] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)971worker2 # [ 0.419041] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned972client # [ 0.350086] io scheduler mq-deadline registered973client # [ 0.350096] io scheduler kyber registered974worker2 # [ 0.419110] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned975worker2 # [ 0.419156] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned976worker2 # [ 0.419178] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned977worker2 # [ 0.419199] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned978worker1 # [ 0.353515] pl061_gpio 9030000.pl061: PL061 GPIO chip registered979worker2 # [ 0.419221] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned980worker2 # [ 0.419246] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned981lb # [ 0.340469] Key type asymmetric registered982worker2 # [ 0.419268] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned983lb # [ 0.340496] Asymmetric key parser 'x509' registered984worker2 # [ 0.419290] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned985worker1 # [ 0.354830] ledtrig-cpu: registered to indicate activity on CPUs986worker2 # [ 0.419312] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned987worker1 # [ 0.355183] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:988worker2 # [ 0.419334] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned989worker1 # [ 0.355344] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000990worker2 # [ 0.419356] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned991worker2 # [ 0.419378] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned992client # [ 0.359212] pl061_gpio 9030000.pl061: PL061 GPIO chip registered993worker1 # [ 0.355357] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000994worker2 # [ 0.419401] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned995worker1 # [ 0.355365] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000996worker2 # [ 0.419422] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned997worker1 # [ 0.355386] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits998worker2 # [ 0.419444] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned999worker2 # [ 0.419465] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1000worker1 # [ 0.355415] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1001client # [ 0.360593] ledtrig-cpu: registered to indicate activity on CPUs1002worker2 # [ 0.419487] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1003worker1 # [ 0.355518] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001004lb # [ 0.348596] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)1005client # [ 0.360986] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1006worker2 # [ 0.419508] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1007worker1 # [ 0.355527] pci_bus 0000:00: root bus resource [bus 00-ff]1008lb # [ 0.349608] io scheduler mq-deadline registered1009worker2 # [ 0.419530] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1010worker1 # [ 0.355533] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1011client # [ 0.361004] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001012lb # [ 0.349618] io scheduler kyber registered1013worker2 # [ 0.419552] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1014worker1 # [ 0.355538] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1015client # [ 0.361016] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001016worker2 # [ 0.419573] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1017worker1 # [ 0.355543] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1018worker2 # [ 0.419596] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1019client # [ 0.361024] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001020worker1 # [ 0.355602] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1021worker2 # [ 0.419622] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1022client # [ 0.361046] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1023worker2 # [ 0.419632] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1024worker1 # [ 0.356051] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1025worker2 # [ 0.419636] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1026worker1 # [ 0.356264] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1027client # [ 0.361069] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1028worker2 # [ 0.420813] pci 0000:00:0a.0: enabling device (0000 -> 0002)1029worker1 # [ 0.356281] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1030client # [ 0.361143] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001031worker1 # [ 0.356310] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1032client # [ 0.361152] pci_bus 0000:00: root bus resource [bus 00-ff]1033worker1 # [ 0.356326] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1034client # [ 0.361158] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1035client # [ 0.361163] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1036worker1 # [ 0.356780] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1037worker1 # [ 0.356962] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1038client # [ 0.361168] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1039worker1 # [ 0.356977] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1040client # [ 0.361224] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1041worker1 # [ 0.357007] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1042client # [ 0.361670] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1043client # [ 0.361860] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1044lb # [ 0.360685] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1045client # [ 0.361877] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1046client # [ 0.361907] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1047client # [ 0.361924] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1048client # [ 0.362383] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1049lb # [ 0.362066] ledtrig-cpu: registered to indicate activity on CPUs1050client # [ 0.362570] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1051lb # [ 0.362430] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1052client # [ 0.362586] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1053client # [ 0.362616] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1054lb # [ 0.362447] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001055client # [ 0.363071] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1056lb # [ 0.362471] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001057lb # [ 0.362480] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001058lb # [ 0.362499] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1059lb # [ 0.362523] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1060lb # [ 0.362595] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001061lb # [ 0.362605] pci_bus 0000:00: root bus resource [bus 00-ff]1062lb # [ 0.362610] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1063lb # [ 0.362616] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1064worker1 # [ 0.381529] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1065worker1 # [ 0.381735] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]1066lb # [ 0.362620] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1067worker1 # [ 0.381752] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1068lb # [ 0.362710] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1069worker1 # [ 0.381782] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1070lb # [ 0.363145] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1071worker1 # [ 0.382277] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1072lb # [ 0.363330] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1073worker1 # [ 0.382459] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1074lb # [ 0.363346] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1075worker1 # [ 0.382475] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1076lb # [ 0.363376] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1077worker1 # [ 0.382508] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1078lb # [ 0.363392] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1079worker1 # [ 0.382992] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1080lb # [ 0.363855] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1081worker1 # [ 0.383173] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1082lb # [ 0.364037] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1083worker1 # [ 0.383189] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1084worker2 # [ 0.486700] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1085lb # [ 0.364052] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1086worker1 # [ 0.383218] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1087lb # [ 0.364081] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1088worker1 # [ 0.383666] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1089client # [ 0.391324] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]1090lb # [ 0.364539] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1091worker1 # [ 0.383849] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1092client # [ 0.391354] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1093worker1 # [ 0.383865] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1094client # [ 0.391385] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1095worker1 # [ 0.383895] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1096worker2 # [ 0.489825] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1097client # [ 0.391891] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1098worker1 # [ 0.384355] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1099worker2 # [ 0.491786] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1100client # [ 0.392079] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1101worker1 # [ 0.384537] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1102client # [ 0.392096] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1103worker1 # [ 0.384552] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1104lb # [ 0.384910] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]1105worker1 # [ 0.384581] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1106client # [ 0.392129] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1107lb # [ 0.384929] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1108worker1 # [ 0.384597] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1109client # [ 0.392597] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1110lb # [ 0.384958] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1111client # [ 0.392783] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1112worker1 # [ 0.385052] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1113lb # [ 0.385418] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1114client # [ 0.392800] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1115worker1 # [ 0.385236] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1116lb # [ 0.385598] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1117worker1 # [ 0.385266] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1118client # [ 0.392830] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1119lb # [ 0.385613] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1120worker1 # [ 0.385727] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1121client # [ 0.393284] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1122lb # [ 0.385645] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1123worker1 # [ 0.385911] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1124client # [ 0.393469] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1125lb # [ 0.386102] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1126client # [ 0.393486] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1127worker1 # [ 0.385940] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1128lb # [ 0.386282] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1129client # [ 0.393516] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1130lb # [ 0.386297] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1131worker1 # [ 0.386333] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1132worker1 # [ 0.386519] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1133lb # [ 0.386326] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1134client # [ 0.394013] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1135client # [ 0.394209] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1136lb # [ 0.386775] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1137worker1 # [ 0.386767] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1138client # [ 0.394225] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1139worker1 # [ 0.387053] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1140lb # [ 0.386955] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1141worker2 # [ 0.501794] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1142worker1 # [ 0.387070] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1143client # [ 0.394256] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1144lb # [ 0.386970] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1145worker2 # [ 0.503769] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1146lb # [ 0.386999] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1147client # [ 0.394273] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1148worker1 # [ 0.387099] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1149worker1 # [ 0.387550] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1150lb # [ 0.387449] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1151client # [ 0.394745] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1152worker1 # [ 0.387730] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1153lb # [ 0.387631] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1154client # [ 0.394936] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1155worker1 # [ 0.387746] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1156lb # [ 0.387646] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1157client # [ 0.394968] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1158worker2 # [ 0.507283] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1159lb # [ 0.387675] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1160worker1 # [ 0.387774] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1161client # [ 0.395439] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1162lb # [ 0.387691] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1163worker1 # [ 0.388357] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1164client # [ 0.395638] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1165worker1 # [ 0.388368] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1166lb # [ 0.388184] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1167client # [ 0.395669] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1168lb # [ 0.388369] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1169worker1 # [ 0.388374] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1170client # [ 0.396072] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1171lb # [ 0.388399] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1172worker1 # [ 0.388418] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1173client # [ 0.396265] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1174worker1 # [ 0.388465] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1175client # [ 0.396517] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1176worker1 # [ 0.388511] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1177client # [ 0.396800] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1178client # [ 0.396818] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1179worker1 # [ 0.388556] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1180client # [ 0.396849] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1181worker1 # [ 0.388601] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1182client # [ 0.397313] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1183worker2 # [ 0.515903] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1184worker1 # [ 0.388648] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1185client # [ 0.397502] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1186worker1 # [ 0.388694] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1187client # [ 0.397518] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1188worker1 # [ 0.388740] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1189client # [ 0.397548] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1190client # [ 0.398139] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1191worker1 # [ 0.388785] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1192client # [ 0.398151] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1193worker1 # [ 0.388846] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1194worker1 # [ 0.388891] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1195client # [ 0.398156] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1196worker1 # [ 0.388912] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1197client # [ 0.398202] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1198worker1 # [ 0.388934] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1199lb # [ 0.412910] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1200client # [ 0.398250] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1201lb # [ 0.413111] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1202worker1 # [ 0.388955] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1203client # [ 0.398298] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1204lb # [ 0.413140] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1205worker1 # [ 0.388982] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1206client # [ 0.398344] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1207worker1 # [ 0.389003] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1208lb # [ 0.413546] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1209worker1 # [ 0.389026] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1210lb # [ 0.413726] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1211client # [ 0.398392] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1212worker1 # [ 0.389048] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1213lb # [ 0.413975] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1214client # [ 0.398439] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1215worker1 # [ 0.389070] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1216lb # [ 0.414258] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1217worker2 # [ 0.525707] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1218client # [ 0.398486] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1219lb # [ 0.414275] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1220worker1 # [ 0.389092] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1221lb # [ 0.414303] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1222worker1 # [ 0.389115] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1223client # [ 0.398533] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1224worker1 # [ 0.389137] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1225lb # [ 0.414763] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1226client # [ 0.398581] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1227worker1 # [ 0.389160] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1228lb # [ 0.414945] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1229client # [ 0.398642] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1230worker1 # [ 0.389181] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1231lb # [ 0.414960] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1232client # [ 0.398688] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1233worker1 # [ 0.389203] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1234lb # [ 0.414989] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1235worker1 # [ 0.389225] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1236client # [ 0.398710] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1237lb # [ 0.415553] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1238worker1 # [ 0.389248] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1239client # [ 0.398733] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1240lb # [ 0.415564] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1241worker1 # [ 0.389270] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1242client # [ 0.398755] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1243lb # [ 0.415570] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1244worker1 # [ 0.389294] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1245client # [ 0.398780] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1246worker1 # [ 0.389317] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1247lb # [ 0.415613] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1248worker2 # [ 0.533248] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1249client # [ 0.398802] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1250worker1 # [ 0.389341] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1251lb # [ 0.415659] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1252worker2 # [ 0.535263] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1253client # [ 0.398824] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1254worker1 # [ 0.389369] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1255lb # [ 0.415705] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1256worker1 # [ 0.389379] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1257client # [ 0.398847] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1258worker1 # [ 0.389383] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1259client # [ 0.398869] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1260lb # [ 0.415751] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1261client # [ 0.398892] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1262lb # [ 0.415798] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1263client # [ 0.398915] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1264lb # [ 0.415844] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1265client # [ 0.398938] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1266lb # [ 0.415891] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1267client # [ 0.398960] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1268lb # [ 0.415937] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1269client # [ 0.398982] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1270client # [ 0.399004] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1271lb # [ 0.415983] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1272client # [ 0.399026] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1273lb # [ 0.416044] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1274client # [ 0.399048] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1275lb # [ 0.416089] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1276lb # [ 0.416110] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1277worker1 # [ 0.454597] pci 0000:00:0a.0: enabling device (0000 -> 0002)1278lb # [ 0.416132] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1279lb # [ 0.416153] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1280worker2 # [ 0.546646] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1281lb # [ 0.416177] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1282lb # [ 0.416198] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1283lb # [ 0.416220] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1284lb # [ 0.416241] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1285lb # [ 0.416263] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1286lb # [ 0.416284] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1287client # [ 0.459114] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1288lb # [ 0.416306] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1289client # [ 0.459146] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1290lb # [ 0.416328] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1291client # [ 0.459169] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1292lb # [ 0.416349] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1293client # [ 0.459192] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1294lb # [ 0.416370] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1295client # [ 0.459223] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1296lb # [ 0.416391] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1297client # [ 0.459232] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1298lb # [ 0.416412] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1299worker2 # [ 0.557546] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1300client # [ 0.459237] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1301lb # [ 0.416433] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1302client # [ 0.460463] pci 0000:00:0a.0: enabling device (0000 -> 0002)1303lb # [ 0.416454] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1304lb # [ 0.416475] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1305worker2 # [ 0.560074] msm_serial: driver initialized1306lb # [ 0.416503] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1307worker2 # [ 0.560211] SuperH (H)SCI(F) driver initialized1308worker2 # [ 0.560265] STM32 USART driver initialized1309lb # [ 0.416524] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1310lb # [ 0.416550] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1311lb # [ 0.460590] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1312lb # [ 0.460597] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1313lb # [ 0.461782] pci 0000:00:0a.0: enabling device (0000 -> 0002)1314worker1 # [ 0.479528] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1315worker1 # [ 0.485627] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1316worker1 # [ 0.487596] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1317client # [ 0.493477] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1318client # [ 0.496574] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1319lb # [ 0.483919] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1320client # [ 0.498544] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1321worker1 # [ 0.497581] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1322worker2 # [ 0.591198] loop: module loaded1323worker1 # [ 0.499665] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1324worker2 # [ 0.591375] virtio_blk virtio5: 1/0/0 default/read/poll queues1325worker2 # [ 0.592151] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1326worker2 # [ 0.596898] megasas: 07.734.00.00-rc11327worker2 # [ 0.597599] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1328client # [ 0.508516] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1329lb # [ 0.493960] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1330client # [ 0.510490] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1331client # [ 0.514102] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1332worker1 # [ 0.509554] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1333worker1 # [ 0.512254] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1334worker1 # [ 0.514333] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1335worker1 # [ 0.516414] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1336lb # [ 0.498114] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1337lb # [ 0.500069] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1338lb # [ 0.502210] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1339worker2 # [ 0.609004] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001340worker2 # [ 0.609033] Intel/Sharp Extended Query Table at 0x00311341worker2 # [ 0.610714] Using buffer write method1342client # [ 0.522772] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1343lb # [ 0.504304] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1344worker2 # [ 0.610768] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1345worker1 # [ 0.526654] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1346worker1 # [ 0.529254] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1347client # [ 0.532398] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1348lb # [ 0.517962] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1349lb # [ 0.520172] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1350worker2 # [ 0.625794] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001351worker2 # [ 0.625827] Intel/Sharp Extended Query Table at 0x00311352worker2 # [ 0.627689] Using buffer write method1353worker2 # [ 0.627716] Concatenating MTD devices:1354worker2 # [ 0.627720] (0): "0.flash"1355worker2 # [ 0.627724] (1): "0.flash"1356worker2 # [ 0.627727] into device "0.flash"1357worker1 # [ 0.546631] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1358client # [ 0.543982] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1359lb # [ 0.531596] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1360client # [ 0.546066] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1361worker1 # [ 0.549163] msm_serial: driver initialized1362worker1 # [ 0.549296] SuperH (H)SCI(F) driver initialized1363worker1 # [ 0.549349] STM32 USART driver initialized1364client # [ 0.549933] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1365lb # [ 0.537796] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1366lb # [ 0.539951] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1367client # [ 0.564317] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1368client # [ 0.566939] msm_serial: driver initialized1369client # [ 0.567642] SuperH (H)SCI(F) driver initialized1370client # [ 0.567704] STM32 USART driver initialized1371lb # [ 0.557739] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1372lb # [ 0.560424] msm_serial: driver initialized1373lb # [ 0.561124] SuperH (H)SCI(F) driver initialized1374lb # [ 0.561187] STM32 USART driver initialized1375worker1 # [ 0.583234] loop: module loaded1376worker1 # [ 0.583425] virtio_blk virtio5: 1/0/0 default/read/poll queues1377worker1 # [ 0.584272] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1378worker1 # [ 0.590036] megasas: 07.734.00.00-rc11379worker1 # [ 0.590773] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1380client # [ 0.601535] loop: module loaded1381client # [ 0.601740] virtio_blk virtio5: 1/0/0 default/read/poll queues1382client # [ 0.602469] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1383worker1 # [ 0.601431] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001384worker1 # [ 0.601458] Intel/Sharp Extended Query Table at 0x00311385worker1 # [ 0.603642] Using buffer write method1386worker1 # [ 0.603708] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1387lb # [ 0.593289] loop: module loaded1388lb # [ 0.593474] virtio_blk virtio5: 1/0/0 default/read/poll queues1389worker1 # [ 0.605484] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001390worker1 # [ 0.605538] Intel/Sharp Extended Query Table at 0x00311391lb # [ 0.594221] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1392client # [ 0.607722] megasas: 07.734.00.00-rc11393worker1 # [ 0.607200] Using buffer write method1394worker1 # [ 0.607223] Concatenating MTD devices:1395client # [ 0.608449] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1396worker1 # [ 0.607227] (0): "0.flash"1397worker1 # [ 0.607231] (1): "0.flash"1398worker1 # [ 0.607234] into device "0.flash"1399lb # [ 0.601156] megasas: 07.734.00.00-rc11400client # [ 0.620447] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001401lb # [ 0.601829] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1402client # [ 0.620481] Intel/Sharp Extended Query Table at 0x00311403client # [ 0.622633] Using buffer write method1404lb # [ 0.603685] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001405client # [ 0.622707] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1406lb # [ 0.603709] Intel/Sharp Extended Query Table at 0x00311407lb # [ 0.605433] Using buffer write method1408lb # [ 0.605501] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1409lb # [ 0.607302] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001410lb # [ 0.607327] Intel/Sharp Extended Query Table at 0x00311411client # [ 0.632085] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001412client # [ 0.632112] Intel/Sharp Extended Query Table at 0x00311413client # [ 0.634291] Using buffer write method1414client # [ 0.634318] Concatenating MTD devices:1415client # [ 0.634321] (0): "0.flash"1416client # [ 0.634325] (1): "0.flash"1417client # [ 0.634329] into device "0.flash"1418lb # [ 0.625258] Using buffer write method1419lb # [ 0.625292] Concatenating MTD devices:1420lb # [ 0.625296] (0): "0.flash"1421lb # [ 0.625301] (1): "0.flash"1422lb # [ 0.625304] into device "0.flash"1423worker2 # [ 0.852830] Freeing initrd memory: 26148K1424worker2 # [ 0.858744] tun: Universal TUN/TAP device driver, 1.61425worker2 # [ 0.862483] thunder_xcv, ver 1.01426worker2 # [ 0.862528] thunder_bgx, ver 1.01427worker2 # [ 0.862550] nicpf, ver 1.01428worker2 # [ 0.863097] e1000: Intel(R) PRO/1000 Network Driver1429worker2 # [ 0.863104] e1000: Copyright (c) 1999-2006 Intel Corporation.1430worker2 # [ 0.863128] e1000e: Intel(R) PRO/1000 Network Driver1431worker2 # [ 0.863137] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1432worker2 # [ 0.863165] igb: Intel(R) Gigabit Ethernet Network Driver1433worker2 # [ 0.863171] igb: Copyright (c) 2007-2014 Intel Corporation.1434worker2 # [ 0.863192] igbvf: Intel(R) Gigabit Virtual Function Network Driver1435worker2 # [ 0.863198] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1436worker2 # [ 0.863328] sky2: driver version 1.301437worker2 # [ 0.872004] usbcore: registered new interface driver usb-storage1438worker2 # [ 0.872063] usbcore: registered new interface driver usbserial_generic1439worker2 # [ 0.872077] usbserial: USB Serial support registered for generic1440worker2 # [ 0.873028] ehci-pci 0000:00:0a.0: EHCI Host Controller1441worker2 # [ 0.873052] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11442worker2 # [ 0.873267] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001443worker2 # [ 0.878098] hv_vmbus: registering driver hyperv_keyboard1444worker2 # [ 0.879595] rtc-pl031 9010000.pl031: registered as rtc01445worker2 # [ 0.879622] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T16:48:48 UTC (1790009328)1446worker2 # [ 0.879912] i2c_dev: i2c /dev entries driver1447worker2 # [ 0.884844] sdhci: Secure Digital Host Controller Interface driver1448worker2 # [ 0.884853] sdhci: Copyright(c) Pierre Ossman1449worker2 # [ 0.885118] Synopsys Designware Multimedia Card Interface Driver1450worker2 # [ 0.885481] sdhci-pltfm: SDHCI platform and OF driver helper1451worker2 # [ 0.887202] hid: raw HID events driver (C) Jiri Kosina1452worker2 # [ 0.887451] usbcore: registered new interface driver usbhid1453worker2 # [ 0.887458] usbhid: USB HID core driver1454worker2 # [ 0.888403] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001455worker2 # [ 0.888714] hub 1-0:1.0: USB hub found1456worker2 # [ 0.888752] hub 1-0:1.0: 6 ports detected1457worker2 # [ 0.895286] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1458worker2 # [ 0.896834] drop_monitor: Initializing network drop monitor service1459worker2 # [ 0.896988] NET: Registered PF_INET6 protocol family1460worker2 # [ 0.899969] Segment Routing with IPv61461worker2 # [ 0.899999] In-situ OAM (IOAM) with IPv61462worker2 # [ 0.900027] NET: Registered PF_PACKET protocol family1463worker2 # [ 0.901670] 9pnet: Installing 9P2000 support1464worker2 # [ 0.903799] Key type dns_resolver registered1465worker2 # [ 0.910582] registered taskstats version 11466worker2 # [ 0.910723] Loading compiled-in X.509 certificates1467worker2 # [ 0.919411] Demotion targets for Node 0: null1468worker2 # [ 0.919516] Key type .fscrypt registered1469worker2 # [ 0.919523] Key type fscrypt-provisioning registered1470worker2 # [ 0.919615] ima: No TPM chip found, activating TPM-bypass!1471worker2 # [ 0.919635] ima: Allocated hash algorithm: sha11472worker2 # [ 0.919655] ima: No architecture policies found1473worker2 # [ 0.923583] input: gpio-keys as /devices/platform/gpio-keys/input/input01474worker2 # [ 0.940870] clk: Disabling unused clocks1475worker2 # [ 0.940895] PM: genpd: Disabling unused power domains1476worker1 # [ 0.853719] Freeing initrd memory: 26148K1477worker2 # [ 0.945098] Freeing unused kernel memory: 4736K1478worker2 # [ 0.945301] Run /init as init process1479worker1 # [ 0.859572] tun: Universal TUN/TAP device driver, 1.61480worker1 # [ 0.863179] thunder_xcv, ver 1.01481worker1 # [ 0.863223] thunder_bgx, ver 1.01482worker1 # [ 0.863245] nicpf, ver 1.01483worker1 # [ 0.863774] e1000: Intel(R) PRO/1000 Network Driver1484worker1 # [ 0.863781] e1000: Copyright (c) 1999-2006 Intel Corporation.1485worker1 # [ 0.863804] e1000e: Intel(R) PRO/1000 Network Driver1486worker1 # [ 0.863813] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1487worker1 # [ 0.863841] igb: Intel(R) Gigabit Ethernet Network Driver1488worker1 # [ 0.863847] igb: Copyright (c) 2007-2014 Intel Corporation.1489client # [ 0.870730] Freeing initrd memory: 26160K1490worker1 # [ 0.863869] igbvf: Intel(R) Gigabit Virtual Function Network Driver1491worker2 # [ 0.962564] systemd[1]: Successfully made /usr/ read-only.1492worker1 # [ 0.863875] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1493worker1 # [ 0.864004] sky2: driver version 1.301494worker1 # [ 0.866049] ehci-pci 0000:00:0a.0: EHCI Host Controller1495worker1 # [ 0.866074] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11496worker1 # [ 0.866296] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001497worker1 # [ 0.875774] usbcore: registered new interface driver usb-storage1498client # [ 0.876748] tun: Universal TUN/TAP device driver, 1.61499worker1 # [ 0.875903] usbcore: registered new interface driver usbserial_generic1500worker1 # [ 0.875917] usbserial: USB Serial support registered for generic1501worker1 # [ 0.876519] hv_vmbus: registering driver hyperv_keyboard1502lb # [ 0.862658] Freeing initrd memory: 26140K1503worker1 # [ 0.880058] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001504worker1 # [ 0.880437] hub 1-0:1.0: USB hub found1505worker1 # [ 0.880458] hub 1-0:1.0: 6 ports detected1506worker1 # [ 0.882701] rtc-pl031 9010000.pl031: registered as rtc01507lb # [ 0.868659] tun: Universal TUN/TAP device driver, 1.61508client # [ 0.880637] thunder_xcv, ver 1.01509worker1 # [ 0.882729] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T16:48:48 UTC (1790009328)1510client # [ 0.880681] thunder_bgx, ver 1.01511client # [ 0.880704] nicpf, ver 1.01512worker1 # [ 0.883034] i2c_dev: i2c /dev entries driver1513client # [ 0.881243] e1000: Intel(R) PRO/1000 Network Driver1514client # [ 0.881251] e1000: Copyright (c) 1999-2006 Intel Corporation.1515client # [ 0.881275] e1000e: Intel(R) PRO/1000 Network Driver1516client # [ 0.881283] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1517lb # [ 0.872302] thunder_xcv, ver 1.01518lb # [ 0.872346] thunder_bgx, ver 1.01519client # [ 0.881311] igb: Intel(R) Gigabit Ethernet Network Driver1520lb # [ 0.872368] nicpf, ver 1.01521client # [ 0.881317] igb: Copyright (c) 2007-2014 Intel Corporation.1522worker1 # [ 0.887908] sdhci: Secure Digital Host Controller Interface driver1523worker1 # [ 0.887922] sdhci: Copyright(c) Pierre Ossman1524client # [ 0.881341] igbvf: Intel(R) Gigabit Virtual Function Network Driver1525worker1 # [ 0.888194] Synopsys Designware Multimedia Card Interface Driver1526client # [ 0.881347] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1527client # [ 0.881483] sky2: driver version 1.301528worker1 # [ 0.888548] sdhci-pltfm: SDHCI platform and OF driver helper1529client # [ 0.883044] usbcore: registered new interface driver usb-storage1530client # [ 0.883837] ehci-pci 0000:00:0a.0: EHCI Host Controller1531lb # [ 0.874169] e1000: Intel(R) PRO/1000 Network Driver1532worker1 # [ 0.892863] hid: raw HID events driver (C) Jiri Kosina1533client # [ 0.883862] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11534lb # [ 0.874180] e1000: Copyright (c) 1999-2006 Intel Corporation.1535worker1 # [ 0.893093] usbcore: registered new interface driver usbhid1536client # [ 0.884164] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001537lb # [ 0.874204] e1000e: Intel(R) PRO/1000 Network Driver1538worker1 # [ 0.893099] usbhid: USB HID core driver1539lb # [ 0.874213] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1540lb # [ 0.874242] igb: Intel(R) Gigabit Ethernet Network Driver1541lb # [ 0.874248] igb: Copyright (c) 2007-2014 Intel Corporation.1542client # [ 0.894358] usbcore: registered new interface driver usbserial_generic1543lb # [ 0.874271] igbvf: Intel(R) Gigabit Virtual Function Network Driver1544client # [ 0.894379] usbserial: USB Serial support registered for generic1545worker1 # [ 0.895900] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1546client # [ 0.895020] hv_vmbus: registering driver hyperv_keyboard1547lb # [ 0.874277] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1548lb # [ 0.874409] sky2: driver version 1.301549worker1 # [ 0.897380] drop_monitor: Initializing network drop monitor service1550client # [ 0.895124] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001551client # [ 0.895436] hub 1-0:1.0: USB hub found1552lb # [ 0.875973] usbcore: registered new interface driver usb-storage1553worker1 # [ 0.899304] NET: Registered PF_INET6 protocol family1554client # [ 0.895471] hub 1-0:1.0: 6 ports detected1555lb # [ 0.876066] usbcore: registered new interface driver usbserial_generic1556worker1 # [ 0.900437] Segment Routing with IPv61557lb # [ 0.876081] usbserial: USB Serial support registered for generic1558worker1 # [ 0.900469] In-situ OAM (IOAM) with IPv61559lb # [ 0.877248] ehci-pci 0000:00:0a.0: EHCI Host Controller1560worker1 # [ 0.900498] NET: Registered PF_PACKET protocol family1561client # [ 0.900471] rtc-pl031 9010000.pl031: registered as rtc01562lb # [ 0.877273] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11563lb # [ 0.877544] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001564client # [ 0.900499] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T16:48:48 UTC (1790009328)1565client # [ 0.900801] i2c_dev: i2c /dev entries driver1566lb # [ 0.888018] hv_vmbus: registering driver hyperv_keyboard1567lb # [ 0.888605] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001568worker1 # [ 0.902320] 9pnet: Installing 9P2000 support1569lb # [ 0.890282] hub 1-0:1.0: USB hub found1570worker1 # [ 0.904393] Key type dns_resolver registered1571lb # [ 0.890770] hub 1-0:1.0: 6 ports detected1572client # [ 0.905724] sdhci: Secure Digital Host Controller Interface driver1573lb # [ 0.891959] rtc-pl031 9010000.pl031: registered as rtc01574client # [ 0.905736] sdhci: Copyright(c) Pierre Ossman1575client # [ 0.905999] Synopsys Designware Multimedia Card Interface Driver1576lb # [ 0.891984] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T16:48:48 UTC (1790009328)1577client # [ 0.906367] sdhci-pltfm: SDHCI platform and OF driver helper1578lb # [ 0.892288] i2c_dev: i2c /dev entries driver1579client # [ 0.910862] hid: raw HID events driver (C) Jiri Kosina1580worker1 # [ 0.910973] registered taskstats version 11581client # [ 0.911814] usbcore: registered new interface driver usbhid1582client # [ 0.911823] usbhid: USB HID core driver1583worker1 # [ 0.911118] Loading compiled-in X.509 certificates1584client # [ 0.914062] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1585lb # [ 0.897192] sdhci: Secure Digital Host Controller Interface driver1586lb # [ 0.897202] sdhci: Copyright(c) Pierre Ossman1587lb # [ 0.897468] Synopsys Designware Multimedia Card Interface Driver1588client # [ 0.916665] drop_monitor: Initializing network drop monitor service1589lb # [ 0.897837] sdhci-pltfm: SDHCI platform and OF driver helper1590client # [ 0.916837] NET: Registered PF_INET6 protocol family1591lb # [ 0.899455] hid: raw HID events driver (C) Jiri Kosina1592client # [ 0.918759] Segment Routing with IPv61593lb # [ 0.899681] usbcore: registered new interface driver usbhid1594client # [ 0.918778] In-situ OAM (IOAM) with IPv61595worker1 # [ 0.919565] Demotion targets for Node 0: null1596lb # [ 0.899688] usbhid: USB HID core driver1597worker1 # [ 0.919666] Key type .fscrypt registered1598client # [ 0.918807] NET: Registered PF_PACKET protocol family1599worker1 # [ 0.919672] Key type fscrypt-provisioning registered1600worker1 # [ 0.919765] ima: No TPM chip found, activating TPM-bypass!1601worker1 # [ 0.919785] ima: Allocated hash algorithm: sha11602worker1 # [ 0.919805] ima: No architecture policies found1603lb # [ 0.905204] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1604client # [ 0.920691] 9pnet: Installing 9P2000 support1605lb # [ 0.906701] drop_monitor: Initializing network drop monitor service1606client # [ 0.922815] Key type dns_resolver registered1607worker1 # [ 0.923832] input: gpio-keys as /devices/platform/gpio-keys/input/input01608lb # [ 0.906873] NET: Registered PF_INET6 protocol family1609lb # [ 0.909828] Segment Routing with IPv61610lb # [ 0.909848] In-situ OAM (IOAM) with IPv61611lb # [ 0.909877] NET: Registered PF_PACKET protocol family1612lb # [ 0.911493] 9pnet: Installing 9P2000 support1613client # [ 0.929521] registered taskstats version 11614lb # [ 0.914276] Key type dns_resolver registered1615client # [ 0.929682] Loading compiled-in X.509 certificates1616client # [ 0.938163] Demotion targets for Node 0: null1617client # [ 0.938268] Key type .fscrypt registered1618lb # [ 0.920578] registered taskstats version 11619client # [ 0.938275] Key type fscrypt-provisioning registered1620lb # [ 0.920731] Loading compiled-in X.509 certificates1621client # [ 0.938371] ima: No TPM chip found, activating TPM-bypass!1622worker1 # [ 0.941364] clk: Disabling unused clocks1623client # [ 0.938390] ima: Allocated hash algorithm: sha11624worker1 # [ 0.941388] PM: genpd: Disabling unused power domains1625client # [ 0.938410] ima: No architecture policies found1626client # [ 0.942627] input: gpio-keys as /devices/platform/gpio-keys/input/input01627lb # [ 0.929031] Demotion targets for Node 0: null1628lb # [ 0.929135] Key type .fscrypt registered1629worker1 # [ 0.945640] Freeing unused kernel memory: 4736K1630lb # [ 0.929142] Key type fscrypt-provisioning registered1631worker1 # [ 0.945836] Run /init as init process1632lb # [ 0.929231] ima: No TPM chip found, activating TPM-bypass!1633lb # [ 0.929250] ima: Allocated hash algorithm: sha11634lb # [ 0.929272] ima: No architecture policies found1635lb # [ 0.933544] input: gpio-keys as /devices/platform/gpio-keys/input/input01636worker1 # [ 0.959603] systemd[1]: Successfully made /usr/ read-only.1637client # [ 0.960789] clk: Disabling unused clocks1638client # [ 0.960814] PM: genpd: Disabling unused power domains1639client # [ 0.965058] Freeing unused kernel memory: 4736K1640client # [ 0.965242] Run /init as init process1641lb # [ 0.951273] clk: Disabling unused clocks1642lb # [ 0.951297] PM: genpd: Disabling unused power domains1643lb # [ 0.955493] Freeing unused kernel memory: 4736K1644lb # [ 0.955692] Run /init as init process1645client # [ 0.981319] systemd[1]: Successfully made /usr/ read-only.1646lb # [ 0.970747] systemd[1]: Successfully made /usr/ read-only.1647worker2 # [ 1.140401] usb 1-1: new high-speed USB device number 2 using ehci-pci1648worker1 # [ 1.125461] usb 1-1: new high-speed USB device number 2 using ehci-pci1649client # [ 1.143162] usb 1-1: new high-speed USB device number 2 using ehci-pci1650lb # [ 1.136647] usb 1-1: new high-speed USB device number 2 using ehci-pci1651worker2 # [ 1.289969] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11652worker2 # [ 1.297248] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1653worker2 # [ 1.304346] systemd[1]: Detected virtualization qemu.1654worker2 # [ 1.305572] systemd[1]: Detected architecture arm64.1655worker2 # [ 1.306624] systemd[1]: Running in initrd.1656worker2 # [ 1.308121] systemd[1]: Initializing machine ID from random generator.1657worker2 # [ 1.309836] systemd[1]: Hostname set to <worker2>.1658worker1 # [ 1.277747] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11659worker2 # [ 1.372506] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01660client # [ 1.292918] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11661worker1 # [ 1.294427] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1662worker1 # [ 1.306498] systemd[1]: Detected virtualization qemu.1663lb # [ 1.289041] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11664worker1 # [ 1.308522] systemd[1]: Detected architecture arm64.1665worker1 # [ 1.310499] systemd[1]: Running in initrd.1666worker1 # [ 1.313119] systemd[1]: Initializing machine ID from random generator.1667worker1 # [ 1.316080] systemd[1]: Hostname set to <worker1>.1668client # [ 1.316133] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1669client # [ 1.328283] systemd[1]: Detected virtualization qemu.1670client # [ 1.330275] systemd[1]: Detected architecture arm64.1671lb # [ 1.305644] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1672client # [ 1.332285] systemd[1]: Running in initrd.1673lb # [ 1.317535] systemd[1]: Detected virtualization qemu.1674lb # [ 1.319506] systemd[1]: Detected architecture arm64.1675client # [ 1.334839] systemd[1]: Initializing machine ID from random generator.1676lb # [ 1.321557] systemd[1]: Running in initrd.1677client # [ 1.338087] systemd[1]: Hostname set to <client>.1678lb # [ 1.324126] systemd[1]: Initializing machine ID from random generator.1679lb # [ 1.327199] systemd[1]: Hostname set to <lb>.1680worker1 # [ 1.361661] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01681client # [ 1.383415] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01682lb # [ 1.376823] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01683worker2 # [ 1.492400] usb 1-2: new high-speed USB device number 3 using ehci-pci1684worker2 # [ 1.540455] systemd[1]: bpf-restrict-fs: LSM BPF program attached1685worker1 # [ 1.481438] usb 1-2: new high-speed USB device number 3 using ehci-pci1686worker2 # [ 1.595126] systemd[1]: Queued start job for default target Initrd Default Target.1687client # [ 1.507131] usb 1-2: new high-speed USB device number 3 using ehci-pci1688lb # [ 1.496607] usb 1-2: new high-speed USB device number 3 using ehci-pci1689worker2 # [ 1.606337] systemd[1]: Created slice Slice /system/modprobe.1690worker2 # [ 1.607323] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1691worker2 # [ 1.608512] systemd[1]: Expecting device /dev/disk/by-label/nixos...1692worker2 # [ 1.609354] systemd[1]: Reached target Path Units.1693worker2 # [ 1.610002] systemd[1]: Reached target Slice Units.1694worker2 # [ 1.610659] systemd[1]: Reached target Swaps.1695worker2 # [ 1.611269] systemd[1]: Reached target Timer Units.1696worker2 # [ 1.612054] systemd[1]: Listening on D-Bus System Message Bus Socket.1697worker2 # [ 1.613458] systemd[1]: Listening on Journal Socket (/dev/log).1698worker2 # [ 1.614367] systemd[1]: Listening on Journal Sockets.1699worker2 # [ 1.615163] systemd[1]: Listening on udev Control Socket.1700worker2 # [ 1.616043] systemd[1]: Listening on udev Kernel Socket.1701worker2 # [ 1.616823] systemd[1]: Reached target Socket Units.1702worker2 # [ 1.618789] systemd[1]: Starting Create List of Static Device Nodes...1703worker2 # [ 1.629684] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1704worker2 # [ 1.630658] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1705worker2 # [ 1.640464] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21706worker2 # [ 1.642280] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01707worker2 # [ 1.649424] systemd[1]: Mounting Kernel Configuration File System...1708worker2 # [ 1.660550] systemd[1]: Starting Journal Service...1709worker2 # [ 1.662576] systemd[1]: Starting Load Kernel Modules...1710worker2 # [ 1.663352] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1711worker2 # [ 1.676504] systemd[1]: Starting Coldplug All udev Devices...1712worker2 # [ 1.685901] systemd[1]: Finished Create List of Static Device Nodes.1713worker2 # [ 1.687570] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1714worker1 # [ 1.602148] systemd[1]: bpf-restrict-fs: LSM BPF program attached1715worker2 # [ 1.696859] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1716worker2 # [ 1.697229] systemd[1]: Mounted Kernel Configuration File System.1717worker2 # [ 1.704855] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1718worker1 # [ 1.637382] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21719worker1 # [ 1.640399] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01720worker2 # [ 1.733819] systemd-journald[73]: Collecting audit messages is disabled.1721worker2 # [ 1.735304] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1722lb # [ 1.633442] systemd[1]: bpf-restrict-fs: LSM BPF program attached1723worker2 # [ 1.742539] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1724client # [ 1.658871] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21725client # [ 1.661591] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01726worker2 # [ 1.756639] systemd[1]: Starting Create Static Device Nodes in /dev...1727worker2 # [ 1.760490] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1728lb # [ 1.655242] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21729lb # [ 1.661930] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01730worker2 # [ 1.772628] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01731worker2 # [ 1.772866] [drm] features: -virgl +edid -resource_blob -host_visible1732worker2 # [ 1.772875] [drm] features: -context_init1733worker2 # [ 1.773581] [drm] number of scanouts: 11734worker2 # [ 1.773600] [drm] number of cap sets: 01735worker1 # [ 1.688551] systemd[1]: Queued start job for default target Initrd Default Target.1736worker2 # [ 1.788884] systemd[1]: Finished Create Static Device Nodes in /dev.1737worker1 # [ 1.699684] systemd[1]: Created slice Slice /system/modprobe.1738worker2 # [ 1.789211] systemd[1]: Reached target Preparation for Local File Systems.1739worker2 # [ 1.789239] systemd[1]: Reached target Local File Systems.1740worker1 # [ 1.700810] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1741worker1 # [ 1.702096] systemd[1]: Expecting device /dev/disk/by-label/nixos...1742worker2 # [ 1.793090] systemd[1]: Starting Rule-based Manager for Device Events and Files...1743worker1 # [ 1.703084] systemd[1]: Reached target Path Units.1744worker1 # [ 1.703828] systemd[1]: Reached target Slice Units.1745worker1 # [ 1.704597] systemd[1]: Reached target Swaps.1746worker1 # [ 1.705289] systemd[1]: Reached target Timer Units.1747worker1 # [ 1.706278] systemd[1]: Listening on D-Bus System Message Bus Socket.1748worker1 # [ 1.707406] systemd[1]: Listening on Journal Socket (/dev/log).1749worker1 # [ 1.708459] systemd[1]: Listening on Journal Sockets.1750worker1 # [ 1.709376] systemd[1]: Listening on udev Control Socket.1751worker1 # [ 1.710487] systemd[1]: Listening on udev Kernel Socket.1752worker1 # [ 1.711325] systemd[1]: Reached target Socket Units.1753worker1 # [ 1.713535] systemd[1]: Starting Create List of Static Device Nodes...1754worker2 # [ 1.800622] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1755worker2 # [ 1.800647] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01756worker1 # [ 1.721545] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1757worker1 # [ 1.722622] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1758worker1 # [ 1.729655] systemd[1]: Mounting Kernel Configuration File System...1759worker2 # [ 1.828854] Console: switching to colour frame buffer device 160x501760client # [ 1.748927] systemd[1]: bpf-restrict-fs: LSM BPF program attached1761worker2 # [ 1.835930] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1762worker1 # [ 1.753632] systemd[1]: Starting Journal Service...1763lb # [ 1.747850] systemd[1]: Queued start job for default target Initrd Default Target.1764worker1 # [ 1.768376] systemd[1]: Starting Load Kernel Modules...1765worker1 # [ 1.768483] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1766lb # [ 1.754686] systemd[1]: Created slice Slice /system/modprobe.1767lb # [ 1.755734] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1768lb # [ 1.756991] systemd[1]: Expecting device /dev/disk/by-label/nixos...1769lb # [ 1.757024] systemd[1]: Reached target Path Units.1770lb # [ 1.757042] systemd[1]: Reached target Slice Units.1771lb # [ 1.757060] systemd[1]: Reached target Swaps.1772lb # [ 1.757078] systemd[1]: Reached target Timer Units.1773lb # [ 1.757245] systemd[1]: Listening on D-Bus System Message Bus Socket.1774worker2 # [ 1.868920] systemd[1]: Finished Load Kernel Modules.1775lb # [ 1.757384] systemd[1]: Listening on Journal Socket (/dev/log).1776worker1 # [ 1.777813] systemd[1]: Starting Coldplug All udev Devices...1777lb # [ 1.757505] systemd[1]: Listening on Journal Sockets.1778lb # [ 1.757622] systemd[1]: Listening on udev Control Socket.1779lb # [ 1.757728] systemd[1]: Listening on udev Kernel Socket.1780lb # [ 1.757748] systemd[1]: Reached target Socket Units.1781worker2 # [ 1.872690] systemd[1]: Starting Apply Kernel Variables...1782lb # [ 1.767561] systemd[1]: Starting Create List of Static Device Nodes...1783lb # [ 1.776706] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1784lb # [ 1.777722] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1785worker1 # [ 1.793602] systemd[1]: Finished Create List of Static Device Nodes.1786worker1 # [ 1.794454] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1787worker1 # [ 1.803284] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1788lb # [ 1.789984] systemd[1]: Mounting Kernel Configuration File System...1789worker1 # [ 1.809899] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1790worker1 # [ 1.810279] systemd[1]: Mounted Kernel Configuration File System.1791worker1 # [ 1.817850] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1792lb # [ 1.808767] systemd[1]: Starting Journal Service...1793worker1 # [ 1.824665] systemd-journald[73]: Collecting audit messages is disabled.1794client # [ 1.824879] systemd[1]: Queued start job for default target Initrd Default Target.1795lb # [ 1.816512] systemd[1]: Starting Load Kernel Modules...1796worker2 # [ 1.922488] systemd[1]: Finished Apply Kernel Variables.1797lb # [ 1.817522] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1798client # [ 1.833359] systemd[1]: Created slice Slice /system/modprobe.1799client # [ 1.834522] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1800client # [ 1.835983] systemd[1]: Expecting device /dev/disk/by-label/nixos...1801client # [ 1.837006] systemd[1]: Reached target Path Units.1802client # [ 1.837789] systemd[1]: Reached target Slice Units.1803worker2 # [ 1.912833] systemd-modules-load[74]: Inserted module 'dm_mod'1804client # [ 1.838597] systemd[1]: Reached target Swaps.1805client # [ 1.839376] systemd[1]: Reached target Timer Units.1806client # [ 1.840362] systemd[1]: Listening on D-Bus System Message Bus Socket.1807worker2 # [ 1.930251] systemd[1]: Started Journal Service.1808client # [ 1.841546] systemd[1]: Listening on Journal Socket (/dev/log).1809client # [ 1.842642] systemd[1]: Listening on Journal Sockets.1810worker1 # [ 1.837736] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1811client # [ 1.843672] systemd[1]: Listening on udev Control Socket.1812worker1 # [ 1.842483] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01813client # [ 1.844672] systemd[1]: Listening on udev Kernel Socket.1814worker1 # [ 1.842711] [drm] features: -virgl +edid -resource_blob -host_visible1815worker1 # [ 1.842721] [drm] features: -context_init1816client # [ 1.845551] systemd[1]: Reached target Socket Units.1817worker2 # [ 1.920689] systemd-modules-load[74]: Module 'virtio_balloon' is built in1818worker1 # [ 1.843428] [drm] number of scanouts: 11819worker1 # [ 1.843445] [drm] number of cap sets: 01820worker2 # [ 1.921899] systemd-modules-load[74]: Module 'virtio_console' is built in1821client # [ 1.847936] systemd[1]: Starting Create List of Static Device Nodes...1822worker2 # [ 1.925169] systemd-modules-load[74]: Inserted module 'virtio_gpu'1823client # [ 1.855256] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1824lb # [ 1.840745] systemd[1]: Starting Coldplug All udev Devices...1825client # [ 1.856390] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1826worker2 # [ 1.933733] systemd-modules-load[74]: Module 'virtio_rng' is built in1827worker1 # [ 1.857998] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1828worker1 # [ 1.860473] systemd[1]: Starting Create Static Device Nodes in /dev...1829worker2 # [ 1.936481] systemd-udevd[81]: Using default interface naming scheme 'v261'.1830client # [ 1.862784] systemd[1]: Mounting Kernel Configuration File System...1831lb # [ 1.848723] systemd[1]: Finished Create List of Static Device Nodes.1832worker2 # [ 1.941609] systemd[1]: Starting Create System Files and Directories...1833lb # [ 1.849528] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1834lb # [ 1.849823] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1835lb # [ 1.858914] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1836worker1 # [ 1.869716] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1837worker1 # [ 1.869736] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01838client # [ 1.880519] systemd[1]: Starting Journal Service...1839worker2 # [ 1.960240] systemd[1]: Finished Create System Files and Directories.1840lb # [ 1.871575] systemd-journald[73]: Collecting audit messages is disabled.1841lb # [ 1.871650] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1842lb # [ 1.875047] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1843lb # [ 1.875591] systemd[1]: Mounted Kernel Configuration File System.1844worker2 # [ 1.968289] systemd[1]: Started Rule-based Manager for Device Events and Files.1845client # [ 1.902716] systemd[1]: Starting Load Kernel Modules...1846lb # [ 1.884401] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01847client # [ 1.903951] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1848worker1 # [ 1.904429] systemd[1]: Finished Create Static Device Nodes in /dev.1849worker1 # [ 1.904766] systemd[1]: Reached target Preparation for Local File Systems.1850worker1 # [ 1.904790] systemd[1]: Reached target Local File Systems.1851lb # [ 1.896814] [drm] features: -virgl +edid -resource_blob -host_visible1852lb # [ 1.896832] [drm] features: -context_init1853lb # [ 1.897570] [drm] number of scanouts: 11854lb # [ 1.897589] [drm] number of cap sets: 01855worker1 # [ 1.913641] systemd[1]: Starting Rule-based Manager for Device Events and Files...1856client # [ 1.926752] systemd-journald[73]: Collecting audit messages is disabled.1857client # [ 1.926891] systemd[1]: Starting Coldplug All udev Devices...1858lb # [ 1.908893] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1859worker1 # [ 1.930284] Console: switching to colour frame buffer device 160x501860lb # [ 1.908911] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01861worker1 # [ 1.930957] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1862client # [ 1.939227] systemd[1]: Finished Create List of Static Device Nodes.1863client # [ 1.940042] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1864client # [ 1.940319] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1865client # [ 1.940579] systemd[1]: Mounted Kernel Configuration File System.1866lb # [ 1.928734] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1867worker1 # [ 1.946052] systemd[1]: Finished Load Kernel Modules.1868lb # [ 1.931127] Console: switching to colour frame buffer device 160x501869worker1 # [ 1.949758] systemd[1]: Starting Apply Kernel Variables...1870lb # [ 1.936957] systemd[1]: Starting Create Static Device Nodes in /dev...1871client # [ 1.959512] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1872lb # [ 1.949151] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1873worker2 # [ 2.040188] systemd[1]: Starting Virtual Console Setup...1874lb # [ 1.953898] systemd[1]: Finished Load Kernel Modules.1875client # [ 1.982002] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1876lb # [ 1.968912] systemd[1]: Starting Apply Kernel Variables...1877client # [ 1.999683] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1878client # [ 2.002153] systemd[1]: Starting Create Static Device Nodes in /dev...1879worker1 # [ 2.002093] systemd[1]: Finished Apply Kernel Variables.1880client # [ 2.006404] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1881worker1 # [ 2.010538] systemd[1]: Started Journal Service.1882lb # [ 1.996959] systemd[1]: Finished Create Static Device Nodes in /dev.1883lb # [ 1.997182] systemd[1]: Reached target Preparation for Local File Systems.1884lb # [ 1.997216] systemd[1]: Reached target Local File Systems.1885client # [ 2.015075] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01886client # [ 2.015347] [drm] features: -virgl +edid -resource_blob -host_visible1887client # [ 2.015357] [drm] features: -context_init1888client # [ 2.016084] [drm] number of scanouts: 11889client # [ 2.016102] [drm] number of cap sets: 01890lb # [ 2.002073] systemd[1]: Starting Rule-based Manager for Device Events and Files...1891worker1 # [ 2.008275] systemd-modules-load[74]: Inserted module 'dm_mod'1892worker1 # [ 2.012405] systemd-modules-load[74]: Module 'virtio_balloon' is built in1893worker1 # [ 2.013489] systemd-modules-load[74]: Module 'virtio_console' is built in1894worker1 # [ 2.014533] systemd-modules-load[74]: Inserted module 'virtio_gpu'1895worker1 # [ 2.015523] systemd-modules-load[74]: Module 'virtio_rng' is built in1896worker2 # [ 2.108150] systemd-vconsole-setup[105]: Configuration of first virtual console was skipped, ignoring remaining ones.1897worker2 # [ 2.110880] systemd[1]: Finished Virtual Console Setup.1898client # [ 2.031757] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1899client # [ 2.031776] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01900worker1 # [ 2.029834] systemd-udevd[80]: Using default interface naming scheme 'v261'.1901worker1 # [ 2.030960] systemd[1]: Starting Create System Files and Directories...1902lb # [ 2.032727] systemd[1]: Finished Apply Kernel Variables.1903client # [ 2.059740] systemd[1]: Finished Create Static Device Nodes in /dev.1904worker1 # [ 2.048779] systemd[1]: Finished Create System Files and Directories.1905client # [ 2.060105] systemd[1]: Reached target Preparation for Local File Systems.1906client # [ 2.060133] systemd[1]: Reached target Local File Systems.1907client # [ 2.064107] systemd[1]: Starting Rule-based Manager for Device Events and Files...1908lb # [ 2.052769] systemd[1]: Started Journal Service.1909lb # [ 2.040290] systemd-modules-load[74]: Inserted module 'dm_mod'1910worker1 # [ 2.065944] systemd[1]: Started Rule-based Manager for Device Events and Files.1911client # [ 2.071507] Console: switching to colour frame buffer device 160x501912client # [ 2.078097] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1913lb # [ 2.052118] systemd-modules-load[74]: Module 'virtio_balloon' is built in1914lb # [ 2.053426] systemd-modules-load[74]: Module 'virtio_console' is built in1915lb # [ 2.056234] systemd-modules-load[74]: Inserted module 'virtio_gpu'1916lb # [ 2.057247] systemd-modules-load[74]: Module 'virtio_rng' is built in1917lb # [ 2.068304] systemd[1]: Starting Create System Files and Directories...1918client # [ 2.103811] systemd[1]: Finished Load Kernel Modules.1919lb # [ 2.084364] systemd-udevd[81]: Using default interface naming scheme 'v261'.1920client # [ 2.112195] systemd[1]: Starting Apply Kernel Variables...1921lb # [ 2.087675] systemd[1]: Finished Create System Files and Directories.1922client # [ 2.113105] systemd-modules-load[75]: Inserted module 'dm_mod'1923client # [ 2.131232] systemd[1]: Started Journal Service.1924client # [ 2.119072] systemd-modules-load[75]: Module 'virtio_balloon' is built in1925worker1 # [ 2.124169] systemd[1]: Starting Virtual Console Setup...1926client # [ 2.124211] systemd-modules-load[75]: Module 'virtio_console' is built in1927lb # [ 2.114339] systemd[1]: Started Rule-based Manager for Device Events and Files.1928client # [ 2.132343] systemd-modules-load[75]: Inserted module 'virtio_gpu'1929client # [ 2.140311] systemd-modules-load[75]: Module 'virtio_rng' is built in1930client # [ 2.144365] systemd[1]: Starting Create System Files and Directories...1931client # [ 2.164948] systemd[1]: Finished Apply Kernel Variables.1932client # [ 2.169207] systemd[1]: Finished Create System Files and Directories.1933client # [ 2.171022] systemd-udevd[79]: Using default interface naming scheme 'v261'.1934worker1 # [ 2.173914] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1935worker1 # [ 2.175777] systemd[1]: Finished Virtual Console Setup.1936lb # [ 2.172157] systemd[1]: Starting Virtual Console Setup...1937client # [ 2.202230] systemd[1]: Started Rule-based Manager for Device Events and Files.1938lb # [ 2.224509] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1939lb # [ 2.227958] systemd[1]: Finished Virtual Console Setup.1940client # [ 2.259922] systemd[1]: Starting Virtual Console Setup...1941client # [ 2.308462] systemd-vconsole-setup[102]: Configuration of first virtual console was skipped, ignoring remaining ones.1942client # [ 2.311809] systemd[1]: Finished Virtual Console Setup.1943worker2 # [ 2.714827] systemd[1]: Finished Coldplug All udev Devices.1944worker2 # [ 2.715787] systemd[1]: Reached target System Initialization.1945worker2 # [ 2.716677] systemd[1]: Reached target Basic System.1946worker2 # [ 2.829312] (udev-worker)[98]: Network interface NamePolicy= disabled on kernel command line.1947worker1 # [ 2.768995] systemd[1]: Finished Coldplug All udev Devices.1948worker1 # [ 2.769933] systemd[1]: Reached target System Initialization.1949worker1 # [ 2.770758] systemd[1]: Reached target Basic System.1950worker2 # [ 2.899355] (udev-worker)[102]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1951worker2 # [ 2.904272] (udev-worker)[102]: Network interface NamePolicy= disabled on kernel command line.1952lb # [ 2.827653] systemd[1]: Finished Coldplug All udev Devices.1953lb # [ 2.828689] systemd[1]: Reached target System Initialization.1954lb # [ 2.829486] systemd[1]: Reached target Basic System.1955worker2 # [ 2.957956] systemd[1]: Found device /dev/disk/by-label/nixos.1956worker2 # [ 2.964152] systemd[1]: Reached target Initrd Root Device.1957worker2 # [ 2.968781] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1958client # [ 2.907836] systemd[1]: Finished Coldplug All udev Devices.1959client # [ 2.908965] systemd[1]: Reached target System Initialization.1960client # [ 2.909781] systemd[1]: Reached target Basic System.1961worker2 # [ 3.015013] systemd-fsck[114]: nixos: clean, 12/65536 files, 13019/262144 blocks1962worker2 # [ 3.020518] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1963worker2 # [ 3.024771] systemd[1]: Mounting /sysroot...1964worker1 # [ 2.949844] (udev-worker)[92]: Network interface NamePolicy= disabled on kernel command line.1965worker1 # [ 2.954334] (udev-worker)[102]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1966worker1 # [ 2.959044] (udev-worker)[102]: Network interface NamePolicy= disabled on kernel command line.1967worker2 # [ 3.075647] EXT4-fs (vda): mounted filesystem af75b1bc-0c30-4a07-b9d8-a5d001cce35c r/w with ordered data mode. Quota mode: none.1968lb # [ 2.955804] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line.1969worker2 # [ 3.063779] systemd[1]: Mounted /sysroot.1970worker2 # [ 3.065717] systemd[1]: Reached target Initrd Root File System.1971worker2 # [ 3.070237] systemd[1]: Starting Mountpoints Configured in the Real Root...1972worker2 # [ 3.095477] systemd-sysroot-fstab-check[122]: /sysroot should be mounted in the initrd, will request daemon-reload.1973worker2 # [ 3.100097] systemd[1]: Reload requested from client PID 122 ('systemd-sysroot') (unit initrd-parse-etc.service)...1974worker2 # [ 3.104631] systemd[1]: Reloading...1975worker1 # [ 3.028505] systemd[1]: Found device /dev/disk/by-label/nixos.1976lb # [ 3.010689] (udev-worker)[97]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1977worker1 # [ 3.031156] systemd[1]: Reached target Initrd Root Device.1978lb # [ 3.016241] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line.1979worker1 # [ 3.036470] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1980client # [ 3.080511] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line.1981worker1 # [ 3.083632] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks1982worker1 # [ 3.095176] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1983worker1 # [ 3.097167] systemd[1]: Mounting /sysroot...1984lb # [ 3.081486] systemd[1]: Found device /dev/disk/by-label/nixos.1985client # [ 3.097397] (udev-worker)[94]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1986client # [ 3.104168] (udev-worker)[94]: Network interface NamePolicy= disabled on kernel command line.1987lb # [ 3.092779] systemd[1]: Reached target Initrd Root Device.1988lb # [ 3.094630] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1989worker1 # [ 3.145857] EXT4-fs (vda): mounted filesystem bfe28ab3-f4bb-4f60-b1ac-da25a5b20f1e r/w with ordered data mode. Quota mode: none.1990worker1 # [ 3.140743] systemd[1]: Mounted /sysroot.1991worker1 # [ 3.141806] systemd[1]: Reached target Initrd Root File System.1992worker1 # [ 3.145396] systemd[1]: Starting Mountpoints Configured in the Real Root...1993lb # [ 3.141059] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks1994lb # [ 3.147697] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1995worker1 # [ 3.168372] systemd-sysroot-fstab-check[120]: /sysroot should be mounted in the initrd, will request daemon-reload.1996client # [ 3.168432] systemd[1]: Found device /dev/disk/by-label/nixos.1997lb # [ 3.152196] systemd[1]: Mounting /sysroot...1998worker1 # [ 3.173646] systemd[1]: Reload requested from client PID 120 ('systemd-sysroot') (unit initrd-parse-etc.service)...1999client # [ 3.172976] systemd[1]: Reached target Initrd Root Device.2000worker1 # [ 3.177296] systemd[1]: Reloading...
loading 1,902 hidden lines…
3903worker2 # [ 33.729627] nix-grpc-daemon[780]: ts=2026-09-21T16:49:21Z level=info event=scheduler_connected addr=lb:500513904worker2 # [ 33.733753] nix-grpc-daemon[780]: ts=2026-09-21T16:49:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3905worker1 # [ 33.681078] nix-grpc-daemon[775]: ts=2026-09-21T16:49:21Z level=info event=scheduler_connected addr=lb:500513906worker1 # [ 33.686055] nix-grpc-daemon[775]: ts=2026-09-21T16:49:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3907lb: must succeed: curl -sf localhost:9901/clusters3908lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)3909worker2 # [ 33.930589] nix-grpc-daemon[780]: ts=2026-09-21T16:49:21Z level=info event=scheduler_connected addr=lb:500513910worker2 # [ 33.935712] nix-grpc-daemon[780]: ts=2026-09-21T16:49:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3911worker1 # [ 33.886898] nix-grpc-daemon[775]: ts=2026-09-21T16:49:21Z level=info event=scheduler_connected addr=lb:500513912worker1 # [ 33.892424] nix-grpc-daemon[775]: ts=2026-09-21T16:49:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3913worker2 # [ 34.136785] nix-grpc-daemon[780]: ts=2026-09-21T16:49:21Z level=info event=scheduler_connected addr=lb:500513914worker2 # [ 34.141796] nix-grpc-daemon[780]: ts=2026-09-21T16:49:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3915worker1 # [ 34.093243] nix-grpc-daemon[775]: ts=2026-09-21T16:49:21Z level=info event=scheduler_connected addr=lb:500513916worker1 # [ 34.098482] nix-grpc-daemon[775]: ts=2026-09-21T16:49:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3917worker2 # [ 34.342806] nix-grpc-daemon[780]: ts=2026-09-21T16:49:21Z level=info event=scheduler_connected addr=lb:500513918worker2 # [ 34.347901] nix-grpc-daemon[780]: ts=2026-09-21T16:49:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3919worker1 # [ 34.299402] nix-grpc-daemon[775]: ts=2026-09-21T16:49:21Z level=info event=scheduler_connected addr=lb:500513920worker1 # [ 34.304194] nix-grpc-daemon[775]: ts=2026-09-21T16:49:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3921worker2 # [ 34.548692] nix-grpc-daemon[780]: ts=2026-09-21T16:49:22Z level=info event=scheduler_connected addr=lb:500513922worker2 # [ 34.553713] nix-grpc-daemon[780]: ts=2026-09-21T16:49:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3923worker1 # [ 34.502799] nix-grpc-daemon[775]: ts=2026-09-21T16:49:22Z level=info event=scheduler_connected addr=lb:500513924worker1 # [ 34.507019] nix-grpc-daemon[775]: ts=2026-09-21T16:49:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3925worker2 # [ 34.756506] nix-grpc-daemon[780]: ts=2026-09-21T16:49:22Z level=info event=scheduler_connected addr=lb:500513926worker2 # [ 34.760555] nix-grpc-daemon[780]: ts=2026-09-21T16:49:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3927worker1 # [ 34.703098] nix-grpc-daemon[775]: ts=2026-09-21T16:49:22Z level=info event=scheduler_connected addr=lb:500513928worker1 # [ 34.708313] nix-grpc-daemon[775]: ts=2026-09-21T16:49:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3929lb: must succeed: curl -sf localhost:9901/clusters3930worker2 # [ 34.957081] nix-grpc-daemon[780]: ts=2026-09-21T16:49:22Z level=info event=scheduler_connected addr=lb:500513931worker2 # [ 34.962064] nix-grpc-daemon[780]: ts=2026-09-21T16:49:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3932lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)3933worker1 # [ 34.909070] nix-grpc-daemon[775]: ts=2026-09-21T16:49:22Z level=info event=scheduler_connected addr=lb:500513934worker1 # [ 34.914142] nix-grpc-daemon[775]: ts=2026-09-21T16:49:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3935worker2 # [ 35.163121] nix-grpc-daemon[780]: ts=2026-09-21T16:49:22Z level=info event=scheduler_connected addr=lb:500513936worker2 # [ 35.168325] nix-grpc-daemon[780]: ts=2026-09-21T16:49:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3937worker1 # [ 35.115001] nix-grpc-daemon[775]: ts=2026-09-21T16:49:22Z level=info event=scheduler_connected addr=lb:500513938worker1 # [ 35.120211] nix-grpc-daemon[775]: ts=2026-09-21T16:49:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3939worker2 # [ 35.369238] nix-grpc-daemon[780]: ts=2026-09-21T16:49:23Z level=info event=scheduler_connected addr=lb:500513940worker2 # [ 35.374252] nix-grpc-daemon[780]: ts=2026-09-21T16:49:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3941worker1 # [ 35.320772] nix-grpc-daemon[775]: ts=2026-09-21T16:49:22Z level=info event=scheduler_connected addr=lb:500513942worker1 # [ 35.325823] nix-grpc-daemon[775]: ts=2026-09-21T16:49:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3943worker2 # [ 35.575366] nix-grpc-daemon[780]: ts=2026-09-21T16:49:23Z level=info event=scheduler_connected addr=lb:500513944worker2 # [ 35.584850] nix-grpc-daemon[780]: ts=2026-09-21T16:49:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3945worker1 # [ 35.527960] nix-grpc-daemon[775]: ts=2026-09-21T16:49:23Z level=info event=scheduler_connected addr=lb:500513946worker1 # [ 35.532580] nix-grpc-daemon[775]: ts=2026-09-21T16:49:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3947worker2 # [ 35.787689] nix-grpc-daemon[780]: ts=2026-09-21T16:49:23Z level=info event=scheduler_connected addr=lb:500513948worker2 # [ 35.791901] nix-grpc-daemon[780]: ts=2026-09-21T16:49:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3949worker1 # [ 35.729648] nix-grpc-daemon[775]: ts=2026-09-21T16:49:23Z level=info event=scheduler_connected addr=lb:500513950worker1 # [ 35.734891] nix-grpc-daemon[775]: ts=2026-09-21T16:49:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3951lb: must succeed: curl -sf localhost:9901/clusters3952worker2 # [ 35.988456] nix-grpc-daemon[780]: ts=2026-09-21T16:49:23Z level=info event=scheduler_connected addr=lb:500513953worker2 # [ 35.993453] nix-grpc-daemon[780]: ts=2026-09-21T16:49:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3954worker1 # [ 35.935692] nix-grpc-daemon[775]: ts=2026-09-21T16:49:23Z level=info event=scheduler_connected addr=lb:500513955worker1 # [ 35.940981] nix-grpc-daemon[775]: ts=2026-09-21T16:49:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3956lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)3957worker2 # [ 36.194501] nix-grpc-daemon[780]: ts=2026-09-21T16:49:23Z level=info event=scheduler_connected addr=lb:500513958worker2 # [ 36.199541] nix-grpc-daemon[780]: ts=2026-09-21T16:49:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3959worker1 # [ 36.141691] nix-grpc-daemon[775]: ts=2026-09-21T16:49:23Z level=info event=scheduler_connected addr=lb:500513960worker1 # [ 36.146791] nix-grpc-daemon[775]: ts=2026-09-21T16:49:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3961worker2 # [ 36.400635] nix-grpc-daemon[780]: ts=2026-09-21T16:49:24Z level=info event=scheduler_connected addr=lb:500513962worker2 # [ 36.405565] nix-grpc-daemon[780]: ts=2026-09-21T16:49:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3963worker1 # [ 36.347399] nix-grpc-daemon[775]: ts=2026-09-21T16:49:23Z level=info event=scheduler_connected addr=lb:500513964worker1 # [ 36.352633] nix-grpc-daemon[775]: ts=2026-09-21T16:49:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3965worker2 # [ 36.606446] nix-grpc-daemon[780]: ts=2026-09-21T16:49:24Z level=info event=scheduler_connected addr=lb:500513966worker2 # [ 36.611155] nix-grpc-daemon[780]: ts=2026-09-21T16:49:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3967worker1 # [ 36.555104] nix-grpc-daemon[775]: ts=2026-09-21T16:49:24Z level=info event=scheduler_connected addr=lb:500513968worker1 # [ 36.559320] nix-grpc-daemon[775]: ts=2026-09-21T16:49:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3969worker2 # [ 36.814024] nix-grpc-daemon[780]: ts=2026-09-21T16:49:24Z level=info event=scheduler_connected addr=lb:500513970worker2 # [ 36.818143] nix-grpc-daemon[780]: ts=2026-09-21T16:49:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3971worker1 # [ 36.755559] nix-grpc-daemon[775]: ts=2026-09-21T16:49:24Z level=info event=scheduler_connected addr=lb:500513972worker1 # [ 36.760870] nix-grpc-daemon[775]: ts=2026-09-21T16:49:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3973worker2 # [ 37.014663] nix-grpc-daemon[780]: ts=2026-09-21T16:49:24Z level=info event=scheduler_connected addr=lb:500513974worker2 # [ 37.019712] nix-grpc-daemon[780]: ts=2026-09-21T16:49:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3975worker1 # [ 36.961468] nix-grpc-daemon[775]: ts=2026-09-21T16:49:24Z level=info event=scheduler_connected addr=lb:500513976worker1 # [ 36.966655] nix-grpc-daemon[775]: ts=2026-09-21T16:49:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3977lb: must succeed: curl -sf localhost:9901/clusters3978lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)3979worker2 # [ 37.220894] nix-grpc-daemon[780]: ts=2026-09-21T16:49:24Z level=info event=scheduler_connected addr=lb:500513980worker2 # [ 37.225749] nix-grpc-daemon[780]: ts=2026-09-21T16:49:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3981worker1 # [ 37.167389] nix-grpc-daemon[775]: ts=2026-09-21T16:49:24Z level=info event=scheduler_connected addr=lb:500513982worker1 # [ 37.172579] nix-grpc-daemon[775]: ts=2026-09-21T16:49:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3983worker2 # [ 37.426754] nix-grpc-daemon[780]: ts=2026-09-21T16:49:25Z level=info event=scheduler_connected addr=lb:500513984worker2 # [ 37.431742] nix-grpc-daemon[780]: ts=2026-09-21T16:49:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3985worker1 # [ 37.373199] nix-grpc-daemon[775]: ts=2026-09-21T16:49:25Z level=info event=scheduler_connected addr=lb:500513986worker1 # [ 37.378117] nix-grpc-daemon[775]: ts=2026-09-21T16:49:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3987worker2 # [ 37.632721] nix-grpc-daemon[780]: ts=2026-09-21T16:49:25Z level=info event=scheduler_connected addr=lb:500513988worker2 # [ 37.637803] nix-grpc-daemon[780]: ts=2026-09-21T16:49:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3989worker1 # [ 37.580918] nix-grpc-daemon[775]: ts=2026-09-21T16:49:25Z level=info event=scheduler_connected addr=lb:500513990worker1 # [ 37.585014] nix-grpc-daemon[775]: ts=2026-09-21T16:49:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3991worker1 # [ 37.701218] nix-grpc-daemon[775]: ts=2026-09-21T16:49:25Z level=info event=scheduler_take_over3992lb # [ 37.683117] niks3-server[1008]: 2026/09/21 16:49:25 INFO lead: acquired remote=192.168.1.3:500983993worker2 # [ 37.840655] nix-grpc-daemon[780]: ts=2026-09-21T16:49:25Z level=info event=scheduler_connected addr=lb:500513994worker2 # [ 37.844703] nix-grpc-daemon[780]: ts=2026-09-21T16:49:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3995worker1 # [ 37.781242] nix-grpc-daemon[775]: ts=2026-09-21T16:49:25Z level=info event=scheduler_connected addr=lb:500513996worker1 # [ 37.785847] nix-grpc-daemon[775]: ts=2026-09-21T16:49:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3997worker2 # [ 38.041608] nix-grpc-daemon[780]: ts=2026-09-21T16:49:25Z level=info event=scheduler_connected addr=lb:500513998worker2 # [ 38.046617] nix-grpc-daemon[780]: ts=2026-09-21T16:49:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3999worker1 # [ 37.986739] nix-grpc-daemon[775]: ts=2026-09-21T16:49:25Z level=info event=scheduler_connected addr=lb:500514000worker1 # [ 37.991606] nix-grpc-daemon[775]: ts=2026-09-21T16:49:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4001lb: must succeed: curl -sf localhost:9901/clusters4002lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4003worker2 # [ 38.247561] nix-grpc-daemon[780]: ts=2026-09-21T16:49:25Z level=info event=scheduler_connected addr=lb:500514004worker2 # [ 38.252685] nix-grpc-daemon[780]: ts=2026-09-21T16:49:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4005worker1 # [ 38.192356] nix-grpc-daemon[775]: ts=2026-09-21T16:49:25Z level=info event=scheduler_connected addr=lb:500514006worker1 # [ 38.197401] nix-grpc-daemon[775]: ts=2026-09-21T16:49:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4007worker2 # [ 38.453759] nix-grpc-daemon[780]: ts=2026-09-21T16:49:26Z level=info event=scheduler_connected addr=lb:500514008worker1 # [ 38.396128] nix-grpc-daemon[775]: ts=2026-09-21T16:49:26Z level=info event=oidc_jwks provider=mock keys=14009worker1 # [ 38.404401] nix-grpc-daemon[775]: ts=2026-09-21T16:49:26Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:382284010worker1 # [ 38.410417] nix-grpc-daemon[775]: ts=2026-09-21T16:49:26Z level=info event=scheduler_connected addr=lb:500514011worker1 # [ 38.414235] nix-grpc-daemon[775]: ts=2026-09-21T16:49:26Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=04012worker1 # [ 38.419670] nix-grpc-daemon[775]: ts=2026-09-21T16:49:26Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:382284013worker1 # [ 38.424579] nix-grpc-daemon[775]: ts=2026-09-21T16:49:26Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=04014lb: must succeed: curl -sf localhost:9901/clusters4015lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4016worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4017worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)4018lb: must succeed: curl -sf localhost:9901/clusters4019lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4020lb: must succeed: curl -sf localhost:9901/clusters4021lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4022lb: must succeed: curl -sf localhost:9901/clusters4023lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4024lb: must succeed: curl -sf localhost:9901/clusters4025lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4026worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4027worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.06 seconds)4028worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4029worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4030lb: must succeed: curl -sf localhost:9901/clusters4031lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4032lb: must succeed: curl -sf localhost:9901/clusters4033lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4034worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4035worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)4036worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4037worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4038worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4039worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)4040worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4041worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)4042(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 11.72 seconds)4043subtest: balancer auth: client cert, bearer token, nothing, unknown CN4044worker1 # [ 40.445151] nix-grpc-daemon[775]: ts=2026-09-21T16:49:28Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:382444045worker1 # [ 40.465195] systemd[1]: Started Nix Daemon.4046worker1 # [ 40.761721] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4047worker1 # [ 40.790661] nix-grpc-daemon[775]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 11 ms (attempt 1/5)4048worker1 # [ 40.804565] nix-grpc-daemon[775]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 158 ms (attempt 2/5)4049worker1 # [ 40.962837] nix-grpc-daemon[775]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 292 ms (attempt 3/5)4050worker1 # [ 41.262313] nix-grpc-daemon[775]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 283 ms (attempt 4/5)4051worker1 # [ 41.548981] nix-grpc-daemon[775]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4052worker1 # [ 41.555067] nix-grpc-daemon[775]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4053worker1 # [ 41.573582] nix-grpc-daemon[775]: ts=2026-09-21T16:49:29Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:382444054worker1 # [ 41.588775] nix-grpc-daemon[775]: ts=2026-09-21T16:49:29Z level=debug event=path_info_miss path=23mddq5b0nkdsfx6621y4k5jikrlcj3j-nixos-system-client-test error="error: path '/nix/store/23mddq5b0nkdsfx6621y4k5jikrlcj3j-nixos-system-client-test' is required, but there is no substituter that can build it"4055worker2 # [ 41.705121] nix-grpc-daemon[780]: ts=2026-09-21T16:49:29Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:592644056worker2 # [ 41.723212] systemd[1]: Started Nix Daemon.4057worker2 # [ 42.018715] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4058worker2 # [ 42.042916] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 72 ms (attempt 1/5)4059worker2 # [ 42.117593] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 11 ms (attempt 2/5)4060worker2 # [ 42.132677] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 325 ms (attempt 3/5)4061worker2 # [ 42.463479] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 430 ms (attempt 4/5)4062worker2 # [ 42.905582] nix-grpc-daemon[780]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4063worker2 # [ 42.910827] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4064worker2 # [ 42.924166] nix-grpc-daemon[780]: ts=2026-09-21T16:49:30Z level=debug event=path_info_miss path=23mddq5b0nkdsfx6621y4k5jikrlcj3j-nixos-system-client-test error="error: path '/nix/store/23mddq5b0nkdsfx6621y4k5jikrlcj3j-nixos-system-client-test' is required, but there is no substituter that can build it"4065client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4066client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.10 seconds)4067client: must succeed: install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4068client: (finished: must succeed: install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds)4069worker2 # [ 43.364160] nix-grpc-daemon[780]: ts=2026-09-21T16:49:31Z level=info event=oidc_jwks provider=mock keys=14070worker2 # [ 43.370578] nix-grpc-daemon[780]: ts=2026-09-21T16:49:31Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:592644071worker2 # [ 43.390472] nix-grpc-daemon[780]: ts=2026-09-21T16:49:31Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:592644072worker2 # [ 43.402308] nix-grpc-daemon[780]: ts=2026-09-21T16:49:31Z level=debug event=path_info_miss path=23mddq5b0nkdsfx6621y4k5jikrlcj3j-nixos-system-client-test error="error: path '/nix/store/23mddq5b0nkdsfx6621y4k5jikrlcj3j-nixos-system-client-test' is required, but there is no substituter that can build it"4073worker2 # [ 43.417050] nix-grpc-daemon[780]: ts=2026-09-21T16:49:31Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:592644074worker2 # [ 43.425028] nix-grpc-daemon[780]: ts=2026-09-21T16:49:31Z level=debug event=path_info_miss path=23mddq5b0nkdsfx6621y4k5jikrlcj3j-nixos-system-client-test error="error: path '/nix/store/23mddq5b0nkdsfx6621y4k5jikrlcj3j-nixos-system-client-test' is required, but there is no substituter that can build it"4075client: must succeed: rm -r /var/lib/nix-grpc-store4076client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.05 seconds)4077worker2 # [ 45.763590] nix-grpc-daemon[780]: ts=2026-09-21T16:49:33Z level=info event=denied method=QueryMissing cn=- role=none4078worker2 # [ 46.030321] nix-grpc-daemon[780]: ts=2026-09-21T16:49:33Z level=info event=denied method=QueryMissing cn=stranger role=none4079(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 5.91 seconds)4080subtest: DAG build is scheduled across workers and lands in the cache4081client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&24082client # don't know how to build these paths:4083client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4084worker1 # [ 46.197472] nix-grpc-daemon[775]: ts=2026-09-21T16:49:33Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:382444085worker2 # [ 46.291901] nix-grpc-daemon[780]: ts=2026-09-21T16:49:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:592644086worker1 # [ 46.237105] nix-grpc-daemon[775]: ts=2026-09-21T16:49:33Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=04087worker1 # [ 46.243435] nix-grpc-daemon[775]: ts=2026-09-21T16:49:33Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.4:50051 clients=14088worker2 # [ 46.376285] nix-grpc-daemon[780]: ts=2026-09-21T16:49:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:592644089worker2 # [ 46.398239] nix-grpc-daemon[780]: ts=2026-09-21T16:49:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:592644090client # copying 1 paths to 'lb:50051'...4091worker2 # [ 46.414530] nix-grpc-daemon[780]: ts=2026-09-21T16:49:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:592644092client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'...4093worker2 # [ 46.445149] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4094worker2 # [ 46.480459] nix-grpc-daemon[780]: ts=2026-09-21T16:49:34Z level=info event=niks3_push_started pid=8634095worker2 # [ 46.787394] nix-grpc-daemon[863]: time=2026-09-21T16:49:34.422Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""4096worker2 # [ 46.979305] nix-daemon[849]: accepted connection from pid 876, user nix-grpc-daemon (trusted)4097worker2 # [ 46.996845] nix-daemon[849]: reaped child process 879, status = succeeded4098worker2 # [ 47.170624] nix-daemon[849]: accepted connection from pid 881, user nix-grpc-daemon (trusted)4099worker2 # [ 47.186672] nix-daemon[849]: reaped child process 884, status = succeeded4100lb # [ 47.092794] niks3-server[1008]: 2026/09/21 16:49:34 INFO Received uploads request method=POST path=/api/pending_closures4101worker2 # [ 47.224006] nix-grpc-daemon[863]: time=2026-09-21T16:49:34.860Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4102worker2 # [ 47.227271] nix-grpc-daemon[863]: time=2026-09-21T16:49:34.862Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)"4103lb # [ 47.164117] niks3-server[1008]: 2026/09/21 16:49:34 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst4104lb # [ 47.178039] niks3-server[1008]: 2026/09/21 16:49:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4105lb # [ 47.189142] niks3-server[1008]: 2026/09/21 16:49:34 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls4106lb # [ 47.209648] niks3-server[1008]: 2026/09/21 16:49:34 INFO Signed narinfos id=1 count=14107worker2 # [ 47.319963] nix-grpc-daemon[863]: time=2026-09-21T16:49:34.955Z level=INFO msg="Uploading 1 narinfos"4108lb # [ 47.224811] niks3-server[1008]: 2026/09/21 16:49:34 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4109lb # [ 47.233228] niks3-server[1008]: 2026/09/21 16:49:34 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo4110lb # [ 47.252824] niks3-server[1008]: 2026/09/21 16:49:34 INFO Completed upload id=14111worker2 # [ 47.359931] nix-grpc-daemon[863]: time=2026-09-21T16:49:34.996Z level=INFO msg="Upload complete. (564ms)"4112worker2 # [ 47.373013] nix-grpc-daemon[780]: ts=2026-09-21T16:49:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=0 paths=1 nar_bytes_in=5044113worker2 # [ 47.393168] nix-grpc-daemon[780]: ts=2026-09-21T16:49:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:592644114worker2 # [ 47.402266] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4115worker2 # [ 47.406931] nix-daemon[849]: reaped child process 861, status = succeeded4116client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051...4117worker2 # [ 47.418228] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4118client # farm-a> worker2: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4119client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4120client # farm-a> LOG-farm-a-t14121worker2 # [ 48.245542] nix-daemon[849]: reaped child process 888, status = succeeded4122worker2 # [ 48.411134] nix-daemon[849]: accepted connection from pid 904, user nix-grpc-daemon (trusted)4123worker2 # [ 48.428856] nix-daemon[849]: reaped child process 907, status = succeeded4124lb # [ 48.334910] niks3-server[1008]: 2026/09/21 16:49:35 INFO Received uploads request method=POST path=/api/pending_closures4125worker2 # [ 48.457445] nix-grpc-daemon[863]: time=2026-09-21T16:49:36.092Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4126worker2 # [ 48.459238] nix-grpc-daemon[863]: time=2026-09-21T16:49:36.092Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)"4127lb # [ 48.389398] niks3-server[1008]: 2026/09/21 16:49:36 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4128lb # [ 48.400318] niks3-server[1008]: 2026/09/21 16:49:36 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4129lb # [ 48.407989] niks3-server[1008]: 2026/09/21 16:49:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4130lb # [ 48.413496] niks3-server[1008]: 2026/09/21 16:49:36 INFO Signed narinfos id=2 count=14131worker2 # [ 48.521530] nix-grpc-daemon[863]: time=2026-09-21T16:49:36.157Z level=INFO msg="Uploading 1 narinfos"4132lb # [ 48.416440] niks3-server[1008]: 2026/09/21 16:49:36 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls4133lb # [ 48.427441] niks3-server[1008]: 2026/09/21 16:49:36 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4134lb # [ 48.438574] niks3-server[1008]: 2026/09/21 16:49:36 INFO Completed upload id=24135worker2 # [ 48.545623] nix-grpc-daemon[863]: time=2026-09-21T16:49:36.182Z level=INFO msg="Upload complete. (315ms)"4136lb # [ 48.442700] niks3-server[1008]: 2026/09/21 16:49:36 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo4137worker2 # [ 48.557078] nix-grpc-daemon[780]: ts=2026-09-21T16:49:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=14138worker1 # [ 48.482988] nix-grpc-daemon[775]: ts=2026-09-21T16:49:36Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=04139worker2 # [ 48.575963] nix-daemon[849]: reaped child process 886, status = succeeded4140worker1 # [ 48.489002] nix-grpc-daemon[775]: ts=2026-09-21T16:49:36Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.4:50051 clients=14141worker2 # [ 48.620600] nix-grpc-daemon[780]: ts=2026-09-21T16:49:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:592644142worker2 # [ 48.635891] nix-grpc-daemon[780]: ts=2026-09-21T16:49:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:592644143worker2 # [ 48.651163] nix-grpc-daemon[863]: time=2026-09-21T16:49:36.287Z level=INFO msg="All 1 paths already cached"4144client # copying 1 paths to 'lb:50051'...4145worker2 # [ 48.661487] nix-grpc-daemon[780]: ts=2026-09-21T16:49:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:592644146client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'...4147worker2 # [ 48.684384] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4148worker2 # [ 48.896823] nix-daemon[849]: accepted connection from pid 911, user nix-grpc-daemon (trusted)4149worker2 # [ 48.916614] nix-daemon[849]: reaped child process 914, status = succeeded4150worker2 # [ 49.089479] nix-daemon[849]: accepted connection from pid 916, user nix-grpc-daemon (trusted)4151worker2 # [ 49.106783] nix-daemon[849]: reaped child process 919, status = succeeded4152lb # [ 49.015574] niks3-server[1008]: 2026/09/21 16:49:36 INFO Received uploads request method=POST path=/api/pending_closures4153worker2 # [ 49.132055] nix-grpc-daemon[863]: time=2026-09-21T16:49:36.768Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4154worker2 # [ 49.134132] nix-grpc-daemon[863]: time=2026-09-21T16:49:36.770Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)"4155lb # [ 49.048302] niks3-server[1008]: 2026/09/21 16:49:36 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst4156worker2 # [ 49.166710] nix-grpc-daemon[863]: time=2026-09-21T16:49:36.803Z level=INFO msg="Uploading 1 narinfos"4157lb # [ 49.065207] niks3-server[1008]: 2026/09/21 16:49:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4158lb # [ 49.076468] niks3-server[1008]: 2026/09/21 16:49:36 INFO Signed narinfos id=3 count=14159lb # [ 49.079340] niks3-server[1008]: 2026/09/21 16:49:36 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls4160lb # [ 49.087066] niks3-server[1008]: 2026/09/21 16:49:36 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4161worker2 # [ 49.201959] nix-grpc-daemon[863]: time=2026-09-21T16:49:36.838Z level=INFO msg="Upload complete. (500ms)"4162lb # [ 49.095001] niks3-server[1008]: 2026/09/21 16:49:36 INFO Completed upload id=34163lb # [ 49.098540] niks3-server[1008]: 2026/09/21 16:49:36 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo4164worker2 # [ 49.212359] nix-grpc-daemon[780]: ts=2026-09-21T16:49:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=0 paths=1 nar_bytes_in=6244165worker2 # [ 49.223106] nix-grpc-daemon[780]: ts=2026-09-21T16:49:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:592644166worker2 # [ 49.230164] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4167worker2 # [ 49.235026] nix-daemon[849]: reaped child process 909, status = succeeded4168client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051...4169worker2 # [ 49.247480] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4170client # farm-b> worker2: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4171client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4172client # farm-b> LOG-farm-b-t14173worker2 # [ 50.072782] nix-daemon[849]: reaped child process 923, status = succeeded4174worker2 # [ 50.243536] nix-daemon[849]: accepted connection from pid 939, user nix-grpc-daemon (trusted)4175worker2 # [ 50.260216] nix-daemon[849]: reaped child process 942, status = succeeded4176lb # [ 50.169085] niks3-server[1008]: 2026/09/21 16:49:37 INFO Received uploads request method=POST path=/api/pending_closures4177worker2 # [ 50.286982] nix-grpc-daemon[863]: time=2026-09-21T16:49:37.922Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4178worker2 # [ 50.288515] nix-grpc-daemon[863]: time=2026-09-21T16:49:37.922Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)"4179lb # [ 50.209335] niks3-server[1008]: 2026/09/21 16:49:37 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4180worker2 # [ 50.329017] nix-grpc-daemon[863]: time=2026-09-21T16:49:37.965Z level=INFO msg="Uploading 1 narinfos"4181lb # [ 50.222135] niks3-server[1008]: 2026/09/21 16:49:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4182lb # [ 50.233891] niks3-server[1008]: 2026/09/21 16:49:37 INFO Signed narinfos id=4 count=14183lb # [ 50.240857] niks3-server[1008]: 2026/09/21 16:49:37 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls4184lb # [ 50.245793] niks3-server[1008]: 2026/09/21 16:49:37 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst4185lb # [ 50.254777] niks3-server[1008]: 2026/09/21 16:49:37 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4186lb # [ 50.260647] niks3-server[1008]: 2026/09/21 16:49:37 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo4187lb # [ 50.265278] niks3-server[1008]: 2026/09/21 16:49:37 INFO Completed upload id=44188worker2 # [ 50.371958] nix-grpc-daemon[863]: time=2026-09-21T16:49:38.008Z level=INFO msg="Upload complete. (312ms)"4189worker1 # [ 50.299541] nix-grpc-daemon[775]: ts=2026-09-21T16:49:37Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=04190worker1 # [ 50.305244] nix-grpc-daemon[775]: ts=2026-09-21T16:49:37Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.4:50051 clients=14191worker2 # [ 50.384890] nix-grpc-daemon[780]: ts=2026-09-21T16:49:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=14192worker2 # [ 50.400461] nix-daemon[849]: reaped child process 921, status = succeeded4193worker2 # [ 50.430615] nix-grpc-daemon[780]: ts=2026-09-21T16:49:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:592644194worker2 # [ 50.445479] nix-grpc-daemon[780]: ts=2026-09-21T16:49:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:592644195worker2 # [ 50.458968] nix-grpc-daemon[863]: time=2026-09-21T16:49:38.095Z level=INFO msg="All 2 paths already cached"4196client # copying 1 paths to 'lb:50051'...4197client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'...4198worker2 # [ 50.469084] nix-grpc-daemon[780]: ts=2026-09-21T16:49:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:592644199worker2 # [ 50.487069] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4200worker2 # [ 50.695746] nix-daemon[849]: accepted connection from pid 946, user nix-grpc-daemon (trusted)4201worker2 # [ 50.714366] nix-daemon[849]: reaped child process 949, status = succeeded4202worker2 # [ 50.888310] nix-daemon[849]: accepted connection from pid 951, user nix-grpc-daemon (trusted)4203worker2 # [ 50.904345] nix-daemon[849]: reaped child process 954, status = succeeded4204lb # [ 50.811996] niks3-server[1008]: 2026/09/21 16:49:38 INFO Received uploads request method=POST path=/api/pending_closures4205worker2 # [ 50.928894] nix-grpc-daemon[863]: time=2026-09-21T16:49:38.565Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4206worker2 # [ 50.930351] nix-grpc-daemon[863]: time=2026-09-21T16:49:38.565Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)"4207lb # [ 50.837476] niks3-server[1008]: 2026/09/21 16:49:38 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst4208worker2 # [ 50.958300] nix-grpc-daemon[863]: time=2026-09-21T16:49:38.594Z level=INFO msg="Uploading 1 narinfos"4209lb # [ 50.852318] niks3-server[1008]: 2026/09/21 16:49:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4210lb # [ 50.863043] niks3-server[1008]: 2026/09/21 16:49:38 INFO Signed narinfos id=5 count=14211lb # [ 50.877255] envoy[926]: conn peer=[2001:db8:1::1]:33988 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=749 tx=423 ms=24474212lb # [ 50.887856] envoy[926]: conn peer=[2001:db8:1::1]:59316 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=844213lb # [ 50.896562] envoy[926]: conn peer=[2001:db8:1::1]:59332 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=134214lb # [ 50.903725] envoy[926]: conn peer=[2001:db8:1::1]:59346 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=124215lb # [ 50.908956] niks3-server[1008]: 2026/09/21 16:49:38 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4216lb # [ 50.918312] niks3-server[1008]: 2026/09/21 16:49:38 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls4217lb # [ 50.923594] niks3-server[1008]: 2026/09/21 16:49:38 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo4218lb # [ 50.933726] niks3-server[1008]: 2026/09/21 16:49:38 INFO Completed upload id=54219worker2 # [ 51.040939] nix-grpc-daemon[863]: time=2026-09-21T16:49:38.677Z level=INFO msg="Upload complete. (536ms)"4220worker2 # [ 51.052418] nix-grpc-daemon[780]: ts=2026-09-21T16:49:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=0 paths=1 nar_bytes_in=7524221worker2 # [ 51.065061] nix-grpc-daemon[780]: ts=2026-09-21T16:49:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:592644222worker2 # [ 51.078716] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4223worker2 # [ 51.085712] nix-daemon[849]: reaped child process 944, status = succeeded4224client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051...4225client # farm-top> worker2: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4226worker2 # [ 51.100413] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4227client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4228client # farm-top> LOG-farm-top-t14229worker2 # [ 52.141814] nix-daemon[849]: reaped child process 958, status = succeeded4230worker2 # [ 52.314939] nix-daemon[849]: accepted connection from pid 974, user nix-grpc-daemon (trusted)4231worker2 # [ 52.333008] nix-daemon[849]: reaped child process 977, status = succeeded4232lb # [ 52.241334] niks3-server[1008]: 2026/09/21 16:49:39 INFO Received uploads request method=POST path=/api/pending_closures4233worker2 # [ 52.362559] nix-grpc-daemon[863]: time=2026-09-21T16:49:39.998Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4234worker2 # [ 52.365469] nix-grpc-daemon[863]: time=2026-09-21T16:49:39.998Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)"4235lb # [ 52.280955] niks3-server[1008]: 2026/09/21 16:49:39 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4236lb # [ 52.298867] niks3-server[1008]: 2026/09/21 16:49:39 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst4237lb # [ 52.309503] niks3-server[1008]: 2026/09/21 16:49:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4238worker2 # [ 52.421601] nix-grpc-daemon[863]: time=2026-09-21T16:49:40.057Z level=INFO msg="Uploading 1 narinfos"4239lb # [ 52.317711] niks3-server[1008]: 2026/09/21 16:49:39 INFO Signed narinfos id=6 count=14240lb # [ 52.323752] niks3-server[1008]: 2026/09/21 16:49:39 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls4241lb # [ 52.332680] niks3-server[1008]: 2026/09/21 16:49:39 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4242lb # [ 52.341521] niks3-server[1008]: 2026/09/21 16:49:39 INFO Completed upload id=64243worker2 # [ 52.448646] nix-grpc-daemon[863]: time=2026-09-21T16:49:40.085Z level=INFO msg="Upload complete. (336ms)"4244lb # [ 52.345502] niks3-server[1008]: 2026/09/21 16:49:39 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo4245worker2 # [ 52.460621] nix-grpc-daemon[780]: ts=2026-09-21T16:49:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=14246worker2 # [ 52.478611] nix-daemon[849]: reaped child process 956, status = succeeded4247client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2, in 6.46 seconds)4248client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath4249client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath, in 0.23 seconds)4250client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4251client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.03 seconds)4252client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4253client # copying 3 paths...4254client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'...4255client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'...4256client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'...4257client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.30 seconds)4258(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.12 seconds)4259subtest: repeat is answered Cached by the scheduler without building4260worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4261worker1 # finding garbage collector roots...4262worker1 # deleting specified paths...4263worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4264worker1 # deleting unused links...4265worker1 # note: hard linking is currently saving 0.0 KiB4266worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.29 seconds)4267worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4268worker2 # finding garbage collector roots...4269worker2 # deleting specified paths...4270worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4271worker2 # deleting unused links...4272worker2 # note: hard linking is currently saving 0.0 KiB4273worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.27 seconds)4274client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&24275client # don't know how to build these paths:4276client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4277worker1 # [ 53.908461] nix-grpc-daemon[775]: ts=2026-09-21T16:49:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:382444278worker2 # [ 54.001527] nix-grpc-daemon[780]: ts=2026-09-21T16:49:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:592644279worker2 # [ 54.013795] nix-grpc-daemon[863]: time=2026-09-21T16:49:41.649Z level=INFO msg="All 2 paths already cached"4280worker1 # [ 53.944094] nix-grpc-daemon[775]: ts=2026-09-21T16:49:41Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=14281client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2, in 0.33 seconds)4282client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath4283client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath, in 0.21 seconds)4284lb: must succeed: curl -sf localhost:9901/clusters4285lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4286lb: must succeed: curl -sf localhost:9901/clusters4287lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4288worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4289worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4290(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.42 seconds)4291subtest: inputs already in the cache are not sent to the scheduler4292lb: must succeed: curl -sf localhost:9901/clusters4293lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4294lb: must succeed: curl -sf localhost:9901/clusters4295lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4296worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4297worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4298client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b >&24299worker2 # [ 55.041818] nix-grpc-daemon[780]: ts=2026-09-21T16:49:42Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:592644300client # don't know how to build these paths:4301client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4302worker1 # [ 54.966280] nix-grpc-daemon[775]: ts=2026-09-21T16:49:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:382444303worker1 # [ 54.994793] nix-grpc-daemon[775]: ts=2026-09-21T16:49:42Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=04304worker1 # [ 55.000920] nix-grpc-daemon[775]: ts=2026-09-21T16:49:42Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.4:50051 clients=14305worker2 # [ 55.127225] nix-grpc-daemon[780]: ts=2026-09-21T16:49:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:592644306worker2 # [ 55.142425] nix-grpc-daemon[780]: ts=2026-09-21T16:49:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:592644307worker2 # [ 55.158262] nix-grpc-daemon[863]: time=2026-09-21T16:49:42.794Z level=INFO msg="All 2 paths already cached"4308client # copying 1 paths to 'lb:50051'...4309worker2 # [ 55.168727] nix-grpc-daemon[780]: ts=2026-09-21T16:49:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:592644310client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'...4311worker2 # [ 55.187097] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4312worker2 # [ 55.393721] nix-daemon[849]: accepted connection from pid 994, user nix-grpc-daemon (trusted)4313worker2 # [ 55.412812] nix-daemon[849]: reaped child process 997, status = succeeded4314worker2 # [ 55.585251] nix-daemon[849]: accepted connection from pid 999, user nix-grpc-daemon (trusted)4315worker2 # [ 55.603650] nix-daemon[849]: reaped child process 1002, status = succeeded4316lb # [ 55.509459] niks3-server[1008]: 2026/09/21 16:49:43 INFO Received uploads request method=POST path=/api/pending_closures4317worker2 # [ 55.626221] nix-grpc-daemon[863]: time=2026-09-21T16:49:43.262Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4318worker2 # [ 55.627790] nix-grpc-daemon[863]: time=2026-09-21T16:49:43.262Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)"4319lb # [ 55.536284] niks3-server[1008]: 2026/09/21 16:49:43 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst4320lb # [ 55.551505] niks3-server[1008]: 2026/09/21 16:49:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4321worker2 # [ 55.664959] nix-grpc-daemon[863]: time=2026-09-21T16:49:43.301Z level=INFO msg="Uploading 1 narinfos"4322lb # [ 55.564814] niks3-server[1008]: 2026/09/21 16:49:43 INFO Signed narinfos id=7 count=14323lb # [ 55.574794] niks3-server[1008]: 2026/09/21 16:49:43 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls4324worker2 # [ 55.687751] nix-grpc-daemon[863]: time=2026-09-21T16:49:43.324Z level=INFO msg="Upload complete. (482ms)"4325lb # [ 55.581516] niks3-server[1008]: 2026/09/21 16:49:43 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4326lb # [ 55.589048] niks3-server[1008]: 2026/09/21 16:49:43 INFO Completed upload id=74327worker2 # [ 55.694678] nix-grpc-daemon[780]: ts=2026-09-21T16:49:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=0 paths=1 nar_bytes_in=7524328lb # [ 55.591356] niks3-server[1008]: 2026/09/21 16:49:43 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo4329worker2 # [ 55.707773] nix-grpc-daemon[780]: ts=2026-09-21T16:49:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:592644330worker2 # [ 55.714539] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4331worker2 # [ 55.719219] nix-daemon[849]: reaped child process 992, status = succeeded4332client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051...4333worker2 # [ 55.730195] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4334client # farm-top> worker2: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4335client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4336client # farm-top> LOG-farm-top-t1b4337worker2 # [ 56.552813] nix-daemon[849]: reaped child process 1006, status = succeeded4338worker2 # [ 56.711848] nix-daemon[849]: accepted connection from pid 1022, user nix-grpc-daemon (trusted)4339worker2 # [ 56.730223] nix-daemon[849]: reaped child process 1025, status = succeeded4340lb # [ 56.638030] niks3-server[1008]: 2026/09/21 16:49:44 INFO Received uploads request method=POST path=/api/pending_closures4341worker2 # [ 56.755609] nix-grpc-daemon[863]: time=2026-09-21T16:49:44.388Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4342worker2 # [ 56.757205] nix-grpc-daemon[863]: time=2026-09-21T16:49:44.389Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)"4343lb # [ 56.669089] niks3-server[1008]: 2026/09/21 16:49:44 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4344lb # [ 56.686067] niks3-server[1008]: 2026/09/21 16:49:44 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst4345worker2 # [ 56.801672] nix-grpc-daemon[863]: time=2026-09-21T16:49:44.438Z level=INFO msg="Uploading 1 narinfos"4346lb # [ 56.700586] niks3-server[1008]: 2026/09/21 16:49:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4347lb # [ 56.706690] niks3-server[1008]: 2026/09/21 16:49:44 INFO Signed narinfos id=8 count=14348lb # [ 56.712848] niks3-server[1008]: 2026/09/21 16:49:44 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls4349lb # [ 56.715998] niks3-server[1008]: 2026/09/21 16:49:44 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4350lb # [ 56.720687] niks3-server[1008]: 2026/09/21 16:49:44 INFO Completed upload id=84351worker2 # [ 56.827526] nix-grpc-daemon[863]: time=2026-09-21T16:49:44.464Z level=INFO msg="Upload complete. (286ms)"4352lb # [ 56.723141] niks3-server[1008]: 2026/09/21 16:49:44 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo4353worker2 # [ 56.836996] nix-grpc-daemon[780]: ts=2026-09-21T16:49:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=14354worker2 # [ 56.852006] nix-daemon[849]: reaped child process 1004, status = succeeded4355client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.09 seconds)4356client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b outPath4357client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.22 seconds)4358lb: must succeed: curl -sf localhost:9901/clusters4359lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4360lb: must succeed: curl -sf localhost:9901/clusters4361lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4362worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4363worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4364(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.71 seconds)4365subtest: two clients wanting the same drv build it once4366client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup4367client # [ 57.293681] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup.4368client # Running as unit: dedup1.service; invocation ID: d18200f8a57e4f4486452281d22c362e4369client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup, in 0.10 seconds)4370client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup4371client # [ 57.552495] nix[888]: don't know how to build these paths:4372client # [ 57.553949] nix[888]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4373worker1 # [ 57.554090] nix-grpc-daemon[775]: ts=2026-09-21T16:49:45Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:382444374worker1 # [ 57.567601] nix-grpc-daemon[775]: ts=2026-09-21T16:49:45Z level=debug event=want client=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04375worker1 # [ 57.574824] nix-grpc-daemon[775]: ts=2026-09-21T16:49:45Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.3:50051 clients=14376worker1 # [ 57.615650] nix-grpc-daemon[775]: ts=2026-09-21T16:49:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:382444377worker1 # [ 57.636451] nix-grpc-daemon[775]: ts=2026-09-21T16:49:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:382444378client # [ 57.648523] nix[888]: copying 1 paths to 'lb:50051'...4379worker1 # [ 57.652129] nix-grpc-daemon[775]: ts=2026-09-21T16:49:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:382444380client # [ 57.652912] nix[888]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'...4381worker1 # [ 57.667229] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4382worker1 # [ 57.694851] nix-grpc-daemon[775]: ts=2026-09-21T16:49:45Z level=info event=niks3_push_started pid=9104383worker1 # [ 57.961686] nix-grpc-daemon[910]: time=2026-09-21T16:49:45.591Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""4384worker1 # [ 58.153063] nix-daemon[858]: accepted connection from pid 923, user nix-grpc-daemon (trusted)4385worker1 # [ 58.171186] nix-daemon[858]: reaped child process 926, status = succeeded4386worker1 # [ 58.343989] nix-daemon[858]: accepted connection from pid 928, user nix-grpc-daemon (trusted)4387worker1 # [ 58.362485] nix-daemon[858]: reaped child process 931, status = succeeded4388lb # [ 58.355951] niks3-server[1008]: 2026/09/21 16:49:45 INFO Received uploads request method=POST path=/api/pending_closures4389worker1 # [ 58.386133] nix-grpc-daemon[910]: time=2026-09-21T16:49:46.016Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4390worker1 # [ 58.389201] nix-grpc-daemon[910]: time=2026-09-21T16:49:46.018Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)"4391lb # [ 58.393945] niks3-server[1008]: 2026/09/21 16:49:46 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst4392client # [ 58.409435] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup.4393client # Running as unit: dedup2.service; invocation ID: 651d8bd7b16f419d805bdeed68c501854394worker1 # [ 58.432266] nix-grpc-daemon[910]: time=2026-09-21T16:49:46.062Z level=INFO msg="Uploading 1 narinfos"4395lb # [ 58.406314] niks3-server[1008]: 2026/09/21 16:49:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4396lb # [ 58.420258] niks3-server[1008]: 2026/09/21 16:49:46 INFO Signed narinfos id=9 count=14397client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup, in 1.13 seconds)4398lb # [ 58.426703] niks3-server[1008]: 2026/09/21 16:49:46 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls4399lb # [ 58.436941] niks3-server[1008]: 2026/09/21 16:49:46 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4400lb # [ 58.445475] niks3-server[1008]: 2026/09/21 16:49:46 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo4401lb # [ 58.449955] niks3-server[1008]: 2026/09/21 16:49:46 INFO Completed upload id=94402worker1 # [ 58.470110] nix-grpc-daemon[910]: time=2026-09-21T16:49:46.099Z level=INFO msg="Upload complete. (499ms)"4403worker1 # [ 58.477662] nix-grpc-daemon[775]: ts=2026-09-21T16:49:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38244 duration_s=0 paths=1 nar_bytes_in=4324404worker1 # [ 58.485002] nix-grpc-daemon[775]: ts=2026-09-21T16:49:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:382444405worker1 # [ 58.494952] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4406worker1 # [ 58.500408] nix-daemon[858]: reaped child process 908, status = succeeded4407client # [ 58.507060] nix[888]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4408worker1 # [ 58.510692] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4409client # [ 58.776113] nix[904]: don't know how to build these paths:4410client # [ 58.777991] nix[904]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4411worker2 # [ 58.865974] nix-grpc-daemon[780]: ts=2026-09-21T16:49:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:592644412worker1 # [ 58.791842] nix-grpc-daemon[775]: ts=2026-09-21T16:49:46Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04413worker1 # [ 58.806450] nix-grpc-daemon[775]: ts=2026-09-21T16:49:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:382444414client # [ 58.811269] nix[904]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4415lb # [ 60.848890] envoy[926]: conn peer=[2001:db8:1::1]:59352 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9510 tx=4080 ms=62004416lb # [ 60.849205] envoy[926]: conn peer=[2001:db8:1::1]:42812 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=480 ms=574417lb # [ 60.849309] envoy[926]: conn peer=[2001:db8:1::1]:42814 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4172 tx=1809 ms=18094418client: must succeed: sleep 24419client: (finished: must succeed: sleep 2, in 2.07 seconds)4420worker1 # [ 63.137149] nix-daemon[858]: reaped child process 939, status = succeeded4421client: waiting for success: ! systemctl is-active dedup1 dedup24422worker1 # [ 63.300128] nix-daemon[858]: accepted connection from pid 971, user nix-grpc-daemon (trusted)4423worker1 # [ 63.319715] nix-daemon[858]: reaped child process 974, status = succeeded4424lb # [ 63.315963] niks3-server[1008]: 2026/09/21 16:49:50 INFO Received uploads request method=POST path=/api/pending_closures4425worker1 # [ 63.344636] nix-grpc-daemon[910]: time=2026-09-21T16:49:50.973Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4426worker1 # [ 63.346392] nix-grpc-daemon[910]: time=2026-09-21T16:49:50.973Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)"4427lb # [ 63.361441] niks3-server[1008]: 2026/09/21 16:49:50 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4428lb # [ 63.374133] niks3-server[1008]: 2026/09/21 16:49:50 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4429lb # [ 63.385598] niks3-server[1008]: 2026/09/21 16:49:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4430worker1 # [ 63.405710] nix-grpc-daemon[910]: time=2026-09-21T16:49:51.034Z level=INFO msg="Uploading 1 narinfos"4431lb # [ 63.389279] niks3-server[1008]: 2026/09/21 16:49:51 INFO Signed narinfos id=10 count=14432lb # [ 63.392572] niks3-server[1008]: 2026/09/21 16:49:51 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls4433lb # [ 63.402421] niks3-server[1008]: 2026/09/21 16:49:51 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4434worker1 # [ 63.426528] nix-grpc-daemon[910]: time=2026-09-21T16:49:51.057Z level=INFO msg="Upload complete. (300ms)"4435lb # [ 63.408977] niks3-server[1008]: 2026/09/21 16:49:51 INFO Completed upload id=104436lb # [ 63.411669] niks3-server[1008]: 2026/09/21 16:49:51 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo4437worker1 # [ 63.437312] nix-grpc-daemon[775]: ts=2026-09-21T16:49:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38244 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14438worker1 # [ 63.454507] nix-grpc-daemon[775]: ts=2026-09-21T16:49:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38244 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14439worker1 # [ 63.459975] nix-daemon[858]: reaped child process 937, status = succeeded4440client # [ 63.501361] systemd[1]: dedup1.service: Deactivated successfully.4441client # [ 63.509609] systemd[1]: dedup1.service: Consumed 177ms CPU time over 6.213s wall clock time, 43.4M memory peak, 7K incoming IP traffic, 8.4K outgoing IP traffic.4442client # [ 63.545051] systemd[1]: dedup2.service: Deactivated successfully.4443client # [ 63.547169] systemd[1]: dedup2.service: Consumed 211ms CPU time over 5.139s wall clock time, 33.2M memory peak, 5.9K incoming IP traffic, 5.6K outgoing IP traffic.4444client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.15 seconds)4445client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4446client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.07 seconds)4447worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4448worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4449worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4450worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.08 seconds)4451worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4452worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4453worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4454worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4455(finished: subtest: two clients wanting the same drv build it once, in 7.49 seconds)4456subtest: the first client leaving does not fail the second4457client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share4458client # [ 64.773093] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share.4459client # Running as unit: share1.service; invocation ID: 893f0f56bab04af187a630d9a1a737534460client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share, in 0.09 seconds)4461client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share4462worker2 # [ 65.134355] nix-grpc-daemon[780]: ts=2026-09-21T16:49:52Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:592644463client # [ 65.052127] nix[944]: don't know how to build these paths:4464client # [ 65.054293] nix[944]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4465worker1 # [ 65.069258] nix-grpc-daemon[775]: ts=2026-09-21T16:49:52Z level=debug event=want client=6 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=04466worker1 # [ 65.075331] nix-grpc-daemon[775]: ts=2026-09-21T16:49:52Z level=debug event=assigned drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker=192.168.1.4:50051 clients=14467worker2 # [ 65.204112] nix-grpc-daemon[780]: ts=2026-09-21T16:49:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:592644468worker2 # [ 65.220867] nix-grpc-daemon[780]: ts=2026-09-21T16:49:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:592644469client # [ 65.144455] nix[944]: copying 1 paths to 'lb:50051'...4470worker2 # [ 65.235659] nix-grpc-daemon[780]: ts=2026-09-21T16:49:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:592644471client # [ 65.148774] nix[944]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' to 'lb:50051'...4472worker2 # [ 65.255081] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4473worker2 # [ 65.468978] nix-daemon[849]: accepted connection from pid 1054, user nix-grpc-daemon (trusted)4474worker2 # [ 65.489644] nix-daemon[849]: reaped child process 1057, status = succeeded4475worker2 # [ 65.658330] nix-daemon[849]: accepted connection from pid 1059, user nix-grpc-daemon (trusted)4476worker2 # [ 65.675884] nix-daemon[849]: reaped child process 1062, status = succeeded4477lb # [ 65.584994] niks3-server[1008]: 2026/09/21 16:49:53 INFO Received uploads request method=POST path=/api/pending_closures4478worker2 # [ 65.700381] nix-grpc-daemon[863]: time=2026-09-21T16:49:53.336Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4479worker2 # [ 65.701849] nix-grpc-daemon[863]: time=2026-09-21T16:49:53.336Z level=INFO msg="Uploading rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv (432B)"4480lb # [ 65.621686] niks3-server[1008]: 2026/09/21 16:49:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4481worker2 # [ 65.735287] nix-grpc-daemon[863]: time=2026-09-21T16:49:53.371Z level=INFO msg="Uploading 1 narinfos"4482lb # [ 65.636864] niks3-server[1008]: 2026/09/21 16:49:53 INFO Signed narinfos id=11 count=14483lb # [ 65.643270] niks3-server[1008]: 2026/09/21 16:49:53 INFO Registered completed upload object_key=nar/0chdjpc9an2ckdblh08lnyrmzvw3h8qs76cn5ydwjwvgwkrm731l.nar.zst4484worker2 # [ 65.761190] nix-grpc-daemon[863]: time=2026-09-21T16:49:53.397Z level=INFO msg="Upload complete. (487ms)"4485lb # [ 65.654738] niks3-server[1008]: 2026/09/21 16:49:53 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.ls4486lb # [ 65.659443] niks3-server[1008]: 2026/09/21 16:49:53 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4487worker2 # [ 65.768200] nix-grpc-daemon[780]: ts=2026-09-21T16:49:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=0 paths=1 nar_bytes_in=4324488lb # [ 65.664218] niks3-server[1008]: 2026/09/21 16:49:53 INFO Completed upload id=114489lb # [ 65.666577] niks3-server[1008]: 2026/09/21 16:49:53 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.narinfo4490worker2 # [ 65.776248] nix-grpc-daemon[780]: ts=2026-09-21T16:49:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:592644491worker2 # [ 65.786108] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4492worker2 # [ 65.790910] nix-daemon[849]: reaped child process 1052, status = succeeded4493client # [ 65.710238] nix[944]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051...4494worker2 # [ 65.803320] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4495client # [ 65.885928] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share.4496client # Running as unit: share2.service; invocation ID: 120170bb00a7493fac45e13b1aded14e4497client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share, in 1.12 seconds)4498worker1 # [ 66.153042] nix-grpc-daemon[775]: ts=2026-09-21T16:49:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:382444499client # [ 66.159680] nix[960]: don't know how to build these paths:4500client # [ 66.160857] nix[960]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4501worker1 # [ 66.170088] nix-grpc-daemon[775]: ts=2026-09-21T16:49:53Z level=debug event=want client=7 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=04502worker2 # [ 66.261754] nix-grpc-daemon[780]: ts=2026-09-21T16:49:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:592644503client # [ 66.175056] nix[960]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051...4504client: must succeed: sleep 2; systemctl kill -s INT share14505client # [ 69.324948] systemd[1]: share1.service: Sent signal SIGINT to main process 944 (nix) on client request.4506client # [ 69.335473] nix[944]: shutting downrxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)4507client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.10 seconds)4508client: waiting for success: ! systemctl is-active share14509client # [ 69.349406] nix[944]: error: interrupted by the user4510client # [ 69.401353] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE4511client # [ 69.403911] systemd[1]: share1.service: Failed with result 'exit-code'.4512client # [ 69.408703] systemd[1]: share1.service: Consumed 178ms CPU time over 4.632s wall clock time, 41.3M memory peak, 6.3K incoming IP traffic, 8.2K outgoing IP traffic.4513client: (finished: waiting for success: ! systemctl is-active share1, in 0.07 seconds)4514client: waiting for success: ! systemctl is-active share24515worker2 # [ 70.048629] nix-daemon[849]: reaped child process 1066, status = succeeded4516worker2 # [ 70.236933] nix-daemon[849]: accepted connection from pid 1100, user nix-grpc-daemon (trusted)4517worker2 # [ 70.255139] nix-daemon[849]: reaped child process 1103, status = succeeded4518lb # [ 70.161750] niks3-server[1008]: 2026/09/21 16:49:57 INFO Received uploads request method=POST path=/api/pending_closures4519worker2 # [ 70.280403] nix-grpc-daemon[863]: time=2026-09-21T16:49:57.914Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4520worker2 # [ 70.281915] nix-grpc-daemon[863]: time=2026-09-21T16:49:57.914Z level=INFO msg="Uploading fjww0fd4rwk1s35jav3cyal62l24n356-slow-share (120B)"4521lb # [ 70.203236] niks3-server[1008]: 2026/09/21 16:49:57 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst4522lb # [ 70.215894] niks3-server[1008]: 2026/09/21 16:49:57 INFO Registered completed upload object_key=log/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4523lb # [ 70.225742] niks3-server[1008]: 2026/09/21 16:49:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4524worker2 # [ 70.338743] nix-grpc-daemon[863]: time=2026-09-21T16:49:57.974Z level=INFO msg="Uploading 1 narinfos"4525lb # [ 70.233421] niks3-server[1008]: 2026/09/21 16:49:57 INFO Signed narinfos id=12 count=14526lb # [ 70.239211] niks3-server[1008]: 2026/09/21 16:49:57 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.ls4527lb # [ 70.247113] niks3-server[1008]: 2026/09/21 16:49:57 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4528worker2 # [ 70.360174] nix-grpc-daemon[863]: time=2026-09-21T16:49:57.996Z level=INFO msg="Upload complete. (344ms)"4529lb # [ 70.254584] niks3-server[1008]: 2026/09/21 16:49:57 INFO Completed upload id=124530lb # [ 70.257582] niks3-server[1008]: 2026/09/21 16:49:57 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.narinfo4531worker2 # [ 70.372990] nix-grpc-daemon[780]: ts=2026-09-21T16:49:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=4 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=6 outputs=14532worker2 # [ 70.387064] nix-grpc-daemon[780]: ts=2026-09-21T16:49:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=4 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=6 outputs=14533worker2 # [ 70.392876] nix-daemon[849]: reaped child process 1064, status = succeeded4534client # [ 70.368348] systemd[1]: share2.service: Deactivated successfully.4535client # [ 70.372563] systemd[1]: share2.service: Consumed 172ms CPU time over 4.486s wall clock time, 35.2M memory peak, 5.9K incoming IP traffic, 5.8K outgoing IP traffic.4536client: (finished: waiting for success: ! systemctl is-active share2, in 1.14 seconds)4537client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }4538lb # [ 70.847842] envoy[926]: conn peer=[2001:db8:1::1]:42822 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3005 tx=1451 ms=59124539client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.05 seconds)4540(finished: subtest: the first client leaving does not fail the second, in 6.21 seconds)4541subtest: build hook: nix-daemon with builders = grpc://lb4542client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag hook 2>/tmp/hook.log4543client # [ 71.107094] systemd[1]: Started Nix Daemon.4544client # [ 71.294717] nix-daemon[1007]: accepted connection from pid 1004, user root (trusted)4545lb # [ 70.853062] envoy[926]: conn peer=[2001:db8:1::1]:42828 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1430 tx=1085 ms=47324546lb # [ 70.853215] envoy[926]: conn peer=[2001:db8:1::1]:45834 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2977 tx=814 ms=43264547lb # [ 70.853313] envoy[926]: conn peer=[2001:db8:1::1]:45840 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1456 tx=1085 ms=41884548worker1 # [ 71.522137] nix-grpc-daemon[775]: ts=2026-09-21T16:49:59Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:382444549worker1 # [ 71.529577] nix-grpc-daemon[775]: ts=2026-09-21T16:49:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:382444550worker1 # [ 71.542020] nix-grpc-daemon[775]: ts=2026-09-21T16:49:59Z level=debug event=want client=8 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=04551worker1 # [ 71.554204] nix-grpc-daemon[775]: ts=2026-09-21T16:49:59Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.3:50051 clients=14552worker1 # [ 71.587108] nix-grpc-daemon[775]: ts=2026-09-21T16:49:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:382444553worker1 # [ 71.603257] nix-grpc-daemon[775]: ts=2026-09-21T16:49:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:382444554worker1 # [ 71.618597] nix-grpc-daemon[775]: ts=2026-09-21T16:49:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:382444555worker1 # [ 71.634059] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4556worker1 # [ 71.832408] nix-daemon[858]: accepted connection from pid 1009, user nix-grpc-daemon (trusted)4557worker1 # [ 71.852362] nix-daemon[858]: reaped child process 1012, status = succeeded4558worker1 # [ 72.024625] nix-daemon[858]: accepted connection from pid 1014, user nix-grpc-daemon (trusted)4559worker1 # [ 72.043402] nix-daemon[858]: reaped child process 1017, status = succeeded4560lb # [ 72.036909] niks3-server[1008]: 2026/09/21 16:49:59 INFO Received uploads request method=POST path=/api/pending_closures4561worker1 # [ 72.065149] nix-grpc-daemon[910]: time=2026-09-21T16:49:59.694Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4562worker1 # [ 72.066626] nix-grpc-daemon[910]: time=2026-09-21T16:49:59.695Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)"4563lb # [ 72.066018] niks3-server[1008]: 2026/09/21 16:49:59 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst4564lb # [ 72.080304] niks3-server[1008]: 2026/09/21 16:49:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4565worker1 # [ 72.105020] nix-grpc-daemon[910]: time=2026-09-21T16:49:59.735Z level=INFO msg="Uploading 1 narinfos"4566lb # [ 72.092193] niks3-server[1008]: 2026/09/21 16:49:59 INFO Signed narinfos id=13 count=14567lb # [ 72.097280] niks3-server[1008]: 2026/09/21 16:49:59 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls4568lb # [ 72.105948] niks3-server[1008]: 2026/09/21 16:49:59 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4569worker1 # [ 72.130000] nix-grpc-daemon[910]: time=2026-09-21T16:49:59.760Z level=INFO msg="Upload complete. (479ms)"4570lb # [ 72.113459] niks3-server[1008]: 2026/09/21 16:49:59 INFO Completed upload id=134571lb # [ 72.116649] niks3-server[1008]: 2026/09/21 16:49:59 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo4572worker1 # [ 72.136240] nix-grpc-daemon[775]: ts=2026-09-21T16:49:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38244 duration_s=0 paths=1 nar_bytes_in=5124573worker1 # [ 72.150328] nix-grpc-daemon[775]: ts=2026-09-21T16:49:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:382444574worker1 # [ 72.157729] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4575worker1 # [ 72.162983] nix-daemon[858]: reaped child process 1007, status = succeeded4576worker1 # [ 72.176822] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4577worker1 # [ 73.204946] nix-daemon[858]: reaped child process 1021, status = succeeded4578worker1 # [ 73.364541] nix-daemon[858]: accepted connection from pid 1037, user nix-grpc-daemon (trusted)4579worker1 # [ 73.383968] nix-daemon[858]: reaped child process 1040, status = succeeded4580lb # [ 73.378231] niks3-server[1008]: 2026/09/21 16:50:01 INFO Received uploads request method=POST path=/api/pending_closures4581worker1 # [ 73.407875] nix-grpc-daemon[910]: time=2026-09-21T16:50:01.037Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4582worker1 # [ 73.409504] nix-grpc-daemon[910]: time=2026-09-21T16:50:01.037Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)"4583lb # [ 73.407784] niks3-server[1008]: 2026/09/21 16:50:01 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv4584lb # [ 73.420551] niks3-server[1008]: 2026/09/21 16:50:01 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4585lb # [ 73.435229] niks3-server[1008]: 2026/09/21 16:50:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4586worker1 # [ 73.459347] nix-grpc-daemon[910]: time=2026-09-21T16:50:01.089Z level=INFO msg="Uploading 1 narinfos"4587lb # [ 73.448941] niks3-server[1008]: 2026/09/21 16:50:01 INFO Signed narinfos id=14 count=14588lb # [ 73.451682] niks3-server[1008]: 2026/09/21 16:50:01 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls4589lb # [ 73.466814] niks3-server[1008]: 2026/09/21 16:50:01 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4590worker1 # [ 73.492421] nix-grpc-daemon[910]: time=2026-09-21T16:50:01.122Z level=INFO msg="Upload complete. (306ms)"4591lb # [ 73.476064] niks3-server[1008]: 2026/09/21 16:50:01 INFO Completed upload id=144592lb # [ 73.482478] niks3-server[1008]: 2026/09/21 16:50:01 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo4593worker1 # [ 73.504915] nix-grpc-daemon[775]: ts=2026-09-21T16:50:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38244 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=7 outputs=14594worker1 # [ 73.521330] nix-grpc-daemon[775]: ts=2026-09-21T16:50:01Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:382444595worker1 # [ 73.526607] nix-daemon[858]: reaped child process 1019, status = succeeded4596worker1 # [ 73.551146] nix-grpc-daemon[775]: ts=2026-09-21T16:50:01Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:382444597worker1 # [ 73.558369] nix-grpc-daemon[775]: ts=2026-09-21T16:50:01Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:38244 duration_s=0 paths=1 nar_bytes_out=1284598worker2 # [ 73.966749] nix-grpc-daemon[780]: ts=2026-09-21T16:50:01Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:592644599worker1 # [ 73.882333] nix-grpc-daemon[775]: ts=2026-09-21T16:50:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:382444600worker1 # [ 73.893833] nix-grpc-daemon[910]: time=2026-09-21T16:50:01.524Z level=INFO msg="All 1 paths already cached"4601worker1 # [ 73.912129] nix-grpc-daemon[775]: ts=2026-09-21T16:50:01Z level=debug event=want client=9 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=04602worker1 # [ 73.919671] nix-grpc-daemon[775]: ts=2026-09-21T16:50:01Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.3:50051 clients=14603worker1 # [ 73.929051] nix-grpc-daemon[775]: ts=2026-09-21T16:50:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:382444604worker1 # [ 73.940596] nix-grpc-daemon[775]: ts=2026-09-21T16:50:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:382444605worker1 # [ 73.954681] nix-grpc-daemon[910]: time=2026-09-21T16:50:01.585Z level=INFO msg="All 1 paths already cached"4606worker1 # [ 73.964075] nix-grpc-daemon[775]: ts=2026-09-21T16:50:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:382444607worker1 # [ 73.978678] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4608worker1 # [ 74.175285] nix-daemon[858]: accepted connection from pid 1046, user nix-grpc-daemon (trusted)4609worker1 # [ 74.195460] nix-daemon[858]: reaped child process 1049, status = succeeded4610worker1 # [ 74.365165] nix-daemon[858]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)4611worker1 # [ 74.380974] nix-daemon[858]: reaped child process 1054, status = succeeded4612lb # [ 74.374115] niks3-server[1008]: 2026/09/21 16:50:01 INFO Received uploads request method=POST path=/api/pending_closures4613worker1 # [ 74.402642] nix-grpc-daemon[910]: time=2026-09-21T16:50:02.033Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4614worker1 # [ 74.404518] nix-grpc-daemon[910]: time=2026-09-21T16:50:02.033Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)"4615lb # [ 74.408940] niks3-server[1008]: 2026/09/21 16:50:02 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst4616lb # [ 74.419084] niks3-server[1008]: 2026/09/21 16:50:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4617worker1 # [ 74.445866] nix-grpc-daemon[910]: time=2026-09-21T16:50:02.075Z level=INFO msg="Uploading 1 narinfos"4618lb # [ 74.426263] niks3-server[1008]: 2026/09/21 16:50:02 INFO Signed narinfos id=15 count=14619lb # [ 74.434296] niks3-server[1008]: 2026/09/21 16:50:02 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls4620lb # [ 74.444113] niks3-server[1008]: 2026/09/21 16:50:02 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4621worker1 # [ 74.469967] nix-grpc-daemon[910]: time=2026-09-21T16:50:02.100Z level=INFO msg="Upload complete. (473ms)"4622lb # [ 74.452442] niks3-server[1008]: 2026/09/21 16:50:02 INFO Completed upload id=154623lb # [ 74.456067] niks3-server[1008]: 2026/09/21 16:50:02 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo4624worker1 # [ 74.476248] nix-grpc-daemon[775]: ts=2026-09-21T16:50:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38244 duration_s=0 paths=1 nar_bytes_in=6244625worker1 # [ 74.489788] nix-grpc-daemon[775]: ts=2026-09-21T16:50:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:382444626worker1 # [ 74.499332] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4627worker1 # [ 74.501864] nix-daemon[858]: reaped child process 1044, status = succeeded4628worker1 # [ 74.515413] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4629worker1 # [ 75.762426] nix-daemon[858]: reaped child process 1058, status = succeeded4630worker1 # [ 75.942844] nix-daemon[858]: accepted connection from pid 1074, user nix-grpc-daemon (trusted)4631worker1 # [ 75.960441] nix-daemon[858]: reaped child process 1077, status = succeeded4632lb # [ 75.954604] niks3-server[1008]: 2026/09/21 16:50:03 INFO Received uploads request method=POST path=/api/pending_closures4633worker1 # [ 75.984684] nix-grpc-daemon[910]: time=2026-09-21T16:50:03.613Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4634worker1 # [ 75.987060] nix-grpc-daemon[910]: time=2026-09-21T16:50:03.613Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)"4635lb # [ 75.988872] niks3-server[1008]: 2026/09/21 16:50:03 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv4636lb # [ 76.005867] niks3-server[1008]: 2026/09/21 16:50:03 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst4637lb # [ 76.018423] niks3-server[1008]: 2026/09/21 16:50:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4638worker1 # [ 76.043881] nix-grpc-daemon[910]: time=2026-09-21T16:50:03.673Z level=INFO msg="Uploading 1 narinfos"4639lb # [ 76.028311] niks3-server[1008]: 2026/09/21 16:50:03 INFO Signed narinfos id=16 count=14640lb # [ 76.030935] niks3-server[1008]: 2026/09/21 16:50:03 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls4641lb # [ 76.040392] niks3-server[1008]: 2026/09/21 16:50:03 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4642lb # [ 76.049664] niks3-server[1008]: 2026/09/21 16:50:03 INFO Completed upload id=164643worker1 # [ 76.068816] nix-grpc-daemon[910]: time=2026-09-21T16:50:03.699Z level=INFO msg="Upload complete. (342ms)"4644lb # [ 76.052638] niks3-server[1008]: 2026/09/21 16:50:03 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo4645worker2 # [ 76.173839] nix-grpc-daemon[780]: ts=2026-09-21T16:50:03Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:592644646worker1 # [ 76.085139] nix-grpc-daemon[775]: ts=2026-09-21T16:50:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38244 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=8 outputs=14647worker1 # [ 76.100968] nix-daemon[858]: reaped child process 1056, status = succeeded4648worker2 # [ 76.201678] nix-grpc-daemon[780]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'...4649worker2 # [ 76.222130] nix-grpc-daemon[780]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'...4650worker2 # [ 76.260118] nix-grpc-daemon[780]: ts=2026-09-21T16:50:03Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:592644651worker2 # [ 76.267584] nix-grpc-daemon[780]: ts=2026-09-21T16:50:03Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=0 paths=1 nar_bytes_out=1764652worker1 # [ 76.511227] nix-grpc-daemon[775]: ts=2026-09-21T16:50:04Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:382444653worker1 # [ 76.519581] nix-grpc-daemon[775]: ts=2026-09-21T16:50:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:382444654worker1 # [ 76.526465] nix-grpc-daemon[910]: time=2026-09-21T16:50:04.155Z level=INFO msg="All 2 paths already cached"4655worker1 # [ 76.538687] nix-grpc-daemon[775]: ts=2026-09-21T16:50:04Z level=debug event=want client=10 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=04656worker1 # [ 76.545906] nix-grpc-daemon[775]: ts=2026-09-21T16:50:04Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.3:50051 clients=14657worker1 # [ 76.583257] nix-grpc-daemon[775]: ts=2026-09-21T16:50:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:382444658worker1 # [ 76.598909] nix-grpc-daemon[775]: ts=2026-09-21T16:50:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:382444659worker1 # [ 76.613449] nix-grpc-daemon[910]: time=2026-09-21T16:50:04.243Z level=INFO msg="All 2 paths already cached"4660worker1 # [ 76.622864] nix-grpc-daemon[775]: ts=2026-09-21T16:50:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:382444661worker1 # [ 76.636546] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4662worker1 # [ 76.845685] nix-daemon[858]: accepted connection from pid 1083, user nix-grpc-daemon (trusted)4663worker1 # [ 76.864613] nix-daemon[858]: reaped child process 1086, status = succeeded4664worker1 # [ 77.041836] nix-daemon[858]: accepted connection from pid 1088, user nix-grpc-daemon (trusted)4665worker1 # [ 77.059922] nix-daemon[858]: reaped child process 1091, status = succeeded4666lb # [ 77.056858] niks3-server[1008]: 2026/09/21 16:50:04 INFO Received uploads request method=POST path=/api/pending_closures4667worker1 # [ 77.085507] nix-grpc-daemon[910]: time=2026-09-21T16:50:04.715Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4668worker1 # [ 77.087007] nix-grpc-daemon[910]: time=2026-09-21T16:50:04.715Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)"4669lb # [ 77.092808] niks3-server[1008]: 2026/09/21 16:50:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4670worker1 # [ 77.120084] nix-grpc-daemon[910]: time=2026-09-21T16:50:04.750Z level=INFO msg="Uploading 1 narinfos"4671lb # [ 77.105017] niks3-server[1008]: 2026/09/21 16:50:04 INFO Signed narinfos id=17 count=14672lb # [ 77.117420] niks3-server[1008]: 2026/09/21 16:50:04 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4673lb # [ 77.131908] niks3-server[1008]: 2026/09/21 16:50:04 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst4674lb # [ 77.137706] niks3-server[1008]: 2026/09/21 16:50:04 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls4675worker1 # [ 77.161037] nix-grpc-daemon[910]: time=2026-09-21T16:50:04.791Z level=INFO msg="Upload complete. (507ms)"4676lb # [ 77.142925] niks3-server[1008]: 2026/09/21 16:50:04 INFO Completed upload id=174677lb # [ 77.147644] niks3-server[1008]: 2026/09/21 16:50:04 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo4678worker1 # [ 77.168893] nix-grpc-daemon[775]: ts=2026-09-21T16:50:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38244 duration_s=0 paths=1 nar_bytes_in=7524679worker1 # [ 77.182936] nix-grpc-daemon[775]: ts=2026-09-21T16:50:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:382444680worker1 # [ 77.192170] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4681worker1 # [ 77.195095] nix-daemon[858]: reaped child process 1081, status = succeeded4682worker1 # [ 77.206498] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4683worker1 # [ 78.643258] nix-daemon[858]: reaped child process 1095, status = succeeded4684worker1 # [ 78.814625] nix-daemon[858]: accepted connection from pid 1111, user nix-grpc-daemon (trusted)4685worker1 # [ 78.834430] nix-daemon[858]: reaped child process 1114, status = succeeded4686lb # [ 78.833013] niks3-server[1008]: 2026/09/21 16:50:06 INFO Received uploads request method=POST path=/api/pending_closures4687worker1 # [ 78.864466] nix-grpc-daemon[910]: time=2026-09-21T16:50:06.493Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4688worker1 # [ 78.866120] nix-grpc-daemon[910]: time=2026-09-21T16:50:06.493Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)"4689lb # [ 78.880937] niks3-server[1008]: 2026/09/21 16:50:06 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst4690worker1 # [ 78.912196] nix-grpc-daemon[910]: time=2026-09-21T16:50:06.542Z level=INFO msg="Uploading 1 narinfos"4691lb # [ 78.896927] niks3-server[1008]: 2026/09/21 16:50:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign4692lb # [ 78.904312] niks3-server[1008]: 2026/09/21 16:50:06 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv4693lb # [ 78.911138] niks3-server[1008]: 2026/09/21 16:50:06 INFO Signed narinfos id=18 count=14694worker1 # [ 78.929997] nix-grpc-daemon[910]: time=2026-09-21T16:50:06.560Z level=INFO msg="Upload complete. (311ms)"4695lb # [ 78.916593] niks3-server[1008]: 2026/09/21 16:50:06 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls4696lb # [ 78.921078] niks3-server[1008]: 2026/09/21 16:50:06 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete4697lb # [ 78.925044] niks3-server[1008]: 2026/09/21 16:50:06 INFO Completed upload id=184698worker1 # [ 78.941050] nix-grpc-daemon[775]: ts=2026-09-21T16:50:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38244 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=9 outputs=14699lb # [ 78.927059] niks3-server[1008]: 2026/09/21 16:50:06 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo4700worker1 # [ 78.953455] nix-daemon[858]: reaped child process 1093, status = succeeded4701worker2 # [ 79.041233] nix-grpc-daemon[780]: ts=2026-09-21T16:50:06Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:592644702worker2 # [ 79.054520] nix-grpc-daemon[780]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'...4703worker2 # [ 79.098518] nix-grpc-daemon[780]: ts=2026-09-21T16:50:06Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:592644704worker2 # [ 79.103979] nix-grpc-daemon[780]: ts=2026-09-21T16:50:06Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=0 paths=1 nar_bytes_out=2324705client # [ 79.143909] nix-daemon[1007]: reaped child process 1009, status = succeeded4706client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag hook 2>/tmp/hook.log, in 8.25 seconds)4707client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top4708client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.04 seconds)4709client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4710client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds)4711client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4712client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)4713(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 8.37 seconds)4714subtest: an input only one worker has still reaches the builder4715client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag w1only4716client # this derivation will be built:4717client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv4718client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'...4719client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag w1only, in 0.57 seconds)4720client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure4721client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.20 seconds)4722worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4723worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.24 seconds)4724worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only4725worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.04 seconds)4726client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt a >&24727client # [ 80.577167] nix-daemon[1007]: accepted connection from pid 1090, user root (trusted)4728client # this derivation will be built:4729client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4730client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'...4731worker2 # [ 80.879252] nix-grpc-daemon[780]: ts=2026-09-21T16:50:08Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:592644732worker2 # [ 80.887620] nix-grpc-daemon[780]: ts=2026-09-21T16:50:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:592644733client # copying 1 paths...4734client # copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' to 'grpc://lb:50051'...4735worker1 # [ 80.814382] nix-grpc-daemon[775]: ts=2026-09-21T16:50:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:382444736worker1 # [ 80.827702] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4737lb # [ 80.846806] envoy[926]: conn peer=[2001:db8:1::1]:48114 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3414 tx=1718 ms=20944738lb # [ 80.851920] envoy[926]: conn peer=[2001:db8:1::1]:48116 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=333 ms=734739lb # [ 80.856766] envoy[926]: conn peer=[2001:db8:1::1]:48130 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3904 tx=2085 ms=23624740lb # [ 80.861629] envoy[926]: conn peer=[2001:db8:1::1]:48134 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=375 ms=714741lb # [ 80.867388] envoy[926]: conn peer=[2001:db8:1::1]:48144 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4356 tx=2292 ms=25564742lb # [ 80.872426] envoy[926]: conn peer=[2001:db8:1::1]:48158 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=438 tx=406 ms=634743worker1 # [ 81.027589] nix-daemon[858]: accepted connection from pid 1123, user nix-grpc-daemon (trusted)4744worker1 # [ 81.046454] nix-daemon[858]: reaped child process 1126, status = succeeded4745lb # [ 81.039483] niks3-server[1008]: 2026/09/21 16:50:08 INFO Received uploads request method=POST path=/api/pending_closures4746worker1 # [ 81.067951] nix-grpc-daemon[910]: time=2026-09-21T16:50:08.697Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4747worker1 # [ 81.070784] nix-grpc-daemon[910]: time=2026-09-21T16:50:08.697Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)"4748lb # [ 81.064670] niks3-server[1008]: 2026/09/21 16:50:08 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4749worker1 # [ 81.097891] nix-grpc-daemon[910]: time=2026-09-21T16:50:08.728Z level=INFO msg="Uploading 1 narinfos"4750lb # [ 81.079559] niks3-server[1008]: 2026/09/21 16:50:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign4751lb # [ 81.089370] niks3-server[1008]: 2026/09/21 16:50:08 INFO Signed narinfos id=19 count=14752lb # [ 81.098766] niks3-server[1008]: 2026/09/21 16:50:08 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete4753lb # [ 81.105957] niks3-server[1008]: 2026/09/21 16:50:08 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls4754lb # [ 81.110349] niks3-server[1008]: 2026/09/21 16:50:08 INFO Completed upload id=194755worker1 # [ 81.129466] nix-grpc-daemon[910]: time=2026-09-21T16:50:08.759Z level=INFO msg="Upload complete. (286ms)"4756lb # [ 81.113104] niks3-server[1008]: 2026/09/21 16:50:08 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo4757worker1 # [ 81.140364] nix-grpc-daemon[775]: ts=2026-09-21T16:50:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38244 duration_s=0 paths=1 nar_bytes_in=1204758worker1 # [ 81.157856] nix-grpc-daemon[775]: ts=2026-09-21T16:50:08Z level=debug event=want client=11 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=04759worker1 # [ 81.168517] nix-grpc-daemon[775]: ts=2026-09-21T16:50:08Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=14760worker1 # [ 81.174720] nix-daemon[858]: reaped child process 1121, status = succeeded4761worker2 # [ 81.293478] nix-grpc-daemon[780]: ts=2026-09-21T16:50:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:592644762worker2 # [ 81.306371] nix-grpc-daemon[780]: ts=2026-09-21T16:50:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:592644763client # copying 2 paths to 'lb:50051'...4764client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'...4765worker2 # [ 81.325814] nix-grpc-daemon[780]: ts=2026-09-21T16:50:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:592644766client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'...4767worker2 # [ 81.342819] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4768worker2 # [ 81.563708] nix-daemon[849]: accepted connection from pid 1113, user nix-grpc-daemon (trusted)4769worker2 # [ 81.585268] nix-daemon[849]: reaped child process 1116, status = succeeded4770worker2 # [ 81.762618] nix-daemon[849]: accepted connection from pid 1118, user nix-grpc-daemon (trusted)4771worker2 # [ 81.781096] nix-daemon[849]: reaped child process 1121, status = succeeded4772lb # [ 81.685386] niks3-server[1008]: 2026/09/21 16:50:09 INFO Received uploads request method=POST path=/api/pending_closures4773lb # [ 81.694491] niks3-server[1008]: 2026/09/21 16:50:09 INFO Received uploads request method=POST path=/api/pending_closures4774worker2 # [ 81.811689] nix-grpc-daemon[863]: time=2026-09-21T16:50:09.447Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4775worker2 # [ 81.813385] nix-grpc-daemon[863]: time=2026-09-21T16:50:09.447Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)"4776worker2 # [ 81.815134] nix-grpc-daemon[863]: time=2026-09-21T16:50:09.447Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)"4777lb # [ 81.725259] niks3-server[1008]: 2026/09/21 16:50:09 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst4778lb # [ 81.748110] niks3-server[1008]: 2026/09/21 16:50:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign4779worker2 # [ 81.865113] nix-grpc-daemon[863]: time=2026-09-21T16:50:09.501Z level=INFO msg="Uploading 3 narinfos"4780lb # [ 81.760801] niks3-server[1008]: 2026/09/21 16:50:09 INFO Signed narinfos id=20 count=14781lb # [ 81.763690] niks3-server[1008]: 2026/09/21 16:50:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign4782lb # [ 81.776987] niks3-server[1008]: 2026/09/21 16:50:09 INFO Signed narinfos id=21 count=24783lb # [ 81.779531] niks3-server[1008]: 2026/09/21 16:50:09 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls4784lb # [ 81.792326] niks3-server[1008]: 2026/09/21 16:50:09 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst4785lb # [ 81.799367] niks3-server[1008]: 2026/09/21 16:50:09 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls4786lb # [ 81.805270] niks3-server[1008]: 2026/09/21 16:50:09 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo4787lb # [ 81.811175] niks3-server[1008]: 2026/09/21 16:50:09 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4788lb # [ 81.817012] niks3-server[1008]: 2026/09/21 16:50:09 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete4789lb # [ 81.823027] niks3-server[1008]: 2026/09/21 16:50:09 INFO Completed upload id=214790lb # [ 81.825207] niks3-server[1008]: 2026/09/21 16:50:09 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete4791lb # [ 81.828850] niks3-server[1008]: 2026/09/21 16:50:09 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4792lb # [ 81.832521] niks3-server[1008]: 2026/09/21 16:50:09 INFO Completed upload id=204793worker2 # [ 81.939574] nix-grpc-daemon[863]: time=2026-09-21T16:50:09.576Z level=INFO msg="Upload complete. (576ms)"4794worker2 # [ 81.951299] nix-grpc-daemon[780]: ts=2026-09-21T16:50:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=0 paths=2 nar_bytes_in=9844795worker2 # [ 81.968702] nix-grpc-daemon[780]: ts=2026-09-21T16:50:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:592644796worker2 # [ 81.978915] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4797worker2 # [ 81.986412] nix-daemon[849]: reaped child process 1111, status = succeeded4798worker2 # [ 82.003688] nix-grpc-daemon[780]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'...4799client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051...4800worker2 # [ 82.023629] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4801client # uses-input-w1onlya> worker2: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4802client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds4803worker2 # [ 83.470024] nix-daemon[849]: reaped child process 1126, status = succeeded4804worker2 # [ 83.656396] nix-daemon[849]: accepted connection from pid 1142, user nix-grpc-daemon (trusted)4805worker2 # [ 83.677244] nix-daemon[849]: reaped child process 1145, status = succeeded4806lb # [ 83.582671] niks3-server[1008]: 2026/09/21 16:50:11 INFO Received uploads request method=POST path=/api/pending_closures4807worker2 # [ 83.698818] nix-grpc-daemon[863]: time=2026-09-21T16:50:11.335Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4808lb # [ 83.610456] niks3-server[1008]: 2026/09/21 16:50:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign4809worker2 # [ 83.726677] nix-grpc-daemon[863]: time=2026-09-21T16:50:11.363Z level=INFO msg="Uploading 1 narinfos"4810lb # [ 83.622888] niks3-server[1008]: 2026/09/21 16:50:11 INFO Signed narinfos id=22 count=14811lb # [ 83.632323] niks3-server[1008]: 2026/09/21 16:50:11 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4812lb # [ 83.641268] niks3-server[1008]: 2026/09/21 16:50:11 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls4813worker2 # [ 83.757074] nix-grpc-daemon[863]: time=2026-09-21T16:50:11.393Z level=INFO msg="Upload complete. (316ms)"4814lb # [ 83.650840] niks3-server[1008]: 2026/09/21 16:50:11 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete4815lb # [ 83.655714] niks3-server[1008]: 2026/09/21 16:50:11 INFO Completed upload id=224816lb # [ 83.658759] niks3-server[1008]: 2026/09/21 16:50:11 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo4817client # copying 1 paths...4818worker2 # [ 83.768999] nix-grpc-daemon[780]: ts=2026-09-21T16:50:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=10 outputs=14819worker1 # [ 83.689036] nix-grpc-daemon[775]: ts=2026-09-21T16:50:11Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:382444820worker2 # [ 83.781392] nix-daemon[849]: reaped child process 1123, status = succeeded4821worker1 # [ 83.705585] nix-grpc-daemon[775]: copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'http://lb:5751'...4822client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'...4823worker2 # [ 83.843317] nix-grpc-daemon[780]: ts=2026-09-21T16:50:11Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:592644824worker2 # [ 83.850430] nix-grpc-daemon[780]: ts=2026-09-21T16:50:11Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=0 paths=1 nar_bytes_out=1204825client # [ 83.894000] nix-daemon[1007]: reaped child process 1093, status = succeeded4826client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt a >&2, in 3.57 seconds)4827client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt b >&24828client # [ 84.121406] nix-daemon[1007]: accepted connection from pid 1112, user root (trusted)4829client # this derivation will be built:4830client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4831worker1 # [ 84.335584] nix-grpc-daemon[775]: ts=2026-09-21T16:50:11Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:382444832client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'...4833worker2 # [ 84.428242] nix-grpc-daemon[780]: ts=2026-09-21T16:50:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:592644834worker2 # [ 84.569393] nix-grpc-daemon[863]: time=2026-09-21T16:50:12.205Z level=INFO msg="All 1 paths already cached"4835worker1 # [ 84.503866] nix-grpc-daemon[775]: ts=2026-09-21T16:50:12Z level=debug event=want client=12 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=04836worker1 # [ 84.513352] nix-grpc-daemon[775]: ts=2026-09-21T16:50:12Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=14837worker1 # [ 84.521911] nix-grpc-daemon[775]: ts=2026-09-21T16:50:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:382444838worker1 # [ 84.533260] nix-grpc-daemon[775]: ts=2026-09-21T16:50:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:382444839client # copying 1 paths to 'lb:50051'...4840client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'...4841worker1 # [ 84.554625] nix-grpc-daemon[775]: ts=2026-09-21T16:50:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:382444842worker1 # [ 84.569609] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4843worker1 # [ 84.591656] nix-grpc-daemon[775]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 47 ms (attempt 1/5)4844worker1 # [ 84.644725] nix-grpc-daemon[775]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 25 ms (attempt 2/5)4845worker1 # [ 84.677047] nix-grpc-daemon[775]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 36 ms (attempt 3/5)4846worker1 # [ 84.721419] nix-grpc-daemon[775]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 361 ms (attempt 4/5)4847worker1 # [ 85.090525] nix-grpc-daemon[775]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4848worker1 # [ 85.096878] nix-grpc-daemon[775]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4849worker1 # [ 85.103903] nix-grpc-daemon[775]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'...4850worker1 # [ 85.323495] nix-daemon[858]: accepted connection from pid 1140, user nix-grpc-daemon (trusted)4851worker1 # [ 85.343437] nix-daemon[858]: reaped child process 1143, status = succeeded4852worker1 # [ 85.515976] nix-daemon[858]: accepted connection from pid 1145, user nix-grpc-daemon (trusted)4853worker1 # [ 85.531562] nix-daemon[858]: reaped child process 1148, status = succeeded4854lb # [ 85.529519] niks3-server[1008]: 2026/09/21 16:50:13 INFO Received uploads request method=POST path=/api/pending_closures4855worker1 # [ 85.559575] nix-grpc-daemon[910]: time=2026-09-21T16:50:13.189Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4856worker1 # [ 85.561252] nix-grpc-daemon[910]: time=2026-09-21T16:50:13.189Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)"4857lb # [ 85.561643] niks3-server[1008]: 2026/09/21 16:50:13 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst4858worker1 # [ 85.599307] nix-grpc-daemon[910]: time=2026-09-21T16:50:13.229Z level=INFO msg="Uploading 1 narinfos"4859lb # [ 85.572427] niks3-server[1008]: 2026/09/21 16:50:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign4860lb # [ 85.586962] niks3-server[1008]: 2026/09/21 16:50:13 INFO Signed narinfos id=23 count=14861lb # [ 85.592828] niks3-server[1008]: 2026/09/21 16:50:13 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls4862worker1 # [ 85.620388] nix-grpc-daemon[910]: time=2026-09-21T16:50:13.250Z level=INFO msg="Upload complete. (494ms)"4863lb # [ 85.603325] niks3-server[1008]: 2026/09/21 16:50:13 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete4864lb # [ 85.609109] niks3-server[1008]: 2026/09/21 16:50:13 INFO Completed upload id=234865worker1 # [ 85.628120] nix-grpc-daemon[775]: ts=2026-09-21T16:50:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38244 duration_s=1 paths=1 nar_bytes_in=5684866lb # [ 85.612873] niks3-server[1008]: 2026/09/21 16:50:13 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo4867worker1 # [ 85.636677] nix-grpc-daemon[775]: ts=2026-09-21T16:50:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:382444868worker1 # [ 85.648505] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4869worker1 # [ 85.653361] nix-daemon[858]: reaped child process 1133, status = succeeded4870client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051...4871worker1 # [ 85.666571] nix-daemon[858]: accepted connection from pid 775, user nix-grpc-daemon (trusted)4872client # uses-input-w1onlyb> node-a: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4873client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds4874worker1 # [ 86.511695] nix-daemon[858]: reaped child process 1152, status = succeeded4875worker1 # [ 86.696634] nix-daemon[858]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)4876worker1 # [ 86.713649] nix-daemon[858]: reaped child process 1171, status = succeeded4877lb # [ 86.709787] niks3-server[1008]: 2026/09/21 16:50:14 INFO Received uploads request method=POST path=/api/pending_closures4878worker1 # [ 86.740114] nix-grpc-daemon[910]: time=2026-09-21T16:50:14.368Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4879lb # [ 86.737462] niks3-server[1008]: 2026/09/21 16:50:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign4880worker1 # [ 86.766342] nix-grpc-daemon[910]: time=2026-09-21T16:50:14.396Z level=INFO msg="Uploading 1 narinfos"4881lb # [ 86.752068] niks3-server[1008]: 2026/09/21 16:50:14 INFO Signed narinfos id=24 count=14882lb # [ 86.760152] niks3-server[1008]: 2026/09/21 16:50:14 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls4883lb # [ 86.766958] niks3-server[1008]: 2026/09/21 16:50:14 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4884lb # [ 86.775601] niks3-server[1008]: 2026/09/21 16:50:14 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete4885lb # [ 86.780927] niks3-server[1008]: 2026/09/21 16:50:14 INFO Completed upload id=244886worker1 # [ 86.799997] nix-grpc-daemon[910]: time=2026-09-21T16:50:14.430Z level=INFO msg="Upload complete. (323ms)"4887lb # [ 86.783808] niks3-server[1008]: 2026/09/21 16:50:14 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo4888client # copying 1 paths...4889worker1 # [ 86.811497] nix-grpc-daemon[775]: ts=2026-09-21T16:50:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38244 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=11 outputs=14890client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'...4891worker1 # [ 86.827117] nix-grpc-daemon[775]: ts=2026-09-21T16:50:14Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:382444892worker1 # [ 86.831489] nix-daemon[858]: reaped child process 1150, status = succeeded4893worker2 # [ 86.938244] nix-grpc-daemon[780]: ts=2026-09-21T16:50:14Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:592644894worker2 # [ 86.952207] nix-grpc-daemon[780]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'...4895worker2 # [ 86.969406] nix-grpc-daemon[780]: ts=2026-09-21T16:50:14Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=0 paths=1 nar_bytes_out=1204896client # [ 87.013007] nix-daemon[1007]: reaped child process 1115, status = succeeded4897client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.12 seconds)4898client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt c >&24899client # [ 87.228100] nix-daemon[1007]: accepted connection from pid 1133, user root (trusted)4900client # this derivation will be built:4901client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4902worker1 # [ 87.447835] nix-grpc-daemon[775]: ts=2026-09-21T16:50:15Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:382444903client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'...4904worker1 # [ 87.455282] nix-grpc-daemon[775]: ts=2026-09-21T16:50:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:382444905worker1 # [ 87.462050] nix-grpc-daemon[910]: time=2026-09-21T16:50:15.091Z level=INFO msg="All 1 paths already cached"4906worker1 # [ 87.474400] nix-grpc-daemon[775]: ts=2026-09-21T16:50:15Z level=debug event=want client=13 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=04907worker1 # [ 87.480820] nix-grpc-daemon[775]: ts=2026-09-21T16:50:15Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=14908worker2 # [ 87.571993] nix-grpc-daemon[780]: ts=2026-09-21T16:50:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:592644909worker2 # [ 87.587357] nix-grpc-daemon[780]: ts=2026-09-21T16:50:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:592644910worker2 # [ 87.603380] nix-grpc-daemon[863]: time=2026-09-21T16:50:15.239Z level=INFO msg="All 1 paths already cached"4911client # copying 1 paths to 'lb:50051'...4912client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'...4913worker2 # [ 87.613578] nix-grpc-daemon[780]: ts=2026-09-21T16:50:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:592644914worker2 # [ 87.629164] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4915worker2 # [ 87.842015] nix-daemon[849]: accepted connection from pid 1150, user nix-grpc-daemon (trusted)4916worker2 # [ 87.860955] nix-daemon[849]: reaped child process 1153, status = succeeded4917worker2 # [ 88.036765] nix-daemon[849]: accepted connection from pid 1155, user nix-grpc-daemon (trusted)4918worker2 # [ 88.053280] nix-daemon[849]: reaped child process 1158, status = succeeded4919lb # [ 87.961080] niks3-server[1008]: 2026/09/21 16:50:15 INFO Received uploads request method=POST path=/api/pending_closures4920worker2 # [ 88.078158] nix-grpc-daemon[863]: time=2026-09-21T16:50:15.714Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4921worker2 # [ 88.079767] nix-grpc-daemon[863]: time=2026-09-21T16:50:15.714Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)"4922lb # [ 87.990128] niks3-server[1008]: 2026/09/21 16:50:15 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst4923worker2 # [ 88.109079] nix-grpc-daemon[863]: time=2026-09-21T16:50:15.745Z level=INFO msg="Uploading 1 narinfos"4924lb # [ 88.003965] niks3-server[1008]: 2026/09/21 16:50:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign4925lb # [ 88.015564] niks3-server[1008]: 2026/09/21 16:50:15 INFO Signed narinfos id=25 count=14926lb # [ 88.023271] niks3-server[1008]: 2026/09/21 16:50:15 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls4927lb # [ 88.034029] niks3-server[1008]: 2026/09/21 16:50:15 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete4928worker2 # [ 88.146345] nix-grpc-daemon[863]: time=2026-09-21T16:50:15.782Z level=INFO msg="Upload complete. (497ms)"4929lb # [ 88.040537] niks3-server[1008]: 2026/09/21 16:50:15 INFO Completed upload id=254930lb # [ 88.044561] niks3-server[1008]: 2026/09/21 16:50:15 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo4931worker2 # [ 88.157478] nix-grpc-daemon[780]: ts=2026-09-21T16:50:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=0 paths=1 nar_bytes_in=5684932worker2 # [ 88.168556] nix-grpc-daemon[780]: ts=2026-09-21T16:50:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:592644933worker2 # [ 88.175328] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4934worker2 # [ 88.179734] nix-daemon[849]: reaped child process 1148, status = succeeded4935client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051...4936worker2 # [ 88.193152] nix-daemon[849]: accepted connection from pid 780, user nix-grpc-daemon (trusted)4937client # uses-input-w1onlyc> worker2: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4938client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds4939worker2 # [ 89.012563] nix-daemon[849]: reaped child process 1162, status = succeeded4940worker2 # [ 89.183103] nix-daemon[849]: accepted connection from pid 1178, user nix-grpc-daemon (trusted)4941worker2 # [ 89.199585] nix-daemon[849]: reaped child process 1181, status = succeeded4942lb # [ 89.109451] niks3-server[1008]: 2026/09/21 16:50:16 INFO Received uploads request method=POST path=/api/pending_closures4943worker2 # [ 89.228417] nix-grpc-daemon[863]: time=2026-09-21T16:50:16.861Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4944lb # [ 89.137523] niks3-server[1008]: 2026/09/21 16:50:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign4945worker2 # [ 89.254284] nix-grpc-daemon[863]: time=2026-09-21T16:50:16.890Z level=INFO msg="Uploading 1 narinfos"4946lb # [ 89.151083] niks3-server[1008]: 2026/09/21 16:50:16 INFO Signed narinfos id=26 count=14947lb # [ 89.160079] niks3-server[1008]: 2026/09/21 16:50:16 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4948worker2 # [ 89.281359] nix-grpc-daemon[863]: time=2026-09-21T16:50:16.917Z level=INFO msg="Upload complete. (279ms)"4949lb # [ 89.174501] niks3-server[1008]: 2026/09/21 16:50:16 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls4950lb # [ 89.181290] niks3-server[1008]: 2026/09/21 16:50:16 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete4951worker2 # [ 89.288810] nix-grpc-daemon[780]: ts=2026-09-21T16:50:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59264 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=12 outputs=14952lb # [ 89.186339] niks3-server[1008]: 2026/09/21 16:50:16 INFO Completed upload id=264953client # copying 1 paths...4954lb # [ 89.189618] niks3-server[1008]: 2026/09/21 16:50:16 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo4955worker2 # [ 89.300801] nix-daemon[849]: reaped child process 1160, status = succeeded4956worker1 # [ 89.215999] nix-grpc-daemon[775]: ts=2026-09-21T16:50:16Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:382444957worker1 # [ 89.227906] nix-grpc-daemon[775]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'...4958client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'...4959worker1 # [ 89.273832] nix-grpc-daemon[775]: ts=2026-09-21T16:50:16Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:382444960worker1 # [ 89.280995] nix-grpc-daemon[775]: ts=2026-09-21T16:50:16Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:38244 duration_s=0 paths=1 nar_bytes_out=1204961client # [ 89.410198] nix-daemon[1007]: reaped child process 1136, status = succeeded4962client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.40 seconds)4963(finished: subtest: an input only one worker has still reaches the builder, in 10.14 seconds)4964subtest: upload whose reference the worker lacks is completed from the cache4965client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag viacache4966client # this derivation will be built:4967client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv4968client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'...4969client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag viacache, in 0.46 seconds)4970client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A referrer --argstr tag viacache4971client # this derivation will be built:4972client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv4973client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'...4974client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A referrer --argstr tag viacache, in 0.45 seconds)4975client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure4976client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.21 seconds)4977worker1: must succeed: nix-store --import < /tmp/shared/ref.closure4978worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.25 seconds)4979worker2 # [ 90.954885] systemd[1]: Stopping Nix worker-protocol over gRPC...4980lb # [ 90.847702] envoy[926]: conn peer=[2001:db8:1::1]:48172 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4248 tx=1978 ms=30294981lb # [ 90.852800] envoy[926]: conn peer=[2001:db8:1::1]:42560 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=704982lb # [ 90.857814] envoy[926]: conn peer=[2001:db8:1::1]:42570 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3727 tx=1924 ms=26034983lb # [ 90.862730] envoy[926]: conn peer=[2001:db8:1::1]:42586 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=329 ms=954984lb # [ 90.868431] envoy[926]: conn peer=[2001:db8:1::1]:42602 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3741 tx=1990 ms=18914985lb # [ 90.873420] envoy[926]: conn peer=[2001:db8:1::1]:42608 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=714986worker2 # [ 91.501257] nix-grpc-daemon[780]: ts=2026-09-21T16:50:19Z level=info event=signal_exit4987worker2 # [ 92.575682] nix-grpc-daemon[780]: ts=2026-09-21T16:50:20Z level=info event=scheduler_restarting4988worker1 # [ 92.495000] nix-grpc-daemon[775]: ts=2026-09-21T16:50:20Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:382284989worker1 # [ 92.499876] nix-grpc-daemon[775]: ts=2026-09-21T16:50:20Z level=info event=worker_gone addr=192.168.1.4:500514990worker2 # [ 92.611996] nix-grpc-daemon[780]: ts=2026-09-21T16:50:20Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4991worker2 # [ 92.640581] nix-daemon[849]: reaped child process 851, status = succeeded4992worker2 # [ 92.644263] nix-grpc-daemon[780]: E0000 00:00:1790009420.280586 825 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true4993worker2 # [ 92.653299] nix-grpc-daemon[780]: E0000 00:00:1790009420.289751 825 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true4994worker2 # [ 92.862489] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4995worker2 # [ 92.871950] systemd[1]: Stopped Nix worker-protocol over gRPC.4996worker2 # [ 92.875428] systemd[1]: nix-grpc-daemon.service: Consumed 3.254s CPU time over 1min 11.731s wall clock time, 62.5M memory peak, 85.8K incoming IP traffic, 207.4K outgoing IP traffic.4997worker2 # [ 92.888675] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.4998worker2 # [ 92.890773] systemd[1]: Closed Nix worker-protocol over gRPC.4999worker2 # [ 92.893049] systemd[1]: nix-grpc-daemon.socket: Consumed 561us CPU time over 1min 21.634s wall clock time, 256K memory peak, 81.2K incoming IP traffic, 66.9K outgoing IP traffic.5000lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy5001lb: must succeed: curl -sf localhost:9901/clusters5002lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5003lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.09 seconds)5004client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&25005worker1 # [ 93.163301] nix-grpc-daemon[775]: ts=2026-09-21T16:50:20Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:382445006worker1 # [ 93.169773] nix-grpc-daemon[775]: ts=2026-09-21T16:50:20Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:382445007client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache5008client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.29 seconds)5009worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache5010worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.03 seconds)5011worker2 # [ 93.390283] systemd[1]: Listening on Nix worker-protocol over gRPC.5012worker1 # [ 93.377839] systemd[1]: Stopping Nix worker-protocol over gRPC...5013worker2 # [ 93.857082] systemd[1]: Starting scheduler-token.service...5014worker2 # [ 93.981034] systemd[1]: scheduler-token.service: Deactivated successfully.5015worker2 # [ 93.988421] systemd[1]: Finished scheduler-token.service.5016worker2 # [ 93.990004] systemd[1]: scheduler-token.service: Consumed 56ms CPU time over 127ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5017worker2 # [ 93.997937] systemd[1]: Starting Nix worker-protocol over gRPC...5018worker2 # [ 94.203168] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:21Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5019worker2 # [ 94.205795] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:21Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5020worker2 # [ 94.210507] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:21Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15021worker2 # [ 94.220091] nix-grpc-daemon[1205]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5022worker2 # [ 94.221609] nix-grpc-daemon[1205]: E0000 00:00:1790009421.854279 1205 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5023worker2 # [ 94.223960] nix-grpc-daemon[1205]: E0000 00:00:1790009421.856707 1205 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5024worker2 # [ 94.237308] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:21Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5025worker2 # [ 94.242039] systemd[1]: Started Nix worker-protocol over gRPC.5026worker2 # [ 94.248096] nix-grpc-daemon[1205]: E0000 00:00:1790009421.883263 1244 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5027worker2 # [ 94.276085] nix-grpc-daemon[1205]: E0000 00:00:1790009421.902716 1244 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5028worker2 # [ 94.278567] nix-grpc-daemon[1205]: E0000 00:00:1790009421.902906 1244 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5029worker2 # [ 94.287923] nix-grpc-daemon[1205]: E0000 00:00:1790009421.923427 1244 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5030worker2 # [ 94.301816] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:21Z level=info event=scheduler_connected addr=lb:500515031worker1 # [ 94.218163] nix-grpc-daemon[775]: ts=2026-09-21T16:50:21Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:382285032worker1 # [ 94.224133] nix-grpc-daemon[775]: ts=2026-09-21T16:50:21Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05033worker1 # [ 94.281939] nix-grpc-daemon[775]: ts=2026-09-21T16:50:21Z level=info event=signal_exit5034lb # [ 94.542733] niks3-server[1008]: 2026/09/21 16:50:22 INFO lead: released remote=192.168.1.3:500985035worker2 # [ 95.650925] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:23Z level=info event=scheduler_restarting addr=lb:500515036worker1 # [ 95.570227] nix-grpc-daemon[775]: ts=2026-09-21T16:50:23Z level=info event=scheduler_restarting5037worker1 # [ 95.574370] nix-grpc-daemon[775]: ts=2026-09-21T16:50:23Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:382285038worker1 # [ 95.579101] nix-grpc-daemon[775]: ts=2026-09-21T16:50:23Z level=info event=worker_gone addr=192.168.1.4:500515039worker1 # [ 95.583500] nix-grpc-daemon[775]: ts=2026-09-21T16:50:23Z level=info event=scheduler_restarting addr=lb:500515040worker1 # [ 95.587950] nix-grpc-daemon[775]: ts=2026-09-21T16:50:23Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:382285041worker1 # [ 95.592973] nix-grpc-daemon[775]: ts=2026-09-21T16:50:23Z level=info event=worker_gone addr=192.168.1.3:500515042worker1 # [ 95.620100] nix-daemon[858]: reaped child process 860, status = succeeded5043worker2 # [ 95.857433] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:23Z level=info event=scheduler_connected addr=lb:500515044worker2 # [ 95.862424] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5045worker1 # [ 95.790091] nix-grpc-daemon[775]: ts=2026-09-21T16:50:23Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5046worker2 # [ 96.063871] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:23Z level=info event=scheduler_connected addr=lb:500515047worker2 # [ 96.069223] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5048worker2 # [ 96.270010] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:23Z level=info event=scheduler_connected addr=lb:500515049worker2 # [ 96.275190] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5050worker2 # [ 96.476358] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:24Z level=info event=scheduler_connected addr=lb:500515051worker2 # [ 96.481484] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5052worker1 # [ 96.445850] nix-grpc-daemon[775]: E0000 00:00:1790009424.075945 820 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5053worker1 # [ 96.446271] nix-grpc-daemon[775]: E0000 00:00:1790009424.076315 820 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5054worker2 # [ 96.684357] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:24Z level=info event=scheduler_connected addr=lb:500515055worker2 # [ 96.688751] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5056worker1 # [ 96.705522] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5057worker1 # [ 96.711617] systemd[1]: Stopped Nix worker-protocol over gRPC.5058worker1 # [ 96.720794] systemd[1]: nix-grpc-daemon.service: Consumed 2.798s CPU time over 1min 15.835s wall clock time, 59.3M memory peak, 78.2K incoming IP traffic, 159K outgoing IP traffic.5059worker1 # [ 96.734333] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5060worker1 # [ 96.737594] systemd[1]: Closed Nix worker-protocol over gRPC.5061worker1 # [ 96.739280] systemd[1]: nix-grpc-daemon.socket: Consumed 563us CPU time over 1min 25.459s wall clock time, 256K memory peak, 94.7K incoming IP traffic, 79.9K outgoing IP traffic.5062lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy5063lb: must succeed: curl -sf localhost:9901/clusters5064worker2 # [ 96.885185] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:24Z level=info event=scheduler_connected addr=lb:500515065worker2 # [ 96.890407] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5066lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5067lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.09 seconds)5068client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&25069worker2 # [ 97.091399] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:24Z level=info event=scheduler_connected addr=lb:500515070worker2 # [ 97.096833] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5071worker2 # [ 97.195951] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:420985072worker2 # [ 97.212565] nix-daemon[849]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5073worker2 # [ 97.240695] nix-grpc-daemon[1205]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 27 ms (attempt 1/5)5074worker2 # [ 97.274631] nix-grpc-daemon[1205]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 93 ms (attempt 2/5)5075worker2 # [ 97.297841] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:24Z level=info event=scheduler_connected addr=lb:500515076worker2 # [ 97.302979] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5077worker2 # [ 97.375173] nix-grpc-daemon[1205]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 75 ms (attempt 3/5)5078worker2 # [ 97.457450] nix-grpc-daemon[1205]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 426 ms (attempt 4/5)5079worker2 # [ 97.504086] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:25Z level=info event=scheduler_connected addr=lb:500515080worker2 # [ 97.509378] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5081worker2 # [ 97.713544] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:25Z level=info event=scheduler_connected addr=lb:500515082worker2 # [ 97.718070] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5083worker2 # [ 97.891477] nix-grpc-daemon[1205]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5084worker2 # [ 97.897824] nix-grpc-daemon[1205]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5085worker2 # [ 97.914201] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:25Z level=info event=scheduler_connected addr=lb:500515086client # copying 2 paths...5087worker2 # [ 97.921044] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5088client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'...5089worker2 # [ 97.928434] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:420985090client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'...5091worker2 # [ 97.945232] nix-daemon[849]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5092worker2 # [ 97.975731] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:25Z level=info event=niks3_push_started pid=12655093worker2 # [ 98.122013] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:25Z level=info event=scheduler_connected addr=lb:500515094worker2 # [ 98.124385] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5095worker2 # [ 98.190395] nix-grpc-daemon[1265]: time=2026-09-21T16:50:25.826Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""5096worker2 # [ 98.324980] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:25Z level=info event=scheduler_connected addr=lb:500515097worker2 # [ 98.327324] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5098worker2 # [ 98.373913] nix-daemon[849]: accepted connection from pid 1278, user nix-grpc-daemon (trusted)5099worker2 # [ 98.390640] nix-daemon[849]: reaped child process 1281, status = succeeded5100lb # [ 98.297059] niks3-server[1008]: 2026/09/21 16:50:25 INFO Received uploads request method=POST path=/api/pending_closures5101lb # [ 98.310271] niks3-server[1008]: 2026/09/21 16:50:25 INFO Received uploads request method=POST path=/api/pending_closures5102worker2 # [ 98.428074] nix-grpc-daemon[1265]: time=2026-09-21T16:50:26.063Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5103worker2 # [ 98.429544] nix-grpc-daemon[1265]: time=2026-09-21T16:50:26.063Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)"5104worker2 # [ 98.431268] nix-grpc-daemon[1265]: time=2026-09-21T16:50:26.063Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)"5105lb # [ 98.371592] niks3-server[1008]: 2026/09/21 16:50:25 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst5106lb # [ 98.382953] niks3-server[1008]: 2026/09/21 16:50:26 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5107lb # [ 98.389702] niks3-server[1008]: 2026/09/21 16:50:26 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls5108lb # [ 98.394169] niks3-server[1008]: 2026/09/21 16:50:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5109lb # [ 98.399903] niks3-server[1008]: 2026/09/21 16:50:26 INFO Signed narinfos id=27 count=15110worker2 # [ 98.507884] nix-grpc-daemon[1265]: time=2026-09-21T16:50:26.142Z level=INFO msg="Uploading 3 narinfos"5111lb # [ 98.404136] niks3-server[1008]: 2026/09/21 16:50:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5112lb # [ 98.406657] niks3-server[1008]: 2026/09/21 16:50:26 INFO Signed narinfos id=28 count=25113lb # [ 98.413288] niks3-server[1008]: 2026/09/21 16:50:26 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls5114lb # [ 98.424316] niks3-server[1008]: 2026/09/21 16:50:26 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo5115worker2 # [ 98.528846] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:26Z level=info event=scheduler_connected addr=lb:500515116worker2 # [ 98.534968] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5117lb # [ 98.431534] niks3-server[1008]: 2026/09/21 16:50:26 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5118lb # [ 98.438226] niks3-server[1008]: 2026/09/21 16:50:26 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5119lb # [ 98.445252] niks3-server[1008]: 2026/09/21 16:50:26 INFO Completed upload id=285120lb # [ 98.448738] niks3-server[1008]: 2026/09/21 16:50:26 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5121lb # [ 98.451905] niks3-server[1008]: 2026/09/21 16:50:26 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5122worker2 # [ 98.558708] nix-grpc-daemon[1265]: time=2026-09-21T16:50:26.195Z level=INFO msg="Upload complete. (364ms)"5123lb # [ 98.454561] niks3-server[1008]: 2026/09/21 16:50:26 INFO Completed upload id=275124worker2 # [ 98.568435] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42098 duration_s=0 paths=2 nar_bytes_in=3125125worker2 # [ 98.586720] nix-daemon[849]: reaped child process 1263, status = succeeded5126client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.69 seconds)5127worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache5128worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.04 seconds)5129worker1 # [ 98.644772] systemd[1]: Listening on Nix worker-protocol over gRPC.5130worker1 # [ 98.650979] systemd[1]: Starting Nix worker-protocol over gRPC...5131worker2 # [ 98.737481] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:26Z level=info event=scheduler_connected addr=lb:500515132worker2 # [ 98.742171] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5133worker1 # [ 98.847421] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:26Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5134worker1 # [ 98.850108] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:26Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5135worker1 # [ 98.854244] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:26Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15136worker2 # [ 98.943391] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:26Z level=info event=scheduler_connected addr=lb:500515137worker1 # [ 98.863132] nix-grpc-daemon[1197]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5138worker2 # [ 98.948737] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5139worker1 # [ 98.876581] nix-grpc-daemon[1197]: E0000 00:00:1790009426.491406 1197 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5140worker1 # [ 98.881442] nix-grpc-daemon[1197]: E0000 00:00:1790009426.493568 1197 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5141lb # [ 98.860634] niks3-server[1008]: 2026/09/21 16:50:26 INFO lead: acquired remote=192.168.1.3:470685142worker1 # [ 98.888187] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:26Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5143lb: waiting until exactly no endpoints of aarch64-linux are unhealthy5144lb: must succeed: curl -sf localhost:9901/clusters5145worker1 # [ 98.912315] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:26Z level=info event=scheduler_take_over5146worker1 # [ 98.913450] nix-grpc-daemon[1197]: E0000 00:00:1790009426.515378 1237 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5147worker1 # [ 98.924265] systemd[1]: Started Nix worker-protocol over gRPC.5148worker1 # [ 98.925506] nix-grpc-daemon[1197]: E0000 00:00:1790009426.534519 1237 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5149worker1 # [ 98.927893] nix-grpc-daemon[1197]: E0000 00:00:1790009426.534751 1237 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5150worker1 # [ 98.930370] nix-grpc-daemon[1197]: E0000 00:00:1790009426.554026 1237 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5151worker1 # [ 98.941795] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:26Z level=info event=scheduler_connected addr=lb:500515152worker1 # [ 98.946049] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:26Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:570065153worker1 # [ 98.947850] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:26Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05154lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5155lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.09 seconds)5156(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 9.57 seconds)5157subtest: a niks3 restart does not cost the leader its role5158lb: must succeed: curl -sf localhost:9901/clusters5159worker2 # [ 99.149828] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:26Z level=info event=scheduler_connected addr=lb:500515160lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5161lb: must succeed: curl -sf localhost:9901/clusters5162worker1 # [ 99.072000] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:26Z level=info event=oidc_jwks provider=mock keys=15163worker1 # [ 99.075862] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:26Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:570065164worker1 # [ 99.081667] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:26Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05165lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5166lb # [ 99.143420] systemd[1]: Stopping niks3 server...5167worker1 # [ 99.164072] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:26Z level=info event=scheduler_lock_lost status=2005168worker2 # [ 99.252700] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:26Z level=info event=scheduler_lock_lost status=2005169lb # [ 99.149403] niks3-server[1008]: 2026/09/21 16:50:26 INFO Shutdown signal received, draining in-flight requests timeout=10s5170lb # [ 99.160111] niks3-server[1008]: 2026/09/21 16:50:26 INFO lead: released remote=192.168.1.3:470685171lb # [ 99.163472] systemd[1]: niks3.service: Deactivated successfully.5172lb # [ 99.165091] systemd[1]: Stopped niks3 server.5173lb # [ 99.166565] systemd[1]: niks3.service: Consumed 535ms CPU time over 1min 11.888s wall clock time, 22.4M memory peak, 77.7K incoming IP traffic, 110.4K outgoing IP traffic.5174lb # [ 99.176798] systemd[1]: Starting niks3 server...5175lb # [ 99.249817] niks3-server[1147]: 2026/09/21 16:50:26 goose: no migrations to run. current version: 202609200000005176lb # [ 99.256464] niks3-server[1147]: 2026/09/21 16:50:26 OK 1_commit_pending_closure.sql (5ms)5177lb # [ 99.259885] niks3-server[1147]: 2026/09/21 16:50:26 OK 2_object_stats_trigger.sql (1.79ms)5178lb # [ 99.261344] niks3-server[1147]: 2026/09/21 16:50:26 goose: up to current file version: 25179lb # [ 99.263268] niks3-server[1147]: 2026/09/21 16:50:26 INFO Loaded signing key name=farm-test-1 path=/nix/store/8lzz3wcf17v05vfdp1zcfaa9hipw3f70-key5180lb # [ 99.274959] niks3-server[1147]: 2026/09/21 16:50:26 INFO Read proxy enabled — serving cache objects from S35181lb # [ 99.279264] niks3-server[1147]: 2026/09/21 16:50:26 INFO Using socket-activated listener address=0.0.0.0:57515182lb # [ 99.281027] niks3-server[1147]: 2026/09/21 16:50:26 INFO systemd watchdog enabled interval=15s5183lb # [ 99.282277] niks3-server[1147]: 2026/09/21 16:50:26 INFO Starting HTTP server address=0.0.0.0:57515184lb # [ 99.283635] systemd[1]: Started niks3 server.5185lb: waiting for unit niks3.service5186lb: (finished: waiting for unit niks3.service, in 0.04 seconds)5187lb: must succeed: curl -sf localhost:9901/clusters5188lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5189worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5190worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5191lb: must succeed: curl -sf localhost:9901/clusters5192lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5193lb: must succeed: curl -sf localhost:9901/clusters5194lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5195worker1: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5196worker1: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.07 seconds)5197client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag after-niks3-restart >&25198worker1 # [ 99.904588] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:570085199worker1 # [ 99.918534] nix-daemon[858]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5200client # don't know how to build these paths:5201client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5202worker2 # [ 100.027537] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:420985203worker1 # [ 99.970641] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:27Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=05204worker1 # [ 99.978147] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:27Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.3:50051 clients=15205worker1 # [ 100.023514] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:570085206worker1 # [ 100.040745] nix-grpc-daemon[1197]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 37 ms (attempt 1/5)5207worker1 # [ 100.085330] nix-grpc-daemon[1197]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 14 ms (attempt 2/5)5208worker1 # [ 100.099580] nix-grpc-daemon[1197]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 315 ms (attempt 3/5)5209lb # [ 100.151673] niks3-server[1147]: 2026/09/21 16:50:27 INFO lead: acquired remote=192.168.1.3:470785210worker1 # [ 100.421942] nix-grpc-daemon[1197]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 331 ms (attempt 4/5)5211worker1 # [ 100.754040] nix-grpc-daemon[1197]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5212worker1 # [ 100.760908] nix-grpc-daemon[1197]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5213worker1 # [ 100.778920] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:570085214client # copying 1 paths to 'lb:50051'...5215client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'...5216worker1 # [ 100.794041] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:570085217worker1 # [ 100.808911] nix-daemon[858]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5218worker1 # [ 100.836490] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:28Z level=info event=niks3_push_started pid=12695219lb # [ 100.849297] envoy[926]: conn peer=[2001:db8:1::4]:49644 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=64873 tx=3495 ms=710345220lb # [ 100.854342] envoy[926]: conn peer=[2001:db8:1::1]:49184 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=523 ms=275221lb # [ 100.859288] envoy[926]: conn peer=[2001:db8:1::3]:55606 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=12683 tx=3380 ms=744345222lb # [ 100.864430] envoy[926]: conn peer=[2001:db8:1::1]:49196 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1006 tx=305 ms=13895223worker1 # [ 101.050874] nix-grpc-daemon[1269]: time=2026-09-21T16:50:28.680Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""5224worker1 # [ 101.242460] nix-daemon[858]: accepted connection from pid 1282, user nix-grpc-daemon (trusted)5225worker1 # [ 101.258608] nix-daemon[858]: reaped child process 1285, status = succeeded5226worker1 # [ 101.430048] nix-daemon[858]: accepted connection from pid 1287, user nix-grpc-daemon (trusted)5227worker1 # [ 101.445412] nix-daemon[858]: reaped child process 1290, status = succeeded5228lb # [ 101.441951] niks3-server[1147]: 2026/09/21 16:50:29 INFO Received uploads request method=POST path=/api/pending_closures5229worker1 # [ 101.475093] nix-grpc-daemon[1269]: time=2026-09-21T16:50:29.104Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5230worker1 # [ 101.476755] nix-grpc-daemon[1269]: time=2026-09-21T16:50:29.104Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)"5231lb # [ 101.496178] niks3-server[1147]: 2026/09/21 16:50:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5232lb # [ 101.514842] niks3-server[1147]: 2026/09/21 16:50:29 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst5233lb # [ 101.539598] niks3-server[1147]: 2026/09/21 16:50:29 INFO Signed narinfos id=29 count=15234worker1 # [ 101.561666] nix-grpc-daemon[1269]: time=2026-09-21T16:50:29.191Z level=INFO msg="Uploading 1 narinfos"5235lb # [ 101.547409] niks3-server[1147]: 2026/09/21 16:50:29 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls5236lb # [ 101.556083] niks3-server[1147]: 2026/09/21 16:50:29 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5237lb # [ 101.566943] niks3-server[1147]: 2026/09/21 16:50:29 INFO Completed upload id=295238worker1 # [ 101.586250] nix-grpc-daemon[1269]: time=2026-09-21T16:50:29.216Z level=INFO msg="Upload complete. (531ms)"5239lb # [ 101.573338] niks3-server[1147]: 2026/09/21 16:50:29 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.narinfo5240worker1 # [ 101.598301] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57008 duration_s=0 paths=1 nar_bytes_in=5445241worker1 # [ 101.616460] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:570085242worker1 # [ 101.626863] nix-daemon[858]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5243worker1 # [ 101.630213] nix-daemon[858]: reaped child process 1267, status = succeeded5244client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051...5245client # farm-a> node-a: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5246worker1 # [ 101.645961] nix-daemon[858]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5247client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5248client # farm-a> LOG-farm-a-after-niks3-restart5249worker1 # [ 102.690470] nix-daemon[858]: reaped child process 1294, status = succeeded5250worker1 # [ 102.874029] nix-daemon[858]: accepted connection from pid 1310, user nix-grpc-daemon (trusted)5251worker1 # [ 102.890660] nix-daemon[858]: reaped child process 1313, status = succeeded5252lb # [ 102.885478] niks3-server[1147]: 2026/09/21 16:50:30 INFO Received uploads request method=POST path=/api/pending_closures5253worker1 # [ 102.918219] nix-grpc-daemon[1269]: time=2026-09-21T16:50:30.547Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5254worker1 # [ 102.919732] nix-grpc-daemon[1269]: time=2026-09-21T16:50:30.547Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)"5255lb # [ 102.937812] niks3-server[1147]: 2026/09/21 16:50:30 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5256worker1 # [ 102.969233] nix-grpc-daemon[1269]: time=2026-09-21T16:50:30.598Z level=INFO msg="Uploading 1 narinfos"5257lb # [ 102.951164] niks3-server[1147]: 2026/09/21 16:50:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5258lb # [ 102.960490] niks3-server[1147]: 2026/09/21 16:50:30 INFO Signed narinfos id=30 count=15259lb # [ 102.962524] niks3-server[1147]: 2026/09/21 16:50:30 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5260lb # [ 102.967931] niks3-server[1147]: 2026/09/21 16:50:30 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls5261lb # [ 102.976191] niks3-server[1147]: 2026/09/21 16:50:30 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5262lb # [ 102.980242] niks3-server[1147]: 2026/09/21 16:50:30 INFO Completed upload id=305263worker1 # [ 102.999416] nix-grpc-daemon[1269]: time=2026-09-21T16:50:30.629Z level=INFO msg="Upload complete. (339ms)"5264lb # [ 102.982417] niks3-server[1147]: 2026/09/21 16:50:30 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.narinfo5265worker1 # [ 103.019592] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57008 duration_s=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=15266worker1 # [ 103.034127] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:30Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=05267worker1 # [ 103.037864] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:30Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.3:50051 clients=15268worker1 # [ 103.041934] nix-daemon[858]: reaped child process 1292, status = succeeded5269worker1 # [ 103.063580] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:570085270worker1 # [ 103.078470] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:570085271worker1 # [ 103.093450] nix-grpc-daemon[1269]: time=2026-09-21T16:50:30.723Z level=INFO msg="All 1 paths already cached"5272client # copying 1 paths to 'lb:50051'...5273client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'...5274worker1 # [ 103.103275] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:570085275worker1 # [ 103.120341] nix-daemon[858]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5276worker1 # [ 103.320080] nix-daemon[858]: accepted connection from pid 1319, user nix-grpc-daemon (trusted)5277worker1 # [ 103.338938] nix-daemon[858]: reaped child process 1322, status = succeeded5278worker1 # [ 103.508254] nix-daemon[858]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)5279worker1 # [ 103.525373] nix-daemon[858]: reaped child process 1327, status = succeeded5280lb # [ 103.517701] niks3-server[1147]: 2026/09/21 16:50:31 INFO Received uploads request method=POST path=/api/pending_closures5281worker1 # [ 103.547918] nix-grpc-daemon[1269]: time=2026-09-21T16:50:31.178Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5282worker1 # [ 103.549580] nix-grpc-daemon[1269]: time=2026-09-21T16:50:31.178Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)"5283lb # [ 103.546754] niks3-server[1147]: 2026/09/21 16:50:31 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst5284worker1 # [ 103.587076] nix-grpc-daemon[1269]: time=2026-09-21T16:50:31.217Z level=INFO msg="Uploading 1 narinfos"5285lb # [ 103.561761] niks3-server[1147]: 2026/09/21 16:50:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5286lb # [ 103.574331] niks3-server[1147]: 2026/09/21 16:50:31 INFO Signed narinfos id=31 count=15287lb # [ 103.580853] niks3-server[1147]: 2026/09/21 16:50:31 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls5288lb # [ 103.589804] niks3-server[1147]: 2026/09/21 16:50:31 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5289worker1 # [ 103.613085] nix-grpc-daemon[1269]: time=2026-09-21T16:50:31.243Z level=INFO msg="Upload complete. (476ms)"5290lb # [ 103.594594] niks3-server[1147]: 2026/09/21 16:50:31 INFO Completed upload id=315291lb # [ 103.597862] niks3-server[1147]: 2026/09/21 16:50:31 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo5292worker1 # [ 103.620134] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57008 duration_s=0 paths=1 nar_bytes_in=6565293worker1 # [ 103.633481] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:570085294worker1 # [ 103.639760] nix-daemon[858]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5295worker1 # [ 103.644854] nix-daemon[858]: reaped child process 1317, status = succeeded5296client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051...5297worker1 # [ 103.657945] nix-daemon[858]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5298client # farm-b> node-a: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5299client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5300client # farm-b> LOG-farm-b-after-niks3-restart5301worker1 # [ 105.306204] nix-daemon[858]: reaped child process 1331, status = succeeded5302worker1 # [ 105.483896] nix-daemon[858]: accepted connection from pid 1347, user nix-grpc-daemon (trusted)5303worker1 # [ 105.499655] nix-daemon[858]: reaped child process 1350, status = succeeded5304lb # [ 105.497214] niks3-server[1147]: 2026/09/21 16:50:33 INFO Received uploads request method=POST path=/api/pending_closures5305worker1 # [ 105.525645] nix-grpc-daemon[1269]: time=2026-09-21T16:50:33.155Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5306worker1 # [ 105.527396] nix-grpc-daemon[1269]: time=2026-09-21T16:50:33.155Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)"5307lb # [ 105.536756] niks3-server[1147]: 2026/09/21 16:50:33 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst5308worker1 # [ 105.569570] nix-grpc-daemon[1269]: time=2026-09-21T16:50:33.199Z level=INFO msg="Uploading 1 narinfos"5309lb # [ 105.554595] niks3-server[1147]: 2026/09/21 16:50:33 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5310lb # [ 105.563162] niks3-server[1147]: 2026/09/21 16:50:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5311lb # [ 105.567701] niks3-server[1147]: 2026/09/21 16:50:33 INFO Signed narinfos id=32 count=15312lb # [ 105.576179] niks3-server[1147]: 2026/09/21 16:50:33 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls5313worker1 # [ 105.594144] nix-grpc-daemon[1269]: time=2026-09-21T16:50:33.224Z level=INFO msg="Upload complete. (323ms)"5314lb # [ 105.579509] niks3-server[1147]: 2026/09/21 16:50:33 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5315lb # [ 105.584243] niks3-server[1147]: 2026/09/21 16:50:33 INFO Completed upload id=325316lb # [ 105.586536] niks3-server[1147]: 2026/09/21 16:50:33 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.narinfo5317worker1 # [ 105.609834] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57008 duration_s=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=15318worker1 # [ 105.630572] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:33Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=05319worker1 # [ 105.636548] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:33Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.3:50051 clients=15320worker1 # [ 105.641435] nix-daemon[858]: reaped child process 1329, status = succeeded5321worker1 # [ 105.667691] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:570085322worker1 # [ 105.683222] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:570085323worker1 # [ 105.697315] nix-grpc-daemon[1269]: time=2026-09-21T16:50:33.327Z level=INFO msg="All 2 paths already cached"5324client # copying 1 paths to 'lb:50051'...5325client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'...5326worker1 # [ 105.707770] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:570085327worker1 # [ 105.721854] nix-daemon[858]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5328worker1 # [ 105.913071] nix-daemon[858]: accepted connection from pid 1356, user nix-grpc-daemon (trusted)5329worker1 # [ 105.930779] nix-daemon[858]: reaped child process 1359, status = succeeded5330worker1 # [ 106.103405] nix-daemon[858]: accepted connection from pid 1361, user nix-grpc-daemon (trusted)5331worker1 # [ 106.121961] nix-daemon[858]: reaped child process 1364, status = succeeded5332lb # [ 106.113615] niks3-server[1147]: 2026/09/21 16:50:33 INFO Received uploads request method=POST path=/api/pending_closures5333worker1 # [ 106.142473] nix-grpc-daemon[1269]: time=2026-09-21T16:50:33.772Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5334worker1 # [ 106.143902] nix-grpc-daemon[1269]: time=2026-09-21T16:50:33.772Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)"5335lb # [ 106.139931] niks3-server[1147]: 2026/09/21 16:50:33 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst5336worker1 # [ 106.171817] nix-grpc-daemon[1269]: time=2026-09-21T16:50:33.802Z level=INFO msg="Uploading 1 narinfos"5337lb # [ 106.156989] niks3-server[1147]: 2026/09/21 16:50:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5338lb # [ 106.166788] niks3-server[1147]: 2026/09/21 16:50:33 INFO Signed narinfos id=33 count=15339lb # [ 106.176755] niks3-server[1147]: 2026/09/21 16:50:33 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls5340worker1 # [ 106.199562] nix-grpc-daemon[1269]: time=2026-09-21T16:50:33.830Z level=INFO msg="Upload complete. (462ms)"5341lb # [ 106.179998] niks3-server[1147]: 2026/09/21 16:50:33 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5342lb # [ 106.189093] niks3-server[1147]: 2026/09/21 16:50:33 INFO Completed upload id=335343worker1 # [ 106.206814] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57008 duration_s=0 paths=1 nar_bytes_in=7845344lb # [ 106.192205] niks3-server[1147]: 2026/09/21 16:50:33 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo5345worker1 # [ 106.216775] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:570085346worker1 # [ 106.219806] nix-daemon[858]: reaped child process 1354, status = succeeded5347worker1 # [ 106.226648] nix-daemon[858]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5348client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051...5349worker1 # [ 106.237327] nix-daemon[858]: accepted connection from pid 1197, user nix-grpc-daemon (trusted)5350client # farm-top> node-a: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5351client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5352client # farm-top> LOG-farm-top-after-niks3-restart5353worker1 # [ 107.268201] nix-daemon[858]: reaped child process 1368, status = succeeded5354worker1 # [ 107.428273] nix-daemon[858]: accepted connection from pid 1384, user nix-grpc-daemon (trusted)5355worker1 # [ 107.445963] nix-daemon[858]: reaped child process 1387, status = succeeded5356lb # [ 107.439437] niks3-server[1147]: 2026/09/21 16:50:35 INFO Received uploads request method=POST path=/api/pending_closures5357worker1 # [ 107.469658] nix-grpc-daemon[1269]: time=2026-09-21T16:50:35.098Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5358worker1 # [ 107.471286] nix-grpc-daemon[1269]: time=2026-09-21T16:50:35.098Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)"5359lb # [ 107.469995] niks3-server[1147]: 2026/09/21 16:50:35 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5360lb # [ 107.485497] niks3-server[1147]: 2026/09/21 16:50:35 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst5361lb # [ 107.497935] niks3-server[1147]: 2026/09/21 16:50:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5362worker1 # [ 107.521602] nix-grpc-daemon[1269]: time=2026-09-21T16:50:35.151Z level=INFO msg="Uploading 1 narinfos"5363lb # [ 107.505974] niks3-server[1147]: 2026/09/21 16:50:35 INFO Signed narinfos id=34 count=15364lb # [ 107.511709] niks3-server[1147]: 2026/09/21 16:50:35 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls5365lb # [ 107.520263] niks3-server[1147]: 2026/09/21 16:50:35 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5366worker1 # [ 107.544554] nix-grpc-daemon[1269]: time=2026-09-21T16:50:35.175Z level=INFO msg="Upload complete. (296ms)"5367lb # [ 107.526487] niks3-server[1147]: 2026/09/21 16:50:35 INFO Completed upload id=345368lb # [ 107.529602] niks3-server[1147]: 2026/09/21 16:50:35 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo5369worker1 # [ 107.552963] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57008 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=15370worker1 # [ 107.570413] nix-daemon[858]: reaped child process 1366, status = succeeded5371client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag after-niks3-restart >&2, in 7.98 seconds)5372client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag after-niks3-restart outPath5373client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag after-niks3-restart outPath, in 0.23 seconds)5374(finished: subtest: a niks3 restart does not cost the leader its role, in 8.85 seconds)5375subtest: scheduler down: the other node takes the lock and keeps it5376lb: must succeed: curl -sf localhost:9901/clusters5377lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5378lb: must succeed: curl -sf localhost:9901/clusters5379lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5380lb: must succeed: curl -sf localhost:9901/clusters5381lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5382lb: must succeed: curl -sf localhost:9901/clusters5383lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5384worker1 # [ 108.150848] systemd[1]: Stopping Nix worker-protocol over gRPC...5385worker1 # [ 108.875275] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:36Z level=info event=signal_exit5386lb # [ 109.361311] niks3-server[1147]: 2026/09/21 16:50:36 INFO lead: released remote=192.168.1.3:470785387lb # [ 110.152493] niks3-server[1147]: 2026/09/21 16:50:37 INFO lead: acquired remote=192.168.1.4:489825388worker2 # [ 110.265430] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:37Z level=info event=scheduler_take_over5389worker2 # [ 110.467300] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:38Z level=info event=scheduler_restarting addr=lb:500515390worker1 # [ 110.385849] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:38Z level=info event=scheduler_restarting5391worker1 # [ 110.390134] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:38Z level=info event=scheduler_restarting addr=lb:500515392worker1 # [ 110.395037] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:38Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:570065393worker1 # [ 110.400521] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:38Z level=info event=worker_gone addr=192.168.1.3:500515394worker1 # [ 110.405121] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:38Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:570065395worker1 # [ 110.410623] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:38Z level=info event=worker_gone addr=192.168.1.4:500515396worker1 # [ 110.432849] nix-daemon[858]: reaped child process 1257, status = succeeded5397worker2 # [ 110.668772] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:38Z level=info event=scheduler_connected addr=lb:500515398worker1 # [ 110.601005] nix-grpc-daemon[1197]: ts=2026-09-21T16:50:38Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5399worker2 # [ 110.699223] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:38Z level=info event=oidc_jwks provider=mock keys=15400worker2 # [ 110.704378] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:38Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:403605401worker2 # [ 110.710001] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:38Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05402lb # [ 110.850524] envoy[926]: conn peer=[2001:db8:1::1]:49210 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9740 tx=4146 ms=76865403lb # [ 110.855686] envoy[926]: conn peer=[2001:db8:1::3]:33742 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1108 tx=507 ms=116565404worker1 # [ 110.902644] nix-grpc-daemon[1197]: E0000 00:00:1790009438.533026 1242 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true5405worker1 # [ 110.911586] nix-grpc-daemon[1197]: E0000 00:00:1790009438.542012 1242 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5406worker1 # [ 111.519055] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5407worker1 # [ 111.525343] systemd[1]: Stopped Nix worker-protocol over gRPC.5408worker1 # [ 111.527861] systemd[1]: nix-grpc-daemon.service: Consumed 1.381s CPU time over 12.864s wall clock time, 36.1M memory peak, 36.2K incoming IP traffic, 56.8K outgoing IP traffic.5409worker1 # [ 111.541245] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5410worker1 # [ 111.543378] systemd[1]: Closed Nix worker-protocol over gRPC.5411worker1 # [ 111.549130] systemd[1]: nix-grpc-daemon.socket: Consumed 823us CPU time over 12.905s wall clock time, 256K memory peak, 34.9K incoming IP traffic, 25.8K outgoing IP traffic.5412worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5413worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.08 seconds)5414worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5415worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5416client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby >&25417client # don't know how to build these paths:5418client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5419worker2 # [ 112.039506] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:39Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:420985420worker2 # [ 112.047453] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:420985421worker2 # [ 112.070411] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:39Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=05422worker2 # [ 112.077907] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:39Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.4:50051 clients=15423worker2 # [ 112.115382] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420985424worker2 # [ 112.131226] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:420985425client # copying 1 paths to 'lb:50051'...5426client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'...5427worker2 # [ 112.147035] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:420985428worker2 # [ 112.161953] nix-daemon[849]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5429worker2 # [ 112.376385] nix-daemon[849]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)5430worker2 # [ 112.393258] nix-daemon[849]: reaped child process 1304, status = succeeded5431worker2 # [ 112.569504] nix-daemon[849]: accepted connection from pid 1306, user nix-grpc-daemon (trusted)5432worker2 # [ 112.585663] nix-daemon[849]: reaped child process 1309, status = succeeded5433lb # [ 112.492760] niks3-server[1147]: 2026/09/21 16:50:40 INFO Received uploads request method=POST path=/api/pending_closures5434worker2 # [ 112.609193] nix-grpc-daemon[1265]: time=2026-09-21T16:50:40.245Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5435worker2 # [ 112.610709] nix-grpc-daemon[1265]: time=2026-09-21T16:50:40.245Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)"5436lb # [ 112.523348] niks3-server[1147]: 2026/09/21 16:50:40 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst5437worker2 # [ 112.644051] nix-grpc-daemon[1265]: time=2026-09-21T16:50:40.280Z level=INFO msg="Uploading 1 narinfos"5438lb # [ 112.536713] niks3-server[1147]: 2026/09/21 16:50:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5439lb # [ 112.549820] niks3-server[1147]: 2026/09/21 16:50:40 INFO Signed narinfos id=35 count=15440lb # [ 112.556419] niks3-server[1147]: 2026/09/21 16:50:40 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls5441lb # [ 112.566619] niks3-server[1147]: 2026/09/21 16:50:40 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5442lb # [ 112.576300] niks3-server[1147]: 2026/09/21 16:50:40 INFO Completed upload id=355443worker2 # [ 112.683296] nix-grpc-daemon[1265]: time=2026-09-21T16:50:40.319Z level=INFO msg="Upload complete. (504ms)"5444lb # [ 112.580942] niks3-server[1147]: 2026/09/21 16:50:40 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo5445worker2 # [ 112.692282] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42098 duration_s=0 paths=1 nar_bytes_in=5205446worker2 # [ 112.703559] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420985447worker2 # [ 112.711226] nix-daemon[849]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5448worker2 # [ 112.716848] nix-daemon[849]: reaped child process 1299, status = succeeded5449client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051...5450client # farm-a> worker2: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5451worker2 # [ 112.730249] nix-daemon[849]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5452client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5453client # farm-a> LOG-farm-a-on-standby5454worker2 # [ 113.757558] nix-daemon[849]: reaped child process 1313, status = succeeded5455worker2 # [ 113.923358] nix-daemon[849]: accepted connection from pid 1329, user nix-grpc-daemon (trusted)5456worker2 # [ 113.938912] nix-daemon[849]: reaped child process 1332, status = succeeded5457lb # [ 113.847850] niks3-server[1147]: 2026/09/21 16:50:41 INFO Received uploads request method=POST path=/api/pending_closures5458worker2 # [ 113.966188] nix-grpc-daemon[1265]: time=2026-09-21T16:50:41.600Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5459worker2 # [ 113.967746] nix-grpc-daemon[1265]: time=2026-09-21T16:50:41.600Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)"5460lb # [ 113.878164] niks3-server[1147]: 2026/09/21 16:50:41 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5461lb # [ 113.893530] niks3-server[1147]: 2026/09/21 16:50:41 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5462lb # [ 113.905125] niks3-server[1147]: 2026/09/21 16:50:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5463worker2 # [ 114.018503] nix-grpc-daemon[1265]: time=2026-09-21T16:50:41.654Z level=INFO msg="Uploading 1 narinfos"5464lb # [ 113.913113] niks3-server[1147]: 2026/09/21 16:50:41 INFO Signed narinfos id=36 count=15465lb # [ 113.915943] niks3-server[1147]: 2026/09/21 16:50:41 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls5466lb # [ 113.927729] niks3-server[1147]: 2026/09/21 16:50:41 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5467lb # [ 113.939490] niks3-server[1147]: 2026/09/21 16:50:41 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo5468lb # [ 113.944368] niks3-server[1147]: 2026/09/21 16:50:41 INFO Completed upload id=365469worker2 # [ 114.051425] nix-grpc-daemon[1265]: time=2026-09-21T16:50:41.688Z level=INFO msg="Upload complete. (311ms)"5470worker2 # [ 114.076905] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42098 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=15471worker2 # [ 114.088353] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:41Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=05472worker2 # [ 114.091600] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:41Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.4:50051 clients=15473worker2 # [ 114.095558] nix-daemon[849]: reaped child process 1311, status = succeeded5474worker2 # [ 114.115445] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420985475worker2 # [ 114.129746] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:420985476worker2 # [ 114.144601] nix-grpc-daemon[1265]: time=2026-09-21T16:50:41.781Z level=INFO msg="All 1 paths already cached"5477client # copying 1 paths to 'lb:50051'...5478client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'...5479worker2 # [ 114.154772] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:420985480worker2 # [ 114.168310] nix-daemon[849]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5481worker2 # [ 114.371143] nix-daemon[849]: accepted connection from pid 1338, user nix-grpc-daemon (trusted)5482worker2 # [ 114.387641] nix-daemon[849]: reaped child process 1341, status = succeeded5483worker2 # [ 114.562430] nix-daemon[849]: accepted connection from pid 1343, user nix-grpc-daemon (trusted)5484worker2 # [ 114.578988] nix-daemon[849]: reaped child process 1346, status = succeeded5485lb # [ 114.483432] niks3-server[1147]: 2026/09/21 16:50:42 INFO Received uploads request method=POST path=/api/pending_closures5486worker2 # [ 114.600300] nix-grpc-daemon[1265]: time=2026-09-21T16:50:42.236Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5487worker2 # [ 114.601816] nix-grpc-daemon[1265]: time=2026-09-21T16:50:42.236Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)"5488lb # [ 114.516393] niks3-server[1147]: 2026/09/21 16:50:42 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst5489worker2 # [ 114.636380] nix-grpc-daemon[1265]: time=2026-09-21T16:50:42.272Z level=INFO msg="Uploading 1 narinfos"5490lb # [ 114.532310] niks3-server[1147]: 2026/09/21 16:50:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5491lb # [ 114.542707] niks3-server[1147]: 2026/09/21 16:50:42 INFO Signed narinfos id=37 count=15492lb # [ 114.546983] niks3-server[1147]: 2026/09/21 16:50:42 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls5493lb # [ 114.556799] niks3-server[1147]: 2026/09/21 16:50:42 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5494lb # [ 114.562203] niks3-server[1147]: 2026/09/21 16:50:42 INFO Completed upload id=375495worker2 # [ 114.669220] nix-grpc-daemon[1265]: time=2026-09-21T16:50:42.305Z level=INFO msg="Upload complete. (484ms)"5496lb # [ 114.565059] niks3-server[1147]: 2026/09/21 16:50:42 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo5497worker2 # [ 114.676330] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42098 duration_s=0 paths=1 nar_bytes_in=6405498worker2 # [ 114.689953] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420985499worker2 # [ 114.699820] nix-daemon[849]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5500worker2 # [ 114.702877] nix-daemon[849]: reaped child process 1336, status = succeeded5501client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051...5502worker2 # [ 114.714071] nix-daemon[849]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5503client # farm-b> worker2: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5504client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5505client # farm-b> LOG-farm-b-on-standby5506worker2 # [ 115.957743] nix-daemon[849]: reaped child process 1350, status = succeeded5507worker2 # [ 116.132118] nix-daemon[849]: accepted connection from pid 1366, user nix-grpc-daemon (trusted)5508worker2 # [ 116.149604] nix-daemon[849]: reaped child process 1369, status = succeeded5509lb # [ 116.055796] niks3-server[1147]: 2026/09/21 16:50:43 INFO Received uploads request method=POST path=/api/pending_closures5510worker2 # [ 116.172375] nix-grpc-daemon[1265]: time=2026-09-21T16:50:43.808Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5511worker2 # [ 116.173875] nix-grpc-daemon[1265]: time=2026-09-21T16:50:43.808Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)"5512lb # [ 116.093280] niks3-server[1147]: 2026/09/21 16:50:43 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst5513lb # [ 116.105542] niks3-server[1147]: 2026/09/21 16:50:43 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5514lb # [ 116.114872] niks3-server[1147]: 2026/09/21 16:50:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign5515worker2 # [ 116.227593] nix-grpc-daemon[1265]: time=2026-09-21T16:50:43.863Z level=INFO msg="Uploading 1 narinfos"5516lb # [ 116.124816] niks3-server[1147]: 2026/09/21 16:50:43 INFO Signed narinfos id=38 count=15517lb # [ 116.127490] niks3-server[1147]: 2026/09/21 16:50:43 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls5518lb # [ 116.138132] niks3-server[1147]: 2026/09/21 16:50:43 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete5519lb # [ 116.146463] niks3-server[1147]: 2026/09/21 16:50:43 INFO Completed upload id=385520worker2 # [ 116.253558] nix-grpc-daemon[1265]: time=2026-09-21T16:50:43.890Z level=INFO msg="Upload complete. (327ms)"5521lb # [ 116.149807] niks3-server[1147]: 2026/09/21 16:50:43 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo5522worker2 # [ 116.272414] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42098 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=15523worker2 # [ 116.288309] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:43Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=05524worker2 # [ 116.291997] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:43Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.4:50051 clients=15525worker2 # [ 116.297738] nix-daemon[849]: reaped child process 1348, status = succeeded5526worker2 # [ 116.315378] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420985527worker2 # [ 116.329823] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:420985528worker2 # [ 116.343729] nix-grpc-daemon[1265]: time=2026-09-21T16:50:43.980Z level=INFO msg="All 2 paths already cached"5529client # copying 1 paths to 'lb:50051'...5530client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'...5531worker2 # [ 116.354023] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:420985532worker2 # [ 116.368352] nix-daemon[849]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5533worker2 # [ 116.565637] nix-daemon[849]: accepted connection from pid 1375, user nix-grpc-daemon (trusted)5534worker2 # [ 116.583228] nix-daemon[849]: reaped child process 1378, status = succeeded5535worker2 # [ 116.756091] nix-daemon[849]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)5536worker2 # [ 116.772702] nix-daemon[849]: reaped child process 1383, status = succeeded5537lb # [ 116.679972] niks3-server[1147]: 2026/09/21 16:50:44 INFO Received uploads request method=POST path=/api/pending_closures5538worker2 # [ 116.795920] nix-grpc-daemon[1265]: time=2026-09-21T16:50:44.432Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5539worker2 # [ 116.797600] nix-grpc-daemon[1265]: time=2026-09-21T16:50:44.432Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)"5540lb # [ 116.706168] niks3-server[1147]: 2026/09/21 16:50:44 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst5541worker2 # [ 116.825229] nix-grpc-daemon[1265]: time=2026-09-21T16:50:44.461Z level=INFO msg="Uploading 1 narinfos"5542lb # [ 116.722799] niks3-server[1147]: 2026/09/21 16:50:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign5543lb # [ 116.732776] niks3-server[1147]: 2026/09/21 16:50:44 INFO Signed narinfos id=39 count=15544worker2 # [ 116.848650] nix-grpc-daemon[1265]: time=2026-09-21T16:50:44.485Z level=INFO msg="Upload complete. (464ms)"5545lb # [ 116.742116] niks3-server[1147]: 2026/09/21 16:50:44 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls5546worker2 # [ 116.854040] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42098 duration_s=0 paths=1 nar_bytes_in=7685547lb # [ 116.747142] niks3-server[1147]: 2026/09/21 16:50:44 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete5548lb # [ 116.751766] niks3-server[1147]: 2026/09/21 16:50:44 INFO Completed upload id=395549lb # [ 116.754023] niks3-server[1147]: 2026/09/21 16:50:44 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo5550worker2 # [ 116.865271] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420985551worker2 # [ 116.870502] nix-daemon[849]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5552worker2 # [ 116.875159] nix-daemon[849]: reaped child process 1373, status = succeeded5553client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051...5554worker2 # [ 116.886434] nix-daemon[849]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5555client # farm-top> worker2: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5556client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5557client # farm-top> LOG-farm-top-on-standby5558worker2 # [ 117.716539] nix-daemon[849]: reaped child process 1387, status = succeeded5559worker2 # [ 117.891097] nix-daemon[849]: accepted connection from pid 1403, user nix-grpc-daemon (trusted)5560worker2 # [ 117.909524] nix-daemon[849]: reaped child process 1406, status = succeeded5561lb # [ 117.816856] niks3-server[1147]: 2026/09/21 16:50:45 INFO Received uploads request method=POST path=/api/pending_closures5562worker2 # [ 117.934878] nix-grpc-daemon[1265]: time=2026-09-21T16:50:45.569Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5563worker2 # [ 117.936511] nix-grpc-daemon[1265]: time=2026-09-21T16:50:45.569Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)"5564lb # [ 117.846385] niks3-server[1147]: 2026/09/21 16:50:45 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5565lb # [ 117.859111] niks3-server[1147]: 2026/09/21 16:50:45 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst5566lb # [ 117.875037] niks3-server[1147]: 2026/09/21 16:50:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign5567worker2 # [ 117.988843] nix-grpc-daemon[1265]: time=2026-09-21T16:50:45.625Z level=INFO msg="Uploading 1 narinfos"5568lb # [ 117.886714] niks3-server[1147]: 2026/09/21 16:50:45 INFO Signed narinfos id=40 count=15569lb # [ 117.893448] niks3-server[1147]: 2026/09/21 16:50:45 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls5570lb # [ 117.901186] niks3-server[1147]: 2026/09/21 16:50:45 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete5571lb # [ 117.907347] niks3-server[1147]: 2026/09/21 16:50:45 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo5572lb # [ 117.912383] niks3-server[1147]: 2026/09/21 16:50:45 INFO Completed upload id=405573worker2 # [ 118.019455] nix-grpc-daemon[1265]: time=2026-09-21T16:50:45.656Z level=INFO msg="Upload complete. (324ms)"5574worker2 # [ 118.033165] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42098 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=15575worker2 # [ 118.049378] nix-daemon[849]: reaped child process 1385, status = succeeded5576client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby >&2, in 6.30 seconds)5577client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby outPath5578client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby outPath, in 0.23 seconds)5579worker1 # [ 118.305010] systemd[1]: Listening on Nix worker-protocol over gRPC.5580worker1 # [ 118.313768] systemd[1]: Starting Nix worker-protocol over gRPC...5581worker1 # [ 118.505896] nix-grpc-daemon[1405]: ts=2026-09-21T16:50:46Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5582worker1 # [ 118.508401] nix-grpc-daemon[1405]: ts=2026-09-21T16:50:46Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5583worker1 # [ 118.512530] nix-grpc-daemon[1405]: ts=2026-09-21T16:50:46Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15584worker1 # [ 118.520076] nix-grpc-daemon[1405]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5585worker1 # [ 118.521527] nix-grpc-daemon[1405]: E0000 00:00:1790009446.149468 1405 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5586worker1 # [ 118.537846] nix-grpc-daemon[1405]: E0000 00:00:1790009446.155633 1405 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5587worker1 # [ 118.543812] nix-grpc-daemon[1405]: ts=2026-09-21T16:50:46Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5588lb: must succeed: curl -sf localhost:9901/clusters5589worker1 # [ 118.547811] systemd[1]: Started Nix worker-protocol over gRPC.5590worker1 # [ 118.557049] nix-grpc-daemon[1405]: E0000 00:00:1790009446.178775 1446 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5591worker1 # [ 118.581414] nix-grpc-daemon[1405]: E0000 00:00:1790009446.197064 1446 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5592worker1 # [ 118.584309] nix-grpc-daemon[1405]: E0000 00:00:1790009446.197215 1446 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5593worker1 # [ 118.587775] nix-grpc-daemon[1405]: E0000 00:00:1790009446.211371 1446 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5594worker1 # [ 118.595967] nix-grpc-daemon[1405]: ts=2026-09-21T16:50:46Z level=info event=scheduler_connected addr=lb:500515595worker2 # [ 118.685901] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:46Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:403605596worker2 # [ 118.686297] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:46Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05597lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5598worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5599worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5600lb: must succeed: curl -sf localhost:9901/clusters5601lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5602lb: must succeed: curl -sf localhost:9901/clusters5603lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5604worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5605worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5606(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 11.06 seconds)5607subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5608client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs5609client # [ 118.986046] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs.5610client # Running as unit: rs.service; invocation ID: 0d4d528c8c954fb6a8f01abe484b76505611client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs, in 0.10 seconds)5612worker1 # [ 119.237993] nix-grpc-daemon[1405]: ts=2026-09-21T16:50:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:447045613worker1 # [ 119.252333] nix-daemon[858]: accepted connection from pid 1405, user nix-grpc-daemon (trusted)5614client # [ 119.267106] nix[1260]: don't know how to build these paths:5615client # [ 119.269256] nix[1260]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5616worker2 # [ 119.369623] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:47Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=05617worker2 # [ 119.376429] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:47Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.4:50051 clients=15618worker2 # [ 119.423150] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420985619worker2 # [ 119.438360] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:420985620client # [ 119.361461] nix[1260]: copying 1 paths to 'lb:50051'...5621client # [ 119.366590] nix[1260]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'...5622worker2 # [ 119.456487] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:420985623worker2 # [ 119.471295] nix-daemon[849]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5624worker2 # [ 119.686849] nix-daemon[849]: accepted connection from pid 1419, user nix-grpc-daemon (trusted)5625worker2 # [ 119.704689] nix-daemon[849]: reaped child process 1422, status = succeeded5626worker2 # [ 119.882373] nix-daemon[849]: accepted connection from pid 1424, user nix-grpc-daemon (trusted)5627worker2 # [ 119.899496] nix-daemon[849]: reaped child process 1427, status = succeeded5628lb # [ 119.805167] niks3-server[1147]: 2026/09/21 16:50:47 INFO Received uploads request method=POST path=/api/pending_closures5629worker2 # [ 119.921889] nix-grpc-daemon[1265]: time=2026-09-21T16:50:47.557Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5630worker2 # [ 119.923382] nix-grpc-daemon[1265]: time=2026-09-21T16:50:47.557Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)"5631lb # [ 119.834963] niks3-server[1147]: 2026/09/21 16:50:47 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst5632worker2 # [ 119.961997] nix-grpc-daemon[1265]: time=2026-09-21T16:50:47.598Z level=INFO msg="Uploading 1 narinfos"5633lb # [ 119.849086] niks3-server[1147]: 2026/09/21 16:50:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign5634lb # [ 119.859264] niks3-server[1147]: 2026/09/21 16:50:47 INFO Signed narinfos id=41 count=15635lb # [ 119.865168] niks3-server[1147]: 2026/09/21 16:50:47 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls5636lb # [ 119.875985] niks3-server[1147]: 2026/09/21 16:50:47 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete5637lb # [ 119.879859] niks3-server[1147]: 2026/09/21 16:50:47 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo5638worker2 # [ 119.986677] nix-grpc-daemon[1265]: time=2026-09-21T16:50:47.623Z level=INFO msg="Upload complete. (498ms)"5639lb # [ 119.883553] niks3-server[1147]: 2026/09/21 16:50:47 INFO Completed upload id=415640worker2 # [ 119.996978] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42098 duration_s=0 paths=1 nar_bytes_in=4165641worker2 # [ 120.004818] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:420985642worker2 # [ 120.016087] nix-daemon[849]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5643worker2 # [ 120.018131] nix-daemon[849]: reaped child process 1417, status = succeeded5644client # [ 119.937115] nix[1260]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051...5645worker2 # [ 120.030828] nix-daemon[849]: accepted connection from pid 1205, user nix-grpc-daemon (trusted)5646lb # [ 120.850052] envoy[926]: conn peer=[2001:db8:1::1]:45400 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9606 tx=4103 ms=60055647lb: must succeed: curl -sf localhost:9901/clusters5648lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5649lb: must succeed: curl -sf localhost:9901/clusters5650lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5651lb: must succeed: curl -sf localhost:9901/clusters5652lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5653lb: must succeed: curl -sf localhost:9901/clusters5654lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5655worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5656worker1 # [ 121.936123] systemd-journald[275]: Received client request to rotate journal, rotating.5657worker1 # [ 121.957699] systemd-journald[275]: Vacuuming done, freed 0B of archived journals from /var/log/journal/dbc28bec2a6648b8870df220537644ce.5658worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.09 seconds)5659worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service5660worker2 # [ 122.098214] systemd[1]: Stopping Nix worker-protocol over gRPC...5661worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.05 seconds)5662client: must succeed: sleep 5.05663worker2 # [ 122.242209] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:49Z level=info event=draining5664client: (finished: must succeed: sleep 5.0, in 5.06 seconds)5665worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5666worker2 # [ 127.682445] nix-daemon[849]: reaped child process 1431, status = succeeded5667worker2 # [ 127.872037] nix-daemon[849]: accepted connection from pid 1472, user nix-grpc-daemon (trusted)5668worker2 # [ 127.891585] nix-daemon[849]: reaped child process 1475, status = succeeded5669lb # [ 127.798680] niks3-server[1147]: 2026/09/21 16:50:55 INFO Received uploads request method=POST path=/api/pending_closures5670worker2 # [ 127.916397] nix-grpc-daemon[1265]: time=2026-09-21T16:50:55.551Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5671worker2 # [ 127.917935] nix-grpc-daemon[1265]: time=2026-09-21T16:50:55.552Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)"5672lb # [ 127.843144] niks3-server[1147]: 2026/09/21 16:50:55 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5673lb # [ 127.854043] niks3-server[1147]: 2026/09/21 16:50:55 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5674lb # [ 127.865778] niks3-server[1147]: 2026/09/21 16:50:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign5675lb # [ 127.871876] niks3-server[1147]: 2026/09/21 16:50:55 INFO Signed narinfos id=42 count=15676worker2 # [ 127.978860] nix-grpc-daemon[1265]: time=2026-09-21T16:50:55.614Z level=INFO msg="Uploading 1 narinfos"5677lb # [ 127.876905] niks3-server[1147]: 2026/09/21 16:50:55 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls5678lb # [ 127.890845] niks3-server[1147]: 2026/09/21 16:50:55 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete5679lb # [ 127.898308] niks3-server[1147]: 2026/09/21 16:50:55 INFO Completed upload id=425680worker2 # [ 128.005401] nix-grpc-daemon[1265]: time=2026-09-21T16:50:55.641Z level=INFO msg="Upload complete. (360ms)"5681lb # [ 127.901979] niks3-server[1147]: 2026/09/21 16:50:55 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo5682worker2 # [ 128.021146] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42098 duration_s=8 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=15683worker2 # [ 128.039285] nix-daemon[849]: reaped child process 1429, status = succeeded5684client # [ 127.970216] systemd[1]: rs.service: Deactivated successfully.5685client # [ 127.975613] systemd[1]: rs.service: Consumed 173ms CPU time over 8.987s wall clock time, 39.2M memory peak, 7.1K incoming IP traffic, 8.4K outgoing IP traffic.5686worker2 # [ 128.246819] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:55Z level=info event=signal_exit5687lb # [ 128.384129] niks3-server[1147]: 2026/09/21 16:50:56 INFO lead: released remote=192.168.1.4:489825688lb # [ 128.519842] niks3-server[1147]: 2026/09/21 16:50:56 INFO lead: acquired remote=192.168.1.3:421465689worker1 # [ 128.544181] nix-grpc-daemon[1405]: ts=2026-09-21T16:50:56Z level=info event=scheduler_take_over5690worker1 # [ 129.399030] nix-grpc-daemon[1405]: ts=2026-09-21T16:50:57Z level=info event=scheduler_restarting addr=lb:500515691worker2 # [ 129.492899] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:57Z level=info event=scheduler_restarting5692worker2 # [ 129.496785] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:57Z level=info event=scheduler_restarting addr=lb:500515693worker2 # [ 129.501020] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:57Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:403605694worker2 # [ 129.506103] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:57Z level=info event=worker_gone addr=192.168.1.4:500515695worker2 # [ 129.510664] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:57Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:403605696worker2 # [ 129.515853] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:57Z level=info event=worker_gone addr=192.168.1.3:500515697worker2 # [ 129.540140] nix-daemon[849]: reaped child process 1257, status = succeeded5698worker1 # [ 129.600946] nix-grpc-daemon[1405]: ts=2026-09-21T16:50:57Z level=info event=scheduler_connected addr=lb:500515699worker1 # [ 129.611353] nix-grpc-daemon[1405]: ts=2026-09-21T16:50:57Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:602365700worker2 # [ 129.709687] nix-grpc-daemon[1205]: ts=2026-09-21T16:50:57Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5701worker1 # [ 129.619544] nix-grpc-daemon[1405]: ts=2026-09-21T16:50:57Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05702worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.25 seconds)5703client: waiting for success: ! systemctl is-active rs5704client: (finished: waiting for success: ! systemctl is-active rs, in 0.07 seconds)5705client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5706client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds)5707client: must fail: journalctl -u rs | grep -q 'reconnecting'5708client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5709lb: must succeed: curl -sf localhost:9901/clusters5710lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5711worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5712worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5713worker2 # [ 130.326163] nix-grpc-daemon[1205]: E0000 00:00:1790009457.962474 1250 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5714worker2 # [ 130.326604] nix-grpc-daemon[1205]: E0000 00:00:1790009457.962711 1250 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5715worker2 # [ 130.848397] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5716worker2 # [ 130.851577] systemd[1]: Stopped Nix worker-protocol over gRPC.5717worker2 # [ 130.861566] systemd[1]: nix-grpc-daemon.service: Consumed 2.016s CPU time over 36.847s wall clock time, 37M memory peak, 58.1K incoming IP traffic, 119.9K outgoing IP traffic.5718worker2 # [ 130.875414] systemd[1]: Starting scheduler-token.service...5719lb # [ 130.852990] envoy[926]: conn peer=[2001:db8:1::1]:45414 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=1431 ms=87175720lb # [ 130.853239] envoy[926]: conn peer=[2001:db8:1::4]:53666 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=15713 tx=1206 ms=353975721worker2 # [ 130.987621] systemd[1]: scheduler-token.service: Deactivated successfully.5722worker2 # [ 130.996501] systemd[1]: Finished scheduler-token.service.5723worker2 # [ 130.998766] systemd[1]: scheduler-token.service: Consumed 43ms CPU time over 116ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5724worker2 # [ 131.005933] systemd[1]: Starting Nix worker-protocol over gRPC...5725lb: must succeed: curl -sf localhost:9901/clusters5726lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5727worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5728worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5729worker2 # [ 131.194496] nix-grpc-daemon[1485]: ts=2026-09-21T16:50:58Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5730worker2 # [ 131.197093] nix-grpc-daemon[1485]: ts=2026-09-21T16:50:58Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5731worker2 # [ 131.201248] nix-grpc-daemon[1485]: ts=2026-09-21T16:50:58Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15732worker2 # [ 131.208059] nix-grpc-daemon[1485]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5733worker2 # [ 131.209566] nix-grpc-daemon[1485]: E0000 00:00:1790009458.844392 1485 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5734worker2 # [ 131.216881] nix-grpc-daemon[1485]: E0000 00:00:1790009458.850702 1485 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5735worker2 # [ 131.220872] nix-grpc-daemon[1485]: ts=2026-09-21T16:50:58Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5736worker2 # [ 131.223858] systemd[1]: Started Nix worker-protocol over gRPC.5737worker2 # [ 131.230580] nix-grpc-daemon[1485]: E0000 00:00:1790009458.864827 1524 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5738worker2 # [ 131.260136] nix-grpc-daemon[1485]: E0000 00:00:1790009458.882751 1524 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5739worker2 # [ 131.264359] nix-grpc-daemon[1485]: E0000 00:00:1790009458.882970 1524 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5740worker2 # [ 131.267378] nix-grpc-daemon[1485]: E0000 00:00:1790009458.900120 1524 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5741worker2 # [ 131.277255] nix-grpc-daemon[1485]: ts=2026-09-21T16:50:58Z level=info event=scheduler_connected addr=lb:500515742worker1 # [ 131.199436] nix-grpc-daemon[1405]: ts=2026-09-21T16:50:58Z level=info event=oidc_jwks provider=mock keys=15743worker1 # [ 131.203906] nix-grpc-daemon[1405]: ts=2026-09-21T16:50:58Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:602365744worker1 # [ 131.209205] nix-grpc-daemon[1405]: ts=2026-09-21T16:50:58Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05745lb: must succeed: curl -sf localhost:9901/clusters5746lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5747worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5748worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5749(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 13.34 seconds)5750subtest: scheduler restart mid-build: build finishes, no double build5751client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid5752client # [ 132.324801] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid.5753client # Running as unit: mid.service; invocation ID: 2745c9cfb44e49d299bdf898544fa24b5754client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid, in 0.09 seconds)5755worker2 # [ 132.665904] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:00Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:593425756worker2 # [ 132.680764] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)5757client # [ 132.607331] nix[1296]: don't know how to build these paths:5758client # [ 132.610307] nix[1296]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5759worker1 # [ 132.624568] nix-grpc-daemon[1405]: ts=2026-09-21T16:51:00Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05760worker1 # [ 132.629946] nix-grpc-daemon[1405]: ts=2026-09-21T16:51:00Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.3:50051 clients=15761worker1 # [ 132.667717] nix-grpc-daemon[1405]: ts=2026-09-21T16:51:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447045762worker1 # [ 132.684805] nix-grpc-daemon[1405]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 46 ms (attempt 1/5)5763worker1 # [ 132.739000] nix-grpc-daemon[1405]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 171 ms (attempt 2/5)5764worker1 # [ 132.917807] nix-grpc-daemon[1405]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 307 ms (attempt 3/5)5765worker1 # [ 133.225312] nix-grpc-daemon[1405]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 685 ms (attempt 4/5)5766worker1 # [ 133.919299] nix-grpc-daemon[1405]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5767worker1 # [ 133.924754] nix-grpc-daemon[1405]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5768worker1 # [ 133.945136] nix-grpc-daemon[1405]: ts=2026-09-21T16:51:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447045769client # [ 133.957389] nix[1296]: copying 1 paths to 'lb:50051'...5770worker1 # [ 133.962773] nix-grpc-daemon[1405]: ts=2026-09-21T16:51:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:447045771client # [ 133.962445] nix[1296]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'...5772worker1 # [ 133.980655] nix-daemon[858]: accepted connection from pid 1405, user nix-grpc-daemon (trusted)5773worker1 # [ 134.010215] nix-grpc-daemon[1405]: ts=2026-09-21T16:51:01Z level=info event=niks3_push_started pid=15345774worker1 # [ 134.224178] nix-grpc-daemon[1534]: time=2026-09-21T16:51:01.853Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""5775worker1 # [ 134.392956] nix-daemon[858]: accepted connection from pid 1547, user nix-grpc-daemon (trusted)5776worker1 # [ 134.409202] nix-daemon[858]: reaped child process 1550, status = succeeded5777worker1 # [ 134.577967] nix-daemon[858]: accepted connection from pid 1552, user nix-grpc-daemon (trusted)5778worker1 # [ 134.594125] nix-daemon[858]: reaped child process 1555, status = succeeded5779lb # [ 134.595272] niks3-server[1147]: 2026/09/21 16:51:02 INFO Received uploads request method=POST path=/api/pending_closures5780worker1 # [ 134.624304] nix-grpc-daemon[1534]: time=2026-09-21T16:51:02.254Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5781worker1 # [ 134.627281] nix-grpc-daemon[1534]: time=2026-09-21T16:51:02.256Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)"5782lb # [ 134.631264] niks3-server[1147]: 2026/09/21 16:51:02 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst5783worker1 # [ 134.659896] nix-grpc-daemon[1534]: time=2026-09-21T16:51:02.289Z level=INFO msg="Uploading 1 narinfos"5784lb # [ 134.640355] niks3-server[1147]: 2026/09/21 16:51:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign5785lb # [ 134.647743] niks3-server[1147]: 2026/09/21 16:51:02 INFO Signed narinfos id=43 count=15786lb # [ 134.653527] niks3-server[1147]: 2026/09/21 16:51:02 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete5787worker1 # [ 134.677252] nix-grpc-daemon[1534]: time=2026-09-21T16:51:02.307Z level=INFO msg="Upload complete. (449ms)"5788lb # [ 134.660844] niks3-server[1147]: 2026/09/21 16:51:02 INFO Completed upload id=435789worker1 # [ 134.684404] nix-grpc-daemon[1405]: ts=2026-09-21T16:51:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44704 duration_s=0 paths=1 nar_bytes_in=4245790lb # [ 134.682593] niks3-server[1147]: 2026/09/21 16:51:02 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls5791worker1 # [ 134.701862] nix-grpc-daemon[1405]: ts=2026-09-21T16:51:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447045792worker1 # [ 134.707467] nix-daemon[858]: accepted connection from pid 1405, user nix-grpc-daemon (trusted)5793worker1 # [ 134.714653] nix-daemon[858]: reaped child process 1532, status = succeeded5794client # [ 134.721155] nix[1296]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5795worker1 # [ 134.726790] nix-daemon[858]: accepted connection from pid 1405, user nix-grpc-daemon (trusted)5796lb: must succeed: curl -sf localhost:9901/clusters5797lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5798lb: must succeed: curl -sf localhost:9901/clusters5799lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5800worker1: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5801worker1 # [ 137.502238] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1405 (nix-grpc-daemon) on client request.5802worker1 # [ 137.504799] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5803lb # [ 137.493061] niks3-server[1147]: 2026/09/21 16:51:05 INFO lead: released remote=192.168.1.3:421465804client # [ 137.518749] nix[1296]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again5805worker2 # [ 137.608703] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:05Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"5806worker1: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.08 seconds)5807client: must succeed: sleep 5.05808client # [ 137.525260] nix[1296]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5809worker1 # [ 137.539227] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5810worker1 # [ 137.541799] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5811worker1 # [ 137.543035] systemd[1]: nix-grpc-daemon.service: Consumed 561ms CPU time over 19.226s wall clock time, 34.3M memory peak, 15.8K incoming IP traffic, 26.4K outgoing IP traffic.5812worker1 # [ 137.556716] nix-daemon[858]: reaped child process 1560, status = succeeded5813worker1 # [ 137.557802] nix-daemon[858]: reaped child process 1464, status = succeeded5814worker1 # [ 137.568196] nix-daemon[1563]: unexpected Nix daemon error: error: interrupted by the user5815worker1 # [ 137.572141] nix-daemon[858]: reaped child process 1563, status = failed with exit code 15816worker2 # [ 137.809003] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:05Z level=info event=scheduler_connected addr=lb:500515817worker1 # [ 137.807521] systemd[1]: nix-grpc-daemon.service: Scheduled restart job, restart counter is at 1.5818worker1 # [ 137.820638] systemd[1]: Starting Nix worker-protocol over gRPC...5819worker1 # [ 138.032128] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:05Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5820worker1 # [ 138.034683] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:05Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5821worker1 # [ 138.038644] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:05Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15822worker1 # [ 138.046964] nix-grpc-daemon[1600]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5823worker1 # [ 138.057524] nix-grpc-daemon[1600]: E0000 00:00:1790009465.675487 1600 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5824worker1 # [ 138.060808] nix-grpc-daemon[1600]: E0000 00:00:1790009465.677419 1600 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5825worker1 # [ 138.065884] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:05Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5826worker1 # [ 138.068394] systemd[1]: Started Nix worker-protocol over gRPC.5827worker1 # [ 138.069707] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:05Z level=info event=scheduler_take_over5828lb # [ 138.050477] niks3-server[1147]: 2026/09/21 16:51:05 INFO lead: acquired remote=192.168.1.3:562385829worker1 # [ 138.074177] nix-grpc-daemon[1600]: E0000 00:00:1790009465.704174 1640 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5830client # [ 138.072741] nix[1296]: scheduler at lb:50051: scheduler passive, another node serves, reconnecting5831worker2 # [ 138.162896] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:05Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5832worker1 # [ 138.104194] nix-grpc-daemon[1600]: E0000 00:00:1790009465.723461 1640 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5833worker1 # [ 138.106428] nix-grpc-daemon[1600]: E0000 00:00:1790009465.723586 1640 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5834worker1 # [ 138.113267] nix-grpc-daemon[1600]: E0000 00:00:1790009465.743022 1640 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5835worker1 # [ 138.125533] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:05Z level=info event=scheduler_connected addr=lb:500515836worker1 # [ 138.127397] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:05Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:357765837worker1 # [ 138.129034] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:05Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05838worker2 # [ 138.364249] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:06Z level=info event=scheduler_connected addr=lb:500515839worker1 # [ 138.285505] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:05Z level=info event=oidc_jwks provider=mock keys=15840worker1 # [ 138.289853] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:05Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:357765841worker1 # [ 138.295114] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:05Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05842worker1 # [ 139.090142] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:06Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05843worker1 # [ 139.090429] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:06Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.3:50051 clients=15844worker1 # [ 139.139779] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357905845worker1 # [ 139.158411] nix-daemon[858]: accepted connection from pid 1600, user nix-grpc-daemon (trusted)5846worker1 # [ 139.172763] nix-daemon[858]: accepted connection from pid 1600, user nix-grpc-daemon (trusted)5847client # [ 139.188556] nix[1296]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5848worker1 # [ 139.192512] nix-daemon[858]: accepted connection from pid 1600, user nix-grpc-daemon (trusted)5849lb # [ 140.856866] envoy[926]: conn peer=[2001:db8:1::3]:48316 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=666 tx=311 ms=189335850client: (finished: must succeed: sleep 5.0, in 5.06 seconds)5851client: waiting for success: ! systemctl is-active mid5852worker1 # [ 143.022089] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:10Z level=info event=niks3_push_started pid=16835853worker1 # [ 143.071478] nix-daemon[858]: reaped child process 1658, status = succeeded5854worker1 # [ 143.261270] nix-grpc-daemon[1683]: time=2026-09-21T16:51:10.891Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""5855worker1 # [ 143.434849] nix-daemon[858]: accepted connection from pid 1696, user nix-grpc-daemon (trusted)5856worker1 # [ 143.451087] nix-daemon[858]: reaped child process 1699, status = succeeded5857lb # [ 143.445271] niks3-server[1147]: 2026/09/21 16:51:11 INFO Received uploads request method=POST path=/api/pending_closures5858worker1 # [ 143.476094] nix-grpc-daemon[1683]: time=2026-09-21T16:51:11.105Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5859worker1 # [ 143.477561] nix-grpc-daemon[1683]: time=2026-09-21T16:51:11.105Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)"5860lb # [ 143.485788] niks3-server[1147]: 2026/09/21 16:51:11 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5861lb # [ 143.505662] niks3-server[1147]: 2026/09/21 16:51:11 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst5862lb # [ 143.514117] niks3-server[1147]: 2026/09/21 16:51:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign5863worker1 # [ 143.538794] nix-grpc-daemon[1683]: time=2026-09-21T16:51:11.167Z level=INFO msg="Uploading 1 narinfos"5864lb # [ 143.523589] niks3-server[1147]: 2026/09/21 16:51:11 INFO Signed narinfos id=44 count=15865lb # [ 143.530347] niks3-server[1147]: 2026/09/21 16:51:11 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls5866lb # [ 143.538966] niks3-server[1147]: 2026/09/21 16:51:11 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete5867lb # [ 143.548262] niks3-server[1147]: 2026/09/21 16:51:11 INFO Completed upload id=445868worker1 # [ 143.566715] nix-grpc-daemon[1683]: time=2026-09-21T16:51:11.197Z level=INFO msg="Upload complete. (301ms)"5869worker1 # [ 143.580291] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35790 duration_s=4 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=15870lb # [ 143.569379] niks3-server[1147]: 2026/09/21 16:51:11 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo5871worker1 # [ 143.594921] nix-daemon[858]: reaped child process 1656, status = succeeded5872client # [ 143.617813] systemd[1]: mid.service: Deactivated successfully.5873client # [ 143.622030] systemd[1]: mid.service: Consumed 189ms CPU time over 11.297s wall clock time, 41.2M memory peak, 8.9K incoming IP traffic, 10.9K outgoing IP traffic.5874client: (finished: waiting for success: ! systemctl is-active mid, in 1.14 seconds)5875client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }5876client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.06 seconds)5877(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 11.83 seconds)5878subtest: interrupting the client stops the build on the worker5879client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr5880client # [ 144.130779] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr.5881client # Running as unit: intr.service; invocation ID: d078142b823c4c8aae0bb01692acd4025882client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr, in 0.06 seconds)5883worker1 # [ 144.366319] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:357905884client # [ 144.371936] nix[1332]: don't know how to build these paths:5885client # [ 144.373803] nix[1332]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5886worker1 # [ 144.388136] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:12Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=05887worker1 # [ 144.394699] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:12Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.4:50051 clients=15888worker2 # [ 144.531341] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593425889worker2 # [ 144.546174] nix-grpc-daemon[1485]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 73 ms (attempt 1/5)5890worker2 # [ 144.626800] nix-grpc-daemon[1485]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 181 ms (attempt 2/5)5891worker2 # [ 144.815573] nix-grpc-daemon[1485]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 63 ms (attempt 3/5)5892worker2 # [ 144.879648] nix-grpc-daemon[1485]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 629 ms (attempt 4/5)5893worker2 # [ 145.508850] nix-grpc-daemon[1485]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5894worker2 # [ 145.513658] nix-grpc-daemon[1485]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5895worker2 # [ 145.535072] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:593425896worker2 # [ 145.552887] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:593425897client # [ 145.464476] nix[1332]: copying 1 paths to 'lb:50051'...5898client # [ 145.464715] nix[1332]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'...5899worker2 # [ 145.563831] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)5900worker2 # [ 145.589455] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:13Z level=info event=niks3_push_started pid=15855901worker2 # [ 145.791384] nix-grpc-daemon[1585]: time=2026-09-21T16:51:13.427Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca=""5902worker2 # [ 145.968146] nix-daemon[849]: accepted connection from pid 1598, user nix-grpc-daemon (trusted)5903worker2 # [ 145.985547] nix-daemon[849]: reaped child process 1601, status = succeeded5904worker2 # [ 146.158230] nix-daemon[849]: accepted connection from pid 1603, user nix-grpc-daemon (trusted)5905worker2 # [ 146.175962] nix-daemon[849]: reaped child process 1606, status = succeeded5906lb # [ 146.083520] niks3-server[1147]: 2026/09/21 16:51:13 INFO Received uploads request method=POST path=/api/pending_closures5907worker2 # [ 146.204065] nix-grpc-daemon[1585]: time=2026-09-21T16:51:13.840Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5908worker2 # [ 146.205570] nix-grpc-daemon[1585]: time=2026-09-21T16:51:13.840Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)"5909lb # [ 146.125734] niks3-server[1147]: 2026/09/21 16:51:13 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst5910worker2 # [ 146.249849] nix-grpc-daemon[1585]: time=2026-09-21T16:51:13.885Z level=INFO msg="Uploading 1 narinfos"5911lb # [ 146.136357] niks3-server[1147]: 2026/09/21 16:51:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign5912lb # [ 146.149585] niks3-server[1147]: 2026/09/21 16:51:13 INFO Signed narinfos id=45 count=15913lb # [ 146.155780] niks3-server[1147]: 2026/09/21 16:51:13 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls5914lb # [ 146.164280] niks3-server[1147]: 2026/09/21 16:51:13 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete5915lb # [ 146.176080] niks3-server[1147]: 2026/09/21 16:51:13 INFO Completed upload id=455916worker2 # [ 146.283191] nix-grpc-daemon[1585]: time=2026-09-21T16:51:13.919Z level=INFO msg="Upload complete. (487ms)"5917lb # [ 146.179002] niks3-server[1147]: 2026/09/21 16:51:13 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo5918worker2 # [ 146.292317] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59342 duration_s=0 paths=1 nar_bytes_in=4245919worker2 # [ 146.305519] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593425920worker2 # [ 146.311376] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)5921worker2 # [ 146.316208] nix-daemon[849]: reaped child process 1583, status = succeeded5922client # [ 146.235453] nix[1332]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051...5923worker2 # [ 146.329961] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)5924client: must succeed: systemctl kill -s INT intr5925client # [ 147.905220] systemd[1]: intr.service: Sent signal SIGINT to main process 1332 (nix) on client request.5926worker2 # [ 147.999574] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:15Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5927client: (finished: must succeed: systemctl kill -s INT intr, in 0.07 seconds)5928client # [ 147.914232] nix[1332]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5929client # [ 147.915743] nix[1332]: error: interrupted by the user5930client # [ 147.960332] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE5931client # [ 147.971796] systemd[1]: intr.service: Failed with result 'exit-code'.5932client # [ 147.974583] systemd[1]: intr.service: Consumed 159ms CPU time over 3.838s wall clock time, 41.2M memory peak, 6.3K incoming IP traffic, 8.3K outgoing IP traffic.5933worker2 # [ 148.169528] nix-daemon[849]: reaped child process 1608, status = succeeded5934(finished: subtest: interrupting the client stops the build on the worker, in 3.99 seconds)5935subtest: a deploy mid-build drains: the build finishes, the new generation takes over5936client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw5937worker2 # [ 148.172498] nix-daemon[1610]: unexpected Nix daemon error: error: interrupted by the user5938worker2 # [ 148.180410] nix-daemon[849]: reaped child process 1610, status = failed with exit code 15939client # [ 148.146216] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw.5940client # Running as unit: sw.service; invocation ID: f0ffd1d91d544342b5bf4666d30f5c9a5941client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw, in 0.09 seconds)5942worker1 # [ 148.402366] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:16Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:357905943client # [ 148.407960] nix[1352]: don't know how to build these paths:5944client # [ 148.409163] nix[1352]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv5945worker1 # [ 148.418264] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:16Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=05946worker1 # [ 148.424817] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:16Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.3:50051 clients=15947worker1 # [ 148.462764] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357905948worker1 # [ 148.479010] nix-grpc-daemon[1600]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 58 ms (attempt 1/5)5949worker1 # [ 148.544586] nix-grpc-daemon[1600]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 146 ms (attempt 2/5)5950worker1 # [ 148.698113] nix-grpc-daemon[1600]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 156 ms (attempt 3/5)5951worker1 # [ 148.861711] nix-grpc-daemon[1600]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 149 ms (attempt 4/5)5952worker1 # [ 149.018032] nix-grpc-daemon[1600]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5953worker1 # [ 149.023708] nix-grpc-daemon[1600]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5954worker1 # [ 149.042181] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357905955client # [ 149.056217] nix[1352]: copying 1 paths to 'lb:50051'...5956client # [ 149.056630] nix[1352]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'...5957worker1 # [ 149.057915] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357905958worker1 # [ 149.073313] nix-daemon[858]: accepted connection from pid 1600, user nix-grpc-daemon (trusted)5959worker1 # [ 149.276515] nix-daemon[858]: accepted connection from pid 1735, user nix-grpc-daemon (trusted)5960worker1 # [ 149.294858] nix-daemon[858]: reaped child process 1738, status = succeeded5961worker1 # [ 149.475897] nix-daemon[858]: accepted connection from pid 1740, user nix-grpc-daemon (trusted)5962worker1 # [ 149.490680] nix-daemon[858]: reaped child process 1747, status = succeeded5963lb # [ 149.487599] niks3-server[1147]: 2026/09/21 16:51:17 INFO Received uploads request method=POST path=/api/pending_closures5964worker1 # [ 149.517011] nix-grpc-daemon[1683]: time=2026-09-21T16:51:17.146Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5965worker1 # [ 149.518445] nix-grpc-daemon[1683]: time=2026-09-21T16:51:17.146Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)"5966lb # [ 149.522523] niks3-server[1147]: 2026/09/21 16:51:17 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst5967worker1 # [ 149.554249] nix-grpc-daemon[1683]: time=2026-09-21T16:51:17.184Z level=INFO msg="Uploading 1 narinfos"5968lb # [ 149.537751] niks3-server[1147]: 2026/09/21 16:51:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign5969lb # [ 149.547748] niks3-server[1147]: 2026/09/21 16:51:17 INFO Signed narinfos id=46 count=15970worker1 # [ 149.575272] nix-grpc-daemon[1683]: time=2026-09-21T16:51:17.205Z level=INFO msg="Upload complete. (483ms)"5971lb # [ 149.557050] niks3-server[1147]: 2026/09/21 16:51:17 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls5972lb # [ 149.562109] niks3-server[1147]: 2026/09/21 16:51:17 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete5973lb # [ 149.567008] niks3-server[1147]: 2026/09/21 16:51:17 INFO Completed upload id=465974worker1 # [ 149.584996] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35790 duration_s=0 paths=1 nar_bytes_in=4165975lb # [ 149.569168] niks3-server[1147]: 2026/09/21 16:51:17 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo5976worker1 # [ 149.594468] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357905977worker1 # [ 149.603861] nix-daemon[858]: accepted connection from pid 1600, user nix-grpc-daemon (trusted)5978worker1 # [ 149.606469] nix-daemon[858]: reaped child process 1733, status = succeeded5979client # [ 149.614081] nix[1352]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051...5980worker1 # [ 149.618330] nix-daemon[858]: accepted connection from pid 1600, user nix-grpc-daemon (trusted)5981worker1: must succeed: systemctl reload nix-grpc-daemon.service5982worker1 # [ 150.835688] systemd[1]: Reloading Nix worker-protocol over gRPC...5983lb # [ 150.857077] envoy[926]: conn peer=[2001:db8:1::1]:51892 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4127 tx=1931 ms=110165984worker1 # [ 150.882464] systemd[1]: Reloaded Nix worker-protocol over gRPC.5985lb # [ 150.857331] envoy[926]: conn peer=[2001:db8:1::1]:43132 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2987 tx=853 ms=35695986worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.11 seconds)5987worker1: must succeed: pgrep -f 'read -t [9]0 x'5988worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds)5989client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain >&25990worker1 # [ 151.065726] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:18Z level=info event=draining5991worker2 # [ 151.311923] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:18Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:593425992client # don't know how to build these paths:5993client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5994worker2 # [ 151.322462] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:593425995worker1 # [ 151.258151] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:18Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=05996worker1 # [ 151.260282] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:18Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.4:50051 clients=15997worker2 # [ 151.396320] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593425998worker2 # [ 151.411591] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:593425999client # copying 1 paths to 'lb:50051'...6000worker2 # [ 151.427787] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:593426001client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'...6002worker2 # [ 151.451868] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6003worker2 # [ 151.663564] nix-daemon[849]: accepted connection from pid 1657, user nix-grpc-daemon (trusted)6004worker2 # [ 151.683006] nix-daemon[849]: reaped child process 1660, status = succeeded6005worker2 # [ 151.856236] nix-daemon[849]: accepted connection from pid 1662, user nix-grpc-daemon (trusted)6006worker2 # [ 151.872199] nix-daemon[849]: reaped child process 1665, status = succeeded6007lb # [ 151.780931] niks3-server[1147]: 2026/09/21 16:51:19 INFO Received uploads request method=POST path=/api/pending_closures6008worker2 # [ 151.894348] nix-grpc-daemon[1585]: time=2026-09-21T16:51:19.530Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6009worker2 # [ 151.895901] nix-grpc-daemon[1585]: time=2026-09-21T16:51:19.530Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)"6010lb # [ 151.796388] niks3-server[1147]: 2026/09/21 16:51:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6011worker2 # [ 151.908639] nix-grpc-daemon[1585]: time=2026-09-21T16:51:19.545Z level=INFO msg="Uploading 1 narinfos"6012lb # [ 151.802682] niks3-server[1147]: 2026/09/21 16:51:19 INFO Signed narinfos id=47 count=16013lb # [ 151.803900] niks3-server[1147]: 2026/09/21 16:51:19 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst6014lb # [ 151.812811] niks3-server[1147]: 2026/09/21 16:51:19 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6015worker2 # [ 151.923614] nix-grpc-daemon[1585]: time=2026-09-21T16:51:19.560Z level=INFO msg="Upload complete. (454ms)"6016lb # [ 151.818822] niks3-server[1147]: 2026/09/21 16:51:19 INFO Completed upload id=476017worker2 # [ 151.927574] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59342 duration_s=0 paths=1 nar_bytes_in=5286018lb # [ 151.821224] niks3-server[1147]: 2026/09/21 16:51:19 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo6019worker2 # [ 151.936567] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593426020worker2 # [ 151.940263] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6021worker2 # [ 151.943100] nix-daemon[849]: reaped child process 1655, status = succeeded6022client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051...6023worker2 # [ 151.951844] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6024client # farm-a> worker2: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv6025client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6026client # farm-a> LOG-farm-a-during-drain6027worker2 # [ 153.189382] nix-daemon[849]: reaped child process 1669, status = succeeded6028worker2 # [ 153.372805] nix-daemon[849]: accepted connection from pid 1685, user nix-grpc-daemon (trusted)6029worker2 # [ 153.389405] nix-daemon[849]: reaped child process 1688, status = succeeded6030lb # [ 153.296666] niks3-server[1147]: 2026/09/21 16:51:20 INFO Received uploads request method=POST path=/api/pending_closures6031worker2 # [ 153.419859] nix-grpc-daemon[1585]: time=2026-09-21T16:51:21.053Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6032worker2 # [ 153.422153] nix-grpc-daemon[1585]: time=2026-09-21T16:51:21.053Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)"6033lb # [ 153.344594] niks3-server[1147]: 2026/09/21 16:51:20 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6034lb # [ 153.361146] niks3-server[1147]: 2026/09/21 16:51:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6035worker2 # [ 153.474078] nix-grpc-daemon[1585]: time=2026-09-21T16:51:21.109Z level=INFO msg="Uploading 1 narinfos"6036lb # [ 153.367104] niks3-server[1147]: 2026/09/21 16:51:20 INFO Signed narinfos id=48 count=16037lb # [ 153.371051] niks3-server[1147]: 2026/09/21 16:51:20 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv6038lb # [ 153.378480] niks3-server[1147]: 2026/09/21 16:51:20 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls6039lb # [ 153.385037] niks3-server[1147]: 2026/09/21 16:51:21 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6040lb # [ 153.391555] niks3-server[1147]: 2026/09/21 16:51:21 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo6041lb # [ 153.395668] niks3-server[1147]: 2026/09/21 16:51:21 INFO Completed upload id=486042worker2 # [ 153.502537] nix-grpc-daemon[1585]: time=2026-09-21T16:51:21.139Z level=INFO msg="Upload complete. (343ms)"6043worker1 # [ 153.426383] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:21Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=06044worker1 # [ 153.428372] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:21Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.4:50051 clients=16045worker2 # [ 153.516525] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59342 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=16046worker2 # [ 153.531868] nix-daemon[849]: reaped child process 1667, status = succeeded6047worker2 # [ 153.559202] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593426048worker2 # [ 153.574366] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:593426049worker2 # [ 153.588379] nix-grpc-daemon[1585]: time=2026-09-21T16:51:21.224Z level=INFO msg="All 1 paths already cached"6050client # copying 1 paths to 'lb:50051'...6051worker2 # [ 153.598290] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:593426052client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'...6053worker2 # [ 153.621653] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6054worker2 # [ 153.825924] nix-daemon[849]: accepted connection from pid 1692, user nix-grpc-daemon (trusted)6055worker2 # [ 153.843175] nix-daemon[849]: reaped child process 1695, status = succeeded6056worker2 # [ 154.016772] nix-daemon[849]: accepted connection from pid 1697, user nix-grpc-daemon (trusted)6057worker2 # [ 154.034427] nix-daemon[849]: reaped child process 1700, status = succeeded6058lb # [ 153.938927] niks3-server[1147]: 2026/09/21 16:51:21 INFO Received uploads request method=POST path=/api/pending_closures6059worker2 # [ 154.054904] nix-grpc-daemon[1585]: time=2026-09-21T16:51:21.691Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6060worker2 # [ 154.056560] nix-grpc-daemon[1585]: time=2026-09-21T16:51:21.691Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)"6061lb # [ 153.966774] niks3-server[1147]: 2026/09/21 16:51:21 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst6062worker2 # [ 154.088282] nix-grpc-daemon[1585]: time=2026-09-21T16:51:21.724Z level=INFO msg="Uploading 1 narinfos"6063lb # [ 153.982847] niks3-server[1147]: 2026/09/21 16:51:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6064lb # [ 153.992660] niks3-server[1147]: 2026/09/21 16:51:21 INFO Signed narinfos id=49 count=16065lb # [ 153.996361] niks3-server[1147]: 2026/09/21 16:51:21 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls6066lb # [ 154.005579] niks3-server[1147]: 2026/09/21 16:51:21 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6067lb # [ 154.011220] niks3-server[1147]: 2026/09/21 16:51:21 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo6068lb # [ 154.014964] niks3-server[1147]: 2026/09/21 16:51:21 INFO Completed upload id=496069worker2 # [ 154.121974] nix-grpc-daemon[1585]: time=2026-09-21T16:51:21.758Z level=INFO msg="Upload complete. (482ms)"6070worker2 # [ 154.129123] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59342 duration_s=0 paths=1 nar_bytes_in=6406071worker2 # [ 154.142047] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593426072worker2 # [ 154.148472] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6073worker2 # [ 154.153137] nix-daemon[849]: reaped child process 1690, status = succeeded6074client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051...6075client # farm-b> worker2: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv6076worker2 # [ 154.166294] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6077client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6078client # farm-b> LOG-farm-b-during-drain6079worker2 # [ 155.591660] nix-daemon[849]: reaped child process 1704, status = succeeded6080worker2 # [ 155.758372] nix-daemon[849]: accepted connection from pid 1720, user nix-grpc-daemon (trusted)6081worker2 # [ 155.776539] nix-daemon[849]: reaped child process 1723, status = succeeded6082lb # [ 155.684942] niks3-server[1147]: 2026/09/21 16:51:23 INFO Received uploads request method=POST path=/api/pending_closures6083worker2 # [ 155.804396] nix-grpc-daemon[1585]: time=2026-09-21T16:51:23.440Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6084worker2 # [ 155.805890] nix-grpc-daemon[1585]: time=2026-09-21T16:51:23.440Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)"6085lb # [ 155.716503] niks3-server[1147]: 2026/09/21 16:51:23 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv6086lb # [ 155.729657] niks3-server[1147]: 2026/09/21 16:51:23 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst6087lb # [ 155.744376] niks3-server[1147]: 2026/09/21 16:51:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6088worker2 # [ 155.856082] nix-grpc-daemon[1585]: time=2026-09-21T16:51:23.492Z level=INFO msg="Uploading 1 narinfos"6089lb # [ 155.754695] niks3-server[1147]: 2026/09/21 16:51:23 INFO Signed narinfos id=50 count=16090lb # [ 155.761131] niks3-server[1147]: 2026/09/21 16:51:23 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls6091lb # [ 155.770692] niks3-server[1147]: 2026/09/21 16:51:23 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6092lb # [ 155.776066] niks3-server[1147]: 2026/09/21 16:51:23 INFO Completed upload id=506093worker2 # [ 155.882317] nix-grpc-daemon[1585]: time=2026-09-21T16:51:23.518Z level=INFO msg="Upload complete. (305ms)"6094lb # [ 155.779248] niks3-server[1147]: 2026/09/21 16:51:23 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo6095worker1 # [ 155.809668] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:23Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=06096worker1 # [ 155.811513] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:23Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.4:50051 clients=16097worker2 # [ 155.893367] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59342 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=16098worker2 # [ 155.910722] nix-daemon[849]: reaped child process 1702, status = succeeded6099worker2 # [ 155.943198] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593426100worker2 # [ 155.958178] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:593426101worker2 # [ 155.972738] nix-grpc-daemon[1585]: time=2026-09-21T16:51:23.609Z level=INFO msg="All 2 paths already cached"6102client # copying 1 paths to 'lb:50051'...6103worker2 # [ 155.982631] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:593426104client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'...6105worker2 # [ 156.004214] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6106worker2 # [ 156.218700] nix-daemon[849]: accepted connection from pid 1727, user nix-grpc-daemon (trusted)6107worker2 # [ 156.236309] nix-daemon[849]: reaped child process 1730, status = succeeded6108worker2 # [ 156.408846] nix-daemon[849]: accepted connection from pid 1732, user nix-grpc-daemon (trusted)6109worker2 # [ 156.425966] nix-daemon[849]: reaped child process 1735, status = succeeded6110lb # [ 156.332652] niks3-server[1147]: 2026/09/21 16:51:23 INFO Received uploads request method=POST path=/api/pending_closures6111worker2 # [ 156.448750] nix-grpc-daemon[1585]: time=2026-09-21T16:51:24.085Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6112worker2 # [ 156.450291] nix-grpc-daemon[1585]: time=2026-09-21T16:51:24.085Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)"6113lb # [ 156.358338] niks3-server[1147]: 2026/09/21 16:51:23 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst6114worker2 # [ 156.478002] nix-grpc-daemon[1585]: time=2026-09-21T16:51:24.114Z level=INFO msg="Uploading 1 narinfos"6115lb # [ 156.373405] niks3-server[1147]: 2026/09/21 16:51:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6116lb # [ 156.385247] niks3-server[1147]: 2026/09/21 16:51:23 INFO Signed narinfos id=51 count=16117lb # [ 156.397147] niks3-server[1147]: 2026/09/21 16:51:24 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6118worker2 # [ 156.509059] nix-grpc-daemon[1585]: time=2026-09-21T16:51:24.145Z level=INFO msg="Upload complete. (485ms)"6119lb # [ 156.402562] niks3-server[1147]: 2026/09/21 16:51:24 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls6120lb # [ 156.408329] niks3-server[1147]: 2026/09/21 16:51:24 INFO Completed upload id=516121worker2 # [ 156.516097] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59342 duration_s=0 paths=1 nar_bytes_in=7686122lb # [ 156.411941] niks3-server[1147]: 2026/09/21 16:51:24 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo6123worker2 # [ 156.527209] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593426124worker2 # [ 156.533552] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6125worker2 # [ 156.538437] nix-daemon[849]: reaped child process 1725, status = succeeded6126client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051...6127worker2 # [ 156.548801] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6128client # farm-top> worker2: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6129client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6130client # farm-top> LOG-farm-top-during-drain6131worker2 # [ 157.790209] nix-daemon[849]: reaped child process 1739, status = succeeded6132worker2 # [ 157.972468] nix-daemon[849]: accepted connection from pid 1755, user nix-grpc-daemon (trusted)6133worker2 # [ 157.989095] nix-daemon[849]: reaped child process 1758, status = succeeded6134lb # [ 157.898490] niks3-server[1147]: 2026/09/21 16:51:25 INFO Received uploads request method=POST path=/api/pending_closures6135worker2 # [ 158.015273] nix-grpc-daemon[1585]: time=2026-09-21T16:51:25.650Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6136worker2 # [ 158.016943] nix-grpc-daemon[1585]: time=2026-09-21T16:51:25.650Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)"6137lb # [ 157.937932] niks3-server[1147]: 2026/09/21 16:51:25 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst6138lb # [ 157.949033] niks3-server[1147]: 2026/09/21 16:51:25 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6139lb # [ 157.957492] niks3-server[1147]: 2026/09/21 16:51:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6140worker2 # [ 158.069166] nix-grpc-daemon[1585]: time=2026-09-21T16:51:25.705Z level=INFO msg="Uploading 1 narinfos"6141lb # [ 157.965801] niks3-server[1147]: 2026/09/21 16:51:25 INFO Signed narinfos id=52 count=16142lb # [ 157.970917] niks3-server[1147]: 2026/09/21 16:51:25 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls6143lb # [ 157.978426] niks3-server[1147]: 2026/09/21 16:51:25 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6144lb # [ 157.986322] niks3-server[1147]: 2026/09/21 16:51:25 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo6145lb # [ 157.990807] niks3-server[1147]: 2026/09/21 16:51:25 INFO Completed upload id=526146worker2 # [ 158.097931] nix-grpc-daemon[1585]: time=2026-09-21T16:51:25.734Z level=INFO msg="Upload complete. (337ms)"6147worker2 # [ 158.109203] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59342 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=16148worker2 # [ 158.126545] nix-daemon[849]: reaped child process 1737, status = succeeded6149client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain >&2, in 7.13 seconds)6150client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain outPath6151client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain outPath, in 0.23 seconds)6152worker1: must succeed: pkill -f 'read -t [9]0 x'6153worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.06 seconds)6154client: waiting for success: ! systemctl is-active sw6155worker1 # [ 158.452368] nix-daemon[858]: reaped child process 1751, status = succeeded6156worker1 # [ 158.612445] nix-daemon[858]: accepted connection from pid 1794, user nix-grpc-daemon (trusted)6157worker1 # [ 158.631806] nix-daemon[858]: reaped child process 1797, status = succeeded6158lb # [ 158.624176] niks3-server[1147]: 2026/09/21 16:51:26 INFO Received uploads request method=POST path=/api/pending_closures6159worker1 # [ 158.653611] nix-grpc-daemon[1683]: time=2026-09-21T16:51:26.283Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6160worker1 # [ 158.655091] nix-grpc-daemon[1683]: time=2026-09-21T16:51:26.283Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)"6161worker1 # [ 158.697759] nix-grpc-daemon[1683]: time=2026-09-21T16:51:26.327Z level=INFO msg="Uploading 1 narinfos"6162lb # [ 158.669892] niks3-server[1147]: 2026/09/21 16:51:26 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6163lb # [ 158.683268] niks3-server[1147]: 2026/09/21 16:51:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6164lb # [ 158.694916] niks3-server[1147]: 2026/09/21 16:51:26 INFO Signed narinfos id=53 count=16165lb # [ 158.700353] niks3-server[1147]: 2026/09/21 16:51:26 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv6166lb # [ 158.712315] niks3-server[1147]: 2026/09/21 16:51:26 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls6167worker1 # [ 158.737265] nix-grpc-daemon[1683]: time=2026-09-21T16:51:26.366Z level=INFO msg="Upload complete. (299ms)"6168lb # [ 158.717683] niks3-server[1147]: 2026/09/21 16:51:26 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6169lb # [ 158.724218] niks3-server[1147]: 2026/09/21 16:51:26 INFO Completed upload id=536170lb # [ 158.727219] niks3-server[1147]: 2026/09/21 16:51:26 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo6171worker1 # [ 158.747310] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35790 duration_s=9 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=16172worker1 # [ 158.761619] nix-daemon[858]: reaped child process 1749, status = succeeded6173client # [ 158.793149] systemd[1]: sw.service: Deactivated successfully.6174client # [ 158.797789] systemd[1]: sw.service: Consumed 180ms CPU time over 10.650s wall clock time, 41.2M memory peak, 7.3K incoming IP traffic, 8.5K outgoing IP traffic.6175worker1 # [ 159.071491] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:26Z level=info event=signal_exit6176lb # [ 159.269319] niks3-server[1147]: 2026/09/21 16:51:26 INFO lead: released remote=192.168.1.3:562386177client: (finished: waiting for success: ! systemctl is-active sw, in 1.12 seconds)6178client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6179client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.05 seconds)6180worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6181worker1 # [ 160.285613] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:27Z level=info event=scheduler_restarting6182worker2 # [ 160.375490] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:28Z level=info event=scheduler_restarting addr=lb:500516183worker1 # [ 160.295630] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:27Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:357766184worker1 # [ 160.301818] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:27Z level=info event=scheduler_restarting addr=lb:500516185worker1 # [ 160.306197] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:27Z level=info event=worker_gone addr=192.168.1.4:500516186worker1 # [ 160.310822] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:27Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:357766187worker1 # [ 160.315994] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:27Z level=info event=worker_gone addr=192.168.1.3:500516188worker1 # [ 160.340501] nix-daemon[858]: reaped child process 1654, status = succeeded6189worker2 # [ 160.576067] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:28Z level=info event=scheduler_connected addr=lb:500516190worker2 # [ 160.581274] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6191worker1 # [ 160.511654] nix-grpc-daemon[1600]: ts=2026-09-21T16:51:28Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6192worker2 # [ 160.782262] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:28Z level=info event=scheduler_connected addr=lb:500516193worker2 # [ 160.787485] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6194lb # [ 160.859681] envoy[926]: conn peer=[2001:db8:1::1]:58040 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9625 tx=4104 ms=68136195worker2 # [ 160.988478] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:28Z level=info event=scheduler_connected addr=lb:500516196worker2 # [ 160.993624] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6197worker2 # [ 161.194674] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:28Z level=info event=scheduler_connected addr=lb:500516198worker1 # [ 161.105011] nix-grpc-daemon[1600]: E0000 00:00:1790009488.735148 1645 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true6199worker2 # [ 161.199868] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6200worker1 # [ 161.111755] nix-grpc-daemon[1600]: E0000 00:00:1790009488.735498 1645 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6201lb # [ 160.865521] envoy[926]: conn peer=[2001:db8:1::1]:43136 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2995 tx=1470 ms=103636202lb # [ 160.865657] envoy[926]: conn peer=[2001:db8:1::3]:46444 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=894 tx=344 ms=223796203lb # [ 161.120186] niks3-server[1147]: 2026/09/21 16:51:28 INFO lead: acquired remote=192.168.1.4:395466204worker2 # [ 161.240311] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:28Z level=info event=scheduler_take_over6205worker2 # [ 161.402584] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:29Z level=info event=scheduler_connected addr=lb:500516206worker2 # [ 161.406802] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6207worker2 # [ 161.604100] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:29Z level=info event=scheduler_connected addr=lb:500516208worker2 # [ 161.609292] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6209worker2 # [ 161.810366] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:29Z level=info event=scheduler_connected addr=lb:500516210worker2 # [ 161.815816] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6211worker2 # [ 162.017116] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:29Z level=info event=scheduler_connected addr=lb:500516212worker2 # [ 162.042731] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:29Z level=info event=oidc_jwks provider=mock keys=16213worker2 # [ 162.047304] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:29Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:542886214worker2 # [ 162.052738] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:29Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=06215worker1 # [ 162.098823] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6216worker1 # [ 162.103703] systemd[1]: nix-grpc-daemon.service: Consumed 894ms CPU time over 24.559s wall clock time, 35.2M memory peak, 25.1K incoming IP traffic, 31.7K outgoing IP traffic.6217worker1 # [ 162.125153] systemd[1]: Starting Nix worker-protocol over gRPC...6218worker1 # [ 162.318204] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:29Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6219worker1 # [ 162.320721] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:29Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6220worker1 # [ 162.324775] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:29Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16221worker1 # [ 162.332037] nix-grpc-daemon[1811]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6222worker1 # [ 162.333484] nix-grpc-daemon[1811]: E0000 00:00:1790009489.961718 1811 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6223worker1 # [ 162.342899] nix-grpc-daemon[1811]: E0000 00:00:1790009489.967774 1811 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6224worker1 # [ 162.347942] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:29Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6225worker1 # [ 162.350503] systemd[1]: Started Nix worker-protocol over gRPC.6226worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.84 seconds)6227worker1 # [ 162.380283] nix-grpc-daemon[1811]: E0000 00:00:1790009489.983223 1851 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6228lb: waiting until exactly no endpoints of aarch64-linux are unhealthy6229lb: must succeed: curl -sf localhost:9901/clusters6230worker1 # [ 162.382485] nix-grpc-daemon[1811]: E0000 00:00:1790009489.998224 1851 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6231worker1 # [ 162.389168] nix-grpc-daemon[1811]: E0000 00:00:1790009489.998367 1851 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6232worker1 # [ 162.392183] nix-grpc-daemon[1811]: E0000 00:00:1790009490.018780 1851 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6233worker1 # [ 162.405586] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:30Z level=info event=scheduler_connected addr=lb:500516234worker2 # [ 162.498745] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:30Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:542886235worker2 # [ 162.503837] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:30Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=06236lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)6237lb: must succeed: curl -sf localhost:9901/clusters6238lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6239lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.17 seconds)6240lb: must succeed: curl -sf localhost:9901/clusters6241lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6242worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6243worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6244(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 15.61 seconds)6245subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6246client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16247worker2 # [ 164.057422] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:31Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:593426248worker2 # [ 164.067096] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:593426249worker2 # [ 164.112142] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:31Z level=debug event=want client=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=06250worker2 # [ 164.118774] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:31Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=16251worker2 # [ 164.125774] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:31Z level=debug event=want client=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=06252worker2 # [ 164.131619] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:31Z level=debug event=want client=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=06253worker1 # [ 164.081970] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186254worker1 # [ 164.097722] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6255worker1 # [ 164.124928] nix-grpc-daemon[1811]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 87 ms (attempt 1/5)6256worker1 # [ 164.218860] nix-grpc-daemon[1811]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 165 ms (attempt 2/5)6257worker1 # [ 164.386847] nix-grpc-daemon[1811]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 67 ms (attempt 3/5)6258worker1 # [ 164.461547] nix-grpc-daemon[1811]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 671 ms (attempt 4/5)6259worker1 # [ 165.142620] nix-grpc-daemon[1811]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6260worker1 # [ 165.147536] nix-grpc-daemon[1811]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6261worker1 # [ 165.167314] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558186262worker1 # [ 165.182933] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558186263worker1 # [ 165.198268] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6264worker1 # [ 165.226691] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:32Z level=info event=niks3_push_started pid=18716265worker1 # [ 165.440257] nix-grpc-daemon[1871]: time=2026-09-21T16:51:33.070Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca=""6266worker1 # [ 165.611007] nix-daemon[858]: accepted connection from pid 1884, user nix-grpc-daemon (trusted)6267worker1 # [ 165.627449] nix-daemon[858]: reaped child process 1887, status = succeeded6268worker1 # [ 165.797457] nix-daemon[858]: accepted connection from pid 1889, user nix-grpc-daemon (trusted)6269worker1 # [ 165.813746] nix-daemon[858]: reaped child process 1892, status = succeeded6270lb # [ 165.808404] niks3-server[1147]: 2026/09/21 16:51:33 INFO Received uploads request method=POST path=/api/pending_closures6271worker1 # [ 165.836075] nix-grpc-daemon[1871]: time=2026-09-21T16:51:33.465Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6272worker1 # [ 165.837543] nix-grpc-daemon[1871]: time=2026-09-21T16:51:33.465Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)"6273lb # [ 165.844388] niks3-server[1147]: 2026/09/21 16:51:33 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst6274lb # [ 165.857119] niks3-server[1147]: 2026/09/21 16:51:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6275worker1 # [ 165.883486] nix-grpc-daemon[1871]: time=2026-09-21T16:51:33.513Z level=INFO msg="Uploading 1 narinfos"6276lb # [ 165.872200] niks3-server[1147]: 2026/09/21 16:51:33 INFO Signed narinfos id=54 count=16277lb # [ 165.875211] niks3-server[1147]: 2026/09/21 16:51:33 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls6278lb # [ 165.892657] niks3-server[1147]: 2026/09/21 16:51:33 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6279worker1 # [ 165.916203] nix-grpc-daemon[1871]: time=2026-09-21T16:51:33.546Z level=INFO msg="Upload complete. (472ms)"6280lb # [ 165.897141] niks3-server[1147]: 2026/09/21 16:51:33 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo6281lb # [ 165.900926] niks3-server[1147]: 2026/09/21 16:51:33 INFO Completed upload id=546282worker1 # [ 165.924151] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=0 paths=1 nar_bytes_in=5126283worker1 # [ 165.939261] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186284worker1 # [ 165.946750] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6285worker1 # [ 165.951727] nix-daemon[858]: reaped child process 1869, status = succeeded6286worker1 # [ 165.963625] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6287worker1 # [ 167.407203] nix-daemon[858]: reaped child process 1896, status = succeeded6288worker1 # [ 167.588593] nix-daemon[858]: accepted connection from pid 1912, user nix-grpc-daemon (trusted)6289worker1 # [ 167.604657] nix-daemon[858]: reaped child process 1915, status = succeeded6290lb # [ 167.602226] niks3-server[1147]: 2026/09/21 16:51:35 INFO Received uploads request method=POST path=/api/pending_closures6291worker1 # [ 167.628614] nix-grpc-daemon[1871]: time=2026-09-21T16:51:35.258Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6292worker1 # [ 167.634442] nix-grpc-daemon[1871]: time=2026-09-21T16:51:35.261Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)"6293lb # [ 167.630856] niks3-server[1147]: 2026/09/21 16:51:35 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv6294lb # [ 167.650825] niks3-server[1147]: 2026/09/21 16:51:35 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6295lb # [ 167.662510] niks3-server[1147]: 2026/09/21 16:51:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6296lb # [ 167.670235] niks3-server[1147]: 2026/09/21 16:51:35 INFO Signed narinfos id=55 count=16297worker1 # [ 167.689591] nix-grpc-daemon[1871]: time=2026-09-21T16:51:35.318Z level=INFO msg="Uploading 1 narinfos"6298lb # [ 167.677002] niks3-server[1147]: 2026/09/21 16:51:35 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls6299lb # [ 167.691034] niks3-server[1147]: 2026/09/21 16:51:35 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6300lb # [ 167.697433] niks3-server[1147]: 2026/09/21 16:51:35 INFO Completed upload id=556301worker1 # [ 167.715782] nix-grpc-daemon[1871]: time=2026-09-21T16:51:35.346Z level=INFO msg="Upload complete. (338ms)"6302lb # [ 167.700685] niks3-server[1147]: 2026/09/21 16:51:35 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo6303worker2 # [ 167.819911] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:35Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=16304worker1 # [ 167.729123] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=1 outputs=16305worker2 # [ 167.831914] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:35Z level=debug event=want client=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=06306worker1 # [ 167.747201] nix-daemon[858]: reaped child process 1894, status = succeeded6307worker1 # [ 167.775327] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186308worker1 # [ 167.790498] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558186309worker1 # [ 167.805096] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558186310worker1 # [ 167.818754] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6311worker1 # [ 168.014950] nix-daemon[858]: accepted connection from pid 1919, user nix-grpc-daemon (trusted)6312worker1 # [ 168.033081] nix-daemon[858]: reaped child process 1922, status = succeeded6313worker1 # [ 168.204160] nix-daemon[858]: accepted connection from pid 1924, user nix-grpc-daemon (trusted)6314worker1 # [ 168.220002] nix-daemon[858]: reaped child process 1927, status = succeeded6315lb # [ 168.215045] niks3-server[1147]: 2026/09/21 16:51:35 INFO Received uploads request method=POST path=/api/pending_closures6316worker1 # [ 168.242605] nix-grpc-daemon[1871]: time=2026-09-21T16:51:35.872Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6317worker1 # [ 168.244104] nix-grpc-daemon[1871]: time=2026-09-21T16:51:35.872Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)"6318lb # [ 168.242737] niks3-server[1147]: 2026/09/21 16:51:35 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst6319lb # [ 168.253059] niks3-server[1147]: 2026/09/21 16:51:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6320worker1 # [ 168.278319] nix-grpc-daemon[1871]: time=2026-09-21T16:51:35.908Z level=INFO msg="Uploading 1 narinfos"6321lb # [ 168.263484] niks3-server[1147]: 2026/09/21 16:51:35 INFO Signed narinfos id=56 count=16322lb # [ 168.270107] niks3-server[1147]: 2026/09/21 16:51:35 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls6323lb # [ 168.276185] niks3-server[1147]: 2026/09/21 16:51:35 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6324lb # [ 168.284113] niks3-server[1147]: 2026/09/21 16:51:35 INFO Completed upload id=566325worker1 # [ 168.303073] nix-grpc-daemon[1871]: time=2026-09-21T16:51:35.933Z level=INFO msg="Upload complete. (468ms)"6326lb # [ 168.286704] niks3-server[1147]: 2026/09/21 16:51:35 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo6327worker1 # [ 168.312472] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=0 paths=1 nar_bytes_in=5126328worker1 # [ 168.320583] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186329worker1 # [ 168.329697] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6330worker1 # [ 168.332260] nix-daemon[858]: reaped child process 1917, status = succeeded6331worker1 # [ 168.342461] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6332worker1 # [ 169.563934] nix-daemon[858]: reaped child process 1931, status = succeeded6333worker1 # [ 169.719398] nix-daemon[858]: accepted connection from pid 1947, user nix-grpc-daemon (trusted)6334worker1 # [ 169.736485] nix-daemon[858]: reaped child process 1950, status = succeeded6335lb # [ 169.733300] niks3-server[1147]: 2026/09/21 16:51:37 INFO Received uploads request method=POST path=/api/pending_closures6336worker1 # [ 169.761909] nix-grpc-daemon[1871]: time=2026-09-21T16:51:37.391Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6337worker1 # [ 169.763374] nix-grpc-daemon[1871]: time=2026-09-21T16:51:37.391Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)"6338lb # [ 169.772763] niks3-server[1147]: 2026/09/21 16:51:37 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6339worker1 # [ 169.801403] nix-grpc-daemon[1871]: time=2026-09-21T16:51:37.431Z level=INFO msg="Uploading 1 narinfos"6340lb # [ 169.783839] niks3-server[1147]: 2026/09/21 16:51:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6341lb # [ 169.795024] niks3-server[1147]: 2026/09/21 16:51:37 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv6342lb # [ 169.803142] niks3-server[1147]: 2026/09/21 16:51:37 INFO Signed narinfos id=57 count=16343lb # [ 169.811261] niks3-server[1147]: 2026/09/21 16:51:37 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls6344worker1 # [ 169.835152] nix-grpc-daemon[1871]: time=2026-09-21T16:51:37.465Z level=INFO msg="Upload complete. (281ms)"6345lb # [ 169.817631] niks3-server[1147]: 2026/09/21 16:51:37 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6346lb # [ 169.824756] niks3-server[1147]: 2026/09/21 16:51:37 INFO Completed upload id=576347worker1 # [ 169.844473] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=2 outputs=16348lb # [ 169.828434] niks3-server[1147]: 2026/09/21 16:51:37 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo6349worker2 # [ 169.933371] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:37Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=16350worker2 # [ 169.946545] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:37Z level=debug event=want client=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=06351worker1 # [ 169.861374] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186352worker1 # [ 169.866572] nix-daemon[858]: reaped child process 1929, status = succeeded6353worker1 # [ 169.872949] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558186354worker1 # [ 169.884259] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558186355worker1 # [ 169.894040] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6356worker1 # [ 170.082167] nix-daemon[858]: accepted connection from pid 1954, user nix-grpc-daemon (trusted)6357worker1 # [ 170.099545] nix-daemon[858]: reaped child process 1957, status = succeeded6358worker1 # [ 170.273596] nix-daemon[858]: accepted connection from pid 1959, user nix-grpc-daemon (trusted)6359worker1 # [ 170.291763] nix-daemon[858]: reaped child process 1962, status = succeeded6360lb # [ 170.284211] niks3-server[1147]: 2026/09/21 16:51:37 INFO Received uploads request method=POST path=/api/pending_closures6361worker1 # [ 170.311778] nix-grpc-daemon[1871]: time=2026-09-21T16:51:37.942Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6362worker1 # [ 170.313454] nix-grpc-daemon[1871]: time=2026-09-21T16:51:37.942Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)"6363lb # [ 170.312578] niks3-server[1147]: 2026/09/21 16:51:37 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst6364worker1 # [ 170.340156] nix-grpc-daemon[1871]: time=2026-09-21T16:51:37.970Z level=INFO msg="Uploading 1 narinfos"6365lb # [ 170.325021] niks3-server[1147]: 2026/09/21 16:51:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6366lb # [ 170.335206] niks3-server[1147]: 2026/09/21 16:51:37 INFO Signed narinfos id=58 count=16367lb # [ 170.340281] niks3-server[1147]: 2026/09/21 16:51:37 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls6368worker1 # [ 170.366868] nix-grpc-daemon[1871]: time=2026-09-21T16:51:37.997Z level=INFO msg="Upload complete. (461ms)"6369lb # [ 170.349604] niks3-server[1147]: 2026/09/21 16:51:37 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6370worker1 # [ 170.372235] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=0 paths=1 nar_bytes_in=5126371lb # [ 170.355159] niks3-server[1147]: 2026/09/21 16:51:37 INFO Completed upload id=586372lb # [ 170.358886] niks3-server[1147]: 2026/09/21 16:51:37 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo6373worker1 # [ 170.384437] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186374worker1 # [ 170.389584] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6375worker1 # [ 170.394066] nix-daemon[858]: reaped child process 1952, status = succeeded6376worker1 # [ 170.406109] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6377worker1 # [ 171.644202] nix-daemon[858]: reaped child process 1966, status = succeeded6378worker1 # [ 171.815029] nix-daemon[858]: accepted connection from pid 1982, user nix-grpc-daemon (trusted)6379worker1 # [ 171.831962] nix-daemon[858]: reaped child process 1985, status = succeeded6380lb # [ 171.826935] niks3-server[1147]: 2026/09/21 16:51:39 INFO Received uploads request method=POST path=/api/pending_closures6381worker1 # [ 171.856581] nix-grpc-daemon[1871]: time=2026-09-21T16:51:39.486Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6382worker1 # [ 171.860413] nix-grpc-daemon[1871]: time=2026-09-21T16:51:39.489Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)"6383lb # [ 171.866467] niks3-server[1147]: 2026/09/21 16:51:39 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6384worker1 # [ 171.902909] nix-grpc-daemon[1871]: time=2026-09-21T16:51:39.533Z level=INFO msg="Uploading 1 narinfos"6385lb # [ 171.879472] niks3-server[1147]: 2026/09/21 16:51:39 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv6386lb # [ 171.894963] niks3-server[1147]: 2026/09/21 16:51:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6387lb # [ 171.901176] niks3-server[1147]: 2026/09/21 16:51:39 INFO Signed narinfos id=59 count=16388worker1 # [ 171.927397] nix-grpc-daemon[1871]: time=2026-09-21T16:51:39.557Z level=INFO msg="Upload complete. (312ms)"6389lb # [ 171.909089] niks3-server[1147]: 2026/09/21 16:51:39 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls6390lb # [ 171.913202] niks3-server[1147]: 2026/09/21 16:51:39 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6391lb # [ 171.920235] niks3-server[1147]: 2026/09/21 16:51:39 INFO Completed upload id=596392worker2 # [ 172.027646] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:39Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=16393lb # [ 171.924101] niks3-server[1147]: 2026/09/21 16:51:39 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo6394worker1 # [ 171.944145] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=3 outputs=16395worker2 # [ 172.040129] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:39Z level=debug event=want client=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=06396worker1 # [ 171.959519] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186397worker1 # [ 171.966257] nix-daemon[858]: reaped child process 1964, status = succeeded6398worker1 # [ 171.974194] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558186399worker1 # [ 171.987294] nix-grpc-daemon[1871]: time=2026-09-21T16:51:39.617Z level=INFO msg="All 1 paths already cached"6400worker1 # [ 171.995533] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558186401worker1 # [ 172.008390] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6402worker1 # [ 172.211229] nix-daemon[858]: accepted connection from pid 1989, user nix-grpc-daemon (trusted)6403worker1 # [ 172.228682] nix-daemon[858]: reaped child process 1992, status = succeeded6404worker1 # [ 172.401044] nix-daemon[858]: accepted connection from pid 1994, user nix-grpc-daemon (trusted)6405worker1 # [ 172.417604] nix-daemon[858]: reaped child process 1997, status = succeeded6406lb # [ 172.411269] niks3-server[1147]: 2026/09/21 16:51:40 INFO Received uploads request method=POST path=/api/pending_closures6407worker1 # [ 172.439481] nix-grpc-daemon[1871]: time=2026-09-21T16:51:40.069Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6408worker1 # [ 172.441091] nix-grpc-daemon[1871]: time=2026-09-21T16:51:40.069Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)"6409lb # [ 172.437655] niks3-server[1147]: 2026/09/21 16:51:40 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst6410lb # [ 172.453364] niks3-server[1147]: 2026/09/21 16:51:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6411worker1 # [ 172.478870] nix-grpc-daemon[1871]: time=2026-09-21T16:51:40.109Z level=INFO msg="Uploading 1 narinfos"6412lb # [ 172.465202] niks3-server[1147]: 2026/09/21 16:51:40 INFO Signed narinfos id=60 count=16413lb # [ 172.471630] niks3-server[1147]: 2026/09/21 16:51:40 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls6414lb # [ 172.481797] niks3-server[1147]: 2026/09/21 16:51:40 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6415worker1 # [ 172.505856] nix-grpc-daemon[1871]: time=2026-09-21T16:51:40.136Z level=INFO msg="Upload complete. (483ms)"6416lb # [ 172.486750] niks3-server[1147]: 2026/09/21 16:51:40 INFO Completed upload id=606417lb # [ 172.490358] niks3-server[1147]: 2026/09/21 16:51:40 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo6418worker1 # [ 172.512230] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=0 paths=1 nar_bytes_in=6246419worker1 # [ 172.520924] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186420worker1 # [ 172.531878] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6421worker1 # [ 172.536798] nix-daemon[858]: reaped child process 1987, status = succeeded6422worker1 # [ 172.549751] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6423worker1 # [ 173.391000] nix-daemon[858]: reaped child process 2001, status = succeeded6424worker1 # [ 173.563745] nix-daemon[858]: accepted connection from pid 2017, user nix-grpc-daemon (trusted)6425worker1 # [ 173.583123] nix-daemon[858]: reaped child process 2020, status = succeeded6426lb # [ 173.576855] niks3-server[1147]: 2026/09/21 16:51:41 INFO Received uploads request method=POST path=/api/pending_closures6427worker1 # [ 173.605277] nix-grpc-daemon[1871]: time=2026-09-21T16:51:41.235Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6428worker1 # [ 173.609674] nix-grpc-daemon[1871]: time=2026-09-21T16:51:41.237Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)"6429lb # [ 173.614642] niks3-server[1147]: 2026/09/21 16:51:41 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst6430lb # [ 173.627127] niks3-server[1147]: 2026/09/21 16:51:41 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv6431lb # [ 173.636150] niks3-server[1147]: 2026/09/21 16:51:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6432worker1 # [ 173.661920] nix-grpc-daemon[1871]: time=2026-09-21T16:51:41.291Z level=INFO msg="Uploading 1 narinfos"6433lb # [ 173.642458] niks3-server[1147]: 2026/09/21 16:51:41 INFO Signed narinfos id=61 count=16434lb # [ 173.648133] niks3-server[1147]: 2026/09/21 16:51:41 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls6435lb # [ 173.660249] niks3-server[1147]: 2026/09/21 16:51:41 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6436lb # [ 173.666689] niks3-server[1147]: 2026/09/21 16:51:41 INFO Completed upload id=616437worker1 # [ 173.685604] nix-grpc-daemon[1871]: time=2026-09-21T16:51:41.316Z level=INFO msg="Upload complete. (326ms)"6438lb # [ 173.669852] niks3-server[1147]: 2026/09/21 16:51:41 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo6439worker2 # [ 173.786374] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:41Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=16440worker1 # [ 173.697130] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=4 outputs=16441worker2 # [ 173.797203] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:41Z level=debug event=want client=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=06442worker1 # [ 173.717945] nix-daemon[858]: reaped child process 1999, status = succeeded6443worker1 # [ 173.743392] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186444worker1 # [ 173.758982] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558186445worker1 # [ 173.773886] nix-grpc-daemon[1871]: time=2026-09-21T16:51:41.404Z level=INFO msg="All 1 paths already cached"6446worker1 # [ 173.782847] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558186447worker1 # [ 173.796827] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6448worker1 # [ 173.995459] nix-daemon[858]: accepted connection from pid 2024, user nix-grpc-daemon (trusted)6449worker1 # [ 174.014286] nix-daemon[858]: reaped child process 2027, status = succeeded6450worker1 # [ 174.188674] nix-daemon[858]: accepted connection from pid 2029, user nix-grpc-daemon (trusted)6451worker1 # [ 174.204287] nix-daemon[858]: reaped child process 2032, status = succeeded6452lb # [ 174.199256] niks3-server[1147]: 2026/09/21 16:51:41 INFO Received uploads request method=POST path=/api/pending_closures6453worker1 # [ 174.227134] nix-grpc-daemon[1871]: time=2026-09-21T16:51:41.857Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6454worker1 # [ 174.228703] nix-grpc-daemon[1871]: time=2026-09-21T16:51:41.857Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)"6455lb # [ 174.224502] niks3-server[1147]: 2026/09/21 16:51:41 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst6456worker1 # [ 174.256324] nix-grpc-daemon[1871]: time=2026-09-21T16:51:41.886Z level=INFO msg="Uploading 1 narinfos"6457lb # [ 174.239825] niks3-server[1147]: 2026/09/21 16:51:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6458lb # [ 174.251854] niks3-server[1147]: 2026/09/21 16:51:41 INFO Signed narinfos id=62 count=16459lb # [ 174.257040] niks3-server[1147]: 2026/09/21 16:51:41 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls6460lb # [ 174.265924] niks3-server[1147]: 2026/09/21 16:51:41 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6461lb # [ 174.271259] niks3-server[1147]: 2026/09/21 16:51:41 INFO Completed upload id=626462worker1 # [ 174.284444] nix-grpc-daemon[1871]: time=2026-09-21T16:51:41.914Z level=INFO msg="Upload complete. (471ms)"6463lb # [ 174.274637] niks3-server[1147]: 2026/09/21 16:51:41 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo6464worker1 # [ 174.296352] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=0 paths=1 nar_bytes_in=6246465worker1 # [ 174.305429] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186466worker1 # [ 174.312731] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6467worker1 # [ 174.315204] nix-daemon[858]: reaped child process 2022, status = succeeded6468worker1 # [ 174.327174] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6469worker1 # [ 175.375907] nix-daemon[858]: reaped child process 2036, status = succeeded6470worker1 # [ 175.544391] nix-daemon[858]: accepted connection from pid 2052, user nix-grpc-daemon (trusted)6471worker1 # [ 175.561565] nix-daemon[858]: reaped child process 2055, status = succeeded6472lb # [ 175.556717] niks3-server[1147]: 2026/09/21 16:51:43 INFO Received uploads request method=POST path=/api/pending_closures6473worker1 # [ 175.588432] nix-grpc-daemon[1871]: time=2026-09-21T16:51:43.217Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6474worker1 # [ 175.590193] nix-grpc-daemon[1871]: time=2026-09-21T16:51:43.217Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)"6475lb # [ 175.597113] niks3-server[1147]: 2026/09/21 16:51:43 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst6476lb # [ 175.613808] niks3-server[1147]: 2026/09/21 16:51:43 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv6477lb # [ 175.624228] niks3-server[1147]: 2026/09/21 16:51:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6478worker1 # [ 175.648767] nix-grpc-daemon[1871]: time=2026-09-21T16:51:43.278Z level=INFO msg="Uploading 1 narinfos"6479lb # [ 175.633278] niks3-server[1147]: 2026/09/21 16:51:43 INFO Signed narinfos id=63 count=16480lb # [ 175.635981] niks3-server[1147]: 2026/09/21 16:51:43 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls6481lb # [ 175.644116] niks3-server[1147]: 2026/09/21 16:51:43 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6482lb # [ 175.652298] niks3-server[1147]: 2026/09/21 16:51:43 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo6483lb # [ 175.657660] niks3-server[1147]: 2026/09/21 16:51:43 INFO Completed upload id=636484worker1 # [ 175.676801] nix-grpc-daemon[1871]: time=2026-09-21T16:51:43.307Z level=INFO msg="Upload complete. (339ms)"6485worker2 # [ 175.773046] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:43Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=16486worker1 # [ 175.689025] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=5 outputs=16487worker2 # [ 175.785828] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:43Z level=debug event=want client=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=06488worker1 # [ 175.708979] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186489worker1 # [ 175.720265] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558186490worker1 # [ 175.727031] nix-daemon[858]: reaped child process 2034, status = succeeded6491worker1 # [ 175.730992] nix-grpc-daemon[1871]: time=2026-09-21T16:51:43.360Z level=INFO msg="All 1 paths already cached"6492worker1 # [ 175.735541] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558186493worker1 # [ 175.747645] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6494worker1 # [ 175.932597] nix-daemon[858]: accepted connection from pid 2059, user nix-grpc-daemon (trusted)6495worker1 # [ 175.950530] nix-daemon[858]: reaped child process 2062, status = succeeded6496worker1 # [ 176.117814] nix-daemon[858]: accepted connection from pid 2064, user nix-grpc-daemon (trusted)6497worker1 # [ 176.136581] nix-daemon[858]: reaped child process 2067, status = succeeded6498lb # [ 176.131332] niks3-server[1147]: 2026/09/21 16:51:43 INFO Received uploads request method=POST path=/api/pending_closures6499worker1 # [ 176.159334] nix-grpc-daemon[1871]: time=2026-09-21T16:51:43.789Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6500worker1 # [ 176.160909] nix-grpc-daemon[1871]: time=2026-09-21T16:51:43.789Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)"6501lb # [ 176.160792] niks3-server[1147]: 2026/09/21 16:51:43 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst6502lb # [ 176.173370] niks3-server[1147]: 2026/09/21 16:51:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6503worker1 # [ 176.199513] nix-grpc-daemon[1871]: time=2026-09-21T16:51:43.829Z level=INFO msg="Uploading 1 narinfos"6504lb # [ 176.180603] niks3-server[1147]: 2026/09/21 16:51:43 INFO Signed narinfos id=64 count=16505lb # [ 176.187984] niks3-server[1147]: 2026/09/21 16:51:43 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls6506lb # [ 176.194840] niks3-server[1147]: 2026/09/21 16:51:43 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6507lb # [ 176.202745] niks3-server[1147]: 2026/09/21 16:51:43 INFO Completed upload id=646508worker1 # [ 176.221757] nix-grpc-daemon[1871]: time=2026-09-21T16:51:43.852Z level=INFO msg="Upload complete. (460ms)"6509lb # [ 176.205209] niks3-server[1147]: 2026/09/21 16:51:43 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo6510worker1 # [ 176.228235] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=0 paths=1 nar_bytes_in=6246511worker1 # [ 176.241976] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186512worker1 # [ 176.247927] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6513worker1 # [ 176.253105] nix-daemon[858]: reaped child process 2057, status = succeeded6514worker1 # [ 176.266522] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6515worker1 # [ 177.512615] nix-daemon[858]: reaped child process 2071, status = succeeded6516worker1 # [ 177.684667] nix-daemon[858]: accepted connection from pid 2087, user nix-grpc-daemon (trusted)6517worker1 # [ 177.699445] nix-daemon[858]: reaped child process 2090, status = succeeded6518lb # [ 177.697918] niks3-server[1147]: 2026/09/21 16:51:45 INFO Received uploads request method=POST path=/api/pending_closures6519worker1 # [ 177.728261] nix-grpc-daemon[1871]: time=2026-09-21T16:51:45.355Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6520worker1 # [ 177.730121] nix-grpc-daemon[1871]: time=2026-09-21T16:51:45.355Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)"6521lb # [ 177.732219] niks3-server[1147]: 2026/09/21 16:51:45 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv6522lb # [ 177.744385] niks3-server[1147]: 2026/09/21 16:51:45 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst6523lb # [ 177.754420] niks3-server[1147]: 2026/09/21 16:51:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6524worker1 # [ 177.778716] nix-grpc-daemon[1871]: time=2026-09-21T16:51:45.408Z level=INFO msg="Uploading 1 narinfos"6525lb # [ 177.761811] niks3-server[1147]: 2026/09/21 16:51:45 INFO Signed narinfos id=65 count=16526lb # [ 177.767224] niks3-server[1147]: 2026/09/21 16:51:45 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls6527lb # [ 177.774113] niks3-server[1147]: 2026/09/21 16:51:45 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6528worker1 # [ 177.800346] nix-grpc-daemon[1871]: time=2026-09-21T16:51:45.430Z level=INFO msg="Upload complete. (323ms)"6529lb # [ 177.783041] niks3-server[1147]: 2026/09/21 16:51:45 INFO Completed upload id=656530lb # [ 177.786133] niks3-server[1147]: 2026/09/21 16:51:45 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo6531worker2 # [ 177.901924] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:45Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=16532worker1 # [ 177.813297] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=6 outputs=16533worker2 # [ 177.912587] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:45Z level=debug event=want client=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=06534worker1 # [ 177.832305] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186535worker1 # [ 177.839335] nix-daemon[858]: reaped child process 2069, status = succeeded6536worker1 # [ 177.847147] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558186537worker1 # [ 177.860524] nix-grpc-daemon[1871]: time=2026-09-21T16:51:45.491Z level=INFO msg="All 2 paths already cached"6538worker1 # [ 177.868875] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558186539worker1 # [ 177.881608] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6540worker1 # [ 178.074587] nix-daemon[858]: accepted connection from pid 2094, user nix-grpc-daemon (trusted)6541worker1 # [ 178.092834] nix-daemon[858]: reaped child process 2097, status = succeeded6542worker1 # [ 178.262748] nix-daemon[858]: accepted connection from pid 2099, user nix-grpc-daemon (trusted)6543worker1 # [ 178.280158] nix-daemon[858]: reaped child process 2102, status = succeeded6544lb # [ 178.272422] niks3-server[1147]: 2026/09/21 16:51:45 INFO Received uploads request method=POST path=/api/pending_closures6545worker1 # [ 178.300100] nix-grpc-daemon[1871]: time=2026-09-21T16:51:45.930Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6546worker1 # [ 178.301556] nix-grpc-daemon[1871]: time=2026-09-21T16:51:45.930Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)"6547lb # [ 178.297643] niks3-server[1147]: 2026/09/21 16:51:45 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst6548lb # [ 178.311408] niks3-server[1147]: 2026/09/21 16:51:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign6549worker1 # [ 178.336726] nix-grpc-daemon[1871]: time=2026-09-21T16:51:45.967Z level=INFO msg="Uploading 1 narinfos"6550lb # [ 178.322180] niks3-server[1147]: 2026/09/21 16:51:45 INFO Signed narinfos id=66 count=16551lb # [ 178.328723] niks3-server[1147]: 2026/09/21 16:51:45 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls6552lb # [ 178.334545] niks3-server[1147]: 2026/09/21 16:51:45 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete6553lb # [ 178.342218] niks3-server[1147]: 2026/09/21 16:51:45 INFO Completed upload id=666554worker1 # [ 178.361268] nix-grpc-daemon[1871]: time=2026-09-21T16:51:45.991Z level=INFO msg="Upload complete. (465ms)"6555lb # [ 178.344962] niks3-server[1147]: 2026/09/21 16:51:45 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo6556worker1 # [ 178.368294] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=0 paths=1 nar_bytes_in=7526557worker1 # [ 178.381375] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186558worker1 # [ 178.389804] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6559worker1 # [ 178.392383] nix-daemon[858]: reaped child process 2092, status = succeeded6560worker1 # [ 178.404545] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6561worker1 # [ 179.631151] nix-daemon[858]: reaped child process 2106, status = succeeded6562worker1 # [ 179.796404] nix-daemon[858]: accepted connection from pid 2122, user nix-grpc-daemon (trusted)6563worker1 # [ 179.814403] nix-daemon[858]: reaped child process 2125, status = succeeded6564lb # [ 179.807603] niks3-server[1147]: 2026/09/21 16:51:47 INFO Received uploads request method=POST path=/api/pending_closures6565worker1 # [ 179.837468] nix-grpc-daemon[1871]: time=2026-09-21T16:51:47.466Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6566worker1 # [ 179.838911] nix-grpc-daemon[1871]: time=2026-09-21T16:51:47.466Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)"6567lb # [ 179.841581] niks3-server[1147]: 2026/09/21 16:51:47 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv6568lb # [ 179.855423] niks3-server[1147]: 2026/09/21 16:51:47 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst6569worker1 # [ 179.888041] nix-grpc-daemon[1871]: time=2026-09-21T16:51:47.518Z level=INFO msg="Uploading 1 narinfos"6570lb # [ 179.868962] niks3-server[1147]: 2026/09/21 16:51:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign6571lb # [ 179.878613] niks3-server[1147]: 2026/09/21 16:51:47 INFO Signed narinfos id=67 count=16572lb # [ 179.882149] niks3-server[1147]: 2026/09/21 16:51:47 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls6573worker1 # [ 179.908090] nix-grpc-daemon[1871]: time=2026-09-21T16:51:47.538Z level=INFO msg="Upload complete. (294ms)"6574lb # [ 179.890411] niks3-server[1147]: 2026/09/21 16:51:47 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete6575lb # [ 179.895436] niks3-server[1147]: 2026/09/21 16:51:47 INFO Completed upload id=676576lb # [ 179.898472] niks3-server[1147]: 2026/09/21 16:51:47 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo6577worker1 # [ 179.916754] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=7 outputs=16578worker2 # [ 180.004536] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:47Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=16579worker1 # [ 179.932658] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186580worker1 # [ 179.937939] nix-daemon[858]: reaped child process 2104, status = succeeded6581worker1 # [ 179.944596] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558186582worker1 # [ 179.955286] nix-grpc-daemon[1871]: time=2026-09-21T16:51:47.585Z level=INFO msg="All 2 paths already cached"6583worker1 # [ 179.962722] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558186584worker1 # [ 179.973116] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6585worker1 # [ 180.161597] nix-daemon[858]: accepted connection from pid 2129, user nix-grpc-daemon (trusted)6586worker1 # [ 180.178381] nix-daemon[858]: reaped child process 2132, status = succeeded6587worker1 # [ 180.348242] nix-daemon[858]: accepted connection from pid 2134, user nix-grpc-daemon (trusted)6588worker1 # [ 180.364862] nix-daemon[858]: reaped child process 2137, status = succeeded6589lb # [ 180.356892] niks3-server[1147]: 2026/09/21 16:51:47 INFO Received uploads request method=POST path=/api/pending_closures6590worker1 # [ 180.384066] nix-grpc-daemon[1871]: time=2026-09-21T16:51:48.014Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6591worker1 # [ 180.385512] nix-grpc-daemon[1871]: time=2026-09-21T16:51:48.014Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)"6592lb # [ 180.384146] niks3-server[1147]: 2026/09/21 16:51:48 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst6593lb # [ 180.398765] niks3-server[1147]: 2026/09/21 16:51:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign6594worker1 # [ 180.424076] nix-grpc-daemon[1871]: time=2026-09-21T16:51:48.054Z level=INFO msg="Uploading 1 narinfos"6595lb # [ 180.409384] niks3-server[1147]: 2026/09/21 16:51:48 INFO Signed narinfos id=68 count=16596lb # [ 180.416096] niks3-server[1147]: 2026/09/21 16:51:48 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls6597lb # [ 180.423517] niks3-server[1147]: 2026/09/21 16:51:48 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete6598lb # [ 180.429741] niks3-server[1147]: 2026/09/21 16:51:48 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo6599lb # [ 180.435076] niks3-server[1147]: 2026/09/21 16:51:48 INFO Completed upload id=686600worker1 # [ 180.454219] nix-grpc-daemon[1871]: time=2026-09-21T16:51:48.084Z level=INFO msg="Upload complete. (468ms)"6601worker1 # [ 180.464272] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=0 paths=1 nar_bytes_in=7526602worker1 # [ 180.475861] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186603worker1 # [ 180.484891] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6604worker1 # [ 180.487651] nix-daemon[858]: reaped child process 2127, status = succeeded6605worker1 # [ 180.500761] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6606worker1 # [ 181.945929] nix-daemon[858]: reaped child process 2141, status = succeeded6607worker1 # [ 182.127944] nix-daemon[858]: accepted connection from pid 2157, user nix-grpc-daemon (trusted)6608worker1 # [ 182.146952] nix-daemon[858]: reaped child process 2160, status = succeeded6609lb # [ 182.140672] niks3-server[1147]: 2026/09/21 16:51:49 INFO Received uploads request method=POST path=/api/pending_closures6610worker1 # [ 182.169064] nix-grpc-daemon[1871]: time=2026-09-21T16:51:49.799Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6611worker1 # [ 182.171753] nix-grpc-daemon[1871]: time=2026-09-21T16:51:49.801Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)"6612lb # [ 182.174707] niks3-server[1147]: 2026/09/21 16:51:49 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv6613lb # [ 182.187801] niks3-server[1147]: 2026/09/21 16:51:49 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst6614lb # [ 182.198653] niks3-server[1147]: 2026/09/21 16:51:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign6615worker1 # [ 182.224044] nix-grpc-daemon[1871]: time=2026-09-21T16:51:49.854Z level=INFO msg="Uploading 1 narinfos"6616lb # [ 182.208747] niks3-server[1147]: 2026/09/21 16:51:49 INFO Signed narinfos id=69 count=16617lb # [ 182.211767] niks3-server[1147]: 2026/09/21 16:51:49 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls6618lb # [ 182.223766] niks3-server[1147]: 2026/09/21 16:51:49 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete6619lb # [ 182.230586] niks3-server[1147]: 2026/09/21 16:51:49 INFO Completed upload id=696620worker1 # [ 182.249701] nix-grpc-daemon[1871]: time=2026-09-21T16:51:49.880Z level=INFO msg="Upload complete. (338ms)"6621lb # [ 182.233719] niks3-server[1147]: 2026/09/21 16:51:49 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo6622worker2 # [ 182.347355] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:49Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=16623worker1 # [ 182.261067] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=8 outputs=16624worker1 # [ 182.279983] nix-daemon[858]: reaped child process 2139, status = succeeded6625worker1 # [ 182.312217] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186626worker1 # [ 182.326932] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558186627worker1 # [ 182.342608] nix-grpc-daemon[1871]: time=2026-09-21T16:51:49.973Z level=INFO msg="All 2 paths already cached"6628worker1 # [ 182.351592] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558186629worker1 # [ 182.365593] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6630worker1 # [ 182.562272] nix-daemon[858]: accepted connection from pid 2164, user nix-grpc-daemon (trusted)6631worker1 # [ 182.579273] nix-daemon[858]: reaped child process 2167, status = succeeded6632worker1 # [ 182.751261] nix-daemon[858]: accepted connection from pid 2169, user nix-grpc-daemon (trusted)6633worker1 # [ 182.766617] nix-daemon[858]: reaped child process 2172, status = succeeded6634lb # [ 182.758843] niks3-server[1147]: 2026/09/21 16:51:50 INFO Received uploads request method=POST path=/api/pending_closures6635worker1 # [ 182.786332] nix-grpc-daemon[1871]: time=2026-09-21T16:51:50.416Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6636worker1 # [ 182.787772] nix-grpc-daemon[1871]: time=2026-09-21T16:51:50.416Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)"6637lb # [ 182.786876] niks3-server[1147]: 2026/09/21 16:51:50 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst6638lb # [ 182.799040] niks3-server[1147]: 2026/09/21 16:51:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign6639worker1 # [ 182.825268] nix-grpc-daemon[1871]: time=2026-09-21T16:51:50.455Z level=INFO msg="Uploading 1 narinfos"6640lb # [ 182.806448] niks3-server[1147]: 2026/09/21 16:51:50 INFO Signed narinfos id=70 count=16641lb # [ 182.814751] niks3-server[1147]: 2026/09/21 16:51:50 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls6642lb # [ 182.830151] niks3-server[1147]: 2026/09/21 16:51:50 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete6643worker1 # [ 182.854787] nix-grpc-daemon[1871]: time=2026-09-21T16:51:50.485Z level=INFO msg="Upload complete. (474ms)"6644lb # [ 182.838698] niks3-server[1147]: 2026/09/21 16:51:50 INFO Completed upload id=706645lb # [ 182.843143] niks3-server[1147]: 2026/09/21 16:51:50 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo6646worker1 # [ 182.864234] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=0 paths=1 nar_bytes_in=7526647worker1 # [ 182.876821] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558186648worker1 # [ 182.883379] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6649worker1 # [ 182.888807] nix-daemon[858]: reaped child process 2162, status = succeeded6650worker1 # [ 182.902476] nix-daemon[858]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6651worker1 # [ 184.347499] nix-daemon[858]: reaped child process 2176, status = succeeded6652worker1 # [ 184.517418] nix-daemon[858]: accepted connection from pid 2192, user nix-grpc-daemon (trusted)6653worker1 # [ 184.533804] nix-daemon[858]: reaped child process 2195, status = succeeded6654lb # [ 184.527332] niks3-server[1147]: 2026/09/21 16:51:52 INFO Received uploads request method=POST path=/api/pending_closures6655worker1 # [ 184.558797] nix-grpc-daemon[1871]: time=2026-09-21T16:51:52.187Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6656worker1 # [ 184.560567] nix-grpc-daemon[1871]: time=2026-09-21T16:51:52.187Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)"6657lb # [ 184.558341] niks3-server[1147]: 2026/09/21 16:51:52 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv6658lb # [ 184.577317] niks3-server[1147]: 2026/09/21 16:51:52 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst6659lb # [ 184.587354] niks3-server[1147]: 2026/09/21 16:51:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign6660lb # [ 184.592628] niks3-server[1147]: 2026/09/21 16:51:52 INFO Signed narinfos id=71 count=16661worker1 # [ 184.611750] nix-grpc-daemon[1871]: time=2026-09-21T16:51:52.241Z level=INFO msg="Uploading 1 narinfos"6662lb # [ 184.596941] niks3-server[1147]: 2026/09/21 16:51:52 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls6663worker1 # [ 184.625438] nix-grpc-daemon[1871]: time=2026-09-21T16:51:52.255Z level=INFO msg="Upload complete. (312ms)"6664lb # [ 184.606297] niks3-server[1147]: 2026/09/21 16:51:52 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete6665lb # [ 184.612882] niks3-server[1147]: 2026/09/21 16:51:52 INFO Completed upload id=716666lb # [ 184.615455] niks3-server[1147]: 2026/09/21 16:51:52 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo6667worker1 # [ 184.637161] nix-grpc-daemon[1811]: ts=2026-09-21T16:51:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55818 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=9 outputs=16668worker1 # [ 184.653026] nix-daemon[858]: reaped child process 2174, status = succeeded6669client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 21.07 seconds)6670worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6671worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6672client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features '["gpu"]'6673client # [ 184.895777] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6674client # Running as unit: gpu.service; invocation ID: 0f3951748ca745e6bdf5b18ff482cf376675client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds)6676lb: must succeed: curl -sf localhost:9901/clusters6677lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6678lb: must succeed: curl -sf localhost:9901/clusters6679lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6680worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6681worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6682client # [ 185.143711] nix[1432]: don't know how to build these paths:6683client # [ 185.144875] nix[1432]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv6684worker2 # [ 185.233163] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:52Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:593426685worker2 # [ 185.238376] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:593426686client # [ 185.172326] nix[1432]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, waiting up to 30s for a worker6687worker2 # [ 185.261623] nix-grpc-daemon[1485]: ts=2026-09-21T16:51:52Z level=debug event=want client=2 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=06688lb: must succeed: curl -sf localhost:9901/clusters6689lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6690lb: must succeed: curl -sf localhost:9901/clusters6691lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6692worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6693worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6694client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6695lb # [ 190.859069] envoy[926]: conn peer=[2001:db8:1::1]:42268 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27204 tx=11425 ms=207256696client # [ 215.173774] nix[1432]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}6697client # [ 215.183383] nix[1432]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed6698client # [ 215.229835] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6699client # [ 215.237252] systemd[1]: gpu.service: Failed with result 'exit-code'.6700client # [ 215.241047] systemd[1]: gpu.service: Consumed 171ms CPU time over 30.339s wall clock time, 35.4M memory peak, 5.1K incoming IP traffic, 5K outgoing IP traffic.6701client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.24 seconds)6702client: must succeed: journalctl -u gpu -o cat6703client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)6704lb: must succeed: curl -sf localhost:9901/clusters6705lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6706lb: must succeed: curl -sf localhost:9901/clusters6707lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6708worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6709worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6710(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 52.15 seconds)6711subtest: low disk drains a worker and builds go to the other6712worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6713worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.13 seconds)6714worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6715worker1 # [ 217.366502] nix-grpc-daemon[1811]: ts=2026-09-21T16:52:24Z level=info event=unhealthy reason=min_free6716worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.09 seconds)6717client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain >&26718client # don't know how to build these paths:6719client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6720worker2 # [ 218.414469] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:593426721worker1 # [ 218.330502] nix-grpc-daemon[1811]: ts=2026-09-21T16:52:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558186722worker2 # [ 218.446215] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:26Z level=debug event=want client=3 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=06723worker2 # [ 218.453599] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:26Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=16724worker2 # [ 218.490836] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593426725worker2 # [ 218.506011] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:593426726client # copying 1 paths to 'lb:50051'...6727client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'...6728worker2 # [ 218.522088] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:593426729worker2 # [ 218.536890] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6730worker2 # [ 218.739343] nix-daemon[849]: accepted connection from pid 1792, user nix-grpc-daemon (trusted)6731worker2 # [ 218.758187] nix-daemon[849]: reaped child process 1795, status = succeeded6732worker2 # [ 218.928178] nix-daemon[849]: accepted connection from pid 1797, user nix-grpc-daemon (trusted)6733worker2 # [ 218.946177] nix-daemon[849]: reaped child process 1800, status = succeeded6734lb # [ 218.851842] niks3-server[1147]: 2026/09/21 16:52:26 INFO Received uploads request method=POST path=/api/pending_closures6735worker2 # [ 218.970679] nix-grpc-daemon[1585]: time=2026-09-21T16:52:26.606Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6736worker2 # [ 218.972267] nix-grpc-daemon[1585]: time=2026-09-21T16:52:26.606Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)"6737lb # [ 218.884514] niks3-server[1147]: 2026/09/21 16:52:26 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst6738lb # [ 218.898621] niks3-server[1147]: 2026/09/21 16:52:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign6739worker2 # [ 219.013617] nix-grpc-daemon[1585]: time=2026-09-21T16:52:26.650Z level=INFO msg="Uploading 1 narinfos"6740lb # [ 218.912486] niks3-server[1147]: 2026/09/21 16:52:26 INFO Signed narinfos id=72 count=16741lb # [ 218.915408] niks3-server[1147]: 2026/09/21 16:52:26 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls6742lb # [ 218.930984] niks3-server[1147]: 2026/09/21 16:52:26 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete6743lb # [ 218.936414] niks3-server[1147]: 2026/09/21 16:52:26 INFO Completed upload id=726744worker2 # [ 219.042967] nix-grpc-daemon[1585]: time=2026-09-21T16:52:26.679Z level=INFO msg="Upload complete. (489ms)"6745lb # [ 218.939487] niks3-server[1147]: 2026/09/21 16:52:26 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo6746worker2 # [ 219.052295] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59342 duration_s=0 paths=1 nar_bytes_in=5126747worker2 # [ 219.063574] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593426748worker2 # [ 219.073447] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6749worker2 # [ 219.076327] nix-daemon[849]: reaped child process 1790, status = succeeded6750client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051...6751worker2 # [ 219.088503] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6752client # farm-a> worker2: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6753client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6754client # farm-a> LOG-farm-a-drain6755worker2 # [ 219.935572] nix-daemon[849]: reaped child process 1804, status = succeeded6756worker2 # [ 220.113797] nix-daemon[849]: accepted connection from pid 1820, user nix-grpc-daemon (trusted)6757worker2 # [ 220.132505] nix-daemon[849]: reaped child process 1823, status = succeeded6758lb # [ 220.043275] niks3-server[1147]: 2026/09/21 16:52:27 INFO Received uploads request method=POST path=/api/pending_closures6759worker2 # [ 220.160262] nix-grpc-daemon[1585]: time=2026-09-21T16:52:27.793Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6760worker2 # [ 220.162091] nix-grpc-daemon[1585]: time=2026-09-21T16:52:27.793Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)"6761lb # [ 220.078357] niks3-server[1147]: 2026/09/21 16:52:27 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6762worker2 # [ 220.204735] nix-grpc-daemon[1585]: time=2026-09-21T16:52:27.840Z level=INFO msg="Uploading 1 narinfos"6763lb # [ 220.089492] niks3-server[1147]: 2026/09/21 16:52:27 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6764lb # [ 220.102826] niks3-server[1147]: 2026/09/21 16:52:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign6765lb # [ 220.108650] niks3-server[1147]: 2026/09/21 16:52:27 INFO Signed narinfos id=73 count=16766worker2 # [ 220.220411] nix-grpc-daemon[1585]: time=2026-09-21T16:52:27.857Z level=INFO msg="Upload complete. (319ms)"6767lb # [ 220.115058] niks3-server[1147]: 2026/09/21 16:52:27 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls6768lb # [ 220.119221] niks3-server[1147]: 2026/09/21 16:52:27 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete6769lb # [ 220.123662] niks3-server[1147]: 2026/09/21 16:52:27 INFO Completed upload id=736770lb # [ 220.126387] niks3-server[1147]: 2026/09/21 16:52:27 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo6771worker2 # [ 220.231629] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59342 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=10 outputs=16772worker2 # [ 220.253410] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:27Z level=debug event=want client=3 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=06773worker2 # [ 220.260200] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:27Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=16774worker2 # [ 220.264531] nix-daemon[849]: reaped child process 1802, status = succeeded6775worker2 # [ 220.299373] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593426776worker2 # [ 220.314220] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:593426777worker2 # [ 220.329018] nix-grpc-daemon[1585]: time=2026-09-21T16:52:27.965Z level=INFO msg="All 1 paths already cached"6778client # copying 1 paths to 'lb:50051'...6779client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'...6780worker2 # [ 220.338811] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:593426781worker2 # [ 220.352482] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6782worker2 # [ 220.553495] nix-daemon[849]: accepted connection from pid 1829, user nix-grpc-daemon (trusted)6783worker2 # [ 220.570790] nix-daemon[849]: reaped child process 1832, status = succeeded6784worker2 # [ 220.744193] nix-daemon[849]: accepted connection from pid 1834, user nix-grpc-daemon (trusted)6785worker2 # [ 220.759429] nix-daemon[849]: reaped child process 1837, status = succeeded6786lb # [ 220.667720] niks3-server[1147]: 2026/09/21 16:52:28 INFO Received uploads request method=POST path=/api/pending_closures6787worker2 # [ 220.784058] nix-grpc-daemon[1585]: time=2026-09-21T16:52:28.420Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6788worker2 # [ 220.785576] nix-grpc-daemon[1585]: time=2026-09-21T16:52:28.420Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)"6789lb # [ 220.697399] niks3-server[1147]: 2026/09/21 16:52:28 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst6790lb # [ 220.710629] niks3-server[1147]: 2026/09/21 16:52:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign6791worker2 # [ 220.823791] nix-grpc-daemon[1585]: time=2026-09-21T16:52:28.460Z level=INFO msg="Uploading 1 narinfos"6792lb # [ 220.722658] niks3-server[1147]: 2026/09/21 16:52:28 INFO Signed narinfos id=74 count=16793lb # [ 220.728197] niks3-server[1147]: 2026/09/21 16:52:28 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls6794worker2 # [ 220.841371] nix-grpc-daemon[1585]: time=2026-09-21T16:52:28.478Z level=INFO msg="Upload complete. (472ms)"6795lb # [ 220.735333] niks3-server[1147]: 2026/09/21 16:52:28 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete6796lb # [ 220.741605] niks3-server[1147]: 2026/09/21 16:52:28 INFO Completed upload id=746797worker2 # [ 220.848139] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59342 duration_s=0 paths=1 nar_bytes_in=6326798lb # [ 220.745442] niks3-server[1147]: 2026/09/21 16:52:28 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo6799worker2 # [ 220.860379] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593426800worker2 # [ 220.866360] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6801worker2 # [ 220.870787] nix-daemon[849]: reaped child process 1827, status = succeeded6802client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051...6803client # farm-b> worker2: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6804worker2 # [ 220.883429] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6805lb # [ 220.860704] envoy[926]: conn peer=[2001:db8:1::1]:52656 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=897 tx=499 ms=300576806client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6807client # farm-b> LOG-farm-b-drain6808worker2 # [ 221.502939] nix-daemon[849]: reaped child process 1841, status = succeeded6809worker2 # [ 221.662025] nix-daemon[849]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6810worker2 # [ 221.679335] nix-daemon[849]: reaped child process 1860, status = succeeded6811lb # [ 221.589818] niks3-server[1147]: 2026/09/21 16:52:29 INFO Received uploads request method=POST path=/api/pending_closures6812worker2 # [ 221.707624] nix-grpc-daemon[1585]: time=2026-09-21T16:52:29.340Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6813worker2 # [ 221.709247] nix-grpc-daemon[1585]: time=2026-09-21T16:52:29.341Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)"6814lb # [ 221.620285] niks3-server[1147]: 2026/09/21 16:52:29 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst6815lb # [ 221.636572] niks3-server[1147]: 2026/09/21 16:52:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign6816worker2 # [ 221.749869] nix-grpc-daemon[1585]: time=2026-09-21T16:52:29.386Z level=INFO msg="Uploading 1 narinfos"6817lb # [ 221.649486] niks3-server[1147]: 2026/09/21 16:52:29 INFO Signed narinfos id=75 count=16818lb # [ 221.651951] niks3-server[1147]: 2026/09/21 16:52:29 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6819lb # [ 221.658892] niks3-server[1147]: 2026/09/21 16:52:29 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls6820lb # [ 221.666074] niks3-server[1147]: 2026/09/21 16:52:29 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete6821lb # [ 221.671128] niks3-server[1147]: 2026/09/21 16:52:29 INFO Completed upload id=756822worker2 # [ 221.777537] nix-grpc-daemon[1585]: time=2026-09-21T16:52:29.414Z level=INFO msg="Upload complete. (284ms)"6823lb # [ 221.673835] niks3-server[1147]: 2026/09/21 16:52:29 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo6824worker2 # [ 221.792609] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59342 duration_s=0 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=11 outputs=16825worker2 # [ 221.804762] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:29Z level=debug event=want client=3 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=06826worker2 # [ 221.808249] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:29Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=16827worker2 # [ 221.811563] nix-daemon[849]: reaped child process 1839, status = succeeded6828worker2 # [ 221.839116] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593426829worker2 # [ 221.853883] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:593426830worker2 # [ 221.868083] nix-grpc-daemon[1585]: time=2026-09-21T16:52:29.504Z level=INFO msg="All 2 paths already cached"6831client # copying 1 paths to 'lb:50051'...6832client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'...6833worker2 # [ 221.878378] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:593426834worker2 # [ 221.891940] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6835worker2 # [ 222.100915] nix-daemon[849]: accepted connection from pid 1866, user nix-grpc-daemon (trusted)6836worker2 # [ 222.119559] nix-daemon[849]: reaped child process 1869, status = succeeded6837worker2 # [ 222.292121] nix-daemon[849]: accepted connection from pid 1871, user nix-grpc-daemon (trusted)6838worker2 # [ 222.307142] nix-daemon[849]: reaped child process 1874, status = succeeded6839lb # [ 222.215511] niks3-server[1147]: 2026/09/21 16:52:29 INFO Received uploads request method=POST path=/api/pending_closures6840worker2 # [ 222.330969] nix-grpc-daemon[1585]: time=2026-09-21T16:52:29.967Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6841worker2 # [ 222.332563] nix-grpc-daemon[1585]: time=2026-09-21T16:52:29.967Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)"6842lb # [ 222.240556] niks3-server[1147]: 2026/09/21 16:52:29 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst6843worker2 # [ 222.361545] nix-grpc-daemon[1585]: time=2026-09-21T16:52:29.998Z level=INFO msg="Uploading 1 narinfos"6844lb # [ 222.256458] niks3-server[1147]: 2026/09/21 16:52:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign6845lb # [ 222.268382] niks3-server[1147]: 2026/09/21 16:52:29 INFO Signed narinfos id=76 count=16846lb # [ 222.279282] niks3-server[1147]: 2026/09/21 16:52:29 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls6847lb # [ 222.283721] niks3-server[1147]: 2026/09/21 16:52:29 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete6848worker2 # [ 222.395700] nix-grpc-daemon[1585]: time=2026-09-21T16:52:30.032Z level=INFO msg="Upload complete. (487ms)"6849lb # [ 222.289624] niks3-server[1147]: 2026/09/21 16:52:29 INFO Completed upload id=766850lb # [ 222.293624] niks3-server[1147]: 2026/09/21 16:52:29 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo6851worker2 # [ 222.404227] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59342 duration_s=0 paths=1 nar_bytes_in=7606852worker2 # [ 222.412998] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:593426853worker2 # [ 222.421145] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6854worker2 # [ 222.423704] nix-daemon[849]: reaped child process 1864, status = succeeded6855client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051...6856worker2 # [ 222.434176] nix-daemon[849]: accepted connection from pid 1485, user nix-grpc-daemon (trusted)6857client # farm-top> worker2: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6858client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6859client # farm-top> LOG-farm-top-drain6860worker2 # [ 224.064177] nix-daemon[849]: reaped child process 1878, status = succeeded6861worker2 # [ 224.235784] nix-daemon[849]: accepted connection from pid 1894, user nix-grpc-daemon (trusted)6862worker2 # [ 224.253955] nix-daemon[849]: reaped child process 1897, status = succeeded6863lb # [ 224.161964] niks3-server[1147]: 2026/09/21 16:52:31 INFO Received uploads request method=POST path=/api/pending_closures6864worker2 # [ 224.279581] nix-grpc-daemon[1585]: time=2026-09-21T16:52:31.915Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6865worker2 # [ 224.281275] nix-grpc-daemon[1585]: time=2026-09-21T16:52:31.915Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)"6866lb # [ 224.198050] niks3-server[1147]: 2026/09/21 16:52:31 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst6867lb # [ 224.213686] niks3-server[1147]: 2026/09/21 16:52:31 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6868lb # [ 224.224928] niks3-server[1147]: 2026/09/21 16:52:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign6869worker2 # [ 224.336957] nix-grpc-daemon[1585]: time=2026-09-21T16:52:31.973Z level=INFO msg="Uploading 1 narinfos"6870lb # [ 224.235554] niks3-server[1147]: 2026/09/21 16:52:31 INFO Signed narinfos id=77 count=16871lb # [ 224.240656] niks3-server[1147]: 2026/09/21 16:52:31 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls6872worker2 # [ 224.355023] nix-grpc-daemon[1585]: time=2026-09-21T16:52:31.991Z level=INFO msg="Upload complete. (308ms)"6873lb # [ 224.248651] niks3-server[1147]: 2026/09/21 16:52:31 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete6874lb # [ 224.253721] niks3-server[1147]: 2026/09/21 16:52:31 INFO Completed upload id=776875lb # [ 224.256311] niks3-server[1147]: 2026/09/21 16:52:31 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo6876worker2 # [ 224.368994] nix-grpc-daemon[1485]: ts=2026-09-21T16:52:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59342 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=12 outputs=16877worker2 # [ 224.382128] nix-daemon[849]: reaped child process 1876, status = succeeded6878client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain >&2, in 6.27 seconds)6879client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain outPath6880client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain outPath, in 0.23 seconds)6881worker1: must succeed: rm /nix/.rw-store/fill6882worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.16 seconds)6883worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'6884worker1 # [ 225.374517] nix-grpc-daemon[1811]: ts=2026-09-21T16:52:33Z level=info event=healthy reason=min_free6885worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.12 seconds)6886(finished: subtest: low disk drains a worker and builds go to the other, in 10.10 seconds)6887(finished: run the VM test script, in 226.84 seconds)6888test script finished in 227.13s6889cleanup6890kill QemuMachine (pid 49)6891client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6892kill QemuMachine (pid 50)6893lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6894kill QemuMachine (pid 51)6895worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6896kill QemuMachine (pid 52)6897worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6898(finished: cleanup, in 1.11 seconds)6899additionally exposed symbols:6900 client, lb, worker1, worker2,6901 vlan1,6902 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh