vm-test-run-nix-grpc-farm
checks.aarch64-linux.farm
· build #495
· raw
1tribuchet: building on eliza2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker1 # Disk image does not exist, creating the virtualisation disk image...13worker2 # Disk image does not exist, creating the virtualisation disk image...14worker2: QEMU running (pid 52)15worker1: QEMU running (pid 51)16worker1 # Formatting '/build/vm-state-worker1/tmp.xsLVQJybNj', fmt=raw size=107374182417lb: QEMU running (pid 50)18client # Disk image does not exist, creating the virtualisation disk image...19client: QEMU running (pid 49)20worker2 # Formatting '/build/vm-state-worker2/tmp.NMX0iu6wHb', fmt=raw size=107374182421client # Formatting '/build/vm-state-client/tmp.yQbpMxBhEa', fmt=raw size=107374182422lb # Disk image does not exist, creating the virtualisation disk image...23worker1 # mke2fs 1.47.4 (6-Mar-2025)24worker2 # mke2fs 1.47.4 (6-Mar-2025)25(finished: start all VMs, in 0.51 seconds)26worker1 # Discarding device blocks: 0/262144 done27worker2 # Discarding device blocks: 0/262144 done28client: waiting for unit network-addresses-eth1.service29worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes30worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes31client: waiting for the VM to finish booting32worker1 # Filesystem UUID: 85753b96-c7ba-477e-964a-82432a2541e533worker2 # Filesystem UUID: e2934ba0-53e2-4b9d-9be0-9be4a939343634lb # Formatting '/build/vm-state-lb/tmp.8UihXkIwnz', fmt=raw size=107374182435worker1 # Superblock backups stored on blocks:36worker2 # Superblock backups stored on blocks:37client # mke2fs 1.47.4 (6-Mar-2025)38worker2 # 32768, 98304, 163840, 22937639client # Discarding device blocks: 0/262144 done40worker1 # 32768, 98304, 163840, 22937641lb # mke2fs 1.47.4 (6-Mar-2025)42worker2 # 43client # Creating filesystem with 262144 4k blocks and 65536 inodes44worker1 # 45lb # Discarding device blocks: 0/262144 done46worker2 # Allocating group tables: 0/8 done47client # Filesystem UUID: 029cd0ef-f3c7-49c5-af97-28279bde576d48worker1 # Allocating group tables: 0/8 done49lb # Creating filesystem with 262144 4k blocks and 65536 inodes50worker2 # Writing inode tables: 0/8 done51client # Superblock backups stored on blocks:52worker1 # Writing inode tables: 0/8 done53lb # Filesystem UUID: 14220a4a-ec47-41b7-9745-bca8042d2bff54worker2 # Creating journal (8192 blocks): done55client # 32768, 98304, 163840, 22937656worker1 # Creating journal (8192 blocks): done57lb # Superblock backups stored on blocks:58worker2 # Writing superblocks and filesystem accounting information: 0/8 done59client # 60worker1 # Writing superblocks and filesystem accounting information: 0/8 done61lb # 32768, 98304, 163840, 22937662worker2 # 63client # Allocating group tables: 0/8 done64worker1 # 65lb # 66worker2 # Virtualisation disk image created.67client # Writing inode tables: 0/8 done68worker1 # Virtualisation disk image created.69lb # Allocating group tables: 0/8 done70worker2 # Starting virtiofs daemons...71client # Creating journal (8192 blocks): done72worker1 # Starting virtiofs daemons...73lb # Writing inode tables: 0/8 done74worker2 # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)75client # Writing superblocks and filesystem accounting information: 0/8 done76worker1 # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)77lb # Creating journal (8192 blocks): done78worker2 # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether79client # 80worker1 # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether81lb # Writing superblocks and filesystem accounting information: 0/8 done82worker2 # [2026-09-22T11:24:58Z INFO virtiofsd] Waiting for vhost-user socket connection...83client # Virtualisation disk image created.84worker1 # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)85lb # 86worker2 # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)87client # Starting virtiofs daemons...88worker1 # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether89lb # Virtualisation disk image created.90worker2 # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether91client # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)92worker1 # [2026-09-22T11:24:58Z INFO virtiofsd] Waiting for vhost-user socket connection...93lb # Starting virtiofs daemons...94worker2 # [2026-09-22T11:24:58Z INFO virtiofsd] Waiting for vhost-user socket connection...95client # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether96worker1 # [2026-09-22T11:24:58Z INFO virtiofsd] Waiting for vhost-user socket connection...97lb # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)98worker2 # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)99client # [2026-09-22T11:24:58Z INFO virtiofsd] Waiting for vhost-user socket connection...100worker1 # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)101lb # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether102worker2 # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether103client # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)104worker1 # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether105lb # [2026-09-22T11:24:58Z INFO virtiofsd] Waiting for vhost-user socket connection...106worker2 # [2026-09-22T11:24:58Z INFO virtiofsd] Waiting for vhost-user socket connection...107client # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether108worker1 # [2026-09-22T11:24:58Z INFO virtiofsd] Waiting for vhost-user socket connection...109lb # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)110worker2 # [2026-09-22T11:24:58Z INFO virtiofsd] Client connected, servicing requests111client # [2026-09-22T11:24:58Z INFO virtiofsd] Waiting for vhost-user socket connection...112worker1 # [2026-09-22T11:24:58Z INFO virtiofsd] Client connected, servicing requests113lb # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether114worker2 # [2026-09-22T11:24:58Z INFO virtiofsd] Client connected, servicing requests115client # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)116worker1 # [2026-09-22T11:24:58Z INFO virtiofsd] Client connected, servicing requests117lb # [2026-09-22T11:24:58Z INFO virtiofsd] Waiting for vhost-user socket connection...118worker2 # [2026-09-22T11:24:58Z INFO virtiofsd] Client connected, servicing requests119client # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether120worker1 # [2026-09-22T11:24:58Z INFO virtiofsd] Client connected, servicing requests121client # [2026-09-22T11:24:58Z INFO virtiofsd] Waiting for vhost-user socket connection...122lb # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)123client # [2026-09-22T11:24:58Z INFO virtiofsd] Client connected, servicing requests124lb # [2026-09-22T11:24:58Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether125client # [2026-09-22T11:24:58Z INFO virtiofsd] Client connected, servicing requests126lb # [2026-09-22T11:24:58Z INFO virtiofsd] Waiting for vhost-user socket connection...127client # [2026-09-22T11:24:58Z INFO virtiofsd] Client connected, servicing requests128lb # [2026-09-22T11:24:58Z INFO virtiofsd] Client connected, servicing requests129lb # [2026-09-22T11:24:58Z INFO virtiofsd] Client connected, servicing requests130lb # [2026-09-22T11:24:58Z INFO virtiofsd] Client connected, servicing requests131client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]132client # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026133client # [ 0.000000] KASLR enabled134client # [ 0.000000] random: crng init done135client # [ 0.000000] Machine model: linux,dummy-virt136client # [ 0.000000] efi: UEFI not found.137client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT138client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]139client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]140client # [ 0.000000] Zone ranges:141client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]142client # [ 0.000000] DMA32 empty143client # [ 0.000000] Normal empty144client # [ 0.000000] Device empty145client # [ 0.000000] Movable zone start for each node146client # [ 0.000000] Early memory node ranges147client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]148client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]149client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000150client # [ 0.000000] psci: probing for conduit method from DT.151client # [ 0.000000] psci: PSCIv1.3 detected in firmware.152client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs153client # [ 0.000000] psci: Trusted OS migration not required154client # [ 0.000000] psci: SMC Calling Convention v1.1155client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)156client # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296157client # [ 0.000000] Detected PIPT I-cache on CPU0158client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)159client # [ 0.000000] CPU features: detected: GICv3 CPU interface160client # [ 0.000000] CPU features: detected: Spectre-v4161client # [ 0.000000] CPU features: detected: Spectre-BHB162client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38163client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23164client # [ 0.000000] alternatives: applying boot alternatives165client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/ba719awibzsjxd1bw9mhq3k18i64cwhc-nixos-system-client-test/init regInfo=/nix/store/p2989fx14h8v8fmf8spd0dk73x9d2np7-closure-info/registration console=ttyAMA0,115200n8 console=tty0166client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/p2989fx14h8v8fmf8spd0dk73x9d2np7-closure-info/registration", will be passed to user space.167client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes168client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)169client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)170client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB171client # [ 0.000000] software IO TLB: area num 1.172client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB173client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)174client # [ 0.000000] Fallback order for Node 0: 0175client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216176client # [ 0.000000] Policy zone: DMA177client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off178client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1179client # [ 0.000000] allocated 3145728 bytes of page_ext180client # [ 0.000000] ftrace: allocating 74950 entries in 294 pages181client # [ 0.000000] ftrace: allocated 294 pages with 4 groups182client # [ 0.000000] rcu: Hierarchical RCU implementation.183client # [ 0.000000] rcu: RCU event tracing is enabled.184client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.185client # [ 0.000000] Trampoline variant of Tasks RCU enabled.186client # [ 0.000000] Rude variant of Tasks RCU enabled.187client # [ 0.000000] Tracing variant of Tasks RCU enabled.188client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.189client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1190client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.191client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.192client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.193client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0194client # [ 0.000000] GICv3: 256 SPIs implemented195client # [ 0.000000] GICv3: 0 Extended SPIs implemented196client # [ 0.000000] Root IRQ handler: gic_handle_irq197client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI198client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0199client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000200client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]201client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)202client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)203client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000204client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000205client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.206client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns207client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).208client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns209client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns210client # [ 0.000033] arm-pv: using stolen time PV211client # [ 0.000649] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)212client # [ 0.000815] Console: colour dummy device 80x25213client # [ 0.000822] printk: legacy console [tty0] enabled214client # [ 0.001038] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)215client # [ 0.001044] pid_max: default: 32768 minimum: 301216client # [ 0.001131] LSM: initializing lsm=capability,landlock,yama,bpf,ima217client # [ 0.001326] landlock: Up and running.218client # [ 0.001329] Yama: becoming mindful.219client # [ 0.001981] LSM support for eBPF active220client # [ 0.002132] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)221client # [ 0.002165] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)222client # [ 0.003971] rcu: Hierarchical SRCU implementation.223client # [ 0.003975] rcu: Max phase no-delay instances is 1000.224client # [ 0.005249] fsl-mc MSI: its@8080000 domain created225client # [ 0.005339] EFI services will not be available.226client # [ 0.005447] smp: Bringing up secondary CPUs ...227client # [ 0.005455] smp: Brought up 1 node, 1 CPU228client # [ 0.005458] SMP: Total of 1 processors activated.229client # [ 0.005463] CPU: All CPU(s) started at EL1230client # [ 0.005474] CPU features: detected: Branch Target Identification231client # [ 0.005478] CPU features: detected: ARMv8.4 Translation Table Level232client # [ 0.005482] CPU features: detected: Instruction cache invalidation not required for I/D coherence233client # [ 0.005485] CPU features: detected: Data cache clean to the PoU not required for I/D coherence234client # [ 0.005489] CPU features: detected: Common not Private translations235client # [ 0.005492] CPU features: detected: CRC32 instructions236client # [ 0.005495] CPU features: detected: Data cache clean to Point of Deep Persistence237client # [ 0.005499] CPU features: detected: Data cache clean to Point of Persistence238client # [ 0.005502] CPU features: detected: Data independent timing control (DIT)239client # [ 0.005505] CPU features: detected: E0PD240client # [ 0.005507] CPU features: detected: Enhanced Counter Virtualization241worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]242client # [ 0.005510] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)243client # [ 0.005514] CPU features: detected: Enhanced Virtualization Traps244client # [ 0.005517] CPU features: detected: Fine Grained Traps245worker1 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026246worker1 # [ 0.000000] KASLR enabled247worker1 # [ 0.000000] random: crng init done248client # [ 0.005520] CPU features: detected: Generic authentication (architected QARMA5 algorithm)249worker1 # [ 0.000000] Machine model: linux,dummy-virt250worker1 # [ 0.000000] efi: UEFI not found.251client # [ 0.005525] CPU features: detected: RCpc load-acquire (LDAPR)252client # [ 0.005528] CPU features: detected: LSE atomic instructions253worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT254client # [ 0.005531] CPU features: detected: Privileged Access Never255worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]256client # [ 0.005533] CPU features: detected: PMUv3257worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]258client # [ 0.005536] CPU features: detected: RAS Extension Support259worker1 # [ 0.000000] Zone ranges:260client # [ 0.005539] CPU features: detected: RASv1p1 Extension Support261worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]262worker1 # [ 0.000000] DMA32 empty263client # [ 0.005542] CPU features: detected: Random Number Generator264worker1 # [ 0.000000] Normal empty265worker1 # [ 0.000000] Device empty266client # [ 0.005544] CPU features: detected: Speculation barrier (SB)267worker1 # [ 0.000000] Movable zone start for each node268client # [ 0.005547] CPU features: detected: Stage-2 Force Write-Back269worker1 # [ 0.000000] Early memory node ranges270client # [ 0.005550] CPU features: detected: TLB range maintenance instructions271worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]272client # [ 0.005555] CPU features: detected: Speculative Store Bypassing Safe (SSBS)273worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]274worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000275client # [ 0.005595] alternatives: applying system-wide alternatives276worker1 # [ 0.000000] psci: probing for conduit method from DT.277client # [ 0.008530] CPU features: detected: BBM Level 2 without TLB conflict abort278worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware.279worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs280worker1 # [ 0.000000] psci: Trusted OS migration not required281client # [ 0.008713] Memory: 1405876K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124660K reserved, 32768K cma-reserved)282worker1 # [ 0.000000] psci: SMC Calling Convention v1.1283client # [ 0.009074] devtmpfs: initialized284client # [ 0.010923] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)285worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)286worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296287client # [ 0.010946] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).288worker1 # [ 0.000000] Detected PIPT I-cache on CPU0289client # [ 0.011118] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL290client # [ 0.011122] 0 pages in range for non-PLT usage291worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)292client # [ 0.011123] 508272 pages in range for PLT usage293worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface294client # [ 0.011218] pinctrl core: initialized pinctrl subsystem295worker1 # [ 0.000000] CPU features: detected: Spectre-v4296client # [ 0.011985] DMI not present or invalid.297worker1 # [ 0.000000] CPU features: detected: Spectre-BHB298client # [ 0.015615] NET: Registered PF_NETLINK/PF_ROUTE protocol family299worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38300worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23301client # [ 0.017995] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations302worker1 # [ 0.000000] alternatives: applying boot alternatives303client # [ 0.018188] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations304client # [ 0.018389] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations305client # [ 0.018409] audit: initializing netlink subsys (disabled)306client # [ 0.018950] thermal_sys: Registered thermal governor 'fair_share'307client # [ 0.018952] thermal_sys: Registered thermal governor 'bang_bang'308worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/hmsqrrazdjx0ansfssikbgws6gj8896v-nixos-system-worker1-test/init regInfo=/nix/store/8vnpypswqgsjck7hj1wpw5i0fw2n1v8l-closure-info/registration console=ttyAMA0,115200n8 console=tty0309client # [ 0.018955] thermal_sys: Registered thermal governor 'step_wise'310client # [ 0.018958] thermal_sys: Registered thermal governor 'user_space'311worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/8vnpypswqgsjck7hj1wpw5i0fw2n1v8l-closure-info/registration", will be passed to user space.312client # [ 0.018963] thermal_sys: Registered thermal governor 'power_allocator'313worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes314client # [ 0.018990] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1315client # [ 0.018999] cpuidle: using governor ladder316worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)317client # [ 0.019005] cpuidle: using governor menu318worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)319client # [ 0.019210] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.320worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB321client # [ 0.019227] ASID allocator initialised with 65536 entries322worker1 # [ 0.000000] software IO TLB: area num 1.323client # [ 0.020516] Serial: AMBA PL011 UART driver324worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB325client # [ 0.026020] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1326worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)327worker1 # [ 0.000000] Fallback order for Node 0: 0328client # [ 0.026187] printk: console [ttyAMA0] enabled329worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216330worker1 # [ 0.000000] Policy zone: DMA331worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off332worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1333worker1 # [ 0.000000] allocated 3145728 bytes of page_ext334worker1 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages335worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups336worker1 # [ 0.000000] rcu: Hierarchical RCU implementation.337worker1 # [ 0.000000] rcu: RCU event tracing is enabled.338worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.339worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled.340worker1 # [ 0.000000] Rude variant of Tasks RCU enabled.341worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled.342client # [ 0.150830] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages343worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.344client # [ 0.150849] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page345worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1346client # [ 0.150854] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages347client # [ 0.150858] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page348worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.349client # [ 0.150863] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages350worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.351client # [ 0.150867] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page352worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.353client # [ 0.150872] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages354worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0355client # [ 0.150876] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page356worker1 # [ 0.000000] GICv3: 256 SPIs implemented357worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented358worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq359worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI360client # [ 0.158320] fbcon: Taking over console361worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0362client # [ 0.158335] ACPI: Interpreter disabled.363worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000364client # [ 0.160211] iommu: Default domain type: Translated365worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]366client # [ 0.160222] iommu: DMA domain TLB invalidation policy: strict mode367worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)368worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)369worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000370worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000371worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.372worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns373worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).374client # [ 0.167366] SCSI subsystem initialized375worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]376worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns377client # [ 0.168058] usbcore: registered new interface driver usbfs378worker2 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026379client # [ 0.168089] usbcore: registered new interface driver hub380worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns381worker2 # [ 0.000000] KASLR enabled382worker2 # [ 0.000000] random: crng init done383worker1 # [ 0.000032] arm-pv: using stolen time PV384client # [ 0.168104] usbcore: registered new device driver usb385worker2 # [ 0.000000] Machine model: linux,dummy-virt386client # [ 0.168395] pps_core: LinuxPPS API ver. 1 registered387worker2 # [ 0.000000] efi: UEFI not found.388worker1 # [ 0.000516] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)389lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]390worker1 # [ 0.000671] Console: colour dummy device 80x25391client # [ 0.168401] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>392worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT393client # [ 0.168411] PTP clock support registered394worker1 # [ 0.000678] printk: legacy console [tty0] enabled395client # [ 0.168458] EDAC MC: Ver: 3.0.0396worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]397lb # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026398worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]399lb # [ 0.000000] KASLR enabled400client # [ 0.173042] scmi_core: SCMI protocol bus registered401worker1 # [ 0.000877] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)402worker2 # [ 0.000000] Zone ranges:403lb # [ 0.000000] random: crng init done404worker1 # [ 0.000884] pid_max: default: 32768 minimum: 301405worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]406lb # [ 0.000000] Machine model: linux,dummy-virt407client # [ 0.173986] FPGA manager framework408worker2 # [ 0.000000] DMA32 empty409worker1 # [ 0.000959] LSM: initializing lsm=capability,landlock,yama,bpf,ima410lb # [ 0.000000] efi: UEFI not found.411worker2 # [ 0.000000] Normal empty412client # [ 0.174908] vgaarb: loaded413worker1 # [ 0.001124] landlock: Up and running.414worker2 # [ 0.000000] Device empty415lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT416worker1 # [ 0.001127] Yama: becoming mindful.417worker2 # [ 0.000000] Movable zone start for each node418client # [ 0.175547] clocksource: Switched to clocksource arch_sys_counter419worker1 # [ 0.001687] LSM support for eBPF active420worker2 # [ 0.000000] Early memory node ranges421lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]422worker1 # [ 0.001836] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)423worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]424lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]425lb # [ 0.000000] Zone ranges:426worker1 # [ 0.001869] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)427worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]428lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]429worker1 # [ 0.003729] rcu: Hierarchical SRCU implementation.430worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000431lb # [ 0.000000] DMA32 empty432lb # [ 0.000000] Normal empty433worker1 # [ 0.003734] rcu: Max phase no-delay instances is 1000.434worker2 # [ 0.000000] psci: probing for conduit method from DT.435lb # [ 0.000000] Device empty436worker1 # [ 0.005013] fsl-mc MSI: its@8080000 domain created437worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware.438lb # [ 0.000000] Movable zone start for each node439worker1 # [ 0.005105] EFI services will not be available.440lb # [ 0.000000] Early memory node ranges441worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs442worker1 # [ 0.005215] smp: Bringing up secondary CPUs ...443worker2 # [ 0.000000] psci: Trusted OS migration not required444lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]445worker1 # [ 0.005223] smp: Brought up 1 node, 1 CPU446worker2 # [ 0.000000] psci: SMC Calling Convention v1.1447worker1 # [ 0.005226] SMP: Total of 1 processors activated.448lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]449worker1 # [ 0.005232] CPU: All CPU(s) started at EL1450worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)451lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000452client # [ 0.181977] VFS: Disk quotas dquot_6.6.0453worker1 # [ 0.005242] CPU features: detected: Branch Target Identification454lb # [ 0.000000] psci: probing for conduit method from DT.455worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296456client # [ 0.182014] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)457lb # [ 0.000000] psci: PSCIv1.3 detected in firmware.458worker1 # [ 0.005247] CPU features: detected: ARMv8.4 Translation Table Level459worker2 # [ 0.000000] Detected PIPT I-cache on CPU0460lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs461client # [ 0.183597] netfs: FS-Cache loaded462worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)463worker1 # [ 0.005250] CPU features: detected: Instruction cache invalidation not required for I/D coherence464client # [ 0.183716] pnp: PnP ACPI: disabled465lb # [ 0.000000] psci: Trusted OS migration not required466lb # [ 0.000000] psci: SMC Calling Convention v1.1467worker1 # [ 0.005253] CPU features: detected: Data cache clean to the PoU not required for I/D coherence468worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface469lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)470worker2 # [ 0.000000] CPU features: detected: Spectre-v4471worker1 # [ 0.005257] CPU features: detected: Common not Private translations472worker2 # [ 0.000000] CPU features: detected: Spectre-BHB473worker1 # [ 0.005260] CPU features: detected: CRC32 instructions474lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296475worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38476lb # [ 0.000000] Detected PIPT I-cache on CPU0477worker1 # [ 0.005263] CPU features: detected: Data cache clean to Point of Deep Persistence478worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23479client # [ 0.187427] NET: Registered PF_INET protocol family480lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)481worker1 # [ 0.005266] CPU features: detected: Data cache clean to Point of Persistence482worker2 # [ 0.000000] alternatives: applying boot alternatives483lb # [ 0.000000] CPU features: detected: GICv3 CPU interface484worker1 # [ 0.005269] CPU features: detected: Data independent timing control (DIT)485lb # [ 0.000000] CPU features: detected: Spectre-v4486worker1 # [ 0.005272] CPU features: detected: E0PD487lb # [ 0.000000] CPU features: detected: Spectre-BHB488worker1 # [ 0.005275] CPU features: detected: Enhanced Counter Virtualization489lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38490worker1 # [ 0.005278] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)491lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23492worker1 # [ 0.005282] CPU features: detected: Enhanced Virtualization Traps493lb # [ 0.000000] alternatives: applying boot alternatives494worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/bw3mqal5y2l6sysyr6dir665462635vh-nixos-system-worker2-test/init regInfo=/nix/store/xmy09rgpfkkv98has2xhagcrihy5dqwl-closure-info/registration console=ttyAMA0,115200n8 console=tty0495client # [ 0.188280] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)496worker1 # [ 0.005285] CPU features: detected: Fine Grained Traps497worker1 # [ 0.005288] CPU features: detected: Generic authentication (architected QARMA5 algorithm)498worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/xmy09rgpfkkv98has2xhagcrihy5dqwl-closure-info/registration", will be passed to user space.499worker1 # [ 0.005293] CPU features: detected: RCpc load-acquire (LDAPR)500worker1 # [ 0.005296] CPU features: detected: LSE atomic instructions501worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes502worker1 # [ 0.005299] CPU features: detected: Privileged Access Never503lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/hw6sjiya3fqcxqx5sjqhqri1ayky0v1l-nixos-system-lb-test/init regInfo=/nix/store/icv2rkyzy36dhd3gwpgrrp51zhv2f8w0-closure-info/registration console=ttyAMA0,115200n8 console=tty0504worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)505worker1 # [ 0.005302] CPU features: detected: PMUv3506worker1 # [ 0.005305] CPU features: detected: RAS Extension Support507worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)508worker1 # [ 0.005308] CPU features: detected: RASv1p1 Extension Support509worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB510worker2 # [ 0.000000] software IO TLB: area num 1.511worker1 # [ 0.005310] CPU features: detected: Random Number Generator512lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/icv2rkyzy36dhd3gwpgrrp51zhv2f8w0-closure-info/registration", will be passed to user space.513worker1 # [ 0.005313] CPU features: detected: Speculation barrier (SB)514worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB515lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes516worker1 # [ 0.005316] CPU features: detected: Stage-2 Force Write-Back517worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)518lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)519worker1 # [ 0.005319] CPU features: detected: TLB range maintenance instructions520worker2 # [ 0.000000] Fallback order for Node 0: 0521lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)522worker1 # [ 0.005323] CPU features: detected: Speculative Store Bypassing Safe (SSBS)523worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216524worker2 # [ 0.000000] Policy zone: DMA525worker1 # [ 0.005362] alternatives: applying system-wide alternatives526lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB527lb # [ 0.000000] software IO TLB: area num 1.528worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off529worker1 # [ 0.008318] CPU features: detected: BBM Level 2 without TLB conflict abort530worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1531lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB532worker2 # [ 0.000000] allocated 3145728 bytes of page_ext533lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)534worker1 # [ 0.008511] Memory: 1405624K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124664K reserved, 32768K cma-reserved)535worker2 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages536lb # [ 0.000000] Fallback order for Node 0: 0537worker1 # [ 0.008891] devtmpfs: initialized538worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups539lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216540worker2 # [ 0.000000] rcu: Hierarchical RCU implementation.541worker1 # [ 0.010711] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)542lb # [ 0.000000] Policy zone: DMA543worker2 # [ 0.000000] rcu: RCU event tracing is enabled.544lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off545worker1 # [ 0.010733] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).546worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.547lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1548worker1 # [ 0.010937] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL549worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled.550lb # [ 0.000000] allocated 3145728 bytes of page_ext551worker2 # [ 0.000000] Rude variant of Tasks RCU enabled.552lb # [ 0.000000] ftrace: allocating 74950 entries in 294 pages553worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled.554worker1 # [ 0.010942] 0 pages in range for non-PLT usage555lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups556worker1 # [ 0.010943] 508272 pages in range for PLT usage557worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.558lb # [ 0.000000] rcu: Hierarchical RCU implementation.559worker1 # [ 0.011067] pinctrl core: initialized pinctrl subsystem560worker1 # [ 0.011904] DMI not present or invalid.561lb # [ 0.000000] rcu: RCU event tracing is enabled.562worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1563worker1 # [ 0.015269] NET: Registered PF_NETLINK/PF_ROUTE protocol family564lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.565worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.566lb # [ 0.000000] Trampoline variant of Tasks RCU enabled.567worker1 # [ 0.017504] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations568lb # [ 0.000000] Rude variant of Tasks RCU enabled.569worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.570worker1 # [ 0.017696] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations571lb # [ 0.000000] Tracing variant of Tasks RCU enabled.572worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.573lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.574worker1 # [ 0.017896] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations575worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0576worker1 # [ 0.017919] audit: initializing netlink subsys (disabled)577lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1578worker2 # [ 0.000000] GICv3: 256 SPIs implemented579worker1 # [ 0.018475] thermal_sys: Registered thermal governor 'fair_share'580worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented581lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.582worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq583worker1 # [ 0.018477] thermal_sys: Registered thermal governor 'bang_bang'584worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI585worker1 # [ 0.018480] thermal_sys: Registered thermal governor 'step_wise'586lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.587worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0588worker1 # [ 0.018483] thermal_sys: Registered thermal governor 'user_space'589lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.590worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000591worker1 # [ 0.018489] thermal_sys: Registered thermal governor 'power_allocator'592worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]593lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0594lb # [ 0.000000] GICv3: 256 SPIs implemented595worker1 # [ 0.018521] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1596worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)597lb # [ 0.000000] GICv3: 0 Extended SPIs implemented598worker1 # [ 0.018530] cpuidle: using governor ladder599worker1 # [ 0.018536] cpuidle: using governor menu600lb # [ 0.000000] Root IRQ handler: gic_handle_irq601lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI602worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)603worker1 # [ 0.018744] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.604lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0605worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000606worker1 # [ 0.018759] ASID allocator initialised with 65536 entries607worker1 # [ 0.020029] Serial: AMBA PL011 UART driver608lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000609worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000610lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]611worker1 # [ 0.025406] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1612worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.613worker1 # [ 0.025581] printk: console [ttyAMA0] enabled614lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1)615worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns616lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1)617worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).618lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000619client # [ 0.216349] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)620worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns621lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000622client # [ 0.216388] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)623worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns624lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.625worker2 # [ 0.000029] arm-pv: using stolen time PV626client # [ 0.216413] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)627lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns628client # [ 0.216480] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)629worker2 # [ 0.000378] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)630lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).631worker2 # [ 0.000565] Console: colour dummy device 80x25632client # [ 0.216623] TCP: Hash tables configured (established 16384 bind 16384)633worker2 # [ 0.000572] printk: legacy console [tty0] enabled634client # [ 0.216701] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)635lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns636worker1 # [ 0.144141] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages637client # [ 0.216735] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)638worker2 # [ 0.000795] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)639lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns640worker1 # [ 0.144158] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page641worker2 # [ 0.000802] pid_max: default: 32768 minimum: 301642lb # [ 0.000030] arm-pv: using stolen time PV643client # [ 0.216770] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)644worker1 # [ 0.144163] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages645worker2 # [ 0.000881] LSM: initializing lsm=capability,landlock,yama,bpf,ima646client # [ 0.216858] NET: Registered PF_UNIX/PF_LOCAL protocol family647worker1 # [ 0.144168] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page648worker2 # [ 0.001016] landlock: Up and running.649lb # [ 0.000407] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)650client # [ 0.216877] NET: Registered PF_XDP protocol family651worker2 # [ 0.001018] Yama: becoming mindful.652lb # [ 0.000572] Console: colour dummy device 80x25653worker1 # [ 0.144173] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages654client # [ 0.216891] PCI: CLS 0 bytes, default 64655worker2 # [ 0.001450] LSM support for eBPF active656lb # [ 0.000580] printk: legacy console [tty0] enabled657client # [ 0.217126] Trying to unpack rootfs image as initramfs...658worker1 # [ 0.144177] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page659worker2 # [ 0.001583] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)660worker1 # [ 0.144181] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages661lb # [ 0.000794] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)662worker2 # [ 0.001615] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)663worker1 # [ 0.144186] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page664lb # [ 0.000801] pid_max: default: 32768 minimum: 301665worker2 # [ 0.003485] rcu: Hierarchical SRCU implementation.666lb # [ 0.000874] LSM: initializing lsm=capability,landlock,yama,bpf,ima667worker2 # [ 0.003490] rcu: Max phase no-delay instances is 1000.668lb # [ 0.001012] landlock: Up and running.669worker2 # [ 0.004746] fsl-mc MSI: its@8080000 domain created670lb # [ 0.001015] Yama: becoming mindful.671worker2 # [ 0.004835] EFI services will not be available.672lb # [ 0.001470] LSM support for eBPF active673worker1 # [ 0.151251] fbcon: Taking over console674worker2 # [ 0.004906] smp: Bringing up secondary CPUs ...675worker1 # [ 0.151267] ACPI: Interpreter disabled.676worker2 # [ 0.004914] smp: Brought up 1 node, 1 CPU677lb # [ 0.001599] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)678worker2 # [ 0.004917] SMP: Total of 1 processors activated.679lb # [ 0.001632] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)680worker2 # [ 0.004922] CPU: All CPU(s) started at EL1681lb # [ 0.003581] rcu: Hierarchical SRCU implementation.682worker2 # [ 0.004933] CPU features: detected: Branch Target Identification683lb # [ 0.003586] rcu: Max phase no-delay instances is 1000.684worker2 # [ 0.004938] CPU features: detected: ARMv8.4 Translation Table Level685client # [ 0.233065] kvm [1]: HYP mode not available686lb # [ 0.004889] fsl-mc MSI: its@8080000 domain created687lb # [ 0.004987] EFI services will not be available.688worker2 # [ 0.004941] CPU features: detected: Instruction cache invalidation not required for I/D coherence689lb # [ 0.005073] smp: Bringing up secondary CPUs ...690lb # [ 0.005081] smp: Brought up 1 node, 1 CPU691worker2 # [ 0.004945] CPU features: detected: Data cache clean to the PoU not required for I/D coherence692lb # [ 0.005084] SMP: Total of 1 processors activated.693lb # [ 0.005090] CPU: All CPU(s) started at EL1694worker2 # [ 0.004948] CPU features: detected: Common not Private translations695worker2 # [ 0.004952] CPU features: detected: CRC32 instructions696lb # [ 0.005100] CPU features: detected: Branch Target Identification697lb # [ 0.005103] CPU features: detected: ARMv8.4 Translation Table Level698worker2 # [ 0.004954] CPU features: detected: Data cache clean to Point of Deep Persistence699worker2 # [ 0.004958] CPU features: detected: Data cache clean to Point of Persistence700worker1 # [ 0.160263] iommu: Default domain type: Translated701lb # [ 0.005106] CPU features: detected: Instruction cache invalidation not required for I/D coherence702worker2 # [ 0.004961] CPU features: detected: Data independent timing control (DIT)703worker1 # [ 0.160273] iommu: DMA domain TLB invalidation policy: strict mode704lb # [ 0.005110] CPU features: detected: Data cache clean to the PoU not required for I/D coherence705worker2 # [ 0.004964] CPU features: detected: E0PD706worker1 # [ 0.160634] SCSI subsystem initialized707lb # [ 0.005114] CPU features: detected: Common not Private translations708worker2 # [ 0.004966] CPU features: detected: Enhanced Counter Virtualization709lb # [ 0.005117] CPU features: detected: CRC32 instructions710worker1 # [ 0.162623] usbcore: registered new interface driver usbfs711worker2 # [ 0.004970] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)712worker1 # [ 0.162656] usbcore: registered new interface driver hub713lb # [ 0.005120] CPU features: detected: Data cache clean to Point of Deep Persistence714worker2 # [ 0.004973] CPU features: detected: Enhanced Virtualization Traps715worker1 # [ 0.162672] usbcore: registered new device driver usb716lb # [ 0.005124] CPU features: detected: Data cache clean to Point of Persistence717worker1 # [ 0.162970] pps_core: LinuxPPS API ver. 1 registered718worker2 # [ 0.004977] CPU features: detected: Fine Grained Traps719lb # [ 0.005127] CPU features: detected: Data independent timing control (DIT)720worker2 # [ 0.004981] CPU features: detected: Generic authentication (architected QARMA5 algorithm)721worker1 # [ 0.162976] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>722lb # [ 0.005130] CPU features: detected: E0PD723worker1 # [ 0.162987] PTP clock support registered724worker2 # [ 0.004986] CPU features: detected: RCpc load-acquire (LDAPR)725worker1 # [ 0.163037] EDAC MC: Ver: 3.0.0726lb # [ 0.005133] CPU features: detected: Enhanced Counter Virtualization727worker2 # [ 0.004989] CPU features: detected: LSE atomic instructions728worker1 # [ 0.167489] scmi_core: SCMI protocol bus registered729lb # [ 0.005136] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)730worker2 # [ 0.004992] CPU features: detected: Privileged Access Never731worker2 # [ 0.004994] CPU features: detected: PMUv3732lb # [ 0.005140] CPU features: detected: Enhanced Virtualization Traps733worker1 # [ 0.168457] FPGA manager framework734worker2 # [ 0.004997] CPU features: detected: RAS Extension Support735lb # [ 0.005143] CPU features: detected: Fine Grained Traps736worker1 # [ 0.169382] vgaarb: loaded737worker2 # [ 0.005000] CPU features: detected: RASv1p1 Extension Support738lb # [ 0.005147] CPU features: detected: Generic authentication (architected QARMA5 algorithm)739worker1 # [ 0.170001] clocksource: Switched to clocksource arch_sys_counter740worker2 # [ 0.005003] CPU features: detected: Random Number Generator741lb # [ 0.005152] CPU features: detected: RCpc load-acquire (LDAPR)742worker2 # [ 0.005005] CPU features: detected: Speculation barrier (SB)743lb # [ 0.005155] CPU features: detected: LSE atomic instructions744worker2 # [ 0.005008] CPU features: detected: Stage-2 Force Write-Back745lb # [ 0.005158] CPU features: detected: Privileged Access Never746lb # [ 0.005161] CPU features: detected: PMUv3747lb # [ 0.005164] CPU features: detected: RAS Extension Support748lb # [ 0.005167] CPU features: detected: RASv1p1 Extension Support749lb # [ 0.005169] CPU features: detected: Random Number Generator750lb # [ 0.005172] CPU features: detected: Speculation barrier (SB)751lb # [ 0.005175] CPU features: detected: Stage-2 Force Write-Back752worker1 # [ 0.176036] VFS: Disk quotas dquot_6.6.0753lb # [ 0.005178] CPU features: detected: TLB range maintenance instructions754worker1 # [ 0.176078] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)755lb # [ 0.005183] CPU features: detected: Speculative Store Bypassing Safe (SSBS)756worker1 # [ 0.177742] netfs: FS-Cache loaded757worker1 # [ 0.177881] pnp: PnP ACPI: disabled758lb # [ 0.005221] alternatives: applying system-wide alternatives759lb # [ 0.008190] CPU features: detected: BBM Level 2 without TLB conflict abort760worker2 # [ 0.005011] CPU features: detected: TLB range maintenance instructions761lb # [ 0.008348] Memory: 1405680K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124636K reserved, 32768K cma-reserved)762lb # [ 0.008723] devtmpfs: initialized763worker2 # [ 0.005016] CPU features: detected: Speculative Store Bypassing Safe (SSBS)764worker1 # [ 0.181729] NET: Registered PF_INET protocol family765worker2 # [ 0.005053] alternatives: applying system-wide alternatives766worker2 # [ 0.008009] CPU features: detected: BBM Level 2 without TLB conflict abort767worker1 # [ 0.181993] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)768worker2 # [ 0.008209] Memory: 1405680K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124632K reserved, 32768K cma-reserved)769worker2 # [ 0.008560] devtmpfs: initialized770worker2 # [ 0.010339] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)771worker2 # [ 0.010361] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).772worker2 # [ 0.010535] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL773worker2 # [ 0.010539] 0 pages in range for non-PLT usage774worker2 # [ 0.010540] 508272 pages in range for PLT usage775worker2 # [ 0.010645] pinctrl core: initialized pinctrl subsystem776worker2 # [ 0.011403] DMI not present or invalid.777lb # [ 0.010529] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)778worker2 # [ 0.014448] NET: Registered PF_NETLINK/PF_ROUTE protocol family779lb # [ 0.010552] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).780worker2 # [ 0.016677] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations781lb # [ 0.010727] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL782worker2 # [ 0.016847] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations783lb # [ 0.010731] 0 pages in range for non-PLT usage784lb # [ 0.010732] 508272 pages in range for PLT usage785worker2 # [ 0.017045] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations786lb # [ 0.010836] pinctrl core: initialized pinctrl subsystem787worker2 # [ 0.017068] audit: initializing netlink subsys (disabled)788lb # [ 0.011643] DMI not present or invalid.789worker2 # [ 0.017645] thermal_sys: Registered thermal governor 'fair_share'790lb # [ 0.014776] NET: Registered PF_NETLINK/PF_ROUTE protocol family791worker2 # [ 0.017647] thermal_sys: Registered thermal governor 'bang_bang'792lb # [ 0.017249] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations793worker2 # [ 0.017651] thermal_sys: Registered thermal governor 'step_wise'794lb # [ 0.017447] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations795worker2 # [ 0.017654] thermal_sys: Registered thermal governor 'user_space'796lb # [ 0.017643] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations797worker2 # [ 0.017659] thermal_sys: Registered thermal governor 'power_allocator'798lb # [ 0.017665] audit: initializing netlink subsys (disabled)799worker2 # [ 0.017691] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1800lb # [ 0.018258] thermal_sys: Registered thermal governor 'fair_share'801worker2 # [ 0.017699] cpuidle: using governor ladder802worker2 # [ 0.017706] cpuidle: using governor menu803lb # [ 0.018260] thermal_sys: Registered thermal governor 'bang_bang'804worker2 # [ 0.017894] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.805lb # [ 0.018264] thermal_sys: Registered thermal governor 'step_wise'806worker2 # [ 0.017910] ASID allocator initialised with 65536 entries807lb # [ 0.018266] thermal_sys: Registered thermal governor 'user_space'808worker2 # [ 0.019145] Serial: AMBA PL011 UART driver809lb # [ 0.018271] thermal_sys: Registered thermal governor 'power_allocator'810worker2 # [ 0.024402] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1811lb # [ 0.018302] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1812worker2 # [ 0.024550] printk: console [ttyAMA0] enabled813lb # [ 0.018310] cpuidle: using governor ladder814lb # [ 0.018316] cpuidle: using governor menu815lb # [ 0.018493] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.816worker2 # [ 0.145672] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages817lb # [ 0.018507] ASID allocator initialised with 65536 entries818worker2 # [ 0.145695] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page819lb # [ 0.019727] Serial: AMBA PL011 UART driver820worker2 # [ 0.145700] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages821lb # [ 0.025013] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1822worker2 # [ 0.145705] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page823lb # [ 0.025163] printk: console [ttyAMA0] enabled824worker2 # [ 0.145709] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages825worker2 # [ 0.145714] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page826lb # [ 0.146651] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages827worker2 # [ 0.145718] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages828lb # [ 0.146669] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page829worker2 # [ 0.145723] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page830lb # [ 0.146674] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages831lb # [ 0.146679] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page832lb # [ 0.146683] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages833worker2 # [ 0.153303] fbcon: Taking over console834lb # [ 0.146688] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page835worker2 # [ 0.153317] ACPI: Interpreter disabled.836lb # [ 0.146692] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages837worker2 # [ 0.155208] iommu: Default domain type: Translated838lb # [ 0.146696] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page839worker2 # [ 0.155219] iommu: DMA domain TLB invalidation policy: strict mode840worker2 # [ 0.156923] SCSI subsystem initialized841lb # [ 0.154257] fbcon: Taking over console842lb # [ 0.154271] ACPI: Interpreter disabled.843lb # [ 0.156215] iommu: Default domain type: Translated844worker1 # [ 0.210944] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)845lb # [ 0.156225] iommu: DMA domain TLB invalidation policy: strict mode846worker1 # [ 0.210985] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)847lb # [ 0.157938] SCSI subsystem initialized848worker1 # [ 0.211012] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)849worker1 # [ 0.211085] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)850worker1 # [ 0.211228] TCP: Hash tables configured (established 16384 bind 16384)851worker1 # [ 0.211303] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)852worker1 # [ 0.211336] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)853worker2 # [ 0.164906] usbcore: registered new interface driver usbfs854worker1 # [ 0.211372] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)855worker2 # [ 0.164948] usbcore: registered new interface driver hub856worker1 # [ 0.211460] NET: Registered PF_UNIX/PF_LOCAL protocol family857worker2 # [ 0.164964] usbcore: registered new device driver usb858lb # [ 0.163061] usbcore: registered new interface driver usbfs859worker1 # [ 0.211513] NET: Registered PF_XDP protocol family860worker2 # [ 0.165243] pps_core: LinuxPPS API ver. 1 registered861lb # [ 0.163094] usbcore: registered new interface driver hub862worker1 # [ 0.211528] PCI: CLS 0 bytes, default 64863lb # [ 0.163109] usbcore: registered new device driver usb864worker1 # [ 0.211760] Trying to unpack rootfs image as initramfs...865worker2 # [ 0.165250] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>866lb # [ 0.163377] pps_core: LinuxPPS API ver. 1 registered867worker2 # [ 0.165260] PTP clock support registered868worker2 # [ 0.165308] EDAC MC: Ver: 3.0.0869lb # [ 0.163383] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>870worker2 # [ 0.169907] scmi_core: SCMI protocol bus registered871lb # [ 0.163393] PTP clock support registered872lb # [ 0.163441] EDAC MC: Ver: 3.0.0873worker2 # [ 0.170934] FPGA manager framework874lb # [ 0.168063] scmi_core: SCMI protocol bus registered875worker2 # [ 0.171935] vgaarb: loaded876lb # [ 0.169038] FPGA manager framework877lb # [ 0.169966] vgaarb: loaded878worker2 # [ 0.172573] clocksource: Switched to clocksource arch_sys_counter879worker2 # [ 0.173199] VFS: Disk quotas dquot_6.6.0880lb # [ 0.170579] clocksource: Switched to clocksource arch_sys_counter881worker2 # [ 0.173232] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)882worker1 # [ 0.227543] kvm [1]: HYP mode not available883worker2 # [ 0.175677] netfs: FS-Cache loaded884worker2 # [ 0.175800] pnp: PnP ACPI: disabled885lb # [ 0.176593] VFS: Disk quotas dquot_6.6.0886lb # [ 0.176634] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)887client # [ 0.324053] Initialise system trusted keyrings888lb # [ 0.178258] netfs: FS-Cache loaded889lb # [ 0.178360] pnp: PnP ACPI: disabled890client # [ 0.324788] workingset: timestamp_bits=42 max_order=19 bucket_order=0891client # [ 0.326051] squashfs: version 4.0 (2009/01/31) Phillip Lougher892worker2 # [ 0.181835] NET: Registered PF_INET protocol family893client # [ 0.326821] 9p: Installing v9fs 9p2000 file system support894worker2 # [ 0.182097] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)895lb # [ 0.182292] NET: Registered PF_INET protocol family896lb # [ 0.182554] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)897client # [ 0.355605] Key type asymmetric registered898client # [ 0.355630] Asymmetric key parser 'x509' registered899client # [ 0.355700] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)900client # [ 0.357820] io scheduler mq-deadline registered901client # [ 0.357831] io scheduler kyber registered902worker2 # [ 0.212826] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)903worker2 # [ 0.212870] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)904lb # [ 0.211545] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)905worker2 # [ 0.212900] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)906lb # [ 0.211591] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)907worker2 # [ 0.212980] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)908lb # [ 0.211618] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)909worker2 # [ 0.213124] TCP: Hash tables configured (established 16384 bind 16384)910lb # [ 0.211693] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)911worker2 # [ 0.213204] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)912client # [ 0.367694] pl061_gpio 9030000.pl061: PL061 GPIO chip registered913lb # [ 0.211846] TCP: Hash tables configured (established 16384 bind 16384)914worker2 # [ 0.213239] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)915lb # [ 0.211924] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)916worker2 # [ 0.213276] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)917lb # [ 0.211959] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)918worker2 # [ 0.213367] NET: Registered PF_UNIX/PF_LOCAL protocol family919lb # [ 0.211995] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)920worker2 # [ 0.213396] NET: Registered PF_XDP protocol family921client # [ 0.369109] ledtrig-cpu: registered to indicate activity on CPUs922worker2 # [ 0.213417] PCI: CLS 0 bytes, default 64923lb # [ 0.212083] NET: Registered PF_UNIX/PF_LOCAL protocol family924worker2 # [ 0.213654] Trying to unpack rootfs image as initramfs...925lb # [ 0.212114] NET: Registered PF_XDP protocol family926client # [ 0.369524] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:927lb # [ 0.212130] PCI: CLS 0 bytes, default 64928lb # [ 0.212356] Trying to unpack rootfs image as initramfs...929client # [ 0.369541] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000930client # [ 0.369554] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000931worker2 # [ 0.230268] kvm [1]: HYP mode not available932client # [ 0.369563] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000933lb # [ 0.228368] kvm [1]: HYP mode not available934client # [ 0.369586] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits935client # [ 0.369610] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]936client # [ 0.369688] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00937client # [ 0.369698] pci_bus 0000:00: root bus resource [bus 00-ff]938client # [ 0.369704] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]939client # [ 0.369709] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]940client # [ 0.369714] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]941client # [ 0.369802] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint942client # [ 0.370251] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint943client # [ 0.370441] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]944client # [ 0.370458] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]945client # [ 0.370488] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]946client # [ 0.370504] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]947client # [ 0.370965] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint948client # [ 0.371152] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]949client # [ 0.371168] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]950client # [ 0.371198] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]951client # [ 0.390859] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint952client # [ 0.391046] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]953client # [ 0.391062] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]954client # [ 0.391092] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]955client # [ 0.394707] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint956client # [ 0.394894] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]957client # [ 0.394909] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]958client # [ 0.394939] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]959client # [ 0.394957] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]960client # [ 0.395411] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint961worker1 # [ 0.318545] Initialise system trusted keyrings962worker1 # [ 0.319277] workingset: timestamp_bits=42 max_order=19 bucket_order=0963worker1 # [ 0.320539] squashfs: version 4.0 (2009/01/31) Phillip Lougher964worker1 # [ 0.321315] 9p: Installing v9fs 9p2000 file system support965client # [ 0.407665] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]966client # [ 0.407709] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]967client # [ 0.408209] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint968client # [ 0.408403] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]969client # [ 0.408433] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]970client # [ 0.408825] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint971client # [ 0.409010] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]972client # [ 0.409297] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint973client # [ 0.409487] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]974client # [ 0.409518] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]975client # [ 0.409979] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint976client # [ 0.410166] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]977client # [ 0.410197] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]978worker1 # [ 0.341974] Key type asymmetric registered979worker1 # [ 0.341998] Asymmetric key parser 'x509' registered980client # [ 0.410647] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint981client # [ 0.410834] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]982client # [ 0.410864] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]983client # [ 0.411333] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint984client # [ 0.411617] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]985client # [ 0.411634] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]986client # [ 0.411664] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]987client # [ 0.412122] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint988client # [ 0.412306] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]989client # [ 0.412322] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]990worker1 # [ 0.350091] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)991client # [ 0.412352] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]992worker1 # [ 0.351164] io scheduler mq-deadline registered993client # [ 0.412947] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned994worker1 # [ 0.351175] io scheduler kyber registered995client # [ 0.412958] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned996client # [ 0.412963] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned997client # [ 0.413008] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned998client # [ 0.413055] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned999client # [ 0.413103] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1000client # [ 0.413151] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1001client # [ 0.413198] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1002client # [ 0.413247] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1003client # [ 0.413293] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1004client # [ 0.413340] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1005client # [ 0.413385] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1006worker1 # [ 0.362143] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1007client # [ 0.413456] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1008client # [ 0.413502] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1009client # [ 0.413523] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1010client # [ 0.413545] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1011worker1 # [ 0.363539] ledtrig-cpu: registered to indicate activity on CPUs1012client # [ 0.413568] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1013worker1 # [ 0.363953] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1014client # [ 0.413589] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1015worker1 # [ 0.363971] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001016client # [ 0.413611] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1017worker1 # [ 0.363984] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001018client # [ 0.413633] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1019client # [ 0.413657] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1020worker1 # [ 0.363992] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001021client # [ 0.413679] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1022worker1 # [ 0.364015] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1023client # [ 0.413701] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1024worker1 # [ 0.364041] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1025client # [ 0.413724] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1026worker1 # [ 0.364116] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001027client # [ 0.413747] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1028worker1 # [ 0.364125] pci_bus 0000:00: root bus resource [bus 00-ff]1029client # [ 0.413780] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1030worker1 # [ 0.364131] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1031client # [ 0.413802] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1032worker1 # [ 0.364137] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1033client # [ 0.413822] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1034worker1 # [ 0.364142] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1035client # [ 0.413843] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1036client # [ 0.413864] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1037worker1 # [ 0.364230] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1038client # [ 0.413885] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1039worker1 # [ 0.364660] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1040client # [ 0.413911] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1041worker1 # [ 0.364843] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1042client # [ 0.413920] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1043worker1 # [ 0.364860] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1044client # [ 0.413924] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1045worker1 # [ 0.364890] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1046client # [ 0.414747] pci 0000:00:07.0: enabling device (0000 -> 0002)1047lb # [ 0.319124] Initialise system trusted keyrings1048worker1 # [ 0.364906] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1049lb # [ 0.319880] workingset: timestamp_bits=42 max_order=19 bucket_order=01050worker1 # [ 0.365354] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1051worker1 # [ 0.365534] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1052worker2 # [ 0.324249] Initialise system trusted keyrings1053lb # [ 0.321143] squashfs: version 4.0 (2009/01/31) Phillip Lougher1054worker1 # [ 0.365550] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1055worker2 # [ 0.325032] workingset: timestamp_bits=42 max_order=19 bucket_order=01056worker1 # [ 0.365580] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1057lb # [ 0.321903] 9p: Installing v9fs 9p2000 file system support1058worker2 # [ 0.326286] squashfs: version 4.0 (2009/01/31) Phillip Lougher1059worker1 # [ 0.385064] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1060worker1 # [ 0.385248] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1061worker2 # [ 0.327072] 9p: Installing v9fs 9p2000 file system support1062worker1 # [ 0.385264] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1063worker1 # [ 0.385294] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1064worker1 # [ 0.385763] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1065worker1 # [ 0.385944] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1066worker1 # [ 0.385960] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1067worker1 # [ 0.385990] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1068worker1 # [ 0.392184] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1069worker1 # [ 0.392633] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1070worker1 # [ 0.392813] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1071worker1 # [ 0.392843] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1072worker1 # [ 0.393278] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1073worker1 # [ 0.393458] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1074worker1 # [ 0.393487] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1075worker1 # [ 0.393865] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1076worker1 # [ 0.394050] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1077worker1 # [ 0.394339] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1078worker1 # [ 0.394519] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1079worker1 # [ 0.394548] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1080client # [ 0.491012] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1081worker1 # [ 0.394988] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1082worker1 # [ 0.395171] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1083worker2 # [ 0.351111] Key type asymmetric registered1084worker1 # [ 0.395201] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1085worker2 # [ 0.351136] Asymmetric key parser 'x509' registered1086worker1 # [ 0.395645] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1087worker2 # [ 0.351217] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)1088worker1 # [ 0.395826] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1089worker2 # [ 0.353441] io scheduler mq-deadline registered1090worker1 # [ 0.395856] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1091lb # [ 0.350647] Key type asymmetric registered1092worker2 # [ 0.353450] io scheduler kyber registered1093lb # [ 0.350674] Asymmetric key parser 'x509' registered1094worker1 # [ 0.396300] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1095lb # [ 0.350744] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)1096worker1 # [ 0.396555] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1097lb # [ 0.352857] io scheduler mq-deadline registered1098worker1 # [ 0.396573] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1099lb # [ 0.352868] io scheduler kyber registered1100worker1 # [ 0.396602] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1101worker1 # [ 0.397045] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1102worker1 # [ 0.397230] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1103worker1 # [ 0.397246] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1104worker1 # [ 0.397275] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1105worker1 # [ 0.397820] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1106client # [ 0.500947] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1107worker1 # [ 0.397831] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1108worker1 # [ 0.397836] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1109worker1 # [ 0.397879] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1110worker1 # [ 0.397924] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1111worker1 # [ 0.397969] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1112lb # [ 0.362702] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1113client # [ 0.505370] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1114client # [ 0.507311] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1115lb # [ 0.364108] ledtrig-cpu: registered to indicate activity on CPUs1116client # [ 0.509340] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1117lb # [ 0.364520] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1118lb # [ 0.364538] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001119lb # [ 0.364563] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001120lb # [ 0.364573] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001121lb # [ 0.364594] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1122lb # [ 0.364618] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1123worker2 # [ 0.368754] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1124lb # [ 0.364697] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001125worker2 # [ 0.369425] ledtrig-cpu: registered to indicate activity on CPUs1126lb # [ 0.364706] pci_bus 0000:00: root bus resource [bus 00-ff]1127worker2 # [ 0.369801] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1128lb # [ 0.364723] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1129worker2 # [ 0.369819] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001130lb # [ 0.364729] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1131lb # [ 0.364735] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1132worker2 # [ 0.369831] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001133client # [ 0.519651] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1134lb # [ 0.364790] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1135worker2 # [ 0.369840] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001136client # [ 0.521466] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1137lb # [ 0.365222] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1138worker2 # [ 0.369861] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1139client # [ 0.523243] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1140lb # [ 0.365406] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1141worker1 # [ 0.438107] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1142worker2 # [ 0.369889] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1143lb # [ 0.365423] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1144worker2 # [ 0.369966] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001145lb # [ 0.365453] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1146worker1 # [ 0.438173] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1147lb # [ 0.365469] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1148worker2 # [ 0.369976] pci_bus 0000:00: root bus resource [bus 00-ff]1149worker1 # [ 0.438222] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1150worker2 # [ 0.369983] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1151lb # [ 0.365927] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1152worker1 # [ 0.438269] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1153worker2 # [ 0.369988] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1154lb # [ 0.366108] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1155worker1 # [ 0.438316] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1156worker2 # [ 0.369994] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1157lb # [ 0.366125] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1158lb # [ 0.366154] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1159worker1 # [ 0.438364] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1160worker2 # [ 0.370071] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1161worker1 # [ 0.438483] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1162worker2 # [ 0.370525] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1163worker2 # [ 0.370718] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1164worker1 # [ 0.438533] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1165lb # [ 0.385723] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1166worker2 # [ 0.370735] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1167lb # [ 0.385909] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1168worker1 # [ 0.438556] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1169lb # [ 0.385925] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1170worker2 # [ 0.370764] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1171worker1 # [ 0.438579] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1172lb # [ 0.385956] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1173worker2 # [ 0.370781] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1174worker1 # [ 0.438601] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1175worker2 # [ 0.371253] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1176lb # [ 0.386437] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1177worker1 # [ 0.438623] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1178worker2 # [ 0.371443] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1179worker1 # [ 0.438645] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1180client # [ 0.533540] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1181worker2 # [ 0.371459] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1182client # [ 0.535475] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1183worker1 # [ 0.438667] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1184worker2 # [ 0.371490] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1185worker1 # [ 0.438696] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1186worker2 # [ 0.371955] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1187worker1 # [ 0.438718] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1188worker2 # [ 0.372140] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1189worker1 # [ 0.438740] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1190worker2 # [ 0.372156] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1191worker1 # [ 0.438763] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1192worker2 # [ 0.372185] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1193worker1 # [ 0.438785] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1194worker2 # [ 0.372663] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1195worker1 # [ 0.438807] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1196worker2 # [ 0.372848] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1197worker1 # [ 0.438828] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1198worker2 # [ 0.372863] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1199worker1 # [ 0.438849] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1200worker2 # [ 0.372893] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1201worker1 # [ 0.438871] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1202worker2 # [ 0.372911] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1203worker1 # [ 0.438892] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1204worker1 # [ 0.438913] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1205worker2 # [ 0.373373] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1206worker1 # [ 0.438942] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1207worker2 # [ 0.373559] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1208worker1 # [ 0.438952] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1209worker2 # [ 0.373589] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1210client # [ 0.545543] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1211worker1 # [ 0.438957] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1212worker2 # [ 0.374048] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1213lb # [ 0.394695] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1214worker1 # [ 0.439772] pci 0000:00:07.0: enabling device (0000 -> 0002)1215worker2 # [ 0.374235] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1216lb # [ 0.394723] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1217worker2 # [ 0.374268] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1218lb # [ 0.394754] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1219lb # [ 0.394774] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1220worker2 # [ 0.374688] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1221worker2 # [ 0.374874] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1222lb # [ 0.395279] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1223lb # [ 0.395474] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1224worker2 # [ 0.375146] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1225worker2 # [ 0.375334] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1226lb # [ 0.395504] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1227worker2 # [ 0.375364] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1228lb # [ 0.395988] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1229worker2 # [ 0.375834] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1230lb # [ 0.396175] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1231worker2 # [ 0.376020] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1232lb # [ 0.396205] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1233worker2 # [ 0.376050] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1234lb # [ 0.396596] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1235worker2 # [ 0.376502] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1236lb # [ 0.396777] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1237lb # [ 0.397054] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1238lb # [ 0.397240] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1239lb # [ 0.397270] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1240client # [ 0.560773] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1241lb # [ 0.397724] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1242lb # [ 0.397909] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1243lb # [ 0.397938] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1244client # [ 0.563287] msm_serial: driver initialized1245client # [ 0.563431] SuperH (H)SCI(F) driver initialized1246lb # [ 0.398386] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1247client # [ 0.563485] STM32 USART driver initialized1248lb # [ 0.398570] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1249lb # [ 0.398616] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1250lb # [ 0.399076] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1251lb # [ 0.399351] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1252worker2 # [ 0.420782] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1253lb # [ 0.399368] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1254worker2 # [ 0.420829] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1255lb # [ 0.399398] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1256worker2 # [ 0.421360] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1257worker2 # [ 0.421679] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1258lb # [ 0.399882] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1259worker1 # [ 0.483937] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1260worker2 # [ 0.421696] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1261lb # [ 0.400064] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1262lb # [ 0.400079] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1263worker2 # [ 0.421726] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1264lb # [ 0.400109] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1265worker2 # [ 0.422185] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1266lb # [ 0.400696] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1267worker2 # [ 0.422368] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1268worker2 # [ 0.422383] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1269lb # [ 0.400707] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1270worker2 # [ 0.422413] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1271lb # [ 0.400713] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1272worker2 # [ 0.422988] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1273lb # [ 0.400758] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1274worker2 # [ 0.423016] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1275lb # [ 0.400803] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1276worker2 # [ 0.423023] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1277lb # [ 0.400850] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1278worker2 # [ 0.423069] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1279lb # [ 0.400896] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1280worker2 # [ 0.423116] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1281lb # [ 0.400942] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1282worker2 # [ 0.423163] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1283lb # [ 0.400989] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1284worker2 # [ 0.423210] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1285lb # [ 0.401035] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1286worker2 # [ 0.423258] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1287lb # [ 0.401081] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1288worker2 # [ 0.423305] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1289lb # [ 0.401127] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1290worker2 # [ 0.423352] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1291lb # [ 0.401198] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1292worker2 # [ 0.423399] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1293lb # [ 0.401244] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1294worker2 # [ 0.423445] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1295worker1 # [ 0.494136] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1296lb # [ 0.401266] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1297worker1 # [ 0.497868] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1298worker2 # [ 0.423513] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1299lb # [ 0.401288] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1300worker1 # [ 0.499970] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1301worker2 # [ 0.423559] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1302lb # [ 0.401309] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1303worker2 # [ 0.423580] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1304lb # [ 0.401331] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1305worker2 # [ 0.423601] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1306lb # [ 0.401353] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1307worker2 # [ 0.423621] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1308lb # [ 0.401375] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1309worker2 # [ 0.423643] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1310lb # [ 0.401399] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1311worker2 # [ 0.423663] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1312lb # [ 0.401421] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1313worker2 # [ 0.423684] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1314lb # [ 0.401443] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1315worker2 # [ 0.423710] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1316lb # [ 0.401466] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1317worker2 # [ 0.423732] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1318lb # [ 0.401488] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1319worker2 # [ 0.423753] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1320lb # [ 0.401509] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1321worker2 # [ 0.423775] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1322lb # [ 0.401531] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1323worker2 # [ 0.423797] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1324lb # [ 0.401552] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1325worker2 # [ 0.423817] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1326lb # [ 0.401574] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1327worker1 # [ 0.510183] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1328worker2 # [ 0.423838] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1329lb # [ 0.401595] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1330worker1 # [ 0.512413] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1331worker2 # [ 0.423858] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1332client # [ 0.597726] loop: module loaded1333lb # [ 0.401617] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1334worker2 # [ 0.423879] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1335lb # [ 0.401644] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1336client # [ 0.597977] virtio_blk virtio2: 1/0/0 default/read/poll queues1337worker2 # [ 0.423899] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1338lb # [ 0.401653] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1339client # [ 0.598806] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1340worker2 # [ 0.423921] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1341lb # [ 0.401658] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1342worker2 # [ 0.423946] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1343lb # [ 0.402460] pci 0000:00:07.0: enabling device (0000 -> 0002)1344worker2 # [ 0.423956] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1345worker2 # [ 0.423960] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1346worker2 # [ 0.464858] pci 0000:00:07.0: enabling device (0000 -> 0002)1347client # [ 0.604107] megasas: 07.734.00.00-rc11348lb # [ 0.462698] pci 0000:00:07.0: quirk_usb_early_handoff+0x0/0xa60 took 58829 usecs1349client # [ 0.604841] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1350worker1 # [ 0.522166] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1351worker1 # [ 0.524527] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1352worker1 # [ 0.526416] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1353worker1 # [ 0.528468] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1354worker1 # [ 0.538591] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1355client # [ 0.624683] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001356client # [ 0.624714] Intel/Sharp Extended Query Table at 0x00311357client # [ 0.626294] Using buffer write method1358client # [ 0.626338] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1359worker2 # [ 0.486033] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1360worker2 # [ 0.488158] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1361client # [ 0.631361] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001362client # [ 0.631384] Intel/Sharp Extended Query Table at 0x00311363client # [ 0.635768] Using buffer write method1364client # [ 0.635791] Concatenating MTD devices:1365client # [ 0.635795] (0): "0.flash"1366lb # [ 0.489135] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1367client # [ 0.635799] (1): "0.flash"1368client # [ 0.635810] into device "0.flash"1369worker1 # [ 0.555271] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1370lb # [ 0.492100] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1371lb # [ 0.494077] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1372worker2 # [ 0.499178] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1373worker1 # [ 0.558221] msm_serial: driver initialized1374worker1 # [ 0.558382] SuperH (H)SCI(F) driver initialized1375worker1 # [ 0.558436] STM32 USART driver initialized1376lb # [ 0.504035] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1377lb # [ 0.506179] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1378worker2 # [ 0.509192] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1379worker2 # [ 0.512866] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1380worker2 # [ 0.514881] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1381lb # [ 0.517664] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1382lb # [ 0.520191] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1383lb # [ 0.522058] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1384worker2 # [ 0.524698] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1385worker2 # [ 0.526901] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1386worker1 # [ 0.588527] loop: module loaded1387worker1 # [ 0.588703] virtio_blk virtio2: 1/0/0 default/read/poll queues1388worker1 # [ 0.589500] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1389lb # [ 0.532191] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1390lb # [ 0.534127] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1391worker2 # [ 0.536777] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1392worker2 # [ 0.539326] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1393worker1 # [ 0.594608] megasas: 07.734.00.00-rc11394worker1 # [ 0.595351] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1395worker2 # [ 0.543123] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1396lb # [ 0.545567] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1397worker2 # [ 0.557807] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1398worker2 # [ 0.560425] msm_serial: driver initialized1399worker2 # [ 0.560571] SuperH (H)SCI(F) driver initialized1400lb # [ 0.559821] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1401worker1 # [ 0.619258] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001402worker1 # [ 0.619294] Intel/Sharp Extended Query Table at 0x00311403worker1 # [ 0.620981] Using buffer write method1404lb # [ 0.562533] msm_serial: driver initialized1405worker1 # [ 0.621048] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1406worker2 # [ 0.561724] STM32 USART driver initialized1407lb # [ 0.563409] SuperH (H)SCI(F) driver initialized1408lb # [ 0.563471] STM32 USART driver initialized1409worker1 # [ 0.626186] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001410worker1 # [ 0.626257] Intel/Sharp Extended Query Table at 0x00311411worker1 # [ 0.628026] Using buffer write method1412worker1 # [ 0.628052] Concatenating MTD devices:1413worker1 # [ 0.628056] (0): "0.flash"1414worker1 # [ 0.628060] (1): "0.flash"1415worker1 # [ 0.628070] into device "0.flash"1416worker2 # [ 0.594794] loop: module loaded1417worker2 # [ 0.594980] virtio_blk virtio2: 1/0/0 default/read/poll queues1418lb # [ 0.593381] loop: module loaded1419worker2 # [ 0.595995] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1420lb # [ 0.593570] virtio_blk virtio2: 1/0/0 default/read/poll queues1421lb # [ 0.594359] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1422worker2 # [ 0.601217] megasas: 07.734.00.00-rc11423worker2 # [ 0.601929] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1424lb # [ 0.599157] megasas: 07.734.00.00-rc11425lb # [ 0.599864] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1426worker2 # [ 0.612616] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001427worker2 # [ 0.612643] Intel/Sharp Extended Query Table at 0x00311428worker2 # [ 0.614322] Using buffer write method1429lb # [ 0.611749] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001430lb # [ 0.611780] Intel/Sharp Extended Query Table at 0x00311431worker2 # [ 0.614385] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1432lb # [ 0.613381] Using buffer write method1433lb # [ 0.613444] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1434worker2 # [ 0.620280] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001435worker2 # [ 0.620305] Intel/Sharp Extended Query Table at 0x00311436lb # [ 0.624856] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001437worker2 # [ 0.630431] Using buffer write method1438worker2 # [ 0.630468] Concatenating MTD devices:1439lb # [ 0.624886] Intel/Sharp Extended Query Table at 0x00311440worker2 # [ 0.630472] (0): "0.flash"1441worker2 # [ 0.630476] (1): "0.flash"1442worker2 # [ 0.630484] into device "0.flash"1443lb # [ 0.634618] Using buffer write method1444lb # [ 0.634644] Concatenating MTD devices:1445lb # [ 0.634648] (0): "0.flash"1446lb # [ 0.634656] (1): "0.flash"1447lb # [ 0.634660] into device "0.flash"1448client # [ 0.867781] Freeing initrd memory: 26972K1449client # [ 0.873784] tun: Universal TUN/TAP device driver, 1.61450client # [ 0.877490] thunder_xcv, ver 1.01451client # [ 0.877532] thunder_bgx, ver 1.01452client # [ 0.877555] nicpf, ver 1.01453client # [ 0.878124] e1000: Intel(R) PRO/1000 Network Driver1454client # [ 0.878131] e1000: Copyright (c) 1999-2006 Intel Corporation.1455client # [ 0.878153] e1000e: Intel(R) PRO/1000 Network Driver1456client # [ 0.878161] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1457client # [ 0.878189] igb: Intel(R) Gigabit Ethernet Network Driver1458client # [ 0.878194] igb: Copyright (c) 2007-2014 Intel Corporation.1459client # [ 0.878216] igbvf: Intel(R) Gigabit Virtual Function Network Driver1460client # [ 0.878222] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1461client # [ 0.878355] sky2: driver version 1.301462client # [ 0.880241] ehci-pci 0000:00:07.0: EHCI Host Controller1463client # [ 0.880266] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11464client # [ 0.880517] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001465client # [ 0.890304] usbcore: registered new interface driver usb-storage1466client # [ 0.890354] usbcore: registered new interface driver usbserial_generic1467client # [ 0.890367] usbserial: USB Serial support registered for generic1468client # [ 0.890948] hv_vmbus: registering driver hyperv_keyboard1469client # [ 0.891605] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001470client # [ 0.891902] hub 1-0:1.0: USB hub found1471client # [ 0.891922] hub 1-0:1.0: 6 ports detected1472client # [ 0.897080] rtc-pl031 9010000.pl031: registered as rtc01473client # [ 0.897109] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T11:24:59 UTC (1790076299)1474client # [ 0.897421] i2c_dev: i2c /dev entries driver1475client # [ 0.902812] sdhci: Secure Digital Host Controller Interface driver1476client # [ 0.902826] sdhci: Copyright(c) Pierre Ossman1477client # [ 0.903085] Synopsys Designware Multimedia Card Interface Driver1478client # [ 0.903450] sdhci-pltfm: SDHCI platform and OF driver helper1479client # [ 0.907710] hid: raw HID events driver (C) Jiri Kosina1480client # [ 0.907951] usbcore: registered new interface driver usbhid1481client # [ 0.907958] usbhid: USB HID core driver1482client # [ 0.910858] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1483client # [ 0.913473] drop_monitor: Initializing network drop monitor service1484client # [ 0.913631] NET: Registered PF_INET6 protocol family1485client # [ 0.915519] Segment Routing with IPv61486client # [ 0.916202] In-situ OAM (IOAM) with IPv61487client # [ 0.916240] NET: Registered PF_PACKET protocol family1488client # [ 0.917388] 9pnet: Installing 9P2000 support1489client # [ 0.917435] Key type dns_resolver registered1490client # [ 0.924124] registered taskstats version 11491client # [ 0.924281] Loading compiled-in X.509 certificates1492client # [ 0.932949] Demotion targets for Node 0: null1493client # [ 0.933052] Key type .fscrypt registered1494client # [ 0.933058] Key type fscrypt-provisioning registered1495client # [ 0.933149] ima: No TPM chip found, activating TPM-bypass!1496client # [ 0.933168] ima: Allocated hash algorithm: sha11497client # [ 0.933190] ima: No architecture policies found1498client # [ 0.937255] input: gpio-keys as /devices/platform/gpio-keys/input/input01499client # [ 0.955064] clk: Disabling unused clocks1500client # [ 0.955094] PM: genpd: Disabling unused power domains1501worker1 # [ 0.872968] Freeing initrd memory: 26976K1502client # [ 0.959254] Freeing unused kernel memory: 4736K1503client # [ 0.959452] Run /init as init process1504worker1 # [ 0.879223] tun: Universal TUN/TAP device driver, 1.61505worker1 # [ 0.883027] thunder_xcv, ver 1.01506worker1 # [ 0.883073] thunder_bgx, ver 1.01507worker1 # [ 0.883095] nicpf, ver 1.01508worker1 # [ 0.883725] e1000: Intel(R) PRO/1000 Network Driver1509worker1 # [ 0.883732] e1000: Copyright (c) 1999-2006 Intel Corporation.1510worker1 # [ 0.883757] e1000e: Intel(R) PRO/1000 Network Driver1511client # [ 0.973796] systemd[1]: Successfully made /usr/ read-only.1512worker1 # [ 0.883765] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1513worker1 # [ 0.883794] igb: Intel(R) Gigabit Ethernet Network Driver1514worker1 # [ 0.883799] igb: Copyright (c) 2007-2014 Intel Corporation.1515worker1 # [ 0.883823] igbvf: Intel(R) Gigabit Virtual Function Network Driver1516worker1 # [ 0.883829] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1517worker1 # [ 0.883965] sky2: driver version 1.301518worker1 # [ 0.885536] usbcore: registered new interface driver usb-storage1519worker1 # [ 0.885619] usbcore: registered new interface driver usbserial_generic1520worker1 # [ 0.885634] usbserial: USB Serial support registered for generic1521worker1 # [ 0.886727] ehci-pci 0000:00:07.0: EHCI Host Controller1522worker1 # [ 0.886752] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11523worker1 # [ 0.887025] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001524worker1 # [ 0.898539] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001525worker1 # [ 0.898863] hub 1-0:1.0: USB hub found1526worker1 # [ 0.898879] hub 1-0:1.0: 6 ports detected1527worker1 # [ 0.901164] hv_vmbus: registering driver hyperv_keyboard1528worker1 # [ 0.902806] rtc-pl031 9010000.pl031: registered as rtc01529worker1 # [ 0.902832] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T11:24:59 UTC (1790076299)1530worker1 # [ 0.903138] i2c_dev: i2c /dev entries driver1531worker1 # [ 0.908484] sdhci: Secure Digital Host Controller Interface driver1532worker1 # [ 0.908495] sdhci: Copyright(c) Pierre Ossman1533worker1 # [ 0.908759] Synopsys Designware Multimedia Card Interface Driver1534worker1 # [ 0.909130] sdhci-pltfm: SDHCI platform and OF driver helper1535worker1 # [ 0.913346] hid: raw HID events driver (C) Jiri Kosina1536worker1 # [ 0.913623] usbcore: registered new interface driver usbhid1537worker1 # [ 0.913629] usbhid: USB HID core driver1538worker1 # [ 0.916489] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1539worker1 # [ 0.919158] drop_monitor: Initializing network drop monitor service1540worker1 # [ 0.919321] NET: Registered PF_INET6 protocol family1541worker1 # [ 0.921194] Segment Routing with IPv61542worker1 # [ 0.921213] In-situ OAM (IOAM) with IPv61543worker1 # [ 0.921240] NET: Registered PF_PACKET protocol family1544worker1 # [ 0.923076] 9pnet: Installing 9P2000 support1545worker1 # [ 0.923126] Key type dns_resolver registered1546worker1 # [ 0.929900] registered taskstats version 11547worker2 # [ 0.872259] Freeing initrd memory: 26944K1548worker1 # [ 0.930575] Loading compiled-in X.509 certificates1549worker2 # [ 0.878299] tun: Universal TUN/TAP device driver, 1.61550worker1 # [ 0.938783] Demotion targets for Node 0: null1551worker1 # [ 0.938893] Key type .fscrypt registered1552worker1 # [ 0.938901] Key type fscrypt-provisioning registered1553worker1 # [ 0.939000] ima: No TPM chip found, activating TPM-bypass!1554worker1 # [ 0.939019] ima: Allocated hash algorithm: sha11555worker1 # [ 0.939041] ima: No architecture policies found1556worker1 # [ 0.943011] input: gpio-keys as /devices/platform/gpio-keys/input/input01557worker2 # [ 0.881991] thunder_xcv, ver 1.01558worker2 # [ 0.882033] thunder_bgx, ver 1.01559worker2 # [ 0.882056] nicpf, ver 1.01560worker2 # [ 0.882627] e1000: Intel(R) PRO/1000 Network Driver1561worker2 # [ 0.882634] e1000: Copyright (c) 1999-2006 Intel Corporation.1562lb # [ 0.883015] Freeing initrd memory: 26948K1563worker2 # [ 0.882659] e1000e: Intel(R) PRO/1000 Network Driver1564worker2 # [ 0.882667] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1565worker2 # [ 0.882696] igb: Intel(R) Gigabit Ethernet Network Driver1566worker2 # [ 0.882702] igb: Copyright (c) 2007-2014 Intel Corporation.1567worker2 # [ 0.882725] igbvf: Intel(R) Gigabit Virtual Function Network Driver1568worker2 # [ 0.882730] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1569worker2 # [ 0.882862] sky2: driver version 1.301570lb # [ 0.889040] tun: Universal TUN/TAP device driver, 1.61571worker2 # [ 0.884460] usbcore: registered new interface driver usb-storage1572worker2 # [ 0.884543] usbcore: registered new interface driver usbserial_generic1573worker2 # [ 0.884557] usbserial: USB Serial support registered for generic1574worker2 # [ 0.885344] ehci-pci 0000:00:07.0: EHCI Host Controller1575worker2 # [ 0.885370] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11576worker2 # [ 0.885648] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001577lb # [ 0.892832] thunder_xcv, ver 1.01578worker2 # [ 0.897198] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001579lb # [ 0.892879] thunder_bgx, ver 1.01580worker2 # [ 0.897505] hub 1-0:1.0: USB hub found1581lb # [ 0.892901] nicpf, ver 1.01582worker2 # [ 0.897531] hub 1-0:1.0: 6 ports detected1583lb # [ 0.893480] e1000: Intel(R) PRO/1000 Network Driver1584lb # [ 0.893487] e1000: Copyright (c) 1999-2006 Intel Corporation.1585lb # [ 0.893512] e1000e: Intel(R) PRO/1000 Network Driver1586worker2 # [ 0.900112] hv_vmbus: registering driver hyperv_keyboard1587lb # [ 0.893519] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1588lb # [ 0.893552] igb: Intel(R) Gigabit Ethernet Network Driver1589lb # [ 0.893557] igb: Copyright (c) 2007-2014 Intel Corporation.1590worker2 # [ 0.901668] rtc-pl031 9010000.pl031: registered as rtc01591lb # [ 0.893580] igbvf: Intel(R) Gigabit Virtual Function Network Driver1592worker1 # [ 0.961257] clk: Disabling unused clocks1593lb # [ 0.893585] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1594worker1 # [ 0.961285] PM: genpd: Disabling unused power domains1595worker2 # [ 0.901698] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T11:24:59 UTC (1790076299)1596lb # [ 0.893719] sky2: driver version 1.301597worker2 # [ 0.902006] i2c_dev: i2c /dev entries driver1598lb # [ 0.902425] usbcore: registered new interface driver usb-storage1599worker1 # [ 0.965383] Freeing unused kernel memory: 4736K1600lb # [ 0.902550] usbcore: registered new interface driver usbserial_generic1601worker1 # [ 0.965606] Run /init as init process1602lb # [ 0.902564] usbserial: USB Serial support registered for generic1603worker2 # [ 0.907298] sdhci: Secure Digital Host Controller Interface driver1604lb # [ 0.903211] ehci-pci 0000:00:07.0: EHCI Host Controller1605worker2 # [ 0.907307] sdhci: Copyright(c) Pierre Ossman1606lb # [ 0.903234] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11607worker2 # [ 0.907568] Synopsys Designware Multimedia Card Interface Driver1608lb # [ 0.903397] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001609worker2 # [ 0.907935] sdhci-pltfm: SDHCI platform and OF driver helper1610lb # [ 0.908670] hv_vmbus: registering driver hyperv_keyboard1611worker2 # [ 0.912151] hid: raw HID events driver (C) Jiri Kosina1612worker2 # [ 0.912385] usbcore: registered new interface driver usbhid1613worker2 # [ 0.912391] usbhid: USB HID core driver1614lb # [ 0.910246] rtc-pl031 9010000.pl031: registered as rtc01615lb # [ 0.910276] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T11:24:59 UTC (1790076299)1616lb # [ 0.912314] i2c_dev: i2c /dev entries driver1617worker2 # [ 0.915282] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1618worker2 # [ 0.917899] drop_monitor: Initializing network drop monitor service1619worker2 # [ 0.918089] NET: Registered PF_INET6 protocol family1620worker2 # [ 0.920030] Segment Routing with IPv61621worker2 # [ 0.920049] In-situ OAM (IOAM) with IPv61622worker2 # [ 0.920077] NET: Registered PF_PACKET protocol family1623worker1 # [ 0.980201] systemd[1]: Successfully made /usr/ read-only.1624lb # [ 0.916070] sdhci: Secure Digital Host Controller Interface driver1625lb # [ 0.916081] sdhci: Copyright(c) Pierre Ossman1626lb # [ 0.916352] Synopsys Designware Multimedia Card Interface Driver1627lb # [ 0.916723] sdhci-pltfm: SDHCI platform and OF driver helper1628lb # [ 0.918280] hid: raw HID events driver (C) Jiri Kosina1629worker2 # [ 0.921886] 9pnet: Installing 9P2000 support1630worker2 # [ 0.921931] Key type dns_resolver registered1631lb # [ 0.918510] usbcore: registered new interface driver usbhid1632lb # [ 0.918516] usbhid: USB HID core driver1633lb # [ 0.923114] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001634lb # [ 0.924240] hub 1-0:1.0: USB hub found1635lb # [ 0.924754] hub 1-0:1.0: 6 ports detected1636worker2 # [ 0.928558] registered taskstats version 11637lb # [ 0.926708] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1638worker2 # [ 0.929243] Loading compiled-in X.509 certificates1639lb # [ 0.928243] drop_monitor: Initializing network drop monitor service1640lb # [ 0.928367] NET: Registered PF_INET6 protocol family1641lb # [ 0.931403] Segment Routing with IPv61642lb # [ 0.931420] In-situ OAM (IOAM) with IPv61643lb # [ 0.931450] NET: Registered PF_PACKET protocol family1644lb # [ 0.933112] 9pnet: Installing 9P2000 support1645worker2 # [ 0.937359] Demotion targets for Node 0: null1646worker2 # [ 0.937471] Key type .fscrypt registered1647lb # [ 0.933162] Key type dns_resolver registered1648worker2 # [ 0.937478] Key type fscrypt-provisioning registered1649worker2 # [ 0.937573] ima: No TPM chip found, activating TPM-bypass!1650worker2 # [ 0.937591] ima: Allocated hash algorithm: sha11651worker2 # [ 0.937613] ima: No architecture policies found1652worker2 # [ 0.941596] input: gpio-keys as /devices/platform/gpio-keys/input/input01653lb # [ 0.940031] registered taskstats version 11654lb # [ 0.940176] Loading compiled-in X.509 certificates1655lb # [ 0.949085] Demotion targets for Node 0: null1656lb # [ 0.949188] Key type .fscrypt registered1657lb # [ 0.949194] Key type fscrypt-provisioning registered1658lb # [ 0.949292] ima: No TPM chip found, activating TPM-bypass!1659lb # [ 0.949311] ima: Allocated hash algorithm: sha11660lb # [ 0.949333] ima: No architecture policies found1661lb # [ 0.953419] input: gpio-keys as /devices/platform/gpio-keys/input/input01662worker2 # [ 0.959835] clk: Disabling unused clocks1663worker2 # [ 0.959859] PM: genpd: Disabling unused power domains1664worker2 # [ 0.963980] Freeing unused kernel memory: 4736K1665worker2 # [ 0.964200] Run /init as init process1666lb # [ 0.971697] clk: Disabling unused clocks1667lb # [ 0.971720] PM: genpd: Disabling unused power domains1668worker2 # [ 0.978767] systemd[1]: Successfully made /usr/ read-only.1669lb # [ 0.975900] Freeing unused kernel memory: 4736K1670lb # [ 0.976099] Run /init as init process1671lb # [ 0.992830] systemd[1]: Successfully made /usr/ read-only.1672client # [ 1.139623] usb 1-1: new high-speed USB device number 2 using ehci-pci1673worker1 # [ 1.146090] usb 1-1: new high-speed USB device number 2 using ehci-pci1674worker2 # [ 1.144614] usb 1-1: new high-speed USB device number 2 using ehci-pci1675client # [ 1.289659] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11676lb # [ 1.170668] usb 1-1: new high-speed USB device number 2 using ehci-pci1677client # [ 1.308621] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1678client # [ 1.320920] systemd[1]: Detected virtualization qemu.1679client # [ 1.322939] systemd[1]: Detected architecture arm64.1680client # [ 1.324934] systemd[1]: Running in initrd.1681client # [ 1.327677] systemd[1]: Initializing machine ID from random generator.1682client # [ 1.330521] systemd[1]: Hostname set to <client>.1683client # [ 1.375837] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01684worker1 # [ 1.296001] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11685worker1 # [ 1.315136] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1686worker1 # [ 1.321157] systemd[1]: Detected virtualization qemu.1687worker1 # [ 1.322093] systemd[1]: Detected architecture arm64.1688worker1 # [ 1.322940] systemd[1]: Running in initrd.1689worker1 # [ 1.324170] systemd[1]: Initializing machine ID from random generator.1690worker1 # [ 1.325424] systemd[1]: Hostname set to <worker1>.1691worker2 # [ 1.297140] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11692worker1 # [ 1.378246] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01693worker2 # [ 1.313408] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1694worker2 # [ 1.325665] systemd[1]: Detected virtualization qemu.1695worker2 # [ 1.327664] systemd[1]: Detected architecture arm64.1696worker2 # [ 1.329624] systemd[1]: Running in initrd.1697lb # [ 1.323109] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11698worker2 # [ 1.332211] systemd[1]: Initializing machine ID from random generator.1699worker2 # [ 1.335336] systemd[1]: Hostname set to <worker2>.1700lb # [ 1.329343] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1701lb # [ 1.341811] systemd[1]: Detected virtualization qemu.1702lb # [ 1.341910] systemd[1]: Detected architecture arm64.1703lb # [ 1.341935] systemd[1]: Running in initrd.1704lb # [ 1.348519] systemd[1]: Initializing machine ID from random generator.1705lb # [ 1.351392] systemd[1]: Hostname set to <lb>.1706client # [ 1.499598] usb 1-2: new high-speed USB device number 3 using ehci-pci1707worker2 # [ 1.380876] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01708lb # [ 1.406836] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01709worker1 # [ 1.498092] usb 1-2: new high-speed USB device number 3 using ehci-pci1710client # [ 1.642628] systemd[1]: bpf-restrict-fs: LSM BPF program attached1711worker2 # [ 1.504673] usb 1-2: new high-speed USB device number 3 using ehci-pci1712client # [ 1.662321] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21713lb # [ 1.526621] usb 1-2: new high-speed USB device number 3 using ehci-pci1714client # [ 1.671739] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01715worker1 # [ 1.616791] systemd[1]: bpf-restrict-fs: LSM BPF program attached1716worker1 # [ 1.654659] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21717worker1 # [ 1.658970] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01718client # [ 1.756696] systemd[1]: Queued start job for default target Initrd Default Target.1719client # [ 1.766366] systemd[1]: Created slice Slice /system/modprobe.1720client # [ 1.767666] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1721client # [ 1.767713] systemd[1]: Expecting device /dev/disk/by-label/nixos...1722client # [ 1.767745] systemd[1]: Reached target Path Units.1723client # [ 1.767768] systemd[1]: Reached target Slice Units.1724client # [ 1.767789] systemd[1]: Reached target Swaps.1725client # [ 1.767807] systemd[1]: Reached target Timer Units.1726client # [ 1.768019] systemd[1]: Listening on D-Bus System Message Bus Socket.1727client # [ 1.768179] systemd[1]: Listening on Journal Socket (/dev/log).1728client # [ 1.768326] systemd[1]: Listening on Journal Sockets.1729client # [ 1.768459] systemd[1]: Listening on udev Control Socket.1730client # [ 1.768582] systemd[1]: Listening on udev Kernel Socket.1731client # [ 1.768603] systemd[1]: Reached target Socket Units.1732client # [ 1.780936] systemd[1]: Starting Create List of Static Device Nodes...1733client # [ 1.782085] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1734worker2 # [ 1.645313] systemd[1]: bpf-restrict-fs: LSM BPF program attached1735client # [ 1.791741] systemd[1]: Mounting Kernel Configuration File System...1736client # [ 1.799886] systemd[1]: Starting Journal Service...1737lb # [ 1.658175] systemd[1]: bpf-restrict-fs: LSM BPF program attached1738worker1 # [ 1.723912] systemd[1]: Queued start job for default target Initrd Default Target.1739worker2 # [ 1.665151] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21740worker2 # [ 1.670281] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01741worker1 # [ 1.732373] systemd[1]: Created slice Slice /system/modprobe.1742worker1 # [ 1.733582] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1743worker1 # [ 1.735088] systemd[1]: Expecting device /dev/disk/by-label/nixos...1744worker1 # [ 1.735130] systemd[1]: Reached target Path Units.1745worker1 # [ 1.735154] systemd[1]: Reached target Slice Units.1746worker1 # [ 1.735175] systemd[1]: Reached target Swaps.1747client # [ 1.823940] systemd[1]: Starting Load Kernel Modules...1748worker1 # [ 1.735197] systemd[1]: Reached target Timer Units.1749client # [ 1.824921] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1750worker1 # [ 1.735395] systemd[1]: Listening on D-Bus System Message Bus Socket.1751worker1 # [ 1.735550] systemd[1]: Listening on Journal Socket (/dev/log).1752worker1 # [ 1.735691] systemd[1]: Listening on Journal Sockets.1753worker1 # [ 1.735828] systemd[1]: Listening on udev Control Socket.1754worker1 # [ 1.735944] systemd[1]: Listening on udev Kernel Socket.1755worker1 # [ 1.735967] systemd[1]: Reached target Socket Units.1756lb # [ 1.681274] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21757worker1 # [ 1.746936] systemd[1]: Starting Create List of Static Device Nodes...1758worker1 # [ 1.748184] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1759lb # [ 1.686453] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01760worker1 # [ 1.755753] systemd[1]: Mounting Kernel Configuration File System...1761client # [ 1.846021] systemd-journald[72]: Collecting audit messages is disabled.1762worker1 # [ 1.766294] systemd[1]: Starting Journal Service...1763client # [ 1.847171] systemd[1]: Starting Coldplug All udev Devices...1764client # [ 1.867749] systemd[1]: Finished Create List of Static Device Nodes.1765client # [ 1.869327] systemd[1]: Mounted Kernel Configuration File System.1766worker1 # [ 1.790176] systemd[1]: Starting Load Kernel Modules...1767worker1 # [ 1.791172] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1768client # [ 1.884003] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1769worker1 # [ 1.809637] systemd[1]: Starting Coldplug All udev Devices...1770worker2 # [ 1.751202] systemd[1]: Queued start job for default target Initrd Default Target.1771worker1 # [ 1.818988] systemd-journald[72]: Collecting audit messages is disabled.1772worker2 # [ 1.762893] systemd[1]: Created slice Slice /system/modprobe.1773worker2 # [ 1.764088] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1774worker2 # [ 1.765415] systemd[1]: Expecting device /dev/disk/by-label/nixos...1775worker2 # [ 1.766427] systemd[1]: Reached target Path Units.1776worker2 # [ 1.767227] systemd[1]: Reached target Slice Units.1777worker2 # [ 1.768025] systemd[1]: Reached target Swaps.1778worker2 # [ 1.768807] systemd[1]: Reached target Timer Units.1779worker2 # [ 1.769772] systemd[1]: Listening on D-Bus System Message Bus Socket.1780lb # [ 1.766977] systemd[1]: Queued start job for default target Initrd Default Target.1781worker2 # [ 1.771075] systemd[1]: Listening on Journal Socket (/dev/log).1782worker2 # [ 1.772166] systemd[1]: Listening on Journal Sockets.1783worker2 # [ 1.773169] systemd[1]: Listening on udev Control Socket.1784worker1 # [ 1.830250] systemd[1]: Finished Create List of Static Device Nodes.1785client # [ 1.915193] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1786worker1 # [ 1.830970] systemd[1]: Mounted Kernel Configuration File System.1787worker2 # [ 1.774157] systemd[1]: Listening on udev Kernel Socket.1788worker2 # [ 1.775038] systemd[1]: Reached target Socket Units.1789worker2 # [ 1.777407] systemd[1]: Starting Create List of Static Device Nodes...1790client # [ 1.920234] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1791worker2 # [ 1.778538] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1792lb # [ 1.776970] systemd[1]: Created slice Slice /system/modprobe.1793client # [ 1.922754] systemd[1]: Starting Create Static Device Nodes in /dev...1794lb # [ 1.778162] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1795lb # [ 1.779519] systemd[1]: Expecting device /dev/disk/by-label/nixos...1796lb # [ 1.780557] systemd[1]: Reached target Path Units.1797lb # [ 1.781341] systemd[1]: Reached target Slice Units.1798lb # [ 1.782133] systemd[1]: Reached target Swaps.1799lb # [ 1.782896] systemd[1]: Reached target Timer Units.1800worker2 # [ 1.785932] systemd[1]: Mounting Kernel Configuration File System...1801lb # [ 1.783894] systemd[1]: Listening on D-Bus System Message Bus Socket.1802worker1 # [ 1.842446] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1803lb # [ 1.785114] systemd[1]: Listening on Journal Socket (/dev/log).1804lb # [ 1.786199] systemd[1]: Listening on Journal Sockets.1805client # [ 1.931773] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1806lb # [ 1.787224] systemd[1]: Listening on udev Control Socket.1807lb # [ 1.787356] systemd[1]: Listening on udev Kernel Socket.1808lb # [ 1.787379] systemd[1]: Reached target Socket Units.1809lb # [ 1.791484] systemd[1]: Starting Create List of Static Device Nodes...1810lb # [ 1.792644] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1811worker2 # [ 1.796854] systemd[1]: Starting Journal Service...1812client # [ 1.940690] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01813client # [ 1.940931] [drm] features: -virgl +edid -resource_blob -host_visible1814client # [ 1.940941] [drm] features: -context_init1815client # [ 1.941658] [drm] number of scanouts: 11816client # [ 1.941675] [drm] number of cap sets: 01817lb # [ 1.800107] systemd[1]: Mounting Kernel Configuration File System...1818lb # [ 1.810874] systemd[1]: Starting Journal Service...1819worker2 # [ 1.821217] systemd[1]: Starting Load Kernel Modules...1820worker1 # [ 1.878633] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1821worker2 # [ 1.822242] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1822worker1 # [ 1.881146] systemd[1]: Starting Create Static Device Nodes in /dev...1823client # [ 1.968148] systemd[1]: Finished Create Static Device Nodes in /dev.1824client # [ 1.968486] systemd[1]: Reached target Preparation for Local File Systems.1825client # [ 1.968515] systemd[1]: Reached target Local File Systems.1826worker1 # [ 1.889277] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1827client # [ 1.972439] systemd[1]: Starting Rule-based Manager for Device Events and Files...1828client # [ 1.976950] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1829client # [ 1.976973] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01830lb # [ 1.835028] systemd[1]: Starting Load Kernel Modules...1831lb # [ 1.836097] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1832worker1 # [ 1.898075] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1833worker2 # [ 1.845397] systemd[1]: Starting Coldplug All udev Devices...1834worker1 # [ 1.903986] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01835worker1 # [ 1.904228] [drm] features: -virgl +edid -resource_blob -host_visible1836worker1 # [ 1.904238] [drm] features: -context_init1837worker1 # [ 1.904917] [drm] number of scanouts: 11838worker1 # [ 1.904934] [drm] number of cap sets: 01839worker2 # [ 1.850690] systemd-journald[72]: Collecting audit messages is disabled.1840lb # [ 1.854755] systemd[1]: Starting Coldplug All udev Devices...1841worker2 # [ 1.860714] systemd[1]: Finished Create List of Static Device Nodes.1842worker2 # [ 1.861464] systemd[1]: Mounted Kernel Configuration File System.1843client # [ 1.999837] Console: switching to colour frame buffer device 160x501844lb # [ 1.864755] systemd-journald[72]: Collecting audit messages is disabled.1845lb # [ 1.871194] systemd[1]: Finished Create List of Static Device Nodes.1846lb # [ 1.871890] systemd[1]: Mounted Kernel Configuration File System.1847worker2 # [ 1.876775] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1848worker1 # [ 1.930349] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1849worker1 # [ 1.930367] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01850worker1 # [ 1.938863] systemd[1]: Finished Create Static Device Nodes in /dev.1851worker1 # [ 1.939225] systemd[1]: Reached target Preparation for Local File Systems.1852worker1 # [ 1.939254] systemd[1]: Reached target Local File Systems.1853client # [ 2.028186] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1854worker1 # [ 1.944697] systemd[1]: Starting Rule-based Manager for Device Events and Files...1855lb # [ 1.891005] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1856worker2 # [ 1.896698] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1857client # [ 2.040335] systemd[1]: Finished Load Kernel Modules.1858worker2 # [ 1.898435] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1859lb # [ 1.900651] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1860client # [ 2.033831] systemd-modules-load[74]: Inserted module 'dm_mod'1861client # [ 2.052099] systemd[1]: Starting Apply Kernel Variables...1862client # [ 2.052304] systemd[1]: Started Journal Service.1863worker2 # [ 1.913980] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01864worker2 # [ 1.914220] [drm] features: -virgl +edid -resource_blob -host_visible1865worker2 # [ 1.914230] [drm] features: -context_init1866worker2 # [ 1.914943] [drm] number of scanouts: 11867client # [ 2.044283] systemd-modules-load[74]: Module 'virtio_balloon' is built in1868worker2 # [ 1.914961] [drm] number of cap sets: 01869client # [ 2.045448] systemd-modules-load[74]: Module 'virtio_console' is built in1870client # [ 2.046579] systemd-modules-load[74]: Inserted module 'virtio_gpu'1871client # [ 2.047569] systemd-modules-load[74]: Module 'virtio_rng' is built in1872lb # [ 1.922724] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1873lb # [ 1.925681] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1874worker1 # [ 1.986098] Console: switching to colour frame buffer device 160x501875worker1 # [ 1.986770] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1876worker2 # [ 1.933227] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1877worker2 # [ 1.935736] systemd[1]: Starting Create Static Device Nodes in /dev...1878client # [ 2.064319] systemd-udevd[79]: Using default interface naming scheme 'v261'.1879lb # [ 1.935032] systemd[1]: Starting Create Static Device Nodes in /dev...1880client # [ 2.070025] systemd[1]: Starting Create System Files and Directories...1881worker1 # [ 2.002937] systemd[1]: Finished Load Kernel Modules.1882worker2 # [ 1.940366] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1883worker2 # [ 1.940388] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01884lb # [ 1.944594] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01885lb # [ 1.944840] [drm] features: -virgl +edid -resource_blob -host_visible1886lb # [ 1.944850] [drm] features: -context_init1887lb # [ 1.945575] [drm] number of scanouts: 11888lb # [ 1.945594] [drm] number of cap sets: 01889worker1 # [ 2.010369] systemd[1]: Starting Apply Kernel Variables...1890worker1 # [ 2.004303] systemd-modules-load[74]: Inserted module 'dm_mod'1891worker1 # [ 2.005370] systemd-modules-load[74]: Module 'virtio_balloon' is built in1892worker1 # [ 2.006448] systemd-modules-load[74]: Module 'virtio_console' is built in1893worker1 # [ 2.007518] systemd-modules-load[74]: Inserted module 'virtio_gpu'1894worker1 # [ 2.026247] systemd[1]: Started Journal Service.1895worker1 # [ 2.016510] systemd-modules-load[74]: Module 'virtio_rng' is built in1896worker2 # [ 1.964887] Console: switching to colour frame buffer device 160x501897lb # [ 1.966996] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1898lb # [ 1.967019] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01899lb # [ 1.975667] systemd[1]: Finished Create Static Device Nodes in /dev.1900lb # [ 1.976065] systemd[1]: Reached target Preparation for Local File Systems.1901client # [ 2.108709] systemd[1]: Finished Create System Files and Directories.1902lb # [ 1.976092] systemd[1]: Reached target Local File Systems.1903worker1 # [ 2.028633] systemd-udevd[79]: Using default interface naming scheme 'v261'.1904client # [ 2.113525] systemd[1]: Finished Apply Kernel Variables.1905worker2 # [ 1.989224] systemd[1]: Finished Create Static Device Nodes in /dev.1906worker2 # [ 1.989585] systemd[1]: Reached target Preparation for Local File Systems.1907worker1 # [ 2.036345] systemd[1]: Starting Create System Files and Directories...1908worker2 # [ 1.989610] systemd[1]: Reached target Local File Systems.1909worker2 # [ 1.993587] systemd[1]: Starting Rule-based Manager for Device Events and Files...1910lb # [ 1.987073] systemd[1]: Starting Rule-based Manager for Device Events and Files...1911client # [ 2.125591] systemd[1]: Started Rule-based Manager for Device Events and Files.1912worker2 # [ 1.997295] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1913worker1 # [ 2.053869] systemd[1]: Finished Apply Kernel Variables.1914worker2 # [ 2.009824] systemd[1]: Finished Load Kernel Modules.1915lb # [ 2.000277] Console: switching to colour frame buffer device 160x501916worker2 # [ 2.005558] systemd-modules-load[74]: Inserted module 'dm_mod'1917worker2 # [ 2.008219] systemd-modules-load[74]: Module 'virtio_balloon' is built in1918worker2 # [ 2.011060] systemd-modules-load[74]: Module 'virtio_console' is built in1919lb # [ 2.023159] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1920worker1 # [ 2.077040] systemd[1]: Finished Create System Files and Directories.1921worker2 # [ 2.024348] systemd-modules-load[74]: Inserted module 'virtio_gpu'1922lb # [ 2.039238] systemd[1]: Finished Load Kernel Modules.1923worker1 # [ 2.087437] systemd[1]: Started Rule-based Manager for Device Events and Files.1924worker2 # [ 2.041929] systemd[1]: Starting Apply Kernel Variables...1925worker2 # [ 2.042133] systemd[1]: Started Journal Service.1926worker2 # [ 2.029362] systemd-modules-load[74]: Module 'virtio_rng' is built in1927lb # [ 2.042939] systemd[1]: Starting Apply Kernel Variables...1928client # [ 2.184102] systemd[1]: Starting Virtual Console Setup...1929worker2 # [ 2.048201] systemd[1]: Starting Create System Files and Directories...1930lb # [ 2.048338] systemd-modules-load[74]: Inserted module 'dm_mod'1931lb # [ 2.049457] systemd-modules-load[74]: Module 'virtio_balloon' is built in1932lb # [ 2.050552] systemd-modules-load[74]: Module 'virtio_console' is built in1933lb # [ 2.051618] systemd-modules-load[74]: Inserted module 'virtio_gpu'1934worker2 # [ 2.058202] systemd[1]: Finished Apply Kernel Variables.1935lb # [ 2.070849] systemd[1]: Started Journal Service.1936lb # [ 2.064466] systemd-modules-load[74]: Module 'virtio_rng' is built in1937lb # [ 2.072109] systemd[1]: Starting Create System Files and Directories...1938worker2 # [ 2.083691] systemd[1]: Finished Create System Files and Directories.1939worker1 # [ 2.148153] systemd[1]: Starting Virtual Console Setup...1940client # [ 2.232443] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1941lb # [ 2.088375] systemd-udevd[78]: Using default interface naming scheme 'v261'.1942client # [ 2.235841] systemd[1]: Finished Virtual Console Setup.1943lb # [ 2.090882] systemd[1]: Finished Apply Kernel Variables.1944worker2 # [ 2.092537] systemd-udevd[79]: Using default interface naming scheme 'v261'.1945lb # [ 2.113278] systemd[1]: Finished Create System Files and Directories.1946worker2 # [ 2.117319] systemd[1]: Started Rule-based Manager for Device Events and Files.1947lb # [ 2.130121] systemd[1]: Started Rule-based Manager for Device Events and Files.1948worker1 # [ 2.196489] systemd-vconsole-setup[102]: Configuration of first virtual console was skipped, ignoring remaining ones.1949worker1 # [ 2.199964] systemd[1]: Finished Virtual Console Setup.1950worker2 # [ 2.168608] systemd[1]: Starting Virtual Console Setup...1951lb # [ 2.184393] systemd[1]: Starting Virtual Console Setup...1952worker2 # [ 2.224468] systemd-vconsole-setup[102]: Configuration of first virtual console was skipped, ignoring remaining ones.1953worker2 # [ 2.227781] systemd[1]: Finished Virtual Console Setup.1954lb # [ 2.236471] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1955lb # [ 2.239732] systemd[1]: Finished Virtual Console Setup.1956client # [ 2.854717] systemd[1]: Finished Coldplug All udev Devices.1957client # [ 2.855651] systemd[1]: Reached target System Initialization.1958client # [ 2.860160] systemd[1]: Reached target Basic System.1959worker1 # [ 2.811121] systemd[1]: Finished Coldplug All udev Devices.1960worker1 # [ 2.812097] systemd[1]: Reached target System Initialization.1961worker1 # [ 2.812940] systemd[1]: Reached target Basic System.1962worker2 # [ 2.824591] systemd[1]: Finished Coldplug All udev Devices.1963worker2 # [ 2.825534] systemd[1]: Reached target System Initialization.1964worker2 # [ 2.826335] systemd[1]: Reached target Basic System.1965client # [ 2.996129] (udev-worker)[96]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1966lb # [ 2.865566] systemd[1]: Finished Coldplug All udev Devices.1967lb # [ 2.866518] systemd[1]: Reached target System Initialization.1968lb # [ 2.867328] systemd[1]: Reached target Basic System.1969worker1 # [ 2.944054] (udev-worker)[98]: Network interface NamePolicy= disabled on kernel command line.1970client # [ 3.032876] (udev-worker)[96]: Network interface NamePolicy= disabled on kernel command line.1971client # [ 3.037014] (udev-worker)[98]: Network interface NamePolicy= disabled on kernel command line.1972worker1 # [ 2.977943] (udev-worker)[92]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1973worker1 # [ 2.984181] (udev-worker)[92]: Network interface NamePolicy= disabled on kernel command line.1974client # [ 3.085953] systemd[1]: Found device /dev/disk/by-label/nixos.1975client # [ 3.088917] systemd[1]: Reached target Initrd Root Device.1976client # [ 3.092363] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1977worker2 # [ 2.997473] (udev-worker)[94]: Network interface NamePolicy= disabled on kernel command line.1978lb # [ 2.999850] (udev-worker)[104]: Network interface NamePolicy= disabled on kernel command line.1979client # [ 3.145630] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks1980worker2 # [ 3.003676] (udev-worker)[101]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1981client # [ 3.151219] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1982worker2 # [ 3.009813] (udev-worker)[101]: Network interface NamePolicy= disabled on kernel command line.1983worker1 # [ 3.076985] systemd[1]: Found device /dev/disk/by-label/nixos.1984worker1 # [ 3.080942] systemd[1]: Reached target Initrd Root Device.1985client # [ 3.164233] systemd[1]: Mounting /sysroot...1986worker1 # [ 3.084076] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1987lb # [ 3.045489] (udev-worker)[95]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1988lb # [ 3.050246] (udev-worker)[95]: Network interface NamePolicy= disabled on kernel command line.1989client # [ 3.214651] EXT4-fs (vda): mounted filesystem 029cd0ef-f3c7-49c5-af97-28279bde576d r/w with ordered data mode. Quota mode: none.1990client # [ 3.205868] systemd[1]: Mounted /sysroot.1991client # [ 3.207605] systemd[1]: Reached target Initrd Root File System.1992worker1 # [ 3.126539] systemd-fsck[111]: nixos: clean, 12/65536 files, 13019/262144 blocks1993client # [ 3.212453] systemd[1]: Starting Mountpoints Configured in the Real Root...1994worker1 # [ 3.131708] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1995worker2 # [ 3.073056] systemd[1]: Found device /dev/disk/by-label/nixos.1996worker1 # [ 3.136380] systemd[1]: Mounting /sysroot...1997worker2 # [ 3.076252] systemd[1]: Reached target Initrd Root Device.1998worker2 # [ 3.080424] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1999client # [ 3.239696] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload.2000lb # [ 3.103909] systemd[1]: Found device /dev/disk/by-label/nixos.
loading 1,986 hidden lines…
3987client: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key3988client: (finished: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds)3989worker2 # [ 35.592082] nix-grpc-daemon[769]: ts=2026-09-22T11:25:34Z level=info event=oidc_jwks provider=mock keys=13990worker2 # [ 35.595872] nix-grpc-daemon[769]: ts=2026-09-22T11:25:34Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:468323991worker1 # [ 35.675530] nix-grpc-daemon[766]: ts=2026-09-22T11:25:34Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:558443992worker1 # [ 35.688149] nix-grpc-daemon[766]: ts=2026-09-22T11:25:34Z level=debug event=path_info_miss path=ba719awibzsjxd1bw9mhq3k18i64cwhc-nixos-system-client-test error="error: path '/nix/store/ba719awibzsjxd1bw9mhq3k18i64cwhc-nixos-system-client-test' is required, but there is no substituter that can build it"3993worker1 # [ 35.703038] nix-grpc-daemon[766]: ts=2026-09-22T11:25:34Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:558443994worker1 # [ 35.711041] nix-grpc-daemon[766]: ts=2026-09-22T11:25:34Z level=debug event=path_info_miss path=ba719awibzsjxd1bw9mhq3k18i64cwhc-nixos-system-client-test error="error: path '/nix/store/ba719awibzsjxd1bw9mhq3k18i64cwhc-nixos-system-client-test' is required, but there is no substituter that can build it"3995client: must succeed: rm -r /var/lib/nix-grpc-store3996client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.04 seconds)3997worker2 # [ 37.910103] nix-grpc-daemon[769]: ts=2026-09-22T11:25:36Z level=info event=denied method=QueryMissing cn=- role=none3998worker2 # [ 38.074362] nix-grpc-daemon[769]: ts=2026-09-22T11:25:36Z level=info event=denied method=QueryMissing cn=stranger role=none3999(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 4.95 seconds)4000subtest: DAG build is scheduled across workers and lands in the cache4001client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&24002client # don't know how to build these paths:4003client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4004worker1 # [ 38.301495] nix-grpc-daemon[766]: ts=2026-09-22T11:25:36Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:558444005worker2 # [ 38.242273] nix-grpc-daemon[769]: ts=2026-09-22T11:25:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:468324006worker1 # [ 38.342099] nix-grpc-daemon[766]: ts=2026-09-22T11:25:36Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=04007worker1 # [ 38.347924] nix-grpc-daemon[766]: ts=2026-09-22T11:25:36Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.4:50051 clients=14008worker2 # [ 38.325918] nix-grpc-daemon[769]: ts=2026-09-22T11:25:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324009worker2 # [ 38.353889] nix-grpc-daemon[769]: ts=2026-09-22T11:25:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:468324010client # copying 1 paths to 'lb:50051'...4011client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'...4012worker2 # [ 38.370351] nix-grpc-daemon[769]: ts=2026-09-22T11:25:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:468324013worker2 # [ 38.392722] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4014worker2 # [ 38.429647] nix-grpc-daemon[769]: ts=2026-09-22T11:25:37Z level=info event=niks3_push_started pid=8504015lb # [ 38.559272] envoy[908]: conn peer=[2001:db8:1::1]:56758 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=749 tx=423 ms=18554016worker2 # [ 38.648967] nix-grpc-daemon[850]: time=2026-09-22T11:25:37.263Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""4017worker2 # [ 38.760073] nix-daemon[836]: accepted connection from pid 863, user nix-grpc-daemon (trusted)4018worker2 # [ 38.780237] nix-daemon[836]: reaped child process 866, status = succeeded4019worker2 # [ 38.883977] nix-daemon[836]: accepted connection from pid 868, user nix-grpc-daemon (trusted)4020worker2 # [ 38.903402] nix-daemon[836]: reaped child process 871, status = succeeded4021lb # [ 38.564418] envoy[908]: conn peer=[2001:db8:1::1]:56774 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=864022lb # [ 38.564544] envoy[908]: conn peer=[2001:db8:1::1]:56782 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=124023lb # [ 38.564643] envoy[908]: conn peer=[2001:db8:1::1]:56798 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=134024lb # [ 38.900987] niks3-server[960]: 2026/09/22 11:25:37 INFO Received uploads request method=POST path=/api/pending_closures4025worker2 # [ 38.939909] nix-grpc-daemon[850]: time=2026-09-22T11:25:37.553Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4026worker2 # [ 38.942531] nix-grpc-daemon[850]: time=2026-09-22T11:25:37.553Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)"4027lb # [ 38.986802] niks3-server[960]: 2026/09/22 11:25:37 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst4028lb # [ 39.002707] niks3-server[960]: 2026/09/22 11:25:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4029worker2 # [ 39.012600] nix-grpc-daemon[850]: time=2026-09-22T11:25:37.626Z level=INFO msg="Uploading 1 narinfos"4030lb # [ 39.014338] niks3-server[960]: 2026/09/22 11:25:37 INFO Signed narinfos id=1 count=14031lb # [ 39.028629] niks3-server[960]: 2026/09/22 11:25:37 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls4032lb # [ 39.036142] niks3-server[960]: 2026/09/22 11:25:37 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4033lb # [ 39.047194] niks3-server[960]: 2026/09/22 11:25:37 INFO Completed upload id=14034worker2 # [ 39.048904] nix-grpc-daemon[850]: time=2026-09-22T11:25:37.663Z level=INFO msg="Upload complete. (396ms)"4035lb # [ 39.050428] niks3-server[960]: 2026/09/22 11:25:37 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo4036worker2 # [ 39.061178] nix-grpc-daemon[769]: ts=2026-09-22T11:25:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=0 paths=1 nar_bytes_in=5044037worker2 # [ 39.076854] nix-grpc-daemon[769]: ts=2026-09-22T11:25:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324038worker2 # [ 39.090649] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4039worker2 # [ 39.094418] nix-daemon[836]: reaped child process 848, status = succeeded4040client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051...4041worker2 # [ 39.106553] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4042client # farm-a> worker2: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4043client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4044client # farm-a> LOG-farm-a-t14045worker2 # [ 40.214027] nix-daemon[836]: reaped child process 875, status = succeeded4046worker2 # [ 40.286141] nix-daemon[836]: accepted connection from pid 891, user nix-grpc-daemon (trusted)4047worker2 # [ 40.303041] nix-daemon[836]: reaped child process 894, status = succeeded4048lb # [ 40.310195] niks3-server[960]: 2026/09/22 11:25:38 INFO Received uploads request method=POST path=/api/pending_closures4049worker2 # [ 40.324467] nix-grpc-daemon[850]: time=2026-09-22T11:25:38.937Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4050worker2 # [ 40.326256] nix-grpc-daemon[850]: time=2026-09-22T11:25:38.937Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)"4051lb # [ 40.357009] niks3-server[960]: 2026/09/22 11:25:38 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4052lb # [ 40.368752] niks3-server[960]: 2026/09/22 11:25:38 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4053lb # [ 40.381275] niks3-server[960]: 2026/09/22 11:25:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4054worker2 # [ 40.388113] nix-grpc-daemon[850]: time=2026-09-22T11:25:39.001Z level=INFO msg="Uploading 1 narinfos"4055lb # [ 40.391300] niks3-server[960]: 2026/09/22 11:25:38 INFO Signed narinfos id=2 count=14056lb # [ 40.396763] niks3-server[960]: 2026/09/22 11:25:38 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls4057lb # [ 40.409752] niks3-server[960]: 2026/09/22 11:25:39 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4058lb # [ 40.421517] niks3-server[960]: 2026/09/22 11:25:39 INFO Completed upload id=24059worker2 # [ 40.421310] nix-grpc-daemon[850]: time=2026-09-22T11:25:39.036Z level=INFO msg="Upload complete. (299ms)"4060lb # [ 40.426386] niks3-server[960]: 2026/09/22 11:25:39 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo4061worker2 # [ 40.433178] nix-grpc-daemon[769]: ts=2026-09-22T11:25:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=14062worker1 # [ 40.503062] nix-grpc-daemon[766]: ts=2026-09-22T11:25:39Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=04063worker1 # [ 40.509015] nix-grpc-daemon[766]: ts=2026-09-22T11:25:39Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.4:50051 clients=14064worker2 # [ 40.453573] nix-daemon[836]: reaped child process 873, status = succeeded4065worker2 # [ 40.486366] nix-grpc-daemon[769]: ts=2026-09-22T11:25:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324066worker2 # [ 40.501308] nix-grpc-daemon[769]: ts=2026-09-22T11:25:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:468324067worker2 # [ 40.515850] nix-grpc-daemon[850]: time=2026-09-22T11:25:39.130Z level=INFO msg="All 1 paths already cached"4068client # copying 1 paths to 'lb:50051'...4069worker2 # [ 40.525721] nix-grpc-daemon[769]: ts=2026-09-22T11:25:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:468324070client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'...4071worker2 # [ 40.547286] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4072worker2 # [ 40.710635] nix-daemon[836]: accepted connection from pid 898, user nix-grpc-daemon (trusted)4073worker2 # [ 40.736175] nix-daemon[836]: reaped child process 901, status = succeeded4074worker2 # [ 40.832416] nix-daemon[836]: accepted connection from pid 903, user nix-grpc-daemon (trusted)4075worker2 # [ 40.849109] nix-daemon[836]: reaped child process 906, status = succeeded4076lb # [ 40.852973] niks3-server[960]: 2026/09/22 11:25:39 INFO Received uploads request method=POST path=/api/pending_closures4077worker2 # [ 40.863429] nix-grpc-daemon[850]: time=2026-09-22T11:25:39.477Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4078worker2 # [ 40.865177] nix-grpc-daemon[850]: time=2026-09-22T11:25:39.477Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)"4079lb # [ 40.884765] niks3-server[960]: 2026/09/22 11:25:39 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst4080worker2 # [ 40.894817] nix-grpc-daemon[850]: time=2026-09-22T11:25:39.509Z level=INFO msg="Uploading 1 narinfos"4081lb # [ 40.899789] niks3-server[960]: 2026/09/22 11:25:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4082lb # [ 40.909703] niks3-server[960]: 2026/09/22 11:25:39 INFO Signed narinfos id=3 count=14083lb # [ 40.918400] niks3-server[960]: 2026/09/22 11:25:39 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls4084worker2 # [ 40.925399] nix-grpc-daemon[850]: time=2026-09-22T11:25:39.540Z level=INFO msg="Upload complete. (360ms)"4085lb # [ 40.926594] niks3-server[960]: 2026/09/22 11:25:39 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4086lb # [ 40.931760] niks3-server[960]: 2026/09/22 11:25:39 INFO Completed upload id=34087worker2 # [ 40.932290] nix-grpc-daemon[769]: ts=2026-09-22T11:25:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=0 paths=1 nar_bytes_in=6244088lb # [ 40.935372] niks3-server[960]: 2026/09/22 11:25:39 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo4089worker2 # [ 40.945033] nix-grpc-daemon[769]: ts=2026-09-22T11:25:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324090worker2 # [ 40.953359] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4091worker2 # [ 40.957381] nix-daemon[836]: reaped child process 896, status = succeeded4092client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051...4093worker2 # [ 40.968176] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4094client # farm-b> worker2: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4095client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4096client # farm-b> LOG-farm-b-t14097worker2 # [ 42.069854] nix-daemon[836]: reaped child process 910, status = succeeded4098worker2 # [ 42.150554] nix-daemon[836]: accepted connection from pid 927, user nix-grpc-daemon (trusted)4099worker2 # [ 42.166836] nix-daemon[836]: reaped child process 930, status = succeeded4100lb # [ 42.173063] niks3-server[960]: 2026/09/22 11:25:40 INFO Received uploads request method=POST path=/api/pending_closures4101worker2 # [ 42.186337] nix-grpc-daemon[850]: time=2026-09-22T11:25:40.800Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4102worker2 # [ 42.188083] nix-grpc-daemon[850]: time=2026-09-22T11:25:40.800Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)"4103lb # [ 42.204420] niks3-server[960]: 2026/09/22 11:25:40 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4104lb # [ 42.217627] niks3-server[960]: 2026/09/22 11:25:40 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst4105worker2 # [ 42.241701] nix-grpc-daemon[850]: time=2026-09-22T11:25:40.856Z level=INFO msg="Uploading 1 narinfos"4106lb # [ 42.233101] niks3-server[960]: 2026/09/22 11:25:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4107lb # [ 42.244977] niks3-server[960]: 2026/09/22 11:25:40 INFO Signed narinfos id=4 count=14108lb # [ 42.247471] niks3-server[960]: 2026/09/22 11:25:40 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls4109lb # [ 42.265064] niks3-server[960]: 2026/09/22 11:25:40 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4110lb # [ 42.273968] niks3-server[960]: 2026/09/22 11:25:40 INFO Completed upload id=44111worker2 # [ 42.275514] nix-grpc-daemon[850]: time=2026-09-22T11:25:40.890Z level=INFO msg="Upload complete. (295ms)"4112lb # [ 42.276728] niks3-server[960]: 2026/09/22 11:25:40 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo4113worker1 # [ 42.355440] nix-grpc-daemon[766]: ts=2026-09-22T11:25:40Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=04114worker2 # [ 42.289213] nix-grpc-daemon[769]: ts=2026-09-22T11:25:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=14115worker1 # [ 42.361256] nix-grpc-daemon[766]: ts=2026-09-22T11:25:40Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.4:50051 clients=14116worker2 # [ 42.307303] nix-daemon[836]: reaped child process 908, status = succeeded4117worker2 # [ 42.338197] nix-grpc-daemon[769]: ts=2026-09-22T11:25:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324118worker2 # [ 42.353070] nix-grpc-daemon[769]: ts=2026-09-22T11:25:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:468324119worker2 # [ 42.367341] nix-grpc-daemon[850]: time=2026-09-22T11:25:40.982Z level=INFO msg="All 2 paths already cached"4120client # copying 1 paths to 'lb:50051'...4121client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'...4122worker2 # [ 42.377563] nix-grpc-daemon[769]: ts=2026-09-22T11:25:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:468324123worker2 # [ 42.395584] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4124worker2 # [ 42.561937] nix-daemon[836]: accepted connection from pid 934, user nix-grpc-daemon (trusted)4125worker2 # [ 42.578722] nix-daemon[836]: reaped child process 937, status = succeeded4126worker2 # [ 42.682217] nix-daemon[836]: accepted connection from pid 939, user nix-grpc-daemon (trusted)4127worker2 # [ 42.697830] nix-daemon[836]: reaped child process 942, status = succeeded4128lb # [ 42.703370] niks3-server[960]: 2026/09/22 11:25:41 INFO Received uploads request method=POST path=/api/pending_closures4129worker2 # [ 42.717182] nix-grpc-daemon[850]: time=2026-09-22T11:25:41.331Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4130worker2 # [ 42.718698] nix-grpc-daemon[850]: time=2026-09-22T11:25:41.331Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)"4131lb # [ 42.734231] niks3-server[960]: 2026/09/22 11:25:41 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst4132worker2 # [ 42.759965] nix-grpc-daemon[850]: time=2026-09-22T11:25:41.374Z level=INFO msg="Uploading 1 narinfos"4133lb # [ 42.751331] niks3-server[960]: 2026/09/22 11:25:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4134lb # [ 42.765624] niks3-server[960]: 2026/09/22 11:25:41 INFO Signed narinfos id=5 count=14135lb # [ 42.772632] niks3-server[960]: 2026/09/22 11:25:41 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls4136lb # [ 42.786093] niks3-server[960]: 2026/09/22 11:25:41 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4137worker2 # [ 42.793086] nix-grpc-daemon[850]: time=2026-09-22T11:25:41.407Z level=INFO msg="Upload complete. (380ms)"4138lb # [ 42.792828] niks3-server[960]: 2026/09/22 11:25:41 INFO Completed upload id=54139lb # [ 42.799360] niks3-server[960]: 2026/09/22 11:25:41 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo4140worker2 # [ 42.804333] nix-grpc-daemon[769]: ts=2026-09-22T11:25:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=0 paths=1 nar_bytes_in=7524141worker2 # [ 42.817773] nix-grpc-daemon[769]: ts=2026-09-22T11:25:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324142worker2 # [ 42.824336] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4143worker2 # [ 42.829318] nix-daemon[836]: reaped child process 932, status = succeeded4144client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051...4145worker2 # [ 42.841813] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4146client # farm-top> worker2: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4147client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4148client # farm-top> LOG-farm-top-t14149worker2 # [ 43.917186] nix-daemon[836]: reaped child process 946, status = succeeded4150worker2 # [ 44.025094] nix-daemon[836]: accepted connection from pid 962, user nix-grpc-daemon (trusted)4151worker2 # [ 44.043861] nix-daemon[836]: reaped child process 965, status = succeeded4152lb # [ 44.049033] niks3-server[960]: 2026/09/22 11:25:42 INFO Received uploads request method=POST path=/api/pending_closures4153worker2 # [ 44.060503] nix-grpc-daemon[850]: time=2026-09-22T11:25:42.674Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4154worker2 # [ 44.062354] nix-grpc-daemon[850]: time=2026-09-22T11:25:42.674Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)"4155lb # [ 44.097125] niks3-server[960]: 2026/09/22 11:25:42 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst4156worker2 # [ 44.108867] nix-grpc-daemon[850]: time=2026-09-22T11:25:42.723Z level=INFO msg="Uploading 1 narinfos"4157lb # [ 44.109721] niks3-server[960]: 2026/09/22 11:25:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4158lb # [ 44.119929] niks3-server[960]: 2026/09/22 11:25:42 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4159lb # [ 44.130977] niks3-server[960]: 2026/09/22 11:25:42 INFO Signed narinfos id=6 count=14160worker2 # [ 44.140344] nix-grpc-daemon[850]: time=2026-09-22T11:25:42.755Z level=INFO msg="Upload complete. (288ms)"4161lb # [ 44.142343] niks3-server[960]: 2026/09/22 11:25:42 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls4162worker2 # [ 44.152643] nix-grpc-daemon[769]: ts=2026-09-22T11:25:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=14163lb # [ 44.153263] niks3-server[960]: 2026/09/22 11:25:42 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4164lb # [ 44.161009] niks3-server[960]: 2026/09/22 11:25:42 INFO Completed upload id=64165lb # [ 44.164517] niks3-server[960]: 2026/09/22 11:25:42 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo4166worker2 # [ 44.170464] nix-daemon[836]: reaped child process 944, status = succeeded4167client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 6.11 seconds)4168client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath4169client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.19 seconds)4170client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4171client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.03 seconds)4172client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4173client # copying 3 paths...4174client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'...4175client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'...4176client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'...4177client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.23 seconds)4178(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 6.64 seconds)4179subtest: repeat is answered Cached by the scheduler without building4180worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4181worker1 # finding garbage collector roots...4182worker1 # deleting specified paths...4183worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4184worker1 # deleting unused links...4185worker1 # note: hard linking is currently saving 0.0 KiB4186worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.23 seconds)4187worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4188worker2 # finding garbage collector roots...4189worker2 # deleting specified paths...4190worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4191worker2 # deleting unused links...4192worker2 # note: hard linking is currently saving 0.0 KiB4193worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.22 seconds)4194client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&24195worker2 # [ 45.394959] nix-grpc-daemon[769]: ts=2026-09-22T11:25:44Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:468324196client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4197client # /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4198worker2 # [ 45.406987] nix-grpc-daemon[769]: ts=2026-09-22T11:25:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:468324199worker2 # [ 45.418842] nix-grpc-daemon[850]: time=2026-09-22T11:25:44.032Z level=INFO msg="All 2 paths already cached"4200worker1 # [ 45.498413] nix-grpc-daemon[766]: ts=2026-09-22T11:25:44Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=14201client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 0.28 seconds)4202client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath4203client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.12 seconds)4204lb: must succeed: curl -sf localhost:9901/clusters4205lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4206lb: must succeed: curl -sf localhost:9901/clusters4207lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4208worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4209worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4210(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.12 seconds)4211subtest: inputs already in the cache are not sent to the scheduler4212lb: must succeed: curl -sf localhost:9901/clusters4213lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4214lb: must succeed: curl -sf localhost:9901/clusters4215lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4216worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4217worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4218client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&24219worker1 # [ 46.291734] nix-grpc-daemon[766]: ts=2026-09-22T11:25:44Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:558444220client # don't know how to build these paths:4221client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4222worker2 # [ 46.241147] nix-grpc-daemon[769]: ts=2026-09-22T11:25:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:468324223worker2 # [ 46.251288] nix-grpc-daemon[850]: time=2026-09-22T11:25:44.866Z level=INFO msg="All 2 paths already cached"4224worker1 # [ 46.327858] nix-grpc-daemon[766]: ts=2026-09-22T11:25:44Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=04225worker1 # [ 46.333978] nix-grpc-daemon[766]: ts=2026-09-22T11:25:44Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.4:50051 clients=14226worker2 # [ 46.313817] nix-grpc-daemon[769]: ts=2026-09-22T11:25:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324227worker2 # [ 46.329144] nix-grpc-daemon[769]: ts=2026-09-22T11:25:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:468324228worker2 # [ 46.344657] nix-grpc-daemon[850]: time=2026-09-22T11:25:44.959Z level=INFO msg="All 2 paths already cached"4229client # copying 1 paths to 'lb:50051'...4230client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'...4231worker2 # [ 46.354850] nix-grpc-daemon[769]: ts=2026-09-22T11:25:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:468324232worker2 # [ 46.370077] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4233worker2 # [ 46.533732] nix-daemon[836]: accepted connection from pid 983, user nix-grpc-daemon (trusted)4234worker2 # [ 46.557790] nix-daemon[836]: reaped child process 986, status = succeeded4235worker2 # [ 46.652107] nix-daemon[836]: accepted connection from pid 988, user nix-grpc-daemon (trusted)4236worker2 # [ 46.670767] nix-daemon[836]: reaped child process 991, status = succeeded4237lb # [ 46.673589] niks3-server[960]: 2026/09/22 11:25:45 INFO Received uploads request method=POST path=/api/pending_closures4238worker2 # [ 46.684904] nix-grpc-daemon[850]: time=2026-09-22T11:25:45.299Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4239worker2 # [ 46.686337] nix-grpc-daemon[850]: time=2026-09-22T11:25:45.299Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)"4240lb # [ 46.702709] niks3-server[960]: 2026/09/22 11:25:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4241worker2 # [ 46.712658] nix-grpc-daemon[850]: time=2026-09-22T11:25:45.327Z level=INFO msg="Uploading 1 narinfos"4242lb # [ 46.714554] niks3-server[960]: 2026/09/22 11:25:45 INFO Signed narinfos id=7 count=14243lb # [ 46.727289] niks3-server[960]: 2026/09/22 11:25:45 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls4244lb # [ 46.739216] niks3-server[960]: 2026/09/22 11:25:45 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst4245lb # [ 46.754197] niks3-server[960]: 2026/09/22 11:25:45 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4246lb # [ 46.762311] niks3-server[960]: 2026/09/22 11:25:45 INFO Completed upload id=74247worker2 # [ 46.763853] nix-grpc-daemon[850]: time=2026-09-22T11:25:45.378Z level=INFO msg="Upload complete. (375ms)"4248lb # [ 46.766254] niks3-server[960]: 2026/09/22 11:25:45 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo4249worker2 # [ 46.773155] nix-grpc-daemon[769]: ts=2026-09-22T11:25:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=0 paths=1 nar_bytes_in=7524250worker2 # [ 46.790618] nix-grpc-daemon[769]: ts=2026-09-22T11:25:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324251worker2 # [ 46.800335] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4252worker2 # [ 46.804822] nix-daemon[836]: reaped child process 981, status = succeeded4253client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051...4254client # farm-top> worker2: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4255worker2 # [ 46.819737] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4256client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4257client # farm-top> LOG-farm-top-t1b4258worker2 # [ 47.660379] nix-daemon[836]: reaped child process 995, status = succeeded4259worker2 # [ 47.763134] nix-daemon[836]: accepted connection from pid 1011, user nix-grpc-daemon (trusted)4260worker2 # [ 47.782658] nix-daemon[836]: reaped child process 1014, status = succeeded4261lb # [ 47.786939] niks3-server[960]: 2026/09/22 11:25:46 INFO Received uploads request method=POST path=/api/pending_closures4262worker2 # [ 47.800402] nix-grpc-daemon[850]: time=2026-09-22T11:25:46.414Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4263worker2 # [ 47.801918] nix-grpc-daemon[850]: time=2026-09-22T11:25:46.414Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)"4264lb # [ 47.830150] niks3-server[960]: 2026/09/22 11:25:46 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst4265lb # [ 47.843550] niks3-server[960]: 2026/09/22 11:25:46 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4266worker2 # [ 47.855980] nix-grpc-daemon[850]: time=2026-09-22T11:25:46.470Z level=INFO msg="Uploading 1 narinfos"4267lb # [ 47.860273] niks3-server[960]: 2026/09/22 11:25:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4268lb # [ 47.868825] niks3-server[960]: 2026/09/22 11:25:46 INFO Signed narinfos id=8 count=14269lb # [ 47.871529] niks3-server[960]: 2026/09/22 11:25:46 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls4270lb # [ 47.883755] niks3-server[960]: 2026/09/22 11:25:46 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4271lb # [ 47.888477] niks3-server[960]: 2026/09/22 11:25:46 INFO Completed upload id=84272worker2 # [ 47.884550] nix-grpc-daemon[850]: time=2026-09-22T11:25:46.499Z level=INFO msg="Upload complete. (255ms)"4273lb # [ 47.891716] niks3-server[960]: 2026/09/22 11:25:46 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo4274worker2 # [ 47.897501] nix-grpc-daemon[769]: ts=2026-09-22T11:25:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=14275worker2 # [ 47.916135] nix-daemon[836]: reaped child process 993, status = succeeded4276client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2, in 1.93 seconds)4277client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath4278client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.18 seconds)4279lb: must succeed: curl -sf localhost:9901/clusters4280lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4281lb: must succeed: curl -sf localhost:9901/clusters4282lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4283worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4284worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4285(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.44 seconds)4286subtest: two clients wanting the same drv build it once4287client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup4288client # [ 48.498448] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup.4289client # Running as unit: dedup1.service; invocation ID: 89740d3a28fe4bd8806530d946f054354290client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 0.09 seconds)4291client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup4292lb # [ 48.558900] envoy[908]: conn peer=[2001:db8:1::1]:56800 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9506 tx=4080 ms=59404293worker2 # [ 48.563119] nix-grpc-daemon[769]: ts=2026-09-22T11:25:47Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:468324294lb # [ 48.565201] envoy[908]: conn peer=[2001:db8:1::1]:48122 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=480 ms=654295client # [ 48.716216] nix[935]: don't know how to build these paths:4296client # [ 48.717356] nix[935]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4297lb # [ 48.571148] envoy[908]: conn peer=[2001:db8:1::1]:48134 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4198 tx=1809 ms=16824298worker1 # [ 48.651067] nix-grpc-daemon[766]: ts=2026-09-22T11:25:47Z level=debug event=want client=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04299worker1 # [ 48.656891] nix-grpc-daemon[766]: ts=2026-09-22T11:25:47Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.3:50051 clients=14300worker1 # [ 48.663785] nix-grpc-daemon[766]: ts=2026-09-22T11:25:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558444301worker1 # [ 48.678654] nix-grpc-daemon[766]: ts=2026-09-22T11:25:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558444302client # [ 48.779458] nix[935]: copying 1 paths to 'lb:50051'...4303client # [ 48.781468] nix[935]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'...4304worker1 # [ 48.697878] nix-grpc-daemon[766]: ts=2026-09-22T11:25:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558444305worker1 # [ 48.716685] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4306worker1 # [ 48.744734] nix-grpc-daemon[766]: ts=2026-09-22T11:25:47Z level=info event=niks3_push_started pid=9024307worker1 # [ 48.872115] nix-grpc-daemon[766]: E0000 00:00:1790076347.480778 808 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4308worker1 # [ 48.910995] nix-grpc-daemon[766]: E0000 00:00:1790076347.520688 808 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4309worker1 # [ 48.940239] nix-grpc-daemon[766]: E0000 00:00:1790076347.527329 808 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4310worker1 # [ 48.968203] nix-grpc-daemon[766]: E0000 00:00:1790076347.563740 808 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4311worker1 # [ 48.992997] nix-grpc-daemon[902]: time=2026-09-22T11:25:47.602Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""4312worker1 # [ 49.119275] nix-daemon[847]: accepted connection from pid 915, user nix-grpc-daemon (trusted)4313worker1 # [ 49.143913] nix-daemon[847]: reaped child process 918, status = succeeded4314worker1 # [ 49.268173] nix-daemon[847]: accepted connection from pid 920, user nix-grpc-daemon (trusted)4315worker1 # [ 49.287653] nix-daemon[847]: reaped child process 923, status = succeeded4316lb # [ 49.235402] niks3-server[960]: 2026/09/22 11:25:47 INFO Received uploads request method=POST path=/api/pending_closures4317worker1 # [ 49.309816] nix-grpc-daemon[902]: time=2026-09-22T11:25:47.919Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4318worker1 # [ 49.311721] nix-grpc-daemon[902]: time=2026-09-22T11:25:47.919Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)"4319worker1 # [ 49.359037] nix-grpc-daemon[902]: time=2026-09-22T11:25:47.969Z level=INFO msg="Uploading 1 narinfos"4320lb # [ 49.283208] niks3-server[960]: 2026/09/22 11:25:47 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst4321lb # [ 49.302112] niks3-server[960]: 2026/09/22 11:25:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4322lb # [ 49.311676] niks3-server[960]: 2026/09/22 11:25:47 INFO Signed narinfos id=9 count=14323lb # [ 49.320970] niks3-server[960]: 2026/09/22 11:25:47 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls4324lb # [ 49.333489] niks3-server[960]: 2026/09/22 11:25:47 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4325lb # [ 49.348399] niks3-server[960]: 2026/09/22 11:25:47 INFO Completed upload id=94326worker1 # [ 49.411650] nix-grpc-daemon[902]: time=2026-09-22T11:25:48.022Z level=INFO msg="Upload complete. (415ms)"4327lb # [ 49.352168] niks3-server[960]: 2026/09/22 11:25:47 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo4328worker1 # [ 49.424617] nix-grpc-daemon[766]: ts=2026-09-22T11:25:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=0 paths=1 nar_bytes_in=4324329worker1 # [ 49.444666] nix-grpc-daemon[766]: ts=2026-09-22T11:25:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558444330worker1 # [ 49.463587] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4331worker1 # [ 49.472614] nix-daemon[847]: reaped child process 900, status = succeeded4332worker1 # [ 49.486390] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4333client # [ 49.569060] nix[935]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4334client # [ 49.625246] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup.4335client # Running as unit: dedup2.service; invocation ID: c79d39bd783b4eafb3eb7432ac8b7c064336client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 1.14 seconds)4337client # [ 49.905857] nix[951]: don't know how to build these paths:4338client # [ 49.907650] nix[951]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4339worker2 # [ 49.761916] nix-grpc-daemon[769]: ts=2026-09-22T11:25:48Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:468324340worker1 # [ 49.846356] nix-grpc-daemon[766]: ts=2026-09-22T11:25:48Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04341client # [ 49.938710] nix[951]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4342worker1 # [ 49.854852] nix-grpc-daemon[766]: ts=2026-09-22T11:25:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558444343client: must succeed: sleep 24344client: (finished: must succeed: sleep 2, in 2.06 seconds)4345client: waiting for success: ! systemctl is-active dedup1 dedup24346worker1 # [ 53.145851] nix-daemon[847]: reaped child process 927, status = succeeded4347worker1 # [ 53.251556] nix-daemon[847]: accepted connection from pid 959, user nix-grpc-daemon (trusted)4348worker1 # [ 53.272785] nix-daemon[847]: reaped child process 962, status = succeeded4349lb # [ 53.217621] niks3-server[960]: 2026/09/22 11:25:51 INFO Received uploads request method=POST path=/api/pending_closures4350worker1 # [ 53.296972] nix-grpc-daemon[902]: time=2026-09-22T11:25:51.903Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4351worker1 # [ 53.298636] nix-grpc-daemon[902]: time=2026-09-22T11:25:51.903Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)"4352lb # [ 53.261020] niks3-server[960]: 2026/09/22 11:25:51 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4353lb # [ 53.299308] niks3-server[960]: 2026/09/22 11:25:51 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4354lb # [ 53.317581] niks3-server[960]: 2026/09/22 11:25:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4355worker1 # [ 53.380157] nix-grpc-daemon[902]: time=2026-09-22T11:25:51.989Z level=INFO msg="Uploading 1 narinfos"4356lb # [ 53.326434] niks3-server[960]: 2026/09/22 11:25:51 INFO Signed narinfos id=10 count=14357lb # [ 53.331811] niks3-server[960]: 2026/09/22 11:25:51 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls4358lb # [ 53.342060] niks3-server[960]: 2026/09/22 11:25:51 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4359lb # [ 53.350537] niks3-server[960]: 2026/09/22 11:25:51 INFO Completed upload id=104360worker1 # [ 53.411967] nix-grpc-daemon[902]: time=2026-09-22T11:25:52.022Z level=INFO msg="Upload complete. (307ms)"4361lb # [ 53.353569] niks3-server[960]: 2026/09/22 11:25:51 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo4362worker1 # [ 53.429515] nix-grpc-daemon[766]: ts=2026-09-22T11:25:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14363worker1 # [ 53.448567] nix-grpc-daemon[766]: ts=2026-09-22T11:25:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14364worker1 # [ 53.457991] nix-daemon[847]: reaped child process 925, status = succeeded4365client # [ 53.567679] systemd[1]: dedup1.service: Deactivated successfully.4366client # [ 53.576424] systemd[1]: dedup1.service: Consumed 195ms CPU time over 5.076s wall clock time, 39.4M memory peak, 7.3K incoming IP traffic, 8.4K outgoing IP traffic.4367client # [ 53.603802] systemd[1]: dedup2.service: Deactivated successfully.4368client # [ 53.606915] systemd[1]: dedup2.service: Consumed 249ms CPU time over 3.982s wall clock time, 33.2M memory peak, 6K incoming IP traffic, 5.6K outgoing IP traffic.4369client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.13 seconds)4370client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4371client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.07 seconds)4372worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4373worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4374worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4375worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.05 seconds)4376worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4377worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4378worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4379worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4380(finished: subtest: two clients wanting the same drv build it once, in 6.20 seconds)4381subtest: --timeout and --max-silent-time reach the builder4382worker1 # [ 54.782931] nix-grpc-daemon[766]: ts=2026-09-22T11:25:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:558444383worker1 # [ 54.798545] nix-grpc-daemon[766]: ts=2026-09-22T11:25:53Z level=debug event=want client=6 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv cached=04384worker1 # [ 54.801741] nix-grpc-daemon[766]: ts=2026-09-22T11:25:53Z level=debug event=assigned drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv worker=192.168.1.4:50051 clients=14385worker2 # [ 54.785230] nix-grpc-daemon[769]: ts=2026-09-22T11:25:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324386worker2 # [ 54.793248] nix-grpc-daemon[769]: ts=2026-09-22T11:25:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:468324387worker2 # [ 54.804311] nix-grpc-daemon[769]: ts=2026-09-22T11:25:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:468324388worker2 # [ 54.812437] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4389worker2 # [ 54.957709] nix-daemon[836]: accepted connection from pid 1039, user nix-grpc-daemon (trusted)4390worker2 # [ 54.986384] nix-daemon[836]: reaped child process 1042, status = succeeded4391worker2 # [ 55.113823] nix-daemon[836]: accepted connection from pid 1044, user nix-grpc-daemon (trusted)4392worker2 # [ 55.141214] nix-daemon[836]: reaped child process 1047, status = succeeded4393lb # [ 55.141648] niks3-server[960]: 2026/09/22 11:25:53 INFO Received uploads request method=POST path=/api/pending_closures4394worker2 # [ 55.154083] nix-grpc-daemon[850]: time=2026-09-22T11:25:53.767Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4395worker2 # [ 55.155538] nix-grpc-daemon[850]: time=2026-09-22T11:25:53.768Z level=INFO msg="Uploading sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv (424B)"4396lb # [ 55.165921] niks3-server[960]: 2026/09/22 11:25:53 INFO Registered completed upload object_key=nar/1m12xmj6yzmb9y2j8ly0yvh5fyzy1nqk7ajhqwr4wn3b5h0bsch5.nar.zst4397worker2 # [ 55.171947] nix-grpc-daemon[850]: time=2026-09-22T11:25:53.786Z level=INFO msg="Uploading 1 narinfos"4398lb # [ 55.171921] niks3-server[960]: 2026/09/22 11:25:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4399lb # [ 55.175921] niks3-server[960]: 2026/09/22 11:25:53 INFO Signed narinfos id=11 count=14400lb # [ 55.179571] niks3-server[960]: 2026/09/22 11:25:53 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.ls4401lb # [ 55.185614] niks3-server[960]: 2026/09/22 11:25:53 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4402lb # [ 55.188724] niks3-server[960]: 2026/09/22 11:25:53 INFO Completed upload id=114403worker2 # [ 55.189952] nix-grpc-daemon[850]: time=2026-09-22T11:25:53.804Z level=INFO msg="Upload complete. (367ms)"4404lb # [ 55.191965] niks3-server[960]: 2026/09/22 11:25:53 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.narinfo4405worker2 # [ 55.195502] nix-grpc-daemon[769]: ts=2026-09-22T11:25:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=0 paths=1 nar_bytes_in=4244406worker2 # [ 55.206883] nix-grpc-daemon[769]: ts=2026-09-22T11:25:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324407worker2 # [ 55.213602] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4408worker2 # [ 55.217818] nix-daemon[836]: reaped child process 1037, status = succeeded4409worker2 # [ 55.230635] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4410lb # [ 58.560443] envoy[908]: conn peer=[2001:db8:1::1]:48136 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3018 tx=1451 ms=48244411lb # [ 58.563304] envoy[908]: conn peer=[2001:db8:1::1]:48140 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1430 tx=1085 ms=36704412worker2 # [ 61.247346] nix-grpc-daemon[769]: ts=2026-09-22T11:25:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=6 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv assign_id=6 outputs=04413worker2 # [ 61.284763] nix-daemon[836]: reaped child process 1049, status = succeeded4414worker2 # [ 61.286750] nix-daemon[836]: reaped child process 1051, status = succeeded4415worker2 # [ 61.527200] nix-grpc-daemon[769]: ts=2026-09-22T11:26:00Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:468324416worker1 # [ 61.602873] nix-grpc-daemon[766]: ts=2026-09-22T11:26:00Z level=debug event=want client=7 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv cached=04417worker1 # [ 61.603328] nix-grpc-daemon[766]: ts=2026-09-22T11:26:00Z level=debug event=assigned drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv worker=192.168.1.3:50051 clients=14418worker1 # [ 61.616593] nix-grpc-daemon[766]: ts=2026-09-22T11:26:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558444419worker1 # [ 61.632297] nix-grpc-daemon[766]: ts=2026-09-22T11:26:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558444420worker1 # [ 61.653031] nix-grpc-daemon[766]: ts=2026-09-22T11:26:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558444421worker1 # [ 61.667978] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4422worker1 # [ 61.828379] nix-daemon[847]: accepted connection from pid 979, user nix-grpc-daemon (trusted)4423worker1 # [ 61.853585] nix-daemon[847]: reaped child process 982, status = succeeded4424worker1 # [ 61.962876] nix-daemon[847]: accepted connection from pid 984, user nix-grpc-daemon (trusted)4425worker1 # [ 61.987857] nix-daemon[847]: reaped child process 987, status = succeeded4426lb # [ 61.930844] niks3-server[960]: 2026/09/22 11:26:00 INFO Received uploads request method=POST path=/api/pending_closures4427worker1 # [ 62.002231] nix-grpc-daemon[902]: time=2026-09-22T11:26:00.612Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4428worker1 # [ 62.003827] nix-grpc-daemon[902]: time=2026-09-22T11:26:00.612Z level=INFO msg="Uploading pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv (424B)"4429lb # [ 61.956056] niks3-server[960]: 2026/09/22 11:26:00 INFO Registered completed upload object_key=nar/14f5yrnllfav3svmw4v4svw15r4zlf5yv1p972jy14mmq6gdw943.nar.zst4430lb # [ 61.972764] niks3-server[960]: 2026/09/22 11:26:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4431worker1 # [ 62.039825] nix-grpc-daemon[902]: time=2026-09-22T11:26:00.650Z level=INFO msg="Uploading 1 narinfos"4432lb # [ 61.979681] niks3-server[960]: 2026/09/22 11:26:00 INFO Signed narinfos id=12 count=14433lb # [ 61.984694] niks3-server[960]: 2026/09/22 11:26:00 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.ls4434lb # [ 61.995282] niks3-server[960]: 2026/09/22 11:26:00 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4435worker1 # [ 62.064115] nix-grpc-daemon[902]: time=2026-09-22T11:26:00.675Z level=INFO msg="Upload complete. (381ms)"4436lb # [ 62.003115] niks3-server[960]: 2026/09/22 11:26:00 INFO Completed upload id=124437worker1 # [ 62.070046] nix-grpc-daemon[766]: ts=2026-09-22T11:26:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=0 paths=1 nar_bytes_in=4244438lb # [ 62.009178] niks3-server[960]: 2026/09/22 11:26:00 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.narinfo4439worker1 # [ 62.076368] nix-grpc-daemon[766]: ts=2026-09-22T11:26:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558444440worker1 # [ 62.088366] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4441worker1 # [ 62.090516] nix-daemon[847]: reaped child process 977, status = succeeded4442worker1 # [ 62.101960] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4443worker1 # [ 68.117363] nix-grpc-daemon[766]: ts=2026-09-22T11:26:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=6 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv assign_id=7 outputs=04444worker1 # [ 68.150162] nix-daemon[847]: reaped child process 989, status = succeeded4445worker1 # [ 68.152376] nix-daemon[847]: reaped child process 991, status = succeeded4446(finished: subtest: --timeout and --max-silent-time reach the builder, in 13.74 seconds)4447subtest: the first client leaving does not fail the second4448client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share4449client # [ 68.438385] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share.4450client # Running as unit: share1.service; invocation ID: 267c907587dc48589b5fbf051ed8e7c44451client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 0.09 seconds)4452client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share4453worker1 # [ 68.567908] nix-grpc-daemon[766]: ts=2026-09-22T11:26:07Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:558444454client # [ 68.658027] nix[1025]: don't know how to build these paths:4455client # [ 68.659149] nix[1025]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4456worker1 # [ 68.587623] nix-grpc-daemon[766]: ts=2026-09-22T11:26:07Z level=debug event=want client=8 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=04457worker1 # [ 68.594329] nix-grpc-daemon[766]: ts=2026-09-22T11:26:07Z level=debug event=assigned drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker=192.168.1.4:50051 clients=14458worker2 # [ 68.537950] nix-grpc-daemon[769]: ts=2026-09-22T11:26:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324459worker2 # [ 68.555819] nix-grpc-daemon[769]: ts=2026-09-22T11:26:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:468324460lb # [ 68.557693] envoy[908]: conn peer=[2001:db8:1::1]:55324 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2988 tx=950 ms=65904461lb # [ 68.563074] envoy[908]: conn peer=[2001:db8:1::1]:33790 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2989 tx=976 ms=65554462client # [ 68.720709] nix[1025]: copying 1 paths to 'lb:50051'...4463client # [ 68.725518] nix[1025]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' to 'lb:50051'...4464worker2 # [ 68.583079] nix-grpc-daemon[769]: ts=2026-09-22T11:26:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:468324465worker2 # [ 68.597355] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4466worker2 # [ 68.771358] nix-daemon[836]: accepted connection from pid 1074, user nix-grpc-daemon (trusted)4467worker2 # [ 68.796524] nix-daemon[836]: reaped child process 1077, status = succeeded4468worker2 # [ 68.900332] nix-daemon[836]: accepted connection from pid 1079, user nix-grpc-daemon (trusted)4469worker2 # [ 68.922854] nix-daemon[836]: reaped child process 1082, status = succeeded4470lb # [ 68.923324] niks3-server[960]: 2026/09/22 11:26:07 INFO Received uploads request method=POST path=/api/pending_closures4471worker2 # [ 68.933701] nix-grpc-daemon[850]: time=2026-09-22T11:26:07.547Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4472worker2 # [ 68.935225] nix-grpc-daemon[850]: time=2026-09-22T11:26:07.547Z level=INFO msg="Uploading rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv (432B)"4473lb # [ 68.969278] niks3-server[960]: 2026/09/22 11:26:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4474lb # [ 68.995855] niks3-server[960]: 2026/09/22 11:26:07 INFO Signed narinfos id=13 count=14475worker2 # [ 68.997559] nix-grpc-daemon[850]: time=2026-09-22T11:26:07.612Z level=INFO msg="Uploading 1 narinfos"4476lb # [ 69.005832] niks3-server[960]: 2026/09/22 11:26:07 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4477lb # [ 69.104683] niks3-server[960]: 2026/09/22 11:26:07 INFO Registered completed upload object_key=nar/0chdjpc9an2ckdblh08lnyrmzvw3h8qs76cn5ydwjwvgwkrm731l.nar.zst4478lb # [ 69.123062] niks3-server[960]: 2026/09/22 11:26:07 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.ls4479lb # [ 69.139634] niks3-server[960]: 2026/09/22 11:26:07 INFO Completed upload id=134480worker2 # [ 69.140805] nix-grpc-daemon[850]: time=2026-09-22T11:26:07.755Z level=INFO msg="Upload complete. (525ms)"4481lb # [ 69.147744] niks3-server[960]: 2026/09/22 11:26:07 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.narinfo4482worker2 # [ 69.150674] nix-grpc-daemon[769]: ts=2026-09-22T11:26:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=0 paths=1 nar_bytes_in=4324483worker2 # [ 69.172473] nix-grpc-daemon[769]: ts=2026-09-22T11:26:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324484worker2 # [ 69.181932] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4485worker2 # [ 69.189853] nix-daemon[836]: reaped child process 1072, status = succeeded4486client # [ 69.348231] nix[1025]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051...4487worker2 # [ 69.209441] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4488client # [ 69.546711] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share.4489client # Running as unit: share2.service; invocation ID: 2e5737e2f12c4434a89df882368a42fa4490client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 1.12 seconds)4491worker1 # [ 69.664772] nix-grpc-daemon[766]: ts=2026-09-22T11:26:08Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:558444492client # [ 69.756225] nix[1041]: don't know how to build these paths:4493client # [ 69.760066] nix[1041]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4494worker1 # [ 69.689518] nix-grpc-daemon[766]: ts=2026-09-22T11:26:08Z level=debug event=want client=9 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=04495worker2 # [ 69.633884] nix-grpc-daemon[769]: ts=2026-09-22T11:26:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324496client # [ 69.784982] nix[1041]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051...4497client: must succeed: sleep 2; systemctl kill -s INT share14498client # [ 72.933116] systemd[1]: share1.service: Sent signal SIGINT to main process 1025 (nix) on client request.4499client # [ 72.944640] nix[1025]: shutting downrxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)4500client # [ 72.951513] nix[1025]: [mutex.cc : 451] RAW: Unlock 0xfdfbd69fc788 @ 0xfdffdb1359a4 0xfdffdc20d40c 0xfdfffb6757bc 0xfdfffb677414 0xfdfffb66e1d4 0xfdfffa6b4f7c 0xfdfffa442d804501client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.09 seconds)4502client: waiting for success: ! systemctl is-active share14503client # [ 72.984165] systemd-coredump[1056]: Process 1025 (nix) of user 0 terminated abnormally with signal 11/SEGV, processing...4504client # [ 73.011966] systemd[1]: Created slice Slice /system/systemd-coredump.4505client # [ 73.022123] systemd[1]: Started Process Core Dump (PID 1056/UID 0).4506client # [ 75.417633] systemd-coredump[1061]: Cannot store coredump of 1025 (nix): No space left on device4507client # [ 75.546543] systemd-coredump[1061]: Process 1025 (nix) of user 0 terminated abnormally without generating a coredump.4508client # [ 75.564751] systemd[1]: systemd-coredump@0-1-1056_1057-0.service: Deactivated successfully.4509client # [ 75.566297] systemd[1]: systemd-coredump@0-1-1056_1057-0.service: Consumed 2.060s CPU time over 2.536s wall clock time, 968.8M memory peak, 926.3M written to disk.4510client # [ 75.574593] systemd[1]: share1.service: Main process exited, code=dumped, status=11/SEGV4511client # [ 75.577793] systemd[1]: share1.service: Failed with result 'core-dump'.4512client # [ 75.579034] systemd[1]: share1.service: Consumed 373ms CPU time over 7.139s wall clock time, 43.5M memory peak, 4K written to disk, 6.3K incoming IP traffic, 8.1K outgoing IP traffic.4513client: (finished: waiting for success: ! systemctl is-active share1, in 3.29 seconds)4514client: waiting for success: ! systemctl is-active share24515worker2 # [ 76.504681] nix-daemon[836]: reaped child process 1086, status = succeeded4516worker2 # [ 76.603874] nix-daemon[836]: accepted connection from pid 1120, user nix-grpc-daemon (trusted)4517worker2 # [ 76.626571] nix-daemon[836]: reaped child process 1123, status = succeeded4518lb # [ 76.625558] niks3-server[960]: 2026/09/22 11:26:15 INFO Received uploads request method=POST path=/api/pending_closures4519worker2 # [ 76.639912] nix-grpc-daemon[850]: time=2026-09-22T11:26:15.252Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4520worker2 # [ 76.642251] nix-grpc-daemon[850]: time=2026-09-22T11:26:15.252Z level=INFO msg="Uploading fjww0fd4rwk1s35jav3cyal62l24n356-slow-share (120B)"4521lb # [ 76.663301] niks3-server[960]: 2026/09/22 11:26:15 INFO Registered completed upload object_key=log/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4522lb # [ 76.685355] niks3-server[960]: 2026/09/22 11:26:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4523worker2 # [ 76.692421] nix-grpc-daemon[850]: time=2026-09-22T11:26:15.306Z level=INFO msg="Uploading 1 narinfos"4524lb # [ 76.695309] niks3-server[960]: 2026/09/22 11:26:15 INFO Signed narinfos id=14 count=14525lb # [ 76.701745] niks3-server[960]: 2026/09/22 11:26:15 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst4526lb # [ 76.710226] niks3-server[960]: 2026/09/22 11:26:15 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.ls4527lb # [ 76.717530] niks3-server[960]: 2026/09/22 11:26:15 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4528lb # [ 76.724485] niks3-server[960]: 2026/09/22 11:26:15 INFO Completed upload id=144529worker2 # [ 76.725634] nix-grpc-daemon[850]: time=2026-09-22T11:26:15.340Z level=INFO msg="Upload complete. (296ms)"4530lb # [ 76.728248] niks3-server[960]: 2026/09/22 11:26:15 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.narinfo4531worker2 # [ 76.741430] nix-grpc-daemon[769]: ts=2026-09-22T11:26:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=7 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=8 outputs=14532worker2 # [ 76.758630] nix-grpc-daemon[769]: ts=2026-09-22T11:26:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=7 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=8 outputs=14533worker2 # [ 76.765329] nix-daemon[836]: reaped child process 1084, status = succeeded4534client # [ 76.910640] systemd[1]: share2.service: Deactivated successfully.4535client # [ 76.913612] systemd[1]: share2.service: Consumed 181ms CPU time over 7.365s wall clock time, 33.3M memory peak, 4K written to disk, 5.8K incoming IP traffic, 5.5K outgoing IP traffic.4536client: (finished: waiting for success: ! systemctl is-active share2, in 1.13 seconds)4537client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }4538client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.06 seconds)4539(finished: subtest: the first client leaving does not fail the second, in 9.26 seconds)4540subtest: build hook: nix-daemon with builders = grpc://lb4541client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log4542client # [ 77.775987] systemd[1]: Started Nix Daemon.4543client # [ 77.899558] nix-daemon[1107]: accepted connection from pid 1104, user root (trusted)4544worker2 # [ 77.914063] nix-grpc-daemon[769]: ts=2026-09-22T11:26:16Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:468324545worker1 # [ 77.980441] nix-grpc-daemon[766]: ts=2026-09-22T11:26:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558444546worker1 # [ 77.994927] nix-grpc-daemon[766]: ts=2026-09-22T11:26:16Z level=debug event=want client=10 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=04547worker1 # [ 78.003068] nix-grpc-daemon[766]: ts=2026-09-22T11:26:16Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.3:50051 clients=14548worker1 # [ 78.013050] nix-grpc-daemon[766]: ts=2026-09-22T11:26:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558444549worker1 # [ 78.025886] nix-grpc-daemon[766]: ts=2026-09-22T11:26:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558444550worker1 # [ 78.041863] nix-grpc-daemon[766]: ts=2026-09-22T11:26:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558444551worker1 # [ 78.058925] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4552worker1 # [ 78.211764] nix-daemon[847]: accepted connection from pid 1036, user nix-grpc-daemon (trusted)4553worker1 # [ 78.234534] nix-daemon[847]: reaped child process 1039, status = succeeded4554worker1 # [ 78.337809] nix-daemon[847]: accepted connection from pid 1041, user nix-grpc-daemon (trusted)4555worker1 # [ 78.358143] nix-daemon[847]: reaped child process 1044, status = succeeded4556lb # [ 78.299862] niks3-server[960]: 2026/09/22 11:26:16 INFO Received uploads request method=POST path=/api/pending_closures4557worker1 # [ 78.373032] nix-grpc-daemon[902]: time=2026-09-22T11:26:16.983Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4558worker1 # [ 78.374578] nix-grpc-daemon[902]: time=2026-09-22T11:26:16.983Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)"4559lb # [ 78.328619] niks3-server[960]: 2026/09/22 11:26:16 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst4560lb # [ 78.342198] niks3-server[960]: 2026/09/22 11:26:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4561worker1 # [ 78.411320] nix-grpc-daemon[902]: time=2026-09-22T11:26:17.022Z level=INFO msg="Uploading 1 narinfos"4562lb # [ 78.354619] niks3-server[960]: 2026/09/22 11:26:16 INFO Signed narinfos id=15 count=14563lb # [ 78.360739] niks3-server[960]: 2026/09/22 11:26:16 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls4564lb # [ 78.367659] niks3-server[960]: 2026/09/22 11:26:16 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4565lb # [ 78.377626] niks3-server[960]: 2026/09/22 11:26:16 INFO Completed upload id=154566worker1 # [ 78.439951] nix-grpc-daemon[902]: time=2026-09-22T11:26:17.050Z level=INFO msg="Upload complete. (365ms)"4567lb # [ 78.382245] niks3-server[960]: 2026/09/22 11:26:16 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo4568worker1 # [ 78.453232] nix-grpc-daemon[766]: ts=2026-09-22T11:26:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=0 paths=1 nar_bytes_in=5124569worker1 # [ 78.460351] nix-grpc-daemon[766]: ts=2026-09-22T11:26:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558444570worker1 # [ 78.470419] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4571worker1 # [ 78.475488] nix-daemon[847]: reaped child process 1034, status = succeeded4572worker1 # [ 78.490221] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4573lb # [ 78.557981] envoy[908]: conn peer=[2001:db8:1::1]:33792 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2990 tx=814 ms=69184574lb # [ 78.563255] envoy[908]: conn peer=[2001:db8:1::1]:33808 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1430 tx=1086 ms=71524575worker1 # [ 79.568732] nix-daemon[847]: reaped child process 1048, status = succeeded4576worker1 # [ 79.673325] nix-daemon[847]: accepted connection from pid 1064, user nix-grpc-daemon (trusted)4577worker1 # [ 79.696244] nix-daemon[847]: reaped child process 1067, status = succeeded4578lb # [ 79.638618] niks3-server[960]: 2026/09/22 11:26:18 INFO Received uploads request method=POST path=/api/pending_closures4579worker1 # [ 79.711749] nix-grpc-daemon[902]: time=2026-09-22T11:26:18.321Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4580worker1 # [ 79.713850] nix-grpc-daemon[902]: time=2026-09-22T11:26:18.321Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)"4581lb # [ 79.678180] niks3-server[960]: 2026/09/22 11:26:18 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv4582lb # [ 79.688739] niks3-server[960]: 2026/09/22 11:26:18 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4583worker1 # [ 79.758807] nix-grpc-daemon[902]: time=2026-09-22T11:26:18.369Z level=INFO msg="Uploading 1 narinfos"4584lb # [ 79.698405] niks3-server[960]: 2026/09/22 11:26:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4585lb # [ 79.708686] niks3-server[960]: 2026/09/22 11:26:18 INFO Signed narinfos id=16 count=14586lb # [ 79.710923] niks3-server[960]: 2026/09/22 11:26:18 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls4587lb # [ 79.723473] niks3-server[960]: 2026/09/22 11:26:18 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4588lb # [ 79.730424] niks3-server[960]: 2026/09/22 11:26:18 INFO Completed upload id=164589worker1 # [ 79.792850] nix-grpc-daemon[902]: time=2026-09-22T11:26:18.403Z level=INFO msg="Upload complete. (292ms)"4590lb # [ 79.733968] niks3-server[960]: 2026/09/22 11:26:18 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo4591worker1 # [ 79.805195] nix-grpc-daemon[766]: ts=2026-09-22T11:26:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=9 outputs=14592worker1 # [ 79.826957] nix-grpc-daemon[766]: ts=2026-09-22T11:26:18Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558444593worker1 # [ 79.832349] nix-daemon[847]: reaped child process 1046, status = succeeded4594worker2 # [ 79.781173] nix-grpc-daemon[769]: ts=2026-09-22T11:26:18Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:468324595worker2 # [ 79.797056] nix-grpc-daemon[769]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'...4596worker2 # [ 79.815215] nix-grpc-daemon[769]: ts=2026-09-22T11:26:18Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=0 paths=1 nar_bytes_out=1284597worker1 # [ 80.149602] nix-grpc-daemon[766]: ts=2026-09-22T11:26:18Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:558444598worker1 # [ 80.157525] nix-grpc-daemon[766]: ts=2026-09-22T11:26:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558444599worker1 # [ 80.164696] nix-grpc-daemon[902]: time=2026-09-22T11:26:18.774Z level=INFO msg="All 1 paths already cached"4600worker1 # [ 80.177579] nix-grpc-daemon[766]: ts=2026-09-22T11:26:18Z level=debug event=want client=11 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=04601worker1 # [ 80.187227] nix-grpc-daemon[766]: ts=2026-09-22T11:26:18Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.3:50051 clients=14602worker1 # [ 80.196162] nix-grpc-daemon[766]: ts=2026-09-22T11:26:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558444603worker1 # [ 80.203773] nix-grpc-daemon[766]: ts=2026-09-22T11:26:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558444604worker1 # [ 80.215380] nix-grpc-daemon[902]: time=2026-09-22T11:26:18.826Z level=INFO msg="All 1 paths already cached"4605worker1 # [ 80.224847] nix-grpc-daemon[766]: ts=2026-09-22T11:26:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558444606worker1 # [ 80.239088] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4607worker1 # [ 80.395387] nix-daemon[847]: accepted connection from pid 1073, user nix-grpc-daemon (trusted)4608worker1 # [ 80.417338] nix-daemon[847]: reaped child process 1076, status = succeeded4609worker1 # [ 80.521778] nix-daemon[847]: accepted connection from pid 1078, user nix-grpc-daemon (trusted)4610worker1 # [ 80.541404] nix-daemon[847]: reaped child process 1081, status = succeeded4611lb # [ 80.483188] niks3-server[960]: 2026/09/22 11:26:19 INFO Received uploads request method=POST path=/api/pending_closures4612worker1 # [ 80.558251] nix-grpc-daemon[902]: time=2026-09-22T11:26:19.169Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4613worker1 # [ 80.559781] nix-grpc-daemon[902]: time=2026-09-22T11:26:19.169Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)"4614lb # [ 80.517400] niks3-server[960]: 2026/09/22 11:26:19 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst4615worker1 # [ 80.592823] nix-grpc-daemon[902]: time=2026-09-22T11:26:19.203Z level=INFO msg="Uploading 1 narinfos"4616lb # [ 80.531200] niks3-server[960]: 2026/09/22 11:26:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4617lb # [ 80.543371] niks3-server[960]: 2026/09/22 11:26:19 INFO Signed narinfos id=17 count=14618lb # [ 80.550642] niks3-server[960]: 2026/09/22 11:26:19 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls4619lb # [ 80.564843] niks3-server[960]: 2026/09/22 11:26:19 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4620worker1 # [ 80.626793] nix-grpc-daemon[902]: time=2026-09-22T11:26:19.237Z level=INFO msg="Upload complete. (372ms)"4621lb # [ 80.575009] niks3-server[960]: 2026/09/22 11:26:19 INFO Completed upload id=174622worker1 # [ 80.636483] nix-grpc-daemon[766]: ts=2026-09-22T11:26:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=0 paths=1 nar_bytes_in=6244623lb # [ 80.578097] niks3-server[960]: 2026/09/22 11:26:19 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo4624worker1 # [ 80.650275] nix-grpc-daemon[766]: ts=2026-09-22T11:26:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558444625worker1 # [ 80.660000] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4626worker1 # [ 80.663039] nix-daemon[847]: reaped child process 1071, status = succeeded4627worker1 # [ 80.674329] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4628worker1 # [ 81.944874] nix-daemon[847]: reaped child process 1085, status = succeeded4629worker1 # [ 82.043439] nix-daemon[847]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)4630worker1 # [ 82.063931] nix-daemon[847]: reaped child process 1104, status = succeeded4631lb # [ 82.006465] niks3-server[960]: 2026/09/22 11:26:20 INFO Received uploads request method=POST path=/api/pending_closures4632worker1 # [ 82.080492] nix-grpc-daemon[902]: time=2026-09-22T11:26:20.690Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4633worker1 # [ 82.082064] nix-grpc-daemon[902]: time=2026-09-22T11:26:20.690Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)"4634lb # [ 82.038526] niks3-server[960]: 2026/09/22 11:26:20 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv4635lb # [ 82.055839] niks3-server[960]: 2026/09/22 11:26:20 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst4636lb # [ 82.071134] niks3-server[960]: 2026/09/22 11:26:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign4637worker1 # [ 82.138819] nix-grpc-daemon[902]: time=2026-09-22T11:26:20.749Z level=INFO msg="Uploading 1 narinfos"4638lb # [ 82.077825] niks3-server[960]: 2026/09/22 11:26:20 INFO Signed narinfos id=18 count=14639lb # [ 82.085152] niks3-server[960]: 2026/09/22 11:26:20 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls4640lb # [ 82.096737] niks3-server[960]: 2026/09/22 11:26:20 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete4641worker1 # [ 82.165123] nix-grpc-daemon[902]: time=2026-09-22T11:26:20.776Z level=INFO msg="Upload complete. (277ms)"4642lb # [ 82.104924] niks3-server[960]: 2026/09/22 11:26:20 INFO Completed upload id=184643lb # [ 82.109681] niks3-server[960]: 2026/09/22 11:26:20 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo4644worker1 # [ 82.180457] nix-grpc-daemon[766]: ts=2026-09-22T11:26:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=10 outputs=14645worker1 # [ 82.200189] nix-grpc-daemon[766]: ts=2026-09-22T11:26:20Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558444646worker1 # [ 82.205462] nix-daemon[847]: reaped child process 1083, status = succeeded4647worker1 # [ 82.215803] nix-grpc-daemon[766]: ts=2026-09-22T11:26:20Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:558444648worker1 # [ 82.221939] nix-grpc-daemon[766]: ts=2026-09-22T11:26:20Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=0 paths=1 nar_bytes_out=1764649worker1 # [ 82.508269] nix-grpc-daemon[766]: ts=2026-09-22T11:26:21Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:558444650worker2 # [ 82.454715] nix-grpc-daemon[769]: ts=2026-09-22T11:26:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:468324651worker2 # [ 82.473778] nix-grpc-daemon[850]: time=2026-09-22T11:26:21.087Z level=INFO msg="All 1 paths already cached"4652worker1 # [ 82.555741] nix-grpc-daemon[766]: ts=2026-09-22T11:26:21Z level=debug event=want client=12 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=04653worker1 # [ 82.563383] nix-grpc-daemon[766]: ts=2026-09-22T11:26:21Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.3:50051 clients=14654worker1 # [ 82.572811] nix-grpc-daemon[766]: ts=2026-09-22T11:26:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558444655worker1 # [ 82.585673] nix-grpc-daemon[766]: ts=2026-09-22T11:26:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558444656worker1 # [ 82.601048] nix-grpc-daemon[902]: time=2026-09-22T11:26:21.212Z level=INFO msg="All 2 paths already cached"4657worker1 # [ 82.611422] nix-grpc-daemon[766]: ts=2026-09-22T11:26:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558444658worker1 # [ 82.624656] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4659worker1 # [ 82.781921] nix-daemon[847]: accepted connection from pid 1110, user nix-grpc-daemon (trusted)4660worker1 # [ 82.805395] nix-daemon[847]: reaped child process 1113, status = succeeded4661worker1 # [ 82.917130] nix-daemon[847]: accepted connection from pid 1115, user nix-grpc-daemon (trusted)4662worker1 # [ 82.943059] nix-daemon[847]: reaped child process 1118, status = succeeded4663lb # [ 82.877258] niks3-server[960]: 2026/09/22 11:26:21 INFO Received uploads request method=POST path=/api/pending_closures4664worker1 # [ 82.950330] nix-grpc-daemon[902]: time=2026-09-22T11:26:21.560Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4665worker1 # [ 82.951872] nix-grpc-daemon[902]: time=2026-09-22T11:26:21.561Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)"4666lb # [ 82.909610] niks3-server[960]: 2026/09/22 11:26:21 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst4667lb # [ 82.933063] niks3-server[960]: 2026/09/22 11:26:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign4668worker1 # [ 83.001737] nix-grpc-daemon[902]: time=2026-09-22T11:26:21.612Z level=INFO msg="Uploading 1 narinfos"4669lb # [ 82.942645] niks3-server[960]: 2026/09/22 11:26:21 INFO Signed narinfos id=19 count=14670lb # [ 82.952446] niks3-server[960]: 2026/09/22 11:26:21 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls4671lb # [ 82.967188] niks3-server[960]: 2026/09/22 11:26:21 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete4672worker1 # [ 83.037923] nix-grpc-daemon[902]: time=2026-09-22T11:26:21.647Z level=INFO msg="Upload complete. (396ms)"4673lb # [ 82.977774] niks3-server[960]: 2026/09/22 11:26:21 INFO Completed upload id=194674lb # [ 82.981970] niks3-server[960]: 2026/09/22 11:26:21 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo4675worker1 # [ 83.049074] nix-grpc-daemon[766]: ts=2026-09-22T11:26:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=0 paths=1 nar_bytes_in=7524676worker1 # [ 83.064555] nix-grpc-daemon[766]: ts=2026-09-22T11:26:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558444677worker1 # [ 83.075004] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4678worker1 # [ 83.078870] nix-daemon[847]: reaped child process 1108, status = succeeded4679worker1 # [ 83.091979] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4680worker1 # [ 84.572779] nix-daemon[847]: reaped child process 1122, status = succeeded4681worker1 # [ 84.680068] nix-daemon[847]: accepted connection from pid 1138, user nix-grpc-daemon (trusted)4682worker1 # [ 84.703936] nix-daemon[847]: reaped child process 1141, status = succeeded4683lb # [ 84.641952] niks3-server[960]: 2026/09/22 11:26:23 INFO Received uploads request method=POST path=/api/pending_closures4684worker1 # [ 84.715504] nix-grpc-daemon[902]: time=2026-09-22T11:26:23.324Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4685worker1 # [ 84.717152] nix-grpc-daemon[902]: time=2026-09-22T11:26:23.324Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)"4686lb # [ 84.686429] niks3-server[960]: 2026/09/22 11:26:23 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv4687worker1 # [ 84.759473] nix-grpc-daemon[902]: time=2026-09-22T11:26:23.369Z level=INFO msg="Uploading 1 narinfos"4688lb # [ 84.698997] niks3-server[960]: 2026/09/22 11:26:23 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst4689lb # [ 84.711293] niks3-server[960]: 2026/09/22 11:26:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign4690lb # [ 84.723363] niks3-server[960]: 2026/09/22 11:26:23 INFO Signed narinfos id=20 count=14691lb # [ 84.729676] niks3-server[960]: 2026/09/22 11:26:23 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls4692worker1 # [ 84.791854] nix-grpc-daemon[902]: time=2026-09-22T11:26:23.402Z level=INFO msg="Upload complete. (282ms)"4693lb # [ 84.735363] niks3-server[960]: 2026/09/22 11:26:23 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete4694lb # [ 84.741174] niks3-server[960]: 2026/09/22 11:26:23 INFO Completed upload id=204695lb # [ 84.744319] niks3-server[960]: 2026/09/22 11:26:23 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo4696worker1 # [ 84.805242] nix-grpc-daemon[766]: ts=2026-09-22T11:26:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=11 outputs=14697worker1 # [ 84.825425] nix-grpc-daemon[766]: ts=2026-09-22T11:26:23Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558444698worker1 # [ 84.830773] nix-daemon[847]: reaped child process 1120, status = succeeded4699worker2 # [ 84.790555] nix-grpc-daemon[769]: ts=2026-09-22T11:26:23Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:468324700worker2 # [ 84.806150] nix-grpc-daemon[769]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'...4701worker2 # [ 84.822650] nix-grpc-daemon[769]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'...4702worker2 # [ 84.838249] nix-grpc-daemon[769]: ts=2026-09-22T11:26:23Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=0 paths=1 nar_bytes_out=2324703client # [ 85.121639] nix-daemon[1107]: reaped child process 1109, status = succeeded4704client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.51 seconds)4705client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top4706client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.04 seconds)4707client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4708client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds)4709client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4710client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)4711(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.63 seconds)4712subtest: an input only one worker has still reaches the builder4713client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only4714client # this derivation will be built:4715client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv4716client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'...4717client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only, in 0.31 seconds)4718client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure4719client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.18 seconds)4720worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4721worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.23 seconds)4722worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only4723worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.04 seconds)4724client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&24725client # [ 86.285008] nix-daemon[1107]: accepted connection from pid 1190, user root (trusted)4726client # this derivation will be built:4727client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4728worker2 # [ 86.416764] nix-grpc-daemon[769]: ts=2026-09-22T11:26:25Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:468324729client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4730worker2 # [ 86.424165] nix-grpc-daemon[769]: ts=2026-09-22T11:26:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:468324731client # copying 1 paths...4732client # copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' to 'grpc://lb:50051'...4733worker2 # [ 86.442620] nix-grpc-daemon[769]: ts=2026-09-22T11:26:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:468324734worker2 # [ 86.459490] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4735worker2 # [ 86.635198] nix-daemon[836]: accepted connection from pid 1133, user nix-grpc-daemon (trusted)4736worker2 # [ 86.658424] nix-daemon[836]: reaped child process 1136, status = succeeded4737lb # [ 86.660594] niks3-server[960]: 2026/09/22 11:26:25 INFO Received uploads request method=POST path=/api/pending_closures4738worker2 # [ 86.672364] nix-grpc-daemon[850]: time=2026-09-22T11:26:25.286Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4739worker2 # [ 86.674070] nix-grpc-daemon[850]: time=2026-09-22T11:26:25.286Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)"4740lb # [ 86.692247] niks3-server[960]: 2026/09/22 11:26:25 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4741lb # [ 86.709425] niks3-server[960]: 2026/09/22 11:26:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign4742worker2 # [ 86.720748] nix-grpc-daemon[850]: time=2026-09-22T11:26:25.335Z level=INFO msg="Uploading 1 narinfos"4743lb # [ 86.722994] niks3-server[960]: 2026/09/22 11:26:25 INFO Signed narinfos id=21 count=14744lb # [ 86.731632] niks3-server[960]: 2026/09/22 11:26:25 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls4745lb # [ 86.744239] niks3-server[960]: 2026/09/22 11:26:25 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete4746worker2 # [ 86.757517] nix-grpc-daemon[850]: time=2026-09-22T11:26:25.372Z level=INFO msg="Upload complete. (278ms)"4747lb # [ 86.755964] niks3-server[960]: 2026/09/22 11:26:25 INFO Completed upload id=214748worker2 # [ 86.763134] nix-grpc-daemon[769]: ts=2026-09-22T11:26:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=0 paths=1 nar_bytes_in=1204749lb # [ 86.762896] niks3-server[960]: 2026/09/22 11:26:25 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo4750worker2 # [ 86.773480] nix-daemon[836]: reaped child process 1131, status = succeeded4751worker1 # [ 86.859796] nix-grpc-daemon[766]: ts=2026-09-22T11:26:25Z level=debug event=want client=13 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=04752worker1 # [ 86.867157] nix-grpc-daemon[766]: ts=2026-09-22T11:26:25Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=14753worker2 # [ 86.844921] nix-grpc-daemon[769]: ts=2026-09-22T11:26:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324754worker2 # [ 86.866594] nix-grpc-daemon[769]: ts=2026-09-22T11:26:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:468324755client # copying 2 paths to 'lb:50051'...4756worker2 # [ 86.892109] nix-grpc-daemon[769]: ts=2026-09-22T11:26:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:468324757client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'...4758client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'...4759worker2 # [ 86.916207] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4760worker2 # [ 87.096420] nix-daemon[836]: accepted connection from pid 1140, user nix-grpc-daemon (trusted)4761worker2 # [ 87.119319] nix-daemon[836]: reaped child process 1143, status = succeeded4762worker2 # [ 87.230657] nix-daemon[836]: accepted connection from pid 1145, user nix-grpc-daemon (trusted)4763worker2 # [ 87.252436] nix-daemon[836]: reaped child process 1148, status = succeeded4764lb # [ 87.255323] niks3-server[960]: 2026/09/22 11:26:25 INFO Received uploads request method=POST path=/api/pending_closures4765lb # [ 87.266637] niks3-server[960]: 2026/09/22 11:26:25 INFO Received uploads request method=POST path=/api/pending_closures4766worker2 # [ 87.280114] nix-grpc-daemon[850]: time=2026-09-22T11:26:25.894Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4767worker2 # [ 87.281804] nix-grpc-daemon[850]: time=2026-09-22T11:26:25.894Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)"4768worker2 # [ 87.283652] nix-grpc-daemon[850]: time=2026-09-22T11:26:25.894Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)"4769lb # [ 87.307687] niks3-server[960]: 2026/09/22 11:26:25 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst4770lb # [ 87.327183] niks3-server[960]: 2026/09/22 11:26:25 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls4771lb # [ 87.341377] niks3-server[960]: 2026/09/22 11:26:25 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst4772lb # [ 87.355928] niks3-server[960]: 2026/09/22 11:26:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign4773lb # [ 87.359055] niks3-server[960]: 2026/09/22 11:26:25 INFO Signed narinfos id=22 count=14774worker2 # [ 87.366311] nix-grpc-daemon[850]: time=2026-09-22T11:26:25.979Z level=INFO msg="Uploading 3 narinfos"4775lb # [ 87.369380] niks3-server[960]: 2026/09/22 11:26:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign4776lb # [ 87.377393] niks3-server[960]: 2026/09/22 11:26:25 INFO Signed narinfos id=23 count=24777lb # [ 87.382729] niks3-server[960]: 2026/09/22 11:26:25 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls4778lb # [ 87.399876] niks3-server[960]: 2026/09/22 11:26:26 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4779lb # [ 87.404800] niks3-server[960]: 2026/09/22 11:26:26 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4780lb # [ 87.412541] niks3-server[960]: 2026/09/22 11:26:26 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete4781lb # [ 87.414630] niks3-server[960]: 2026/09/22 11:26:26 INFO Completed upload id=224782lb # [ 87.415827] niks3-server[960]: 2026/09/22 11:26:26 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete4783lb # [ 87.419490] niks3-server[960]: 2026/09/22 11:26:26 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo4784lb # [ 87.422603] niks3-server[960]: 2026/09/22 11:26:26 INFO Completed upload id=234785worker2 # [ 87.424488] nix-grpc-daemon[850]: time=2026-09-22T11:26:26.039Z level=INFO msg="Upload complete. (485ms)"4786worker2 # [ 87.440118] nix-grpc-daemon[769]: ts=2026-09-22T11:26:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=0 paths=2 nar_bytes_in=9844787worker2 # [ 87.461916] nix-grpc-daemon[769]: ts=2026-09-22T11:26:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324788worker2 # [ 87.470701] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4789worker2 # [ 87.478403] nix-daemon[836]: reaped child process 1138, status = succeeded4790worker2 # [ 87.488922] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4791client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051...4792client # uses-input-w1onlya> worker2: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4793client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds4794lb # [ 88.562836] envoy[908]: conn peer=[2001:db8:1::1]:49552 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3417 tx=1713 ms=19634795worker2 # [ 88.720413] nix-daemon[836]: reaped child process 1152, status = succeeded4796worker2 # [ 88.855065] nix-daemon[836]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)4797worker2 # [ 88.878350] nix-daemon[836]: reaped child process 1171, status = succeeded4798lb # [ 88.568232] envoy[908]: conn peer=[2001:db8:1::1]:49566 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=334 ms=1004799lb # [ 88.568501] envoy[908]: conn peer=[2001:db8:1::1]:49574 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3899 tx=1956 ms=21274800lb # [ 88.568611] envoy[908]: conn peer=[2001:db8:1::1]:50766 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=375 ms=644801lb # [ 88.568706] envoy[908]: conn peer=[2001:db8:1::1]:50780 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4338 tx=2188 ms=24544802lb # [ 88.568799] envoy[908]: conn peer=[2001:db8:1::1]:50786 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=438 tx=407 ms=1164803lb # [ 88.889888] niks3-server[960]: 2026/09/22 11:26:27 INFO Received uploads request method=POST path=/api/pending_closures4804worker2 # [ 88.936631] nix-grpc-daemon[850]: time=2026-09-22T11:26:27.548Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4805lb # [ 88.965672] niks3-server[960]: 2026/09/22 11:26:27 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls4806lb # [ 88.976638] niks3-server[960]: 2026/09/22 11:26:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign4807lb # [ 88.984581] niks3-server[960]: 2026/09/22 11:26:27 INFO Signed narinfos id=24 count=14808worker2 # [ 88.984680] nix-grpc-daemon[850]: time=2026-09-22T11:26:27.598Z level=INFO msg="Uploading 1 narinfos"4809lb # [ 88.994320] niks3-server[960]: 2026/09/22 11:26:27 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4810lb # [ 89.014462] niks3-server[960]: 2026/09/22 11:26:27 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete4811lb # [ 89.028700] niks3-server[960]: 2026/09/22 11:26:27 INFO Completed upload id=244812lb # [ 89.032212] niks3-server[960]: 2026/09/22 11:26:27 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo4813worker2 # [ 89.030485] nix-grpc-daemon[850]: time=2026-09-22T11:26:27.645Z level=INFO msg="Upload complete. (325ms)"4814worker2 # [ 89.045906] nix-grpc-daemon[769]: ts=2026-09-22T11:26:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=12 outputs=14815worker2 # [ 89.071288] nix-daemon[836]: reaped child process 1150, status = succeeded4816client # copying 1 paths...4817worker1 # [ 89.167760] nix-grpc-daemon[766]: ts=2026-09-22T11:26:27Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558444818worker1 # [ 89.194366] nix-grpc-daemon[766]: copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'http://lb:5751'...4819client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'...4820worker1 # [ 89.277902] nix-grpc-daemon[766]: ts=2026-09-22T11:26:27Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:558444821worker1 # [ 89.286023] nix-grpc-daemon[766]: ts=2026-09-22T11:26:27Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=0 paths=1 nar_bytes_out=1204822client # [ 89.486406] nix-daemon[1107]: reaped child process 1193, status = succeeded4823client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2, in 3.49 seconds)4824client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&24825client # [ 89.732549] nix-daemon[1107]: accepted connection from pid 1212, user root (trusted)4826client # this derivation will be built:4827client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4828worker1 # [ 89.878726] nix-grpc-daemon[766]: ts=2026-09-22T11:26:28Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:558444829client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4830worker2 # [ 89.826311] nix-grpc-daemon[769]: ts=2026-09-22T11:26:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:468324831worker2 # [ 89.842526] nix-grpc-daemon[850]: time=2026-09-22T11:26:28.457Z level=INFO msg="All 1 paths already cached"4832worker1 # [ 89.931781] nix-grpc-daemon[766]: ts=2026-09-22T11:26:28Z level=debug event=want client=14 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=04833worker1 # [ 89.941396] nix-grpc-daemon[766]: ts=2026-09-22T11:26:28Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=14834worker1 # [ 89.954631] nix-grpc-daemon[766]: ts=2026-09-22T11:26:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558444835worker1 # [ 89.971635] nix-grpc-daemon[766]: ts=2026-09-22T11:26:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:558444836client # copying 1 paths to 'lb:50051'...4837client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'...4838worker1 # [ 90.016975] nix-grpc-daemon[766]: ts=2026-09-22T11:26:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:558444839worker1 # [ 90.045998] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4840worker1 # [ 90.084411] nix-grpc-daemon[766]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 62 ms (attempt 1/5)4841worker1 # [ 90.157589] nix-grpc-daemon[766]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 170 ms (attempt 2/5)4842worker1 # [ 90.334796] nix-grpc-daemon[766]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 270 ms (attempt 3/5)4843worker1 # [ 90.607541] nix-grpc-daemon[766]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 617 ms (attempt 4/5)4844worker1 # [ 91.228763] nix-grpc-daemon[766]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4845worker1 # [ 91.236746] nix-grpc-daemon[766]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4846worker1 # [ 91.245247] nix-grpc-daemon[766]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'...4847worker1 # [ 91.558585] nix-daemon[847]: accepted connection from pid 1160, user nix-grpc-daemon (trusted)4848worker1 # [ 91.598489] nix-daemon[847]: reaped child process 1163, status = succeeded4849worker1 # [ 91.695530] nix-daemon[847]: accepted connection from pid 1165, user nix-grpc-daemon (trusted)4850worker1 # [ 91.716846] nix-daemon[847]: reaped child process 1168, status = succeeded4851lb # [ 91.655108] niks3-server[960]: 2026/09/22 11:26:30 INFO Received uploads request method=POST path=/api/pending_closures4852worker1 # [ 91.731761] nix-grpc-daemon[902]: time=2026-09-22T11:26:30.341Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4853worker1 # [ 91.733481] nix-grpc-daemon[902]: time=2026-09-22T11:26:30.341Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)"4854lb # [ 91.692757] niks3-server[960]: 2026/09/22 11:26:30 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst4855lb # [ 91.708530] niks3-server[960]: 2026/09/22 11:26:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign4856worker1 # [ 91.777333] nix-grpc-daemon[902]: time=2026-09-22T11:26:30.387Z level=INFO msg="Uploading 1 narinfos"4857lb # [ 91.714853] niks3-server[960]: 2026/09/22 11:26:30 INFO Signed narinfos id=25 count=14858lb # [ 91.723718] niks3-server[960]: 2026/09/22 11:26:30 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls4859lb # [ 91.734327] niks3-server[960]: 2026/09/22 11:26:30 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete4860worker1 # [ 91.804510] nix-grpc-daemon[902]: time=2026-09-22T11:26:30.415Z level=INFO msg="Upload complete. (518ms)"4861lb # [ 91.745296] niks3-server[960]: 2026/09/22 11:26:30 INFO Completed upload id=254862lb # [ 91.749867] niks3-server[960]: 2026/09/22 11:26:30 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo4863worker1 # [ 91.813077] nix-grpc-daemon[766]: ts=2026-09-22T11:26:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=1 paths=1 nar_bytes_in=5684864worker1 # [ 91.826803] nix-grpc-daemon[766]: ts=2026-09-22T11:26:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:558444865worker1 # [ 91.838806] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4866worker1 # [ 91.848375] nix-daemon[847]: reaped child process 1153, status = succeeded4867client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051...4868worker1 # [ 91.857820] nix-daemon[847]: accepted connection from pid 766, user nix-grpc-daemon (trusted)4869client # uses-input-w1onlyb> node-a: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4870client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds4871worker1 # [ 92.965110] nix-daemon[847]: reaped child process 1172, status = succeeded4872worker1 # [ 93.035813] nix-daemon[847]: accepted connection from pid 1188, user nix-grpc-daemon (trusted)4873worker1 # [ 93.058100] nix-daemon[847]: reaped child process 1191, status = succeeded4874lb # [ 92.994503] niks3-server[960]: 2026/09/22 11:26:31 INFO Received uploads request method=POST path=/api/pending_closures4875worker1 # [ 93.072147] nix-grpc-daemon[902]: time=2026-09-22T11:26:31.679Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4876lb # [ 93.025389] niks3-server[960]: 2026/09/22 11:26:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign4877worker1 # [ 93.097821] nix-grpc-daemon[902]: time=2026-09-22T11:26:31.708Z level=INFO msg="Uploading 1 narinfos"4878lb # [ 93.036976] niks3-server[960]: 2026/09/22 11:26:31 INFO Signed narinfos id=26 count=14879lb # [ 93.048030] niks3-server[960]: 2026/09/22 11:26:31 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls4880lb # [ 93.057097] niks3-server[960]: 2026/09/22 11:26:31 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4881lb # [ 93.067813] niks3-server[960]: 2026/09/22 11:26:31 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete4882worker1 # [ 93.135578] nix-grpc-daemon[902]: time=2026-09-22T11:26:31.746Z level=INFO msg="Upload complete. (268ms)"4883lb # [ 93.074218] niks3-server[960]: 2026/09/22 11:26:31 INFO Completed upload id=264884lb # [ 93.080796] niks3-server[960]: 2026/09/22 11:26:31 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo4885worker1 # [ 93.149138] nix-grpc-daemon[766]: ts=2026-09-22T11:26:31Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=13 outputs=14886worker1 # [ 93.165208] nix-daemon[847]: reaped child process 1170, status = succeeded4887client # copying 1 paths...4888worker2 # [ 93.135145] nix-grpc-daemon[769]: ts=2026-09-22T11:26:31Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:468324889worker2 # [ 93.151371] nix-grpc-daemon[769]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'...4890client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'...4891worker1 # [ 93.263194] nix-grpc-daemon[766]: ts=2026-09-22T11:26:31Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:558444892worker1 # [ 93.268597] nix-grpc-daemon[766]: ts=2026-09-22T11:26:31Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=0 paths=1 nar_bytes_out=1204893client # [ 93.447391] nix-daemon[1107]: reaped child process 1215, status = succeeded4894client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.96 seconds)4895client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&24896client # [ 93.620322] nix-daemon[1107]: accepted connection from pid 1234, user root (trusted)4897client # this derivation will be built:4898client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4899worker1 # [ 93.693356] nix-grpc-daemon[766]: ts=2026-09-22T11:26:32Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:558444900client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4901worker2 # [ 93.636594] nix-grpc-daemon[769]: ts=2026-09-22T11:26:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:468324902worker2 # [ 93.649342] nix-grpc-daemon[850]: time=2026-09-22T11:26:32.263Z level=INFO msg="All 1 paths already cached"4903worker1 # [ 93.729047] nix-grpc-daemon[766]: ts=2026-09-22T11:26:32Z level=debug event=want client=15 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=04904worker1 # [ 93.735934] nix-grpc-daemon[766]: ts=2026-09-22T11:26:32Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=14905worker2 # [ 93.678221] nix-grpc-daemon[769]: ts=2026-09-22T11:26:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324906worker2 # [ 93.694372] nix-grpc-daemon[769]: ts=2026-09-22T11:26:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:468324907worker2 # [ 93.709830] nix-grpc-daemon[850]: time=2026-09-22T11:26:32.324Z level=INFO msg="All 1 paths already cached"4908client # copying 1 paths to 'lb:50051'...4909client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'...4910worker2 # [ 93.721184] nix-grpc-daemon[769]: ts=2026-09-22T11:26:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:468324911worker2 # [ 93.740410] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4912worker2 # [ 93.903286] nix-daemon[836]: accepted connection from pid 1176, user nix-grpc-daemon (trusted)4913worker2 # [ 93.928626] nix-daemon[836]: reaped child process 1179, status = succeeded4914worker2 # [ 94.026353] nix-daemon[836]: accepted connection from pid 1181, user nix-grpc-daemon (trusted)4915worker2 # [ 94.042982] nix-daemon[836]: reaped child process 1184, status = succeeded4916lb # [ 94.050611] niks3-server[960]: 2026/09/22 11:26:32 INFO Received uploads request method=POST path=/api/pending_closures4917worker2 # [ 94.061637] nix-grpc-daemon[850]: time=2026-09-22T11:26:32.675Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4918worker2 # [ 94.063128] nix-grpc-daemon[850]: time=2026-09-22T11:26:32.676Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)"4919lb # [ 94.086446] niks3-server[960]: 2026/09/22 11:26:32 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst4920worker2 # [ 94.100784] nix-grpc-daemon[850]: time=2026-09-22T11:26:32.715Z level=INFO msg="Uploading 1 narinfos"4921lb # [ 94.099549] niks3-server[960]: 2026/09/22 11:26:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign4922lb # [ 94.110278] niks3-server[960]: 2026/09/22 11:26:32 INFO Signed narinfos id=27 count=14923lb # [ 94.115814] niks3-server[960]: 2026/09/22 11:26:32 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls4924lb # [ 94.124932] niks3-server[960]: 2026/09/22 11:26:32 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete4925lb # [ 94.129377] niks3-server[960]: 2026/09/22 11:26:32 INFO Completed upload id=274926worker2 # [ 94.130251] nix-grpc-daemon[850]: time=2026-09-22T11:26:32.745Z level=INFO msg="Upload complete. (369ms)"4927lb # [ 94.131853] niks3-server[960]: 2026/09/22 11:26:32 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo4928worker2 # [ 94.141385] nix-grpc-daemon[769]: ts=2026-09-22T11:26:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=0 paths=1 nar_bytes_in=5684929worker2 # [ 94.154557] nix-grpc-daemon[769]: ts=2026-09-22T11:26:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:468324930worker2 # [ 94.162519] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4931worker2 # [ 94.164732] nix-daemon[836]: reaped child process 1174, status = succeeded4932client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051...4933client # uses-input-w1onlyc> worker2: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4934worker2 # [ 94.177112] nix-daemon[836]: accepted connection from pid 769, user nix-grpc-daemon (trusted)4935client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds4936worker2 # [ 94.852843] nix-daemon[836]: reaped child process 1188, status = succeeded4937worker2 # [ 94.952663] nix-daemon[836]: accepted connection from pid 1204, user nix-grpc-daemon (trusted)4938worker2 # [ 94.972572] nix-daemon[836]: reaped child process 1207, status = succeeded4939lb # [ 94.974407] niks3-server[960]: 2026/09/22 11:26:33 INFO Received uploads request method=POST path=/api/pending_closures4940worker2 # [ 94.986890] nix-grpc-daemon[850]: time=2026-09-22T11:26:33.599Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4941worker2 # [ 95.005347] nix-grpc-daemon[850]: time=2026-09-22T11:26:33.619Z level=INFO msg="Uploading 1 narinfos"4942lb # [ 95.003988] niks3-server[960]: 2026/09/22 11:26:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign4943lb # [ 95.017094] niks3-server[960]: 2026/09/22 11:26:33 INFO Signed narinfos id=28 count=14944lb # [ 95.026064] niks3-server[960]: 2026/09/22 11:26:33 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4945lb # [ 95.038484] niks3-server[960]: 2026/09/22 11:26:33 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls4946worker2 # [ 95.046939] nix-grpc-daemon[850]: time=2026-09-22T11:26:33.661Z level=INFO msg="Upload complete. (257ms)"4947lb # [ 95.047535] niks3-server[960]: 2026/09/22 11:26:33 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete4948lb # [ 95.056618] niks3-server[960]: 2026/09/22 11:26:33 INFO Completed upload id=284949client # copying 1 paths...4950lb # [ 95.061772] niks3-server[960]: 2026/09/22 11:26:33 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo4951worker2 # [ 95.060284] nix-grpc-daemon[769]: ts=2026-09-22T11:26:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46832 duration_s=0 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=14 outputs=14952worker1 # [ 95.126066] nix-grpc-daemon[766]: ts=2026-09-22T11:26:33Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558444953worker2 # [ 95.074863] nix-daemon[836]: reaped child process 1186, status = succeeded4954worker1 # [ 95.139025] nix-grpc-daemon[766]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'...4955client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'...4956worker1 # [ 95.181779] nix-grpc-daemon[766]: ts=2026-09-22T11:26:33Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:558444957worker1 # [ 95.187016] nix-grpc-daemon[766]: ts=2026-09-22T11:26:33Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:55844 duration_s=0 paths=1 nar_bytes_out=1204958client # [ 95.390337] nix-daemon[1107]: reaped child process 1237, status = succeeded4959client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2, in 1.94 seconds)4960(finished: subtest: an input only one worker has still reaches the builder, in 10.14 seconds)4961subtest: upload whose reference the worker lacks is completed from the cache4962client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache4963client # this derivation will be built:4964client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv4965client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'...4966client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache, in 0.34 seconds)4967client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache4968client # this derivation will be built:4969client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv4970client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'...4971client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache, in 0.29 seconds)4972client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure4973client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.12 seconds)4974worker1: must succeed: nix-store --import < /tmp/shared/ref.closure4975worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.19 seconds)4976worker2 # [ 96.271386] systemd[1]: Stopping Nix worker-protocol over gRPC...4977worker2 # [ 96.916583] nix-grpc-daemon[769]: ts=2026-09-22T11:26:35Z level=info event=signal_exit4978worker2 # [ 98.477517] nix-grpc-daemon[769]: ts=2026-09-22T11:26:37Z level=info event=scheduler_restarting4979worker1 # [ 98.552664] nix-grpc-daemon[766]: ts=2026-09-22T11:26:37Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:558284980worker1 # [ 98.552985] nix-grpc-daemon[766]: ts=2026-09-22T11:26:37Z level=info event=worker_gone addr=192.168.1.4:500514981worker2 # [ 98.517025] nix-grpc-daemon[769]: ts=2026-09-22T11:26:37Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4982lb # [ 98.559831] envoy[908]: conn peer=[2001:db8:1::1]:50796 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4273 tx=1977 ms=28344983worker2 # [ 98.563967] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4984worker2 # [ 98.567842] systemd[1]: Stopped Nix worker-protocol over gRPC.4985worker2 # [ 98.581875] systemd[1]: nix-grpc-daemon.service: Consumed 3.881s CPU time over 1min 19.874s wall clock time, 62.1M memory peak, 91.7K incoming IP traffic, 201.3K outgoing IP traffic.4986worker2 # [ 98.594910] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.4987lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy4988lb: must succeed: curl -sf localhost:9901/clusters4989worker2 # [ 98.597469] systemd[1]: Closed Nix worker-protocol over gRPC.4990worker2 # [ 98.598536] systemd[1]: nix-grpc-daemon.socket: Consumed 622us CPU time over 1min 28.234s wall clock time, 256K memory peak, 85.6K incoming IP traffic, 71K outgoing IP traffic.4991worker2 # [ 98.602092] nix-daemon[836]: reaped child process 838, status = succeeded4992lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4993lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.08 seconds)4994client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&24995worker1 # [ 98.952314] nix-grpc-daemon[766]: ts=2026-09-22T11:26:37Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:558444996worker1 # [ 98.961460] nix-grpc-daemon[766]: ts=2026-09-22T11:26:37Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:558444997client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4998client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.25 seconds)4999worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache5000worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.03 seconds)5001worker2 # [ 99.017813] systemd[1]: Listening on Nix worker-protocol over gRPC.5002worker1 # [ 99.153272] systemd[1]: Stopping Nix worker-protocol over gRPC...5003worker2 # [ 99.107220] systemd[1]: Starting scheduler-token.service...5004lb # [ 98.565316] envoy[908]: conn peer=[2001:db8:1::1]:50804 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=329 ms=525005lb # [ 98.565453] envoy[908]: conn peer=[2001:db8:1::1]:42418 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=328 ms=275006lb # [ 98.565593] envoy[908]: conn peer=[2001:db8:1::1]:50812 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3772 tx=2030 ms=34115007lb # [ 98.565709] envoy[908]: conn peer=[2001:db8:1::1]:42422 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3725 tx=1972 ms=15465008lb # [ 98.565810] envoy[908]: conn peer=[2001:db8:1::1]:42428 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=615009lb # [ 98.565910] envoy[908]: conn peer=[2001:db8:1::4]:52832 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=50553 tx=2956 ms=795555010worker2 # [ 99.228188] systemd[1]: scheduler-token.service: Deactivated successfully.5011worker2 # [ 99.236487] systemd[1]: Finished scheduler-token.service.5012worker2 # [ 99.238743] systemd[1]: scheduler-token.service: Consumed 60ms CPU time over 124ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5013worker2 # [ 99.249761] systemd[1]: Starting Nix worker-protocol over gRPC...5014worker2 # [ 99.396729] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:38Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5015worker2 # [ 99.398742] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:38Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5016worker2 # [ 99.403058] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:38Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15017worker2 # [ 99.409271] nix-grpc-daemon[1231]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5018worker2 # [ 99.410810] nix-grpc-daemon[1231]: E0000 00:00:1790076398.023426 1231 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5019worker2 # [ 99.456310] nix-grpc-daemon[1231]: E0000 00:00:1790076398.030077 1231 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5020worker2 # [ 99.458691] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:38Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5021worker2 # [ 99.467694] nix-grpc-daemon[1231]: E0000 00:00:1790076398.042659 1272 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5022worker2 # [ 99.470929] nix-grpc-daemon[1231]: E0000 00:00:1790076398.061745 1272 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5023worker2 # [ 99.473587] nix-grpc-daemon[1231]: E0000 00:00:1790076398.061989 1272 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5024worker2 # [ 99.479168] systemd[1]: Started Nix worker-protocol over gRPC.5025worker2 # [ 99.482564] nix-grpc-daemon[1231]: E0000 00:00:1790076398.081321 1272 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5026worker2 # [ 99.485221] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:38Z level=info event=scheduler_connected addr=lb:500515027worker1 # [ 99.546974] nix-grpc-daemon[766]: ts=2026-09-22T11:26:38Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:558285028worker1 # [ 99.554094] nix-grpc-daemon[766]: ts=2026-09-22T11:26:38Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05029worker1 # [ 99.888959] nix-grpc-daemon[766]: ts=2026-09-22T11:26:38Z level=info event=signal_exit5030lb # [ 100.477396] niks3-server[960]: 2026/09/22 11:26:39 INFO lead: released remote=192.168.1.3:494165031worker2 # [ 101.480755] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:40Z level=info event=scheduler_restarting addr=lb:500515032worker1 # [ 101.549366] nix-grpc-daemon[766]: ts=2026-09-22T11:26:40Z level=info event=scheduler_restarting5033worker1 # [ 101.553609] nix-grpc-daemon[766]: ts=2026-09-22T11:26:40Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:558285034worker1 # [ 101.553759] nix-grpc-daemon[766]: ts=2026-09-22T11:26:40Z level=info event=worker_gone addr=192.168.1.4:500515035worker1 # [ 101.553953] nix-grpc-daemon[766]: ts=2026-09-22T11:26:40Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:558285036worker1 # [ 101.554069] nix-grpc-daemon[766]: ts=2026-09-22T11:26:40Z level=info event=worker_gone addr=192.168.1.3:500515037worker1 # [ 101.554179] nix-grpc-daemon[766]: ts=2026-09-22T11:26:40Z level=info event=scheduler_restarting addr=lb:500515038worker1 # [ 101.580397] nix-daemon[847]: reaped child process 849, status = succeeded5039worker2 # [ 101.680835] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:40Z level=info event=scheduler_connected addr=lb:500515040worker2 # [ 101.686715] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5041worker1 # [ 101.773472] nix-grpc-daemon[766]: ts=2026-09-22T11:26:40Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5042worker2 # [ 101.888162] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:40Z level=info event=scheduler_connected addr=lb:500515043worker2 # [ 101.893573] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5044worker1 # [ 102.047937] nix-grpc-daemon[766]: E0000 00:00:1790076400.658593 811 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5045worker1 # [ 102.054724] nix-grpc-daemon[766]: E0000 00:00:1790076400.658880 811 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5046worker2 # [ 102.094531] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:40Z level=info event=scheduler_connected addr=lb:500515047worker2 # [ 102.098175] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5048worker2 # [ 102.299449] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:40Z level=info event=scheduler_connected addr=lb:500515049worker2 # [ 102.304777] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5050worker1 # [ 102.564541] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5051worker2 # [ 102.507508] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:41Z level=info event=scheduler_connected addr=lb:500515052worker1 # [ 102.571947] systemd[1]: Stopped Nix worker-protocol over gRPC.5053worker2 # [ 102.511861] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5054worker1 # [ 102.578434] systemd[1]: nix-grpc-daemon.service: Consumed 3.376s CPU time over 1min 23.939s wall clock time, 62.1M memory peak, 74.9K incoming IP traffic, 150.2K outgoing IP traffic.5055worker1 # [ 102.594325] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5056worker1 # [ 102.597541] systemd[1]: Closed Nix worker-protocol over gRPC.5057worker1 # [ 102.599531] systemd[1]: nix-grpc-daemon.socket: Consumed 615us CPU time over 1min 32.321s wall clock time, 256K memory peak, 104.8K incoming IP traffic, 88.3K outgoing IP traffic.5058lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy5059lb: must succeed: curl -sf localhost:9901/clusters5060lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5061lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.08 seconds)5062client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&25063worker2 # [ 102.708689] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:41Z level=info event=scheduler_connected addr=lb:500515064worker2 # [ 102.714293] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5065worker2 # [ 102.853969] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:581745066worker2 # [ 102.869275] nix-daemon[836]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5067worker2 # [ 102.896110] nix-grpc-daemon[1231]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 49 ms (attempt 1/5)5068worker2 # [ 102.915428] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:41Z level=info event=scheduler_connected addr=lb:500515069worker2 # [ 102.920728] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5070worker2 # [ 102.953555] nix-grpc-daemon[1231]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 192 ms (attempt 2/5)5071worker2 # [ 103.122073] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:41Z level=info event=scheduler_connected addr=lb:500515072worker2 # [ 103.127368] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5073worker2 # [ 103.147596] nix-grpc-daemon[1231]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 288 ms (attempt 3/5)5074worker2 # [ 103.328802] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:41Z level=info event=scheduler_connected addr=lb:500515075worker2 # [ 103.334070] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5076worker2 # [ 103.437851] nix-grpc-daemon[1231]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 560 ms (attempt 4/5)5077worker2 # [ 103.537067] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:42Z level=info event=scheduler_connected addr=lb:500515078worker2 # [ 103.541285] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5079worker2 # [ 103.738136] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:42Z level=info event=scheduler_connected addr=lb:500515080worker2 # [ 103.743448] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5081worker2 # [ 103.944657] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:42Z level=info event=scheduler_connected addr=lb:500515082worker2 # [ 103.949881] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5083worker2 # [ 104.001919] nix-grpc-daemon[1231]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5084worker2 # [ 104.009309] nix-grpc-daemon[1231]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5085client # copying 2 paths...5086client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'...5087worker2 # [ 104.029968] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:581745088client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'...5089worker2 # [ 104.051771] nix-daemon[836]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5090worker2 # [ 104.080755] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:42Z level=info event=niks3_push_started pid=12915091worker2 # [ 104.152708] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:42Z level=info event=scheduler_connected addr=lb:500515092worker2 # [ 104.155435] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5093worker2 # [ 104.245040] nix-grpc-daemon[1291]: time=2026-09-22T11:26:42.859Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""5094worker2 # [ 104.358045] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:42Z level=info event=scheduler_connected addr=lb:500515095worker2 # [ 104.359812] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5096worker2 # [ 104.365479] nix-daemon[836]: accepted connection from pid 1304, user nix-grpc-daemon (trusted)5097worker2 # [ 104.388429] nix-daemon[836]: reaped child process 1307, status = succeeded5098lb # [ 104.385846] niks3-server[960]: 2026/09/22 11:26:42 INFO Received uploads request method=POST path=/api/pending_closures5099lb # [ 104.397871] niks3-server[960]: 2026/09/22 11:26:43 INFO Received uploads request method=POST path=/api/pending_closures5100worker2 # [ 104.411158] nix-grpc-daemon[1291]: time=2026-09-22T11:26:43.024Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5101worker2 # [ 104.414095] nix-grpc-daemon[1291]: time=2026-09-22T11:26:43.024Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)"5102worker2 # [ 104.415837] nix-grpc-daemon[1291]: time=2026-09-22T11:26:43.024Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)"5103worker2 # [ 104.427341] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:43Z level=info event=scheduler_take_over5104lb # [ 104.426143] niks3-server[960]: 2026/09/22 11:26:43 INFO lead: acquired remote=192.168.1.4:408345105lb # [ 104.470158] niks3-server[960]: 2026/09/22 11:26:43 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst5106lb # [ 104.480360] niks3-server[960]: 2026/09/22 11:26:43 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5107lb # [ 104.489196] niks3-server[960]: 2026/09/22 11:26:43 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls5108lb # [ 104.496078] niks3-server[960]: 2026/09/22 11:26:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5109lb # [ 104.505320] niks3-server[960]: 2026/09/22 11:26:43 INFO Signed narinfos id=29 count=15110worker2 # [ 104.507972] nix-grpc-daemon[1291]: time=2026-09-22T11:26:43.120Z level=INFO msg="Uploading 3 narinfos"5111lb # [ 104.509631] niks3-server[960]: 2026/09/22 11:26:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5112lb # [ 104.515081] niks3-server[960]: 2026/09/22 11:26:43 INFO Signed narinfos id=30 count=25113lb # [ 104.518875] niks3-server[960]: 2026/09/22 11:26:43 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls5114lb # [ 104.537161] niks3-server[960]: 2026/09/22 11:26:43 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo5115lb # [ 104.547745] niks3-server[960]: 2026/09/22 11:26:43 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5116lb # [ 104.554405] niks3-server[960]: 2026/09/22 11:26:43 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5117worker2 # [ 104.562401] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:43Z level=info event=scheduler_connected addr=lb:500515118lb # [ 104.564787] niks3-server[960]: 2026/09/22 11:26:43 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5119lb # [ 104.579666] niks3-server[960]: 2026/09/22 11:26:43 INFO Completed upload id=295120worker2 # [ 104.581622] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:43Z level=info event=oidc_jwks provider=mock keys=15121lb # [ 104.587485] niks3-server[960]: 2026/09/22 11:26:43 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5122lb # [ 104.590491] niks3-server[960]: 2026/09/22 11:26:43 INFO Completed upload id=305123worker2 # [ 104.588350] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:43Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:581785124worker2 # [ 104.600664] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:43Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05125worker2 # [ 104.615998] nix-grpc-daemon[1291]: time=2026-09-22T11:26:43.208Z level=INFO msg="Upload complete. (345ms)"5126worker2 # [ 104.619371] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58174 duration_s=0 paths=2 nar_bytes_in=3125127worker2 # [ 104.624492] nix-daemon[836]: reaped child process 1289, status = succeeded5128client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 2.02 seconds)5129worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache5130worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.03 seconds)5131worker1 # [ 104.811325] systemd[1]: Listening on Nix worker-protocol over gRPC.5132worker1 # [ 104.821153] systemd[1]: Starting Nix worker-protocol over gRPC...5133worker1 # [ 104.980510] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:43Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5134worker1 # [ 104.983485] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:43Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5135worker1 # [ 104.987703] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:43Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15136worker1 # [ 104.995123] nix-grpc-daemon[1217]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5137worker1 # [ 105.012165] nix-grpc-daemon[1217]: E0000 00:00:1790076403.603687 1217 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5138worker1 # [ 105.014486] nix-grpc-daemon[1217]: E0000 00:00:1790076403.606051 1217 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5139lb: waiting until exactly no endpoints of aarch64-linux are unhealthy5140lb: must succeed: curl -sf localhost:9901/clusters5141worker1 # [ 105.044192] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:43Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5142worker1 # [ 105.046379] nix-grpc-daemon[1217]: E0000 00:00:1790076403.616781 1259 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5143worker1 # [ 105.049075] nix-grpc-daemon[1217]: E0000 00:00:1790076403.638343 1259 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5144worker1 # [ 105.051300] nix-grpc-daemon[1217]: E0000 00:00:1790076403.638537 1259 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5145worker1 # [ 105.053767] nix-grpc-daemon[1217]: E0000 00:00:1790076403.654109 1259 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5146worker1 # [ 105.059288] systemd[1]: Started Nix worker-protocol over gRPC.5147worker1 # [ 105.062490] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:43Z level=info event=scheduler_connected addr=lb:500515148worker2 # [ 105.002440] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:43Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:581785149worker2 # [ 105.009119] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:43Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05150lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5151lb: must succeed: curl -sf localhost:9901/clusters5152lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5153lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.15 seconds)5154(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 10.87 seconds)5155subtest: a niks3 restart does not cost the leader its role5156lb: must succeed: curl -sf localhost:9901/clusters5157lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5158lb: must succeed: curl -sf localhost:9901/clusters5159lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5160lb # [ 106.248685] systemd[1]: Stopping niks3 server...5161lb # [ 106.249474] niks3-server[960]: 2026/09/22 11:26:44 INFO Shutdown signal received, draining in-flight requests timeout=10s5162worker1 # [ 106.312514] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:44Z level=info event=scheduler_lock_lost status=2005163worker2 # [ 106.260271] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:44Z level=info event=scheduler_lock_lost status=2005164lb # [ 106.263064] niks3-server[960]: 2026/09/22 11:26:44 INFO lead: released remote=192.168.1.4:408345165lb # [ 106.277148] systemd[1]: niks3.service: Deactivated successfully.5166lb # [ 106.278753] systemd[1]: Stopped niks3 server.5167lb # [ 106.279615] systemd[1]: niks3.service: Consumed 605ms CPU time over 1min 25.059s wall clock time, 26.1M memory peak, 78.5K incoming IP traffic, 114.2K outgoing IP traffic.5168lb # [ 106.290084] systemd[1]: Starting niks3 server...5169lb # [ 106.354591] niks3-server[1098]: 2026/09/22 11:26:44 goose: no migrations to run. current version: 202609200000005170lb # [ 106.361285] niks3-server[1098]: 2026/09/22 11:26:44 OK 1_commit_pending_closure.sql (4.77ms)5171lb # [ 106.364997] niks3-server[1098]: 2026/09/22 11:26:44 OK 2_object_stats_trigger.sql (2.03ms)5172lb # [ 106.366150] niks3-server[1098]: 2026/09/22 11:26:44 goose: up to current file version: 25173lb # [ 106.367882] niks3-server[1098]: 2026/09/22 11:26:44 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key5174lb # [ 106.379478] niks3-server[1098]: 2026/09/22 11:26:44 INFO Read proxy enabled — serving cache objects from S35175lb # [ 106.383396] niks3-server[1098]: 2026/09/22 11:26:44 INFO Using socket-activated listener address=0.0.0.0:57515176lb # [ 106.387792] niks3-server[1098]: 2026/09/22 11:26:44 INFO systemd watchdog enabled interval=15s5177lb # [ 106.389374] niks3-server[1098]: 2026/09/22 11:26:44 INFO Starting HTTP server address=0.0.0.0:57515178lb # [ 106.390792] systemd[1]: Started niks3 server.5179lb: waiting for unit niks3.service5180lb: (finished: waiting for unit niks3.service, in 0.03 seconds)5181lb: must succeed: curl -sf localhost:9901/clusters5182lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5183worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5184worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5185lb: must succeed: curl -sf localhost:9901/clusters5186lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5187lb: must succeed: curl -sf localhost:9901/clusters5188lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5189worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5190worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.07 seconds)5191client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&25192worker2 # [ 106.883918] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:45Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:581745193client # don't know how to build these paths:5194client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5195worker2 # [ 106.895648] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:581745196worker2 # [ 106.926170] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:45Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=05197worker2 # [ 106.933508] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:45Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.4:50051 clients=15198worker2 # [ 106.975166] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:581745199worker2 # [ 106.991205] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:581745200client # copying 1 paths to 'lb:50051'...5201client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'...5202worker2 # [ 107.007279] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:581745203worker2 # [ 107.022434] nix-daemon[836]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5204worker2 # [ 107.182669] nix-daemon[836]: accepted connection from pid 1324, user nix-grpc-daemon (trusted)5205worker2 # [ 107.202189] nix-daemon[836]: reaped child process 1327, status = succeeded5206lb # [ 107.261562] niks3-server[1098]: 2026/09/22 11:26:45 INFO lead: acquired remote=192.168.1.4:371245207worker2 # [ 107.308320] nix-daemon[836]: accepted connection from pid 1329, user nix-grpc-daemon (trusted)5208worker2 # [ 107.325865] nix-daemon[836]: reaped child process 1334, status = succeeded5209lb # [ 107.333231] niks3-server[1098]: 2026/09/22 11:26:45 INFO Received uploads request method=POST path=/api/pending_closures5210worker2 # [ 107.365014] nix-grpc-daemon[1291]: time=2026-09-22T11:26:45.978Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5211worker2 # [ 107.367304] nix-grpc-daemon[1291]: time=2026-09-22T11:26:45.979Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)"5212lb # [ 107.398773] niks3-server[1098]: 2026/09/22 11:26:45 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst5213lb # [ 107.409325] niks3-server[1098]: 2026/09/22 11:26:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5214worker2 # [ 107.415365] nix-grpc-daemon[1291]: time=2026-09-22T11:26:46.029Z level=INFO msg="Uploading 1 narinfos"5215lb # [ 107.418687] niks3-server[1098]: 2026/09/22 11:26:46 INFO Signed narinfos id=31 count=15216lb # [ 107.426688] niks3-server[1098]: 2026/09/22 11:26:46 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls5217lb # [ 107.439738] niks3-server[1098]: 2026/09/22 11:26:46 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5218lb # [ 107.451967] niks3-server[1098]: 2026/09/22 11:26:46 INFO Completed upload id=315219worker2 # [ 107.448733] nix-grpc-daemon[1291]: time=2026-09-22T11:26:46.063Z level=INFO msg="Upload complete. (408ms)"5220worker2 # [ 107.461346] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58174 duration_s=0 paths=1 nar_bytes_in=5445221worker2 # [ 107.480692] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:581745222worker2 # [ 107.488586] nix-daemon[836]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5223worker2 # [ 107.494801] nix-daemon[836]: reaped child process 1322, status = succeeded5224client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051...5225worker2 # [ 107.507180] nix-daemon[836]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5226client # farm-a> worker2: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5227client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5228lb # [ 108.558601] envoy[908]: conn peer=[2001:db8:1::1]:42444 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=519 ms=305229lb # [ 108.564235] envoy[908]: conn peer=[2001:db8:1::3]:50992 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=9355 tx=2705 ms=819705230lb # [ 108.569431] envoy[908]: conn peer=[2001:db8:1::1]:47402 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1008 tx=282 ms=17635231client # farm-a> LOG-farm-a-after-niks3-restart5232worker2 # [ 108.765139] nix-daemon[836]: reaped child process 1338, status = succeeded5233worker2 # [ 108.888741] nix-daemon[836]: accepted connection from pid 1354, user nix-grpc-daemon (trusted)5234worker2 # [ 108.908219] nix-daemon[836]: reaped child process 1357, status = succeeded5235lb # [ 108.914086] niks3-server[1098]: 2026/09/22 11:26:47 INFO Received uploads request method=POST path=/api/pending_closures5236worker2 # [ 108.932463] nix-grpc-daemon[1291]: time=2026-09-22T11:26:47.544Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5237worker2 # [ 108.934248] nix-grpc-daemon[1291]: time=2026-09-22T11:26:47.544Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)"5238lb # [ 108.975657] niks3-server[1098]: 2026/09/22 11:26:47 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5239lb # [ 108.985669] niks3-server[1098]: 2026/09/22 11:26:47 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5240worker2 # [ 108.996116] nix-grpc-daemon[1291]: time=2026-09-22T11:26:47.610Z level=INFO msg="Uploading 1 narinfos"5241lb # [ 108.997642] niks3-server[1098]: 2026/09/22 11:26:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5242lb # [ 109.002070] niks3-server[1098]: 2026/09/22 11:26:47 INFO Signed narinfos id=32 count=15243lb # [ 109.007317] niks3-server[1098]: 2026/09/22 11:26:47 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls5244lb # [ 109.014020] niks3-server[1098]: 2026/09/22 11:26:47 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5245lb # [ 109.024843] niks3-server[1098]: 2026/09/22 11:26:47 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.narinfo5246lb # [ 109.032675] niks3-server[1098]: 2026/09/22 11:26:47 INFO Completed upload id=325247worker2 # [ 109.034284] nix-grpc-daemon[1291]: time=2026-09-22T11:26:47.648Z level=INFO msg="Upload complete. (313ms)"5248worker2 # [ 109.057377] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58174 duration_s=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=15249worker2 # [ 109.076853] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:47Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=05250worker2 # [ 109.081347] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:47Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.4:50051 clients=15251worker2 # [ 109.086300] nix-daemon[836]: reaped child process 1336, status = succeeded5252worker2 # [ 109.098107] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:581745253worker2 # [ 109.114037] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:581745254worker2 # [ 109.131256] nix-grpc-daemon[1291]: time=2026-09-22T11:26:47.745Z level=INFO msg="All 1 paths already cached"5255client # copying 1 paths to 'lb:50051'...5256client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'...5257worker2 # [ 109.141813] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:581745258worker2 # [ 109.155551] nix-daemon[836]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5259worker2 # [ 109.310258] nix-daemon[836]: accepted connection from pid 1363, user nix-grpc-daemon (trusted)5260worker2 # [ 109.334999] nix-daemon[836]: reaped child process 1366, status = succeeded5261worker2 # [ 109.436851] nix-daemon[836]: accepted connection from pid 1368, user nix-grpc-daemon (trusted)5262worker2 # [ 109.458770] nix-daemon[836]: reaped child process 1371, status = succeeded5263lb # [ 109.461056] niks3-server[1098]: 2026/09/22 11:26:48 INFO Received uploads request method=POST path=/api/pending_closures5264worker2 # [ 109.471930] nix-grpc-daemon[1291]: time=2026-09-22T11:26:48.086Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5265worker2 # [ 109.473682] nix-grpc-daemon[1291]: time=2026-09-22T11:26:48.086Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)"5266lb # [ 109.490973] niks3-server[1098]: 2026/09/22 11:26:48 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst5267worker2 # [ 109.507636] nix-grpc-daemon[1291]: time=2026-09-22T11:26:48.122Z level=INFO msg="Uploading 1 narinfos"5268lb # [ 109.507541] niks3-server[1098]: 2026/09/22 11:26:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5269lb # [ 109.519906] niks3-server[1098]: 2026/09/22 11:26:48 INFO Signed narinfos id=33 count=15270lb # [ 109.528882] niks3-server[1098]: 2026/09/22 11:26:48 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls5271worker2 # [ 109.543490] nix-grpc-daemon[1291]: time=2026-09-22T11:26:48.158Z level=INFO msg="Upload complete. (372ms)"5272lb # [ 109.542151] niks3-server[1098]: 2026/09/22 11:26:48 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5273lb # [ 109.548956] niks3-server[1098]: 2026/09/22 11:26:48 INFO Completed upload id=335274worker2 # [ 109.552312] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58174 duration_s=0 paths=1 nar_bytes_in=6565275lb # [ 109.552545] niks3-server[1098]: 2026/09/22 11:26:48 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo5276worker2 # [ 109.567761] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:581745277worker2 # [ 109.574824] nix-daemon[836]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5278worker2 # [ 109.580673] nix-daemon[836]: reaped child process 1361, status = succeeded5279client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051...5280client # farm-b> worker2: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5281worker2 # [ 109.595585] nix-daemon[836]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5282client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5283client # farm-b> LOG-farm-b-after-niks3-restart5284worker2 # [ 110.506080] nix-daemon[836]: reaped child process 1375, status = succeeded5285worker2 # [ 110.580562] nix-daemon[836]: accepted connection from pid 1391, user nix-grpc-daemon (trusted)5286worker2 # [ 110.605930] nix-daemon[836]: reaped child process 1394, status = succeeded5287lb # [ 110.603099] niks3-server[1098]: 2026/09/22 11:26:49 INFO Received uploads request method=POST path=/api/pending_closures5288worker2 # [ 110.620154] nix-grpc-daemon[1291]: time=2026-09-22T11:26:49.233Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5289worker2 # [ 110.622176] nix-grpc-daemon[1291]: time=2026-09-22T11:26:49.233Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)"5290lb # [ 110.649101] niks3-server[1098]: 2026/09/22 11:26:49 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5291lb # [ 110.658203] niks3-server[1098]: 2026/09/22 11:26:49 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst5292lb # [ 110.669135] niks3-server[1098]: 2026/09/22 11:26:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5293worker2 # [ 110.675317] nix-grpc-daemon[1291]: time=2026-09-22T11:26:49.289Z level=INFO msg="Uploading 1 narinfos"5294lb # [ 110.676221] niks3-server[1098]: 2026/09/22 11:26:49 INFO Signed narinfos id=34 count=15295lb # [ 110.684344] niks3-server[1098]: 2026/09/22 11:26:49 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls5296lb # [ 110.694303] niks3-server[1098]: 2026/09/22 11:26:49 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5297lb # [ 110.701354] niks3-server[1098]: 2026/09/22 11:26:49 INFO Completed upload id=345298worker2 # [ 110.705853] nix-grpc-daemon[1291]: time=2026-09-22T11:26:49.320Z level=INFO msg="Upload complete. (298ms)"5299lb # [ 110.707689] niks3-server[1098]: 2026/09/22 11:26:49 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.narinfo5300worker2 # [ 110.724946] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58174 duration_s=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=15301worker2 # [ 110.740480] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:49Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=05302worker2 # [ 110.747564] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:49Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.4:50051 clients=15303worker2 # [ 110.752222] nix-daemon[836]: reaped child process 1373, status = succeeded5304worker2 # [ 110.770237] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:581745305worker2 # [ 110.786642] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:581745306worker2 # [ 110.802120] nix-grpc-daemon[1291]: time=2026-09-22T11:26:49.416Z level=INFO msg="All 2 paths already cached"5307client # copying 1 paths to 'lb:50051'...5308client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'...5309worker2 # [ 110.813061] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:581745310worker2 # [ 110.826849] nix-daemon[836]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5311worker2 # [ 110.978440] nix-daemon[836]: accepted connection from pid 1400, user nix-grpc-daemon (trusted)5312worker2 # [ 111.002195] nix-daemon[836]: reaped child process 1403, status = succeeded5313worker2 # [ 111.106359] nix-daemon[836]: accepted connection from pid 1405, user nix-grpc-daemon (trusted)5314worker2 # [ 111.128420] nix-daemon[836]: reaped child process 1408, status = succeeded5315lb # [ 111.124351] niks3-server[1098]: 2026/09/22 11:26:49 INFO Received uploads request method=POST path=/api/pending_closures5316worker2 # [ 111.134923] nix-grpc-daemon[1291]: time=2026-09-22T11:26:49.749Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5317worker2 # [ 111.136637] nix-grpc-daemon[1291]: time=2026-09-22T11:26:49.749Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)"5318lb # [ 111.153710] niks3-server[1098]: 2026/09/22 11:26:49 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst5319worker2 # [ 111.168209] nix-grpc-daemon[1291]: time=2026-09-22T11:26:49.782Z level=INFO msg="Uploading 1 narinfos"5320lb # [ 111.168630] niks3-server[1098]: 2026/09/22 11:26:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5321lb # [ 111.182038] niks3-server[1098]: 2026/09/22 11:26:49 INFO Signed narinfos id=35 count=15322lb # [ 111.189529] niks3-server[1098]: 2026/09/22 11:26:49 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls5323lb # [ 111.201250] niks3-server[1098]: 2026/09/22 11:26:49 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5324worker2 # [ 111.209526] nix-grpc-daemon[1291]: time=2026-09-22T11:26:49.824Z level=INFO msg="Upload complete. (366ms)"5325lb # [ 111.209498] niks3-server[1098]: 2026/09/22 11:26:49 INFO Completed upload id=355326lb # [ 111.211924] niks3-server[1098]: 2026/09/22 11:26:49 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo5327worker2 # [ 111.221131] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:58174 duration_s=0 paths=1 nar_bytes_in=7845328worker2 # [ 111.232305] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:581745329worker2 # [ 111.239518] nix-daemon[836]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5330worker2 # [ 111.245152] nix-daemon[836]: reaped child process 1398, status = succeeded5331client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051...5332client # farm-top> worker2: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5333worker2 # [ 111.259639] nix-daemon[836]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5334client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5335client # farm-top> LOG-farm-top-after-niks3-restart5336worker2 # [ 112.518163] nix-daemon[836]: reaped child process 1412, status = succeeded5337worker2 # [ 112.624836] nix-daemon[836]: accepted connection from pid 1428, user nix-grpc-daemon (trusted)5338worker2 # [ 112.648738] nix-daemon[836]: reaped child process 1431, status = succeeded5339lb # [ 112.646893] niks3-server[1098]: 2026/09/22 11:26:51 INFO Received uploads request method=POST path=/api/pending_closures5340worker2 # [ 112.672339] nix-grpc-daemon[1291]: time=2026-09-22T11:26:51.284Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5341worker2 # [ 112.674321] nix-grpc-daemon[1291]: time=2026-09-22T11:26:51.284Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)"5342lb # [ 112.691110] niks3-server[1098]: 2026/09/22 11:26:51 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst5343lb # [ 112.708837] niks3-server[1098]: 2026/09/22 11:26:51 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5344worker2 # [ 112.722105] nix-grpc-daemon[1291]: time=2026-09-22T11:26:51.336Z level=INFO msg="Uploading 1 narinfos"5345lb # [ 112.720705] niks3-server[1098]: 2026/09/22 11:26:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5346lb # [ 112.732146] niks3-server[1098]: 2026/09/22 11:26:51 INFO Signed narinfos id=36 count=15347lb # [ 112.736704] niks3-server[1098]: 2026/09/22 11:26:51 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls5348lb # [ 112.747132] niks3-server[1098]: 2026/09/22 11:26:51 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5349lb # [ 112.756297] niks3-server[1098]: 2026/09/22 11:26:51 INFO Completed upload id=365350lb # [ 112.759000] niks3-server[1098]: 2026/09/22 11:26:51 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo5351worker2 # [ 112.756436] nix-grpc-daemon[1291]: time=2026-09-22T11:26:51.371Z level=INFO msg="Upload complete. (286ms)"5352worker2 # [ 112.773648] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:58174 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=15353worker2 # [ 112.790032] nix-daemon[836]: reaped child process 1410, status = succeeded5354client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2, in 6.16 seconds)5355client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath5356client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath, in 0.17 seconds)5357(finished: subtest: a niks3 restart does not cost the leader its role, in 6.86 seconds)5358subtest: scheduler down: the other node takes the lock and keeps it5359lb: must succeed: curl -sf localhost:9901/clusters5360lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5361lb: must succeed: curl -sf localhost:9901/clusters5362lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5363lb: must succeed: curl -sf localhost:9901/clusters5364lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5365lb: must succeed: curl -sf localhost:9901/clusters5366lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5367worker2 # [ 113.237635] systemd[1]: Stopping Nix worker-protocol over gRPC...5368worker2 # [ 113.426295] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:52Z level=info event=signal_exit5369lb # [ 114.273800] niks3-server[1098]: 2026/09/22 11:26:52 INFO lead: released remote=192.168.1.4:371245370worker1 # [ 115.332596] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:53Z level=info event=scheduler_restarting addr=lb:500515371worker2 # [ 115.275125] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:53Z level=info event=scheduler_restarting5372worker2 # [ 115.280231] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:53Z level=info event=scheduler_restarting addr=lb:500515373worker2 # [ 115.284668] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:53Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:581785374worker2 # [ 115.289697] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:53Z level=info event=worker_gone addr=192.168.1.4:500515375worker2 # [ 115.294250] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:53Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:581785376worker2 # [ 115.299465] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:53Z level=info event=worker_gone addr=192.168.1.3:500515377worker2 # [ 115.323161] nix-daemon[836]: reaped child process 1283, status = succeeded5378worker1 # [ 115.532315] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:54Z level=info event=scheduler_connected addr=lb:500515379worker1 # [ 115.537829] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5380worker2 # [ 115.490776] nix-grpc-daemon[1231]: ts=2026-09-22T11:26:54Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5381worker2 # [ 115.497512] nix-grpc-daemon[1231]: E0000 00:00:1790076414.111632 1276 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 19, started true, shutdown true5382worker2 # [ 115.504166] nix-grpc-daemon[1231]: E0000 00:00:1790076414.111792 1276 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5383worker1 # [ 115.738617] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:54Z level=info event=scheduler_connected addr=lb:500515384worker1 # [ 115.744166] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5385worker1 # [ 115.945172] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:54Z level=info event=scheduler_connected addr=lb:500515386worker1 # [ 115.950676] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5387worker1 # [ 116.151520] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:54Z level=info event=scheduler_connected addr=lb:500515388worker1 # [ 116.157092] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5389worker1 # [ 116.359783] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:54Z level=info event=scheduler_connected addr=lb:500515390worker1 # [ 116.364139] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5391worker1 # [ 116.560851] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:55Z level=info event=scheduler_connected addr=lb:500515392worker1 # [ 116.566211] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5393worker2 # [ 116.552908] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5394worker2 # [ 116.559727] systemd[1]: Stopped Nix worker-protocol over gRPC.5395worker2 # [ 116.564643] systemd[1]: nix-grpc-daemon.service: Consumed 1.732s CPU time over 17.303s wall clock time, 38.3M memory peak, 48.8K incoming IP traffic, 91.4K outgoing IP traffic.5396worker2 # [ 116.578452] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5397worker2 # [ 116.586328] systemd[1]: Closed Nix worker-protocol over gRPC.5398worker2 # [ 116.587979] systemd[1]: nix-grpc-daemon.socket: Consumed 738us CPU time over 17.569s wall clock time, 256K memory peak, 39.2K incoming IP traffic, 29.4K outgoing IP traffic.5399worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5400worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.08 seconds)5401worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5402worker1 # [ 116.766634] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:55Z level=info event=scheduler_connected addr=lb:500515403worker1 # [ 116.772762] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5404worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5405worker1 # [ 116.973850] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:55Z level=info event=scheduler_connected addr=lb:500515406worker1 # [ 116.979156] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5407worker1 # [ 117.179943] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:55Z level=info event=scheduler_connected addr=lb:500515408worker1 # [ 117.185089] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5409worker1 # [ 117.318202] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:55Z level=info event=scheduler_take_over5410lb # [ 117.256390] niks3-server[1098]: 2026/09/22 11:26:55 INFO lead: acquired remote=192.168.1.3:598905411worker1 # [ 117.387628] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:55Z level=info event=scheduler_connected addr=lb:500515412worker1 # [ 117.391997] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5413worker1 # [ 117.588670] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:56Z level=info event=scheduler_connected addr=lb:500515414worker1 # [ 117.593998] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5415worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5416worker1 # [ 117.794970] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:56Z level=info event=scheduler_connected addr=lb:500515417worker1 # [ 117.808859] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:56Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:363705418worker1 # [ 117.819370] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:56Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05419worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)5420client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&25421worker1 # [ 118.101676] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:56Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:363845422worker1 # [ 118.117064] nix-daemon[847]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5423client # don't know how to build these paths:5424client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5425worker1 # [ 118.141639] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:363845426worker1 # [ 118.158266] nix-grpc-daemon[1217]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 3 ms (attempt 1/5)5427worker1 # [ 118.170430] nix-grpc-daemon[1217]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 200 ms (attempt 2/5)5428worker1 # [ 118.379899] nix-grpc-daemon[1217]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 115 ms (attempt 3/5)5429worker1 # [ 118.497919] nix-grpc-daemon[1217]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 440 ms (attempt 4/5)5430lb # [ 118.560491] envoy[908]: conn peer=[2001:db8:1::1]:47418 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9766 tx=4079 ms=59045431lb # [ 118.560824] envoy[908]: conn peer=[2001:db8:1::4]:37980 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=13149 tx=995 ms=160035432worker1 # [ 118.941916] nix-grpc-daemon[1217]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5433worker1 # [ 118.947188] nix-grpc-daemon[1217]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5434worker1 # [ 118.979745] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:57Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=05435worker1 # [ 118.986882] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:57Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.3:50051 clients=15436worker1 # [ 119.027614] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:363845437worker1 # [ 119.043295] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:363845438client # copying 1 paths to 'lb:50051'...5439client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'...5440worker1 # [ 119.058019] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:363845441worker1 # [ 119.072742] nix-daemon[847]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5442worker1 # [ 119.101482] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:57Z level=info event=niks3_push_started pid=12935443worker1 # [ 119.250986] nix-grpc-daemon[1293]: time=2026-09-22T11:26:57.861Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""5444worker1 # [ 119.361968] nix-daemon[847]: accepted connection from pid 1306, user nix-grpc-daemon (trusted)5445worker1 # [ 119.384731] nix-daemon[847]: reaped child process 1309, status = succeeded5446worker1 # [ 119.484392] nix-daemon[847]: accepted connection from pid 1311, user nix-grpc-daemon (trusted)5447worker1 # [ 119.501190] nix-daemon[847]: reaped child process 1314, status = succeeded5448lb # [ 119.447064] niks3-server[1098]: 2026/09/22 11:26:58 INFO Received uploads request method=POST path=/api/pending_closures5449worker1 # [ 119.520080] nix-grpc-daemon[1293]: time=2026-09-22T11:26:58.130Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5450worker1 # [ 119.521573] nix-grpc-daemon[1293]: time=2026-09-22T11:26:58.130Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)"5451lb # [ 119.482334] niks3-server[1098]: 2026/09/22 11:26:58 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst5452worker1 # [ 119.565595] nix-grpc-daemon[1293]: time=2026-09-22T11:26:58.176Z level=INFO msg="Uploading 1 narinfos"5453lb # [ 119.495010] niks3-server[1098]: 2026/09/22 11:26:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5454lb # [ 119.509819] niks3-server[1098]: 2026/09/22 11:26:58 INFO Signed narinfos id=37 count=15455lb # [ 119.535403] niks3-server[1098]: 2026/09/22 11:26:58 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5456worker1 # [ 119.606667] nix-grpc-daemon[1293]: time=2026-09-22T11:26:58.217Z level=INFO msg="Upload complete. (351ms)"5457lb # [ 119.549483] niks3-server[1098]: 2026/09/22 11:26:58 INFO Completed upload id=375458worker1 # [ 119.616986] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36384 duration_s=0 paths=1 nar_bytes_in=5205459worker1 # [ 119.634283] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:363845460worker1 # [ 119.643462] nix-daemon[847]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5461worker1 # [ 119.646145] nix-daemon[847]: reaped child process 1291, status = succeeded5462client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051...5463client # farm-a> node-a: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5464worker1 # [ 119.658107] nix-daemon[847]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5465client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5466client # farm-a> LOG-farm-a-on-standby5467worker1 # [ 120.518691] nix-daemon[847]: reaped child process 1318, status = succeeded5468worker1 # [ 120.656220] nix-daemon[847]: accepted connection from pid 1334, user nix-grpc-daemon (trusted)5469worker1 # [ 120.676579] nix-daemon[847]: reaped child process 1337, status = succeeded5470lb # [ 120.617782] niks3-server[1098]: 2026/09/22 11:26:59 INFO Received uploads request method=POST path=/api/pending_closures5471worker1 # [ 120.700206] nix-grpc-daemon[1293]: time=2026-09-22T11:26:59.307Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5472worker1 # [ 120.703934] nix-grpc-daemon[1293]: time=2026-09-22T11:26:59.307Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)"5473lb # [ 120.674243] niks3-server[1098]: 2026/09/22 11:26:59 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5474lb # [ 120.686691] niks3-server[1098]: 2026/09/22 11:26:59 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5475lb # [ 120.696753] niks3-server[1098]: 2026/09/22 11:26:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign5476worker1 # [ 120.759358] nix-grpc-daemon[1293]: time=2026-09-22T11:26:59.368Z level=INFO msg="Uploading 1 narinfos"5477lb # [ 120.706089] niks3-server[1098]: 2026/09/22 11:26:59 INFO Signed narinfos id=38 count=15478lb # [ 120.717715] niks3-server[1098]: 2026/09/22 11:26:59 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls5479lb # [ 120.727358] niks3-server[1098]: 2026/09/22 11:26:59 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete5480lb # [ 120.734830] niks3-server[1098]: 2026/09/22 11:26:59 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo5481lb # [ 120.739461] niks3-server[1098]: 2026/09/22 11:26:59 INFO Completed upload id=385482worker1 # [ 120.802038] nix-grpc-daemon[1293]: time=2026-09-22T11:26:59.413Z level=INFO msg="Upload complete. (331ms)"5483worker1 # [ 120.823884] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36384 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=15484worker1 # [ 120.840415] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:59Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=05485worker1 # [ 120.847923] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:59Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.3:50051 clients=15486worker1 # [ 120.853695] nix-daemon[847]: reaped child process 1316, status = succeeded5487worker1 # [ 120.866623] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:363845488worker1 # [ 120.880827] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:363845489worker1 # [ 120.893485] nix-grpc-daemon[1293]: time=2026-09-22T11:26:59.504Z level=INFO msg="All 1 paths already cached"5490client # copying 1 paths to 'lb:50051'...5491client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'...5492worker1 # [ 120.902677] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:363845493worker1 # [ 120.915996] nix-daemon[847]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5494worker1 # [ 121.062371] nix-daemon[847]: accepted connection from pid 1343, user nix-grpc-daemon (trusted)5495worker1 # [ 121.085456] nix-daemon[847]: reaped child process 1346, status = succeeded5496worker1 # [ 121.183738] nix-daemon[847]: accepted connection from pid 1348, user nix-grpc-daemon (trusted)5497worker1 # [ 121.200775] nix-daemon[847]: reaped child process 1351, status = succeeded5498lb # [ 121.143735] niks3-server[1098]: 2026/09/22 11:26:59 INFO Received uploads request method=POST path=/api/pending_closures5499worker1 # [ 121.214198] nix-grpc-daemon[1293]: time=2026-09-22T11:26:59.824Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5500worker1 # [ 121.215685] nix-grpc-daemon[1293]: time=2026-09-22T11:26:59.824Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)"5501lb # [ 121.164132] niks3-server[1098]: 2026/09/22 11:26:59 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst5502worker1 # [ 121.239618] nix-grpc-daemon[1293]: time=2026-09-22T11:26:59.850Z level=INFO msg="Uploading 1 narinfos"5503lb # [ 121.171977] niks3-server[1098]: 2026/09/22 11:26:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign5504lb # [ 121.181993] niks3-server[1098]: 2026/09/22 11:26:59 INFO Signed narinfos id=39 count=15505lb # [ 121.186223] niks3-server[1098]: 2026/09/22 11:26:59 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls5506lb # [ 121.196629] niks3-server[1098]: 2026/09/22 11:26:59 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete5507lb # [ 121.203648] niks3-server[1098]: 2026/09/22 11:26:59 INFO Completed upload id=395508worker1 # [ 121.266384] nix-grpc-daemon[1293]: time=2026-09-22T11:26:59.877Z level=INFO msg="Upload complete. (335ms)"5509lb # [ 121.206294] niks3-server[1098]: 2026/09/22 11:26:59 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo5510worker1 # [ 121.272235] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36384 duration_s=0 paths=1 nar_bytes_in=6405511worker1 # [ 121.283853] nix-grpc-daemon[1217]: ts=2026-09-22T11:26:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:363845512worker1 # [ 121.289653] nix-daemon[847]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5513worker1 # [ 121.294168] nix-daemon[847]: reaped child process 1341, status = succeeded5514client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051...5515worker1 # [ 121.306373] nix-daemon[847]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5516client # farm-b> node-a: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5517client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5518client # farm-b> LOG-farm-b-on-standby5519worker1 # [ 122.166969] nix-daemon[847]: reaped child process 1355, status = succeeded5520worker1 # [ 122.286211] nix-daemon[847]: accepted connection from pid 1371, user nix-grpc-daemon (trusted)5521worker1 # [ 122.307531] nix-daemon[847]: reaped child process 1374, status = succeeded5522lb # [ 122.249704] niks3-server[1098]: 2026/09/22 11:27:00 INFO Received uploads request method=POST path=/api/pending_closures5523worker1 # [ 122.323492] nix-grpc-daemon[1293]: time=2026-09-22T11:27:00.933Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5524worker1 # [ 122.325261] nix-grpc-daemon[1293]: time=2026-09-22T11:27:00.933Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)"5525lb # [ 122.279833] niks3-server[1098]: 2026/09/22 11:27:00 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5526lb # [ 122.297264] niks3-server[1098]: 2026/09/22 11:27:00 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst5527lb # [ 122.311003] niks3-server[1098]: 2026/09/22 11:27:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign5528worker1 # [ 122.379467] nix-grpc-daemon[1293]: time=2026-09-22T11:27:00.990Z level=INFO msg="Uploading 1 narinfos"5529lb # [ 122.320981] niks3-server[1098]: 2026/09/22 11:27:00 INFO Signed narinfos id=40 count=15530lb # [ 122.323434] niks3-server[1098]: 2026/09/22 11:27:00 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls5531lb # [ 122.338085] niks3-server[1098]: 2026/09/22 11:27:00 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete5532lb # [ 122.345137] niks3-server[1098]: 2026/09/22 11:27:00 INFO Completed upload id=405533worker1 # [ 122.407086] nix-grpc-daemon[1293]: time=2026-09-22T11:27:01.018Z level=INFO msg="Upload complete. (290ms)"5534lb # [ 122.348402] niks3-server[1098]: 2026/09/22 11:27:00 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo5535worker1 # [ 122.425445] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36384 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=15536worker1 # [ 122.442831] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:01Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=05537worker1 # [ 122.447220] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:01Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.3:50051 clients=15538worker1 # [ 122.452132] nix-daemon[847]: reaped child process 1353, status = succeeded5539worker1 # [ 122.471237] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:363845540worker1 # [ 122.487059] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:363845541client # copying 1 paths to 'lb:50051'...5542worker1 # [ 122.501761] nix-grpc-daemon[1293]: time=2026-09-22T11:27:01.112Z level=INFO msg="All 2 paths already cached"5543client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'...5544worker1 # [ 122.508104] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:363845545worker1 # [ 122.521398] nix-daemon[847]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5546worker1 # [ 122.677166] nix-daemon[847]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)5547worker1 # [ 122.697063] nix-daemon[847]: reaped child process 1383, status = succeeded5548worker1 # [ 122.797769] nix-daemon[847]: accepted connection from pid 1385, user nix-grpc-daemon (trusted)5549worker1 # [ 122.815813] nix-daemon[847]: reaped child process 1388, status = succeeded5550lb # [ 122.758424] niks3-server[1098]: 2026/09/22 11:27:01 INFO Received uploads request method=POST path=/api/pending_closures5551worker1 # [ 122.830377] nix-grpc-daemon[1293]: time=2026-09-22T11:27:01.441Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5552worker1 # [ 122.831892] nix-grpc-daemon[1293]: time=2026-09-22T11:27:01.441Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)"5553lb # [ 122.785668] niks3-server[1098]: 2026/09/22 11:27:01 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst5554worker1 # [ 122.861788] nix-grpc-daemon[1293]: time=2026-09-22T11:27:01.472Z level=INFO msg="Uploading 1 narinfos"5555lb # [ 122.801282] niks3-server[1098]: 2026/09/22 11:27:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign5556lb # [ 122.813233] niks3-server[1098]: 2026/09/22 11:27:01 INFO Signed narinfos id=41 count=15557lb # [ 122.815971] niks3-server[1098]: 2026/09/22 11:27:01 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls5558worker1 # [ 122.892493] nix-grpc-daemon[1293]: time=2026-09-22T11:27:01.503Z level=INFO msg="Upload complete. (355ms)"5559lb # [ 122.830545] niks3-server[1098]: 2026/09/22 11:27:01 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete5560lb # [ 122.838534] niks3-server[1098]: 2026/09/22 11:27:01 INFO Completed upload id=415561lb # [ 122.841241] niks3-server[1098]: 2026/09/22 11:27:01 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo5562worker1 # [ 122.900124] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36384 duration_s=0 paths=1 nar_bytes_in=7685563worker1 # [ 122.908738] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:363845564worker1 # [ 122.918498] nix-daemon[847]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5565worker1 # [ 122.922780] nix-daemon[847]: reaped child process 1378, status = succeeded5566client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051...5567worker1 # [ 122.934654] nix-daemon[847]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5568client # farm-top> node-a: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5569client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5570client # farm-top> LOG-farm-top-on-standby5571worker1 # [ 124.216569] nix-daemon[847]: reaped child process 1392, status = succeeded5572worker1 # [ 124.307549] nix-daemon[847]: accepted connection from pid 1408, user nix-grpc-daemon (trusted)5573worker1 # [ 124.330352] nix-daemon[847]: reaped child process 1411, status = succeeded5574lb # [ 124.274103] niks3-server[1098]: 2026/09/22 11:27:02 INFO Received uploads request method=POST path=/api/pending_closures5575worker1 # [ 124.346448] nix-grpc-daemon[1293]: time=2026-09-22T11:27:02.955Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5576worker1 # [ 124.348165] nix-grpc-daemon[1293]: time=2026-09-22T11:27:02.955Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)"5577lb # [ 124.310210] niks3-server[1098]: 2026/09/22 11:27:02 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5578lb # [ 124.323117] niks3-server[1098]: 2026/09/22 11:27:02 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst5579worker1 # [ 124.397679] nix-grpc-daemon[1293]: time=2026-09-22T11:27:03.008Z level=INFO msg="Uploading 1 narinfos"5580lb # [ 124.337979] niks3-server[1098]: 2026/09/22 11:27:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign5581lb # [ 124.348220] niks3-server[1098]: 2026/09/22 11:27:02 INFO Signed narinfos id=42 count=15582lb # [ 124.351340] niks3-server[1098]: 2026/09/22 11:27:02 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls5583lb # [ 124.364109] niks3-server[1098]: 2026/09/22 11:27:02 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete5584lb # [ 124.368533] niks3-server[1098]: 2026/09/22 11:27:02 INFO Completed upload id=425585worker1 # [ 124.430250] nix-grpc-daemon[1293]: time=2026-09-22T11:27:03.041Z level=INFO msg="Upload complete. (285ms)"5586lb # [ 124.370372] niks3-server[1098]: 2026/09/22 11:27:02 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo5587worker1 # [ 124.445284] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36384 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=15588worker1 # [ 124.463374] nix-daemon[847]: reaped child process 1390, status = succeeded5589client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2, in 6.62 seconds)5590client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath5591client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath, in 0.19 seconds)5592worker2 # [ 124.694796] systemd[1]: Listening on Nix worker-protocol over gRPC.5593worker2 # [ 124.702374] systemd[1]: Starting scheduler-token.service...5594worker2 # [ 124.774162] systemd[1]: scheduler-token.service: Deactivated successfully.5595worker2 # [ 124.778541] systemd[1]: Finished scheduler-token.service.5596worker2 # [ 124.783748] systemd[1]: scheduler-token.service: Consumed 41ms CPU time over 74ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5597worker2 # [ 124.789870] systemd[1]: Starting Nix worker-protocol over gRPC...5598worker2 # [ 124.910753] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:03Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5599worker2 # [ 124.913142] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:03Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5600worker2 # [ 124.917083] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:03Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15601worker2 # [ 124.922548] nix-grpc-daemon[1452]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5602worker2 # [ 124.944514] nix-grpc-daemon[1452]: E0000 00:00:1790076423.536553 1452 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5603worker2 # [ 124.946991] nix-grpc-daemon[1452]: E0000 00:00:1790076423.540357 1452 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5604lb: must succeed: curl -sf localhost:9901/clusters5605worker2 # [ 124.972868] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:03Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5606worker2 # [ 124.975075] nix-grpc-daemon[1452]: E0000 00:00:1790076423.548500 1494 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5607worker2 # [ 124.977788] systemd[1]: Started Nix worker-protocol over gRPC.5608worker2 # [ 124.978585] nix-grpc-daemon[1452]: E0000 00:00:1790076423.570760 1494 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5609worker2 # [ 124.980965] nix-grpc-daemon[1452]: E0000 00:00:1790076423.570968 1494 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5610worker2 # [ 124.983271] nix-grpc-daemon[1452]: E0000 00:00:1790076423.586608 1494 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5611worker2 # [ 124.992268] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:03Z level=info event=scheduler_connected addr=lb:500515612worker1 # [ 125.064172] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:03Z level=info event=oidc_jwks provider=mock keys=15613worker1 # [ 125.069635] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:03Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:363705614worker1 # [ 125.075162] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:03Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05615lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5616worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5617worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5618lb: must succeed: curl -sf localhost:9901/clusters5619lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5620lb: must succeed: curl -sf localhost:9901/clusters5621lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5622worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5623worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5624(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 12.30 seconds)5625subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5626client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs5627client # [ 125.508738] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs.5628client # Running as unit: rs.service; invocation ID: 64a8ae1093074b8f9728fc78da4c24195629client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs, in 0.10 seconds)5630worker1 # [ 125.639831] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:04Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:363845631client # [ 125.730631] nix[1361]: don't know how to build these paths:5632client # [ 125.731958] nix[1361]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5633worker1 # [ 125.661572] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:04Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=05634worker1 # [ 125.669217] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:04Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.3:50051 clients=15635worker1 # [ 125.707449] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:363845636worker1 # [ 125.724991] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:363845637client # [ 125.824388] nix[1361]: copying 1 paths to 'lb:50051'...5638worker1 # [ 125.741273] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:363845639client # [ 125.826845] nix[1361]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'...5640worker1 # [ 125.757181] nix-daemon[847]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5641worker1 # [ 125.920531] nix-daemon[847]: accepted connection from pid 1424, user nix-grpc-daemon (trusted)5642worker1 # [ 125.949880] nix-daemon[847]: reaped child process 1427, status = succeeded5643worker1 # [ 126.056156] nix-daemon[847]: accepted connection from pid 1429, user nix-grpc-daemon (trusted)5644worker1 # [ 126.078667] nix-daemon[847]: reaped child process 1432, status = succeeded5645lb # [ 126.015063] niks3-server[1098]: 2026/09/22 11:27:04 INFO Received uploads request method=POST path=/api/pending_closures5646worker1 # [ 126.088422] nix-grpc-daemon[1293]: time=2026-09-22T11:27:04.698Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5647worker1 # [ 126.089893] nix-grpc-daemon[1293]: time=2026-09-22T11:27:04.698Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)"5648lb # [ 126.047920] niks3-server[1098]: 2026/09/22 11:27:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign5649worker1 # [ 126.118928] nix-grpc-daemon[1293]: time=2026-09-22T11:27:04.729Z level=INFO msg="Uploading 1 narinfos"5650lb # [ 126.062682] niks3-server[1098]: 2026/09/22 11:27:04 INFO Signed narinfos id=43 count=15651lb # [ 126.072292] niks3-server[1098]: 2026/09/22 11:27:04 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst5652lb # [ 126.082876] niks3-server[1098]: 2026/09/22 11:27:04 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls5653worker1 # [ 126.158767] nix-grpc-daemon[1293]: time=2026-09-22T11:27:04.769Z level=INFO msg="Upload complete. (383ms)"5654lb # [ 126.097722] niks3-server[1098]: 2026/09/22 11:27:04 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete5655worker1 # [ 126.168276] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36384 duration_s=0 paths=1 nar_bytes_in=4165656lb # [ 126.109769] niks3-server[1098]: 2026/09/22 11:27:04 INFO Completed upload id=435657lb # [ 126.113588] niks3-server[1098]: 2026/09/22 11:27:04 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo5658worker1 # [ 126.183597] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:363845659worker1 # [ 126.189993] nix-daemon[847]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5660worker1 # [ 126.195035] nix-daemon[847]: reaped child process 1422, status = succeeded5661client # [ 126.287944] nix[1361]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051...5662worker1 # [ 126.208196] nix-daemon[847]: accepted connection from pid 1217, user nix-grpc-daemon (trusted)5663lb: must succeed: curl -sf localhost:9901/clusters5664lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5665lb: must succeed: curl -sf localhost:9901/clusters5666lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5667lb: must succeed: curl -sf localhost:9901/clusters5668lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5669lb: must succeed: curl -sf localhost:9901/clusters5670lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5671worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5672worker2 # [ 128.234214] systemd-journald[269]: Received client request to rotate journal, rotating.5673worker2 # [ 128.255106] systemd-journald[269]: Vacuuming done, freed 0B of archived journals from /var/log/journal/96eee89c02c34bcca998924ebf14576d.5674worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.09 seconds)5675worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service5676worker1 # [ 128.364180] systemd[1]: Stopping Nix worker-protocol over gRPC...5677worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds)5678client: must succeed: sleep 5.05679lb # [ 128.558946] envoy[908]: conn peer=[2001:db8:1::1]:60720 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9649 tx=4102 ms=63565680worker1 # [ 129.010344] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:07Z level=info event=draining5681client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5682worker1 # [ 133.637463] nix-daemon[847]: reaped child process 1436, status = succeeded5683worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5684worker1 # [ 133.736828] nix-daemon[847]: accepted connection from pid 1477, user nix-grpc-daemon (trusted)5685worker1 # [ 133.760652] nix-daemon[847]: reaped child process 1480, status = succeeded5686lb # [ 133.697496] niks3-server[1098]: 2026/09/22 11:27:12 INFO Received uploads request method=POST path=/api/pending_closures5687worker1 # [ 133.772399] nix-grpc-daemon[1293]: time=2026-09-22T11:27:12.380Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5688worker1 # [ 133.773865] nix-grpc-daemon[1293]: time=2026-09-22T11:27:12.380Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)"5689lb # [ 133.738822] niks3-server[1098]: 2026/09/22 11:27:12 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5690worker1 # [ 133.824591] nix-grpc-daemon[1293]: time=2026-09-22T11:27:12.435Z level=INFO msg="Uploading 1 narinfos"5691lb # [ 133.751529] niks3-server[1098]: 2026/09/22 11:27:12 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5692lb # [ 133.765861] niks3-server[1098]: 2026/09/22 11:27:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign5693lb # [ 133.772338] niks3-server[1098]: 2026/09/22 11:27:12 INFO Signed narinfos id=44 count=15694lb # [ 133.774278] niks3-server[1098]: 2026/09/22 11:27:12 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls5695lb # [ 133.784608] niks3-server[1098]: 2026/09/22 11:27:12 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete5696lb # [ 133.789894] niks3-server[1098]: 2026/09/22 11:27:12 INFO Completed upload id=445697worker1 # [ 133.852616] nix-grpc-daemon[1293]: time=2026-09-22T11:27:12.463Z level=INFO msg="Upload complete. (229ms)"5698lb # [ 133.792628] niks3-server[1098]: 2026/09/22 11:27:12 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo5699worker1 # [ 133.864851] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36384 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=15700worker1 # [ 133.879425] nix-daemon[847]: reaped child process 1434, status = succeeded5701client # [ 133.978709] systemd[1]: rs.service: Deactivated successfully.5702client # [ 133.983956] systemd[1]: rs.service: Consumed 199ms CPU time over 8.471s wall clock time, 39.2M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.5703worker1 # [ 134.011905] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:12Z level=info event=signal_exit5704lb # [ 134.260232] niks3-server[1098]: 2026/09/22 11:27:12 INFO lead: released remote=192.168.1.3:598905705lb # [ 134.933713] niks3-server[1098]: 2026/09/22 11:27:13 INFO lead: acquired remote=192.168.1.4:345145706worker2 # [ 134.935452] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:13Z level=info event=scheduler_take_over5707worker2 # [ 135.261920] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:13Z level=info event=scheduler_restarting addr=lb:500515708worker1 # [ 135.328421] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:13Z level=info event=scheduler_restarting5709worker1 # [ 135.328834] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:13Z level=info event=scheduler_restarting addr=lb:500515710worker1 # [ 135.328975] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:13Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:363705711worker1 # [ 135.329088] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:13Z level=info event=worker_gone addr=192.168.1.3:500515712worker1 # [ 135.329198] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:13Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:363705713worker1 # [ 135.329305] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:13Z level=info event=worker_gone addr=192.168.1.4:500515714worker1 # [ 135.377597] nix-daemon[847]: reaped child process 1283, status = succeeded5715worker2 # [ 135.463534] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:14Z level=info event=scheduler_connected addr=lb:500515716worker2 # [ 135.469223] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5717worker1 # [ 135.545977] nix-grpc-daemon[1217]: ts=2026-09-22T11:27:14Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5718worker1 # [ 135.648159] nix-grpc-daemon[1217]: E0000 00:00:1790076434.258771 1262 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5719worker1 # [ 135.654744] nix-grpc-daemon[1217]: E0000 00:00:1790076434.259086 1262 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5720worker2 # [ 135.670199] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:14Z level=info event=scheduler_connected addr=lb:500515721worker2 # [ 135.687564] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:14Z level=info event=oidc_jwks provider=mock keys=15722worker2 # [ 135.692250] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:14Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:384645723worker2 # [ 135.697586] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:14Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05724worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.17 seconds)5725client: waiting for success: ! systemctl is-active rs5726client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds)5727client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5728client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds)5729client: must fail: journalctl -u rs | grep -q 'reconnecting'5730client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5731lb: must succeed: curl -sf localhost:9901/clusters5732lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5733worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5734worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5735lb: must succeed: curl -sf localhost:9901/clusters5736lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5737worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5738worker1 # [ 137.214690] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5739worker1 # [ 137.219703] systemd[1]: Stopped Nix worker-protocol over gRPC.5740worker1 # [ 137.224962] systemd[1]: nix-grpc-daemon.service: Consumed 1.986s CPU time over 32.392s wall clock time, 39.1M memory peak, 48.1K incoming IP traffic, 89K outgoing IP traffic.5741worker1 # [ 137.245702] systemd[1]: Starting Nix worker-protocol over gRPC...5742worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5743worker1 # [ 137.398453] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:16Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5744worker1 # [ 137.400757] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:16Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5745worker1 # [ 137.404550] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:16Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15746worker1 # [ 137.409798] nix-grpc-daemon[1487]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5747worker1 # [ 137.411259] nix-grpc-daemon[1487]: E0000 00:00:1790076436.020092 1487 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5748worker1 # [ 137.451770] nix-grpc-daemon[1487]: E0000 00:00:1790076436.026163 1487 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5749worker1 # [ 137.455286] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:16Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5750worker1 # [ 137.457550] nix-grpc-daemon[1487]: E0000 00:00:1790076436.032094 1529 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5751worker1 # [ 137.460113] nix-grpc-daemon[1487]: E0000 00:00:1790076436.047885 1529 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5752worker1 # [ 137.465473] nix-grpc-daemon[1487]: E0000 00:00:1790076436.048049 1529 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5753worker1 # [ 137.469806] nix-grpc-daemon[1487]: E0000 00:00:1790076436.062586 1529 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5754worker1 # [ 137.472304] systemd[1]: Started Nix worker-protocol over gRPC.5755worker1 # [ 137.473146] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:16Z level=info event=scheduler_connected addr=lb:500515756worker2 # [ 137.411342] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:16Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:384645757worker2 # [ 137.416675] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:16Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05758lb: must succeed: curl -sf localhost:9901/clusters5759lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5760worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5761worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5762(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 13.03 seconds)5763subtest: scheduler restart mid-build: build finishes, no double build5764client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid5765client # [ 138.528810] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid.5766client # Running as unit: mid.service; invocation ID: 5a19b1e193fa430abc3037b4762105435767client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid, in 0.09 seconds)5768lb # [ 138.559162] envoy[908]: conn peer=[2001:db8:1::1]:57544 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=1428 ms=82405769worker2 # [ 138.595814] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:17Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:384705770worker2 # [ 138.611883] nix-daemon[836]: accepted connection from pid 1452, user nix-grpc-daemon (trusted)5771client # [ 138.774599] nix[1397]: don't know how to build these paths:5772client # [ 138.776357] nix[1397]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5773worker2 # [ 138.644157] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:17Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05774worker2 # [ 138.650113] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:17Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=15775worker2 # [ 138.699154] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384705776worker2 # [ 138.715446] nix-grpc-daemon[1452]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 12 ms (attempt 1/5)5777worker2 # [ 138.737103] nix-grpc-daemon[1452]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 188 ms (attempt 2/5)5778worker2 # [ 138.927314] nix-grpc-daemon[1452]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 212 ms (attempt 3/5)5779worker2 # [ 139.141367] nix-grpc-daemon[1452]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 89 ms (attempt 4/5)5780lb # [ 138.564453] envoy[908]: conn peer=[2001:db8:1::3]:36150 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2953 tx=971 ms=304795781worker2 # [ 139.232803] nix-grpc-daemon[1452]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5782worker2 # [ 139.239650] nix-grpc-daemon[1452]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5783worker2 # [ 139.257965] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:384705784client # [ 139.416257] nix[1397]: copying 1 paths to 'lb:50051'...5785client # [ 139.421865] nix[1397]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'...5786worker2 # [ 139.278023] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:384705787worker2 # [ 139.292380] nix-daemon[836]: accepted connection from pid 1452, user nix-grpc-daemon (trusted)5788worker2 # [ 139.324806] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:17Z level=info event=niks3_push_started pid=15775789worker2 # [ 139.471112] nix-grpc-daemon[1577]: time=2026-09-22T11:27:18.085Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""5790worker2 # [ 139.580205] nix-daemon[836]: accepted connection from pid 1589, user nix-grpc-daemon (trusted)5791worker2 # [ 139.598027] nix-daemon[836]: reaped child process 1592, status = succeeded5792worker2 # [ 139.731067] nix-daemon[836]: accepted connection from pid 1594, user nix-grpc-daemon (trusted)5793worker2 # [ 139.754933] nix-daemon[836]: reaped child process 1601, status = succeeded5794lb # [ 139.751797] niks3-server[1098]: 2026/09/22 11:27:18 INFO Received uploads request method=POST path=/api/pending_closures5795worker2 # [ 139.763143] nix-grpc-daemon[1577]: time=2026-09-22T11:27:18.377Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5796worker2 # [ 139.766434] nix-grpc-daemon[1577]: time=2026-09-22T11:27:18.379Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)"5797lb # [ 139.786609] niks3-server[1098]: 2026/09/22 11:27:18 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst5798worker2 # [ 139.801000] nix-grpc-daemon[1577]: time=2026-09-22T11:27:18.415Z level=INFO msg="Uploading 1 narinfos"5799lb # [ 139.800677] niks3-server[1098]: 2026/09/22 11:27:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign5800lb # [ 139.810922] niks3-server[1098]: 2026/09/22 11:27:18 INFO Signed narinfos id=45 count=15801lb # [ 139.817289] niks3-server[1098]: 2026/09/22 11:27:18 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls5802lb # [ 139.829797] niks3-server[1098]: 2026/09/22 11:27:18 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete5803lb # [ 139.834444] niks3-server[1098]: 2026/09/22 11:27:18 INFO Completed upload id=455804worker2 # [ 139.830875] nix-grpc-daemon[1577]: time=2026-09-22T11:27:18.445Z level=INFO msg="Upload complete. (356ms)"5805lb # [ 139.839447] niks3-server[1098]: 2026/09/22 11:27:18 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo5806worker2 # [ 139.841337] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38470 duration_s=0 paths=1 nar_bytes_in=4245807worker2 # [ 139.850282] nix-grpc-daemon[1452]: ts=2026-09-22T11:27:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384705808worker2 # [ 139.858824] nix-daemon[836]: accepted connection from pid 1452, user nix-grpc-daemon (trusted)5809worker2 # [ 139.860910] nix-daemon[836]: reaped child process 1575, status = succeeded5810worker2 # [ 139.872629] nix-daemon[836]: accepted connection from pid 1452, user nix-grpc-daemon (trusted)5811client # [ 140.015095] nix[1397]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5812lb: must succeed: curl -sf localhost:9901/clusters5813lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5814lb: must succeed: curl -sf localhost:9901/clusters5815lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5816worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5817worker2 # [ 142.260554] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1452 (nix-grpc-daemon) on client request.5818worker2 # [ 142.262137] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5819lb # [ 142.273148] niks3-server[1098]: 2026/09/22 11:27:20 INFO lead: released remote=192.168.1.4:345145820worker2 # [ 142.280319] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5821worker2 # [ 142.281851] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5822client # [ 142.422507] nix[1397]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again5823worker2 # [ 142.283105] systemd[1]: nix-grpc-daemon.service: Consumed 602ms CPU time over 17.484s wall clock time, 34.6M memory peak, 15.7K incoming IP traffic, 27.7K outgoing IP traffic.5824worker1 # [ 142.339471] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:20Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"5825client # [ 142.428871] nix[1397]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5826worker2 # [ 142.304249] systemd[1]: Starting scheduler-token.service...5827worker2 # [ 142.326783] nix-daemon[836]: reaped child process 1567, status = succeeded5828worker2 # [ 142.333375] nix-daemon[836]: reaped child process 1603, status = succeeded5829worker2 # [ 142.340423] nix-daemon[1605]: unexpected Nix daemon error: error: interrupted by the user5830worker2 # [ 142.348576] nix-daemon[836]: reaped child process 1605, status = failed with exit code 15831lb # [ 142.360600] niks3-server[1098]: 2026/09/22 11:27:20 INFO lead: acquired remote=192.168.1.3:384625832worker1 # [ 142.427378] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:21Z level=info event=scheduler_take_over5833worker2 # [ 142.377179] systemd[1]: nix-grpc-daemon.service: Scheduled restart job, restart counter is at 1.5834worker2 # [ 142.379414] systemd[1]: scheduler-token.service: Deactivated successfully.5835worker2 # [ 142.382902] systemd[1]: Finished scheduler-token.service.5836worker2 # [ 142.383671] systemd[1]: scheduler-token.service: Consumed 23ms CPU time over 82ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5837worker2 # [ 142.386852] systemd[1]: Starting Nix worker-protocol over gRPC...5838worker1 # [ 142.539515] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:21Z level=info event=scheduler_connected addr=lb:500515839worker2 # [ 142.484443] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:21Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5840worker2 # [ 142.486575] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:21Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5841worker2 # [ 142.490160] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:21Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15842worker2 # [ 142.495448] nix-grpc-daemon[1645]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5843worker1 # [ 142.577726] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:21Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5844worker2 # [ 142.520156] nix-grpc-daemon[1645]: E0000 00:00:1790076441.109371 1645 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5845worker2 # [ 142.522552] nix-grpc-daemon[1645]: E0000 00:00:1790076441.112983 1645 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5846worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.32 seconds)5847client: must succeed: sleep 5.05848worker2 # [ 142.547768] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:21Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5849worker2 # [ 142.550003] nix-grpc-daemon[1645]: E0000 00:00:1790076441.122303 1684 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5850worker2 # [ 142.552594] systemd[1]: Started Nix worker-protocol over gRPC.5851worker2 # [ 142.553422] nix-grpc-daemon[1645]: E0000 00:00:1790076441.145987 1684 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5852worker2 # [ 142.558517] nix-grpc-daemon[1645]: E0000 00:00:1790076441.146132 1684 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5853worker2 # [ 142.562939] nix-grpc-daemon[1645]: E0000 00:00:1790076441.161852 1684 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5854worker2 # [ 142.565992] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:21Z level=info event=scheduler_connected addr=lb:500515855worker2 # [ 142.567779] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:21Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5856worker1 # [ 142.778740] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:21Z level=info event=scheduler_connected addr=lb:500515857worker1 # [ 142.783854] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5858worker2 # [ 142.769083] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:21Z level=info event=scheduler_connected addr=lb:500515859worker2 # [ 142.774446] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5860client # [ 142.925621] nix[1397]: scheduler at lb:50051: no healthy upstream, reconnecting5861worker1 # [ 142.984802] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:21Z level=info event=scheduler_connected addr=lb:500515862worker1 # [ 142.990331] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5863worker2 # [ 142.975363] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:21Z level=info event=scheduler_connected addr=lb:500515864worker2 # [ 142.980709] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5865worker1 # [ 143.191164] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:21Z level=info event=scheduler_connected addr=lb:500515866worker1 # [ 143.196946] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5867worker2 # [ 143.181791] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:21Z level=info event=scheduler_connected addr=lb:500515868worker2 # [ 143.186956] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5869worker1 # [ 143.399688] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:22Z level=info event=scheduler_connected addr=lb:500515870worker1 # [ 143.404377] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5871worker2 # [ 143.388072] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:22Z level=info event=scheduler_connected addr=lb:500515872worker2 # [ 143.393139] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5873worker1 # [ 143.600606] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:22Z level=info event=scheduler_connected addr=lb:500515874worker1 # [ 143.606065] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5875worker2 # [ 143.595908] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:22Z level=info event=scheduler_connected addr=lb:500515876worker2 # [ 143.600317] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5877worker1 # [ 143.806827] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:22Z level=info event=scheduler_connected addr=lb:500515878worker1 # [ 143.812404] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5879client # [ 143.929392] nix[1397]: scheduler at lb:50051: no healthy upstream, reconnecting5880worker2 # [ 143.796836] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:22Z level=info event=scheduler_connected addr=lb:500515881worker2 # [ 143.802006] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5882worker1 # [ 144.013283] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:22Z level=info event=scheduler_connected addr=lb:500515883worker1 # [ 144.018729] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5884worker2 # [ 144.003143] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:22Z level=info event=scheduler_connected addr=lb:500515885worker2 # [ 144.008386] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5886worker1 # [ 144.219479] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:22Z level=info event=scheduler_connected addr=lb:500515887worker1 # [ 144.224998] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5888worker2 # [ 144.209286] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:22Z level=info event=scheduler_connected addr=lb:500515889worker1 # [ 144.295434] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:22Z level=info event=oidc_jwks provider=mock keys=15890worker1 # [ 144.299882] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:22Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:460925891worker1 # [ 144.305156] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:22Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05892worker1 # [ 144.428563] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:23Z level=info event=scheduler_connected addr=lb:500515893worker1 # [ 144.432936] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:23Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:460925894worker1 # [ 144.437806] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:23Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05895worker1 # [ 145.857325] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:24Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05896worker1 # [ 145.863362] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:24Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=15897worker2 # [ 145.855863] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:493465898worker2 # [ 145.870835] nix-daemon[836]: accepted connection from pid 1645, user nix-grpc-daemon (trusted)5899worker2 # [ 145.889527] nix-daemon[836]: accepted connection from pid 1645, user nix-grpc-daemon (trusted)5900client # [ 146.052551] nix[1397]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5901worker2 # [ 145.912124] nix-daemon[836]: accepted connection from pid 1645, user nix-grpc-daemon (trusted)5902client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5903client: waiting for success: ! systemctl is-active mid5904worker2 # [ 147.930998] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:26Z level=info event=niks3_push_started pid=17265905worker2 # [ 147.959708] nix-daemon[836]: reaped child process 1701, status = succeeded5906worker2 # [ 148.066509] nix-grpc-daemon[1726]: time=2026-09-22T11:27:26.680Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""5907worker2 # [ 148.172206] nix-daemon[836]: accepted connection from pid 1739, user nix-grpc-daemon (trusted)5908worker2 # [ 148.191502] nix-daemon[836]: reaped child process 1742, status = succeeded5909lb # [ 148.197228] niks3-server[1098]: 2026/09/22 11:27:26 INFO Received uploads request method=POST path=/api/pending_closures5910worker2 # [ 148.209885] nix-grpc-daemon[1726]: time=2026-09-22T11:27:26.822Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5911worker2 # [ 148.211339] nix-grpc-daemon[1726]: time=2026-09-22T11:27:26.822Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)"5912lb # [ 148.239691] niks3-server[1098]: 2026/09/22 11:27:26 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst5913lb # [ 148.252007] niks3-server[1098]: 2026/09/22 11:27:26 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5914lb # [ 148.264551] niks3-server[1098]: 2026/09/22 11:27:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign5915lb # [ 148.281266] niks3-server[1098]: 2026/09/22 11:27:26 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls5916lb # [ 148.289969] niks3-server[1098]: 2026/09/22 11:27:26 INFO Signed narinfos id=46 count=15917worker2 # [ 148.292109] nix-grpc-daemon[1726]: time=2026-09-22T11:27:26.906Z level=INFO msg="Uploading 1 narinfos"5918lb # [ 148.302139] niks3-server[1098]: 2026/09/22 11:27:26 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete5919lb # [ 148.313128] niks3-server[1098]: 2026/09/22 11:27:26 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo5920lb # [ 148.323049] niks3-server[1098]: 2026/09/22 11:27:26 INFO Completed upload id=465921worker2 # [ 148.324729] nix-grpc-daemon[1726]: time=2026-09-22T11:27:26.939Z level=INFO msg="Upload complete. (254ms)"5922worker2 # [ 148.335362] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49346 duration_s=2 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=15923worker2 # [ 148.355662] nix-daemon[836]: reaped child process 1699, status = succeeded5924client # [ 148.514040] systemd[1]: mid.service: Deactivated successfully.5925client # [ 148.518992] systemd[1]: mid.service: Consumed 204ms CPU time over 9.989s wall clock time, 43.4M memory peak, 9.1K incoming IP traffic, 11.3K outgoing IP traffic.5926lb # [ 148.560820] envoy[908]: conn peer=[2001:db8:1::4]:57214 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=2666 tx=376 ms=173005927lb # [ 148.566309] envoy[908]: conn peer=[2001:db8:1::1]:44508 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4288 tx=1948 ms=97695928client: (finished: waiting for success: ! systemctl is-active mid, in 1.12 seconds)5929client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }5930client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.05 seconds)5931(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.69 seconds)5932subtest: interrupting the client stops the build on the worker5933client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr5934client # [ 149.189649] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr.5935client # Running as unit: intr.service; invocation ID: c3af029464d541cf8c6f4a19e27bedb55936client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr, in 0.05 seconds)5937worker2 # [ 149.221301] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:493465938client # [ 149.373289] nix[1433]: don't know how to build these paths:5939client # [ 149.374667] nix[1433]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5940worker1 # [ 149.308475] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:27Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=05941worker1 # [ 149.314268] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:27Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.3:50051 clients=15942worker1 # [ 149.369738] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460945943worker1 # [ 149.391511] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)5944worker1 # [ 149.414542] nix-grpc-daemon[1487]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 47 ms (attempt 1/5)5945worker1 # [ 149.470568] nix-grpc-daemon[1487]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 176 ms (attempt 2/5)5946worker1 # [ 149.648616] nix-grpc-daemon[1487]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 6 ms (attempt 3/5)5947worker1 # [ 149.657870] nix-grpc-daemon[1487]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 175 ms (attempt 4/5)5948worker1 # [ 149.835630] nix-grpc-daemon[1487]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5949worker1 # [ 149.841727] nix-grpc-daemon[1487]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5950worker1 # [ 149.860787] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460945951client # [ 149.960076] nix[1433]: copying 1 paths to 'lb:50051'...5952worker1 # [ 149.877565] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:460945953client # [ 149.962567] nix[1433]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'...5954worker1 # [ 149.892138] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)5955worker1 # [ 149.924860] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:28Z level=info event=niks3_push_started pid=15835956worker1 # [ 150.071896] nix-grpc-daemon[1583]: time=2026-09-22T11:27:28.682Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""5957worker1 # [ 150.185150] nix-daemon[847]: accepted connection from pid 1595, user nix-grpc-daemon (trusted)5958worker1 # [ 150.207749] nix-daemon[847]: reaped child process 1598, status = succeeded5959worker1 # [ 150.315056] nix-daemon[847]: accepted connection from pid 1600, user nix-grpc-daemon (trusted)5960worker1 # [ 150.346666] nix-daemon[847]: reaped child process 1606, status = succeeded5961lb # [ 150.285190] niks3-server[1098]: 2026/09/22 11:27:28 INFO Received uploads request method=POST path=/api/pending_closures5962worker1 # [ 150.362021] nix-grpc-daemon[1583]: time=2026-09-22T11:27:28.971Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5963worker1 # [ 150.363532] nix-grpc-daemon[1583]: time=2026-09-22T11:27:28.971Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)"5964lb # [ 150.329236] niks3-server[1098]: 2026/09/22 11:27:28 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst5965worker1 # [ 150.404604] nix-grpc-daemon[1583]: time=2026-09-22T11:27:29.015Z level=INFO msg="Uploading 1 narinfos"5966lb # [ 150.342923] niks3-server[1098]: 2026/09/22 11:27:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign5967lb # [ 150.355497] niks3-server[1098]: 2026/09/22 11:27:28 INFO Signed narinfos id=47 count=15968lb # [ 150.361654] niks3-server[1098]: 2026/09/22 11:27:28 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls5969lb # [ 150.374350] niks3-server[1098]: 2026/09/22 11:27:28 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete5970worker1 # [ 150.443429] nix-grpc-daemon[1583]: time=2026-09-22T11:27:29.054Z level=INFO msg="Upload complete. (367ms)"5971lb # [ 150.382594] niks3-server[1098]: 2026/09/22 11:27:28 INFO Completed upload id=475972lb # [ 150.385849] niks3-server[1098]: 2026/09/22 11:27:28 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo5973worker1 # [ 150.454250] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=0 paths=1 nar_bytes_in=4245974worker1 # [ 150.465981] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460945975worker1 # [ 150.474820] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)5976worker1 # [ 150.477643] nix-daemon[847]: reaped child process 1581, status = succeeded5977client # [ 150.570774] nix[1433]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051...5978worker1 # [ 150.490322] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)5979client: must succeed: systemctl kill -s INT intr5980client # [ 152.841089] systemd[1]: intr.service: Sent signal SIGINT to main process 1433 (nix) on client request.5981worker1 # [ 152.762705] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:31Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5982client # [ 152.850284] nix[1433]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5983client # [ 152.865352] nix[1433]: error: interrupted by the user5984client: (finished: must succeed: systemctl kill -s INT intr, in 0.08 seconds)5985client # [ 152.895898] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE5986client # [ 152.901524] systemd[1]: intr.service: Failed with result 'exit-code'.5987client # [ 152.903942] systemd[1]: intr.service: Consumed 172ms CPU time over 3.709s wall clock time, 41.4M memory peak, 6.4K incoming IP traffic, 8.3K outgoing IP traffic.5988worker1 # [ 152.918633] nix-daemon[1611]: unexpected Nix daemon error: error: interrupted by the user5989worker1 # [ 152.927166] nix-daemon[847]: reaped child process 1609, status = succeeded5990worker1 # [ 152.928311] nix-daemon[847]: reaped child process 1611, status = failed with exit code 15991(finished: subtest: interrupting the client stops the build on the worker, in 4.98 seconds)5992subtest: a deploy mid-build drains: the build finishes, the new generation takes over5993client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw5994client # [ 154.201247] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw.5995client # Running as unit: sw.service; invocation ID: bbc75801cc824b828fd5538cd4a2c1df5996client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw, in 0.09 seconds)5997worker2 # [ 154.263675] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:32Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:493465998client # [ 154.414950] nix[1454]: don't know how to build these paths:5999client # [ 154.416282] nix[1454]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv6000worker1 # [ 154.345981] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:32Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=06001worker1 # [ 154.351877] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:32Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.4:50051 clients=16002worker2 # [ 154.329897] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:493466003worker2 # [ 154.344690] nix-grpc-daemon[1645]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 56 ms (attempt 1/5)6004worker2 # [ 154.410033] nix-grpc-daemon[1645]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 85 ms (attempt 2/5)6005worker2 # [ 154.496788] nix-grpc-daemon[1645]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 345 ms (attempt 3/5)6006worker2 # [ 154.844113] nix-grpc-daemon[1645]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 130 ms (attempt 4/5)6007worker2 # [ 154.976506] nix-grpc-daemon[1645]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6008worker2 # [ 154.982945] nix-grpc-daemon[1645]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6009worker2 # [ 155.000932] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:493466010client # [ 155.160231] nix[1454]: copying 1 paths to 'lb:50051'...6011worker2 # [ 155.016558] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:493466012client # [ 155.160532] nix[1454]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'...6013worker2 # [ 155.031803] nix-daemon[836]: accepted connection from pid 1645, user nix-grpc-daemon (trusted)6014worker2 # [ 155.193461] nix-daemon[836]: accepted connection from pid 1778, user nix-grpc-daemon (trusted)6015worker2 # [ 155.215737] nix-daemon[836]: reaped child process 1781, status = succeeded6016worker2 # [ 155.317908] nix-daemon[836]: accepted connection from pid 1783, user nix-grpc-daemon (trusted)6017worker2 # [ 155.347543] nix-daemon[836]: reaped child process 1789, status = succeeded6018lb # [ 155.345084] niks3-server[1098]: 2026/09/22 11:27:33 INFO Received uploads request method=POST path=/api/pending_closures6019worker2 # [ 155.356321] nix-grpc-daemon[1726]: time=2026-09-22T11:27:33.970Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6020worker2 # [ 155.357814] nix-grpc-daemon[1726]: time=2026-09-22T11:27:33.970Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)"6021lb # [ 155.378700] niks3-server[1098]: 2026/09/22 11:27:33 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst6022worker2 # [ 155.400989] nix-grpc-daemon[1726]: time=2026-09-22T11:27:34.015Z level=INFO msg="Uploading 1 narinfos"6023lb # [ 155.399285] niks3-server[1098]: 2026/09/22 11:27:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6024lb # [ 155.405822] niks3-server[1098]: 2026/09/22 11:27:34 INFO Signed narinfos id=48 count=16025lb # [ 155.413604] niks3-server[1098]: 2026/09/22 11:27:34 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls6026lb # [ 155.427222] niks3-server[1098]: 2026/09/22 11:27:34 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6027lb # [ 155.437095] niks3-server[1098]: 2026/09/22 11:27:34 INFO Completed upload id=486028worker2 # [ 155.438620] nix-grpc-daemon[1726]: time=2026-09-22T11:27:34.053Z level=INFO msg="Upload complete. (389ms)"6029worker2 # [ 155.447759] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49346 duration_s=0 paths=1 nar_bytes_in=4166030lb # [ 155.444806] niks3-server[1098]: 2026/09/22 11:27:34 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo6031worker2 # [ 155.459604] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:493466032worker2 # [ 155.468218] nix-daemon[836]: accepted connection from pid 1645, user nix-grpc-daemon (trusted)6033worker2 # [ 155.470799] nix-daemon[836]: reaped child process 1776, status = succeeded6034client # [ 155.625128] nix[1454]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051...6035worker2 # [ 155.483318] nix-daemon[836]: accepted connection from pid 1645, user nix-grpc-daemon (trusted)6036worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&26037worker2 # Checking switch inhibitors... done6038worker2 # [ 158.205547] nixos[1826]: switching to system configuration /nix/store/mms4r3ic4vc4scxwax7lpjppqd3q2hgx-nixos-system-worker2-test6039worker2 # [ 158.207378] systemd[1]: Stopped target Remote File Systems.6040worker2 # activating the configuration...6041worker2 # [ 158.213339] systemd[1]: Stopped target Local File Systems.6042worker2 # setting up /etc...6043lb # [ 158.561214] envoy[908]: conn peer=[2001:db8:1::1]:49366 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3000 tx=827 ms=35096044worker2 # [ 158.761548] systemd[1]: Reload requested from client PID 1826 ('.switch-to-conf') (unit backdoor.service)...6045worker2 # [ 158.763104] systemd[1]: Reloading...6046worker2 # [ 163.653673] systemd[1]: Reloading finished in 4884 ms.6047worker2 # restarting sysinit-reactivation.target6048worker2 # [ 163.768728] systemd[1]: Starting Re-setup tmpfiles on a system that is already running....6049worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service[ 163.892457] systemd[1]: Finished Re-setup tmpfiles on a system that is already running..6050worker2 # [ 163.893606] systemd[1]: Reached target Reactivate sysinit units.6051worker2 # 6052worker2 # [ 163.907579] systemd[1]: Reloading D-Bus System Message Bus...6053worker2 # [ 163.910922] dbus-broker-launch[491]: Caught SIGHUP, trigger reload.6054worker2 # [ 163.911890] systemd[1]: Reloading Nix worker-protocol over gRPC...6055worker2 # [ 163.920736] dbus-broker-launch[491]: Looking up NSS user entry for 'systemd-timesync'...6056worker2 # [ 163.922855] dbus-broker-launch[491]: NSS returned no entry for 'systemd-timesync'6057worker2 # [ 163.924257] dbus-broker-launch[491]: Invalid user-name in /nix/store/ng2ydfkv03gvmdws4f4mjf9wj6qj3kq6-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"6058worker2 # [ 163.951913] systemd[1]: Reloaded Nix worker-protocol over gRPC.6059worker2 # [ 163.959029] systemd[1]: Reloaded D-Bus System Message Bus.6060worker2 # [ 163.971306] systemd[1]: Reached target Local File Systems.6061worker2 # [ 163.978376] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore6062worker2 # [ 163.979842] systemd[1]: Starting Create SUID/SGID Wrappers...6063worker2 # [ 163.992228] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met.6064worker2 # [ 163.993490] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc676065worker2 # [ 163.995535] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore6066worker2 # [ 164.003391] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os6067worker2 # [ 164.009873] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os6068worker2 # [ 164.011485] systemd[1]: Reached target Remote File Systems.6069worker2 # [ 164.017046] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os6070worker2 # [ 164.018274] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki6071worker2 # [ 164.307905] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully.6072worker2 # [ 164.309572] systemd[1]: Finished Create SUID/SGID Wrappers.6073worker2 # [ 164.316307] systemd[1]: Starting linger-users.service...6074worker2 # [ 164.359070] systemd[1]: linger-users.service: Deactivated successfully.6075worker2 # [ 164.360191] systemd[1]: Finished linger-users.service.6076worker2 # [ 164.508306] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:43Z level=info event=draining6077worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service6078worker2 # [ 164.612633] nixos[1826]: finished switching to system configuration /nix/store/mms4r3ic4vc4scxwax7lpjppqd3q2hgx-nixos-system-worker2-test6079worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 6.86 seconds)6080worker2: must succeed: pgrep -f 'read -t [9]0 x'6081worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds)6082client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&26083client # don't know how to build these paths:6084client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6085worker1 # [ 164.955999] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:43Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:460946086worker1 # [ 164.963520] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460946087worker1 # [ 164.991289] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:43Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=06088worker1 # [ 164.998377] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:43Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.3:50051 clients=16089worker1 # [ 165.039102] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946090worker1 # [ 165.054425] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460946091client # copying 1 paths to 'lb:50051'...6092client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'...6093worker1 # [ 165.070492] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:460946094worker1 # [ 165.085662] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6095worker1 # [ 165.239170] nix-daemon[847]: accepted connection from pid 1670, user nix-grpc-daemon (trusted)6096worker1 # [ 165.259550] nix-daemon[847]: reaped child process 1673, status = succeeded6097worker1 # [ 165.357574] nix-daemon[847]: accepted connection from pid 1675, user nix-grpc-daemon (trusted)6098worker1 # [ 165.373304] nix-daemon[847]: reaped child process 1678, status = succeeded6099lb # [ 165.317357] niks3-server[1098]: 2026/09/22 11:27:43 INFO Received uploads request method=POST path=/api/pending_closures6100worker1 # [ 165.390937] nix-grpc-daemon[1583]: time=2026-09-22T11:27:44.001Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6101worker1 # [ 165.392607] nix-grpc-daemon[1583]: time=2026-09-22T11:27:44.001Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)"6102lb # [ 165.356513] niks3-server[1098]: 2026/09/22 11:27:43 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst6103worker1 # [ 165.444277] nix-grpc-daemon[1583]: time=2026-09-22T11:27:44.055Z level=INFO msg="Uploading 1 narinfos"6104lb # [ 165.377510] niks3-server[1098]: 2026/09/22 11:27:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6105lb # [ 165.387269] niks3-server[1098]: 2026/09/22 11:27:43 INFO Signed narinfos id=49 count=16106lb # [ 165.392539] niks3-server[1098]: 2026/09/22 11:27:43 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls6107lb # [ 165.404094] niks3-server[1098]: 2026/09/22 11:27:44 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6108lb # [ 165.410291] niks3-server[1098]: 2026/09/22 11:27:44 INFO Completed upload id=496109worker1 # [ 165.473059] nix-grpc-daemon[1583]: time=2026-09-22T11:27:44.084Z level=INFO msg="Upload complete. (370ms)"6110lb # [ 165.413742] niks3-server[1098]: 2026/09/22 11:27:44 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo6111worker1 # [ 165.485373] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=0 paths=1 nar_bytes_in=5286112worker1 # [ 165.500339] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946113worker1 # [ 165.507701] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6114worker1 # [ 165.513486] nix-daemon[847]: reaped child process 1668, status = succeeded6115client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051...6116worker1 # [ 165.528441] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6117client # farm-a> node-a: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv6118client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6119client # farm-a> LOG-farm-a-during-drain6120worker1 # [ 166.773054] nix-daemon[847]: reaped child process 1682, status = succeeded6121worker1 # [ 166.889186] nix-daemon[847]: accepted connection from pid 1698, user nix-grpc-daemon (trusted)6122worker1 # [ 166.911844] nix-daemon[847]: reaped child process 1701, status = succeeded6123lb # [ 166.851577] niks3-server[1098]: 2026/09/22 11:27:45 INFO Received uploads request method=POST path=/api/pending_closures6124worker1 # [ 166.927603] nix-grpc-daemon[1583]: time=2026-09-22T11:27:45.535Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6125worker1 # [ 166.930398] nix-grpc-daemon[1583]: time=2026-09-22T11:27:45.535Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)"6126lb # [ 166.890765] niks3-server[1098]: 2026/09/22 11:27:45 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6127worker1 # [ 166.974725] nix-grpc-daemon[1583]: time=2026-09-22T11:27:45.584Z level=INFO msg="Uploading 1 narinfos"6128lb # [ 166.910872] niks3-server[1098]: 2026/09/22 11:27:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6129lb # [ 166.923458] niks3-server[1098]: 2026/09/22 11:27:45 INFO Signed narinfos id=50 count=16130lb # [ 166.929595] niks3-server[1098]: 2026/09/22 11:27:45 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv6131lb # [ 166.938555] niks3-server[1098]: 2026/09/22 11:27:45 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls6132lb # [ 166.948295] niks3-server[1098]: 2026/09/22 11:27:45 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6133lb # [ 166.955622] niks3-server[1098]: 2026/09/22 11:27:45 INFO Completed upload id=506134worker1 # [ 167.018270] nix-grpc-daemon[1583]: time=2026-09-22T11:27:45.629Z level=INFO msg="Upload complete. (278ms)"6135lb # [ 166.959178] niks3-server[1098]: 2026/09/22 11:27:45 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo6136worker1 # [ 167.037455] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=16137worker1 # [ 167.052327] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:45Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=06138worker1 # [ 167.058202] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:45Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.3:50051 clients=16139worker1 # [ 167.062467] nix-daemon[847]: reaped child process 1680, status = succeeded6140worker1 # [ 167.083302] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946141worker1 # [ 167.098918] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460946142worker1 # [ 167.114428] nix-grpc-daemon[1583]: time=2026-09-22T11:27:45.725Z level=INFO msg="All 1 paths already cached"6143client # copying 1 paths to 'lb:50051'...6144worker1 # [ 167.125009] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:460946145client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'...6146worker1 # [ 167.140501] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6147worker1 # [ 167.299035] nix-daemon[847]: accepted connection from pid 1707, user nix-grpc-daemon (trusted)6148worker1 # [ 167.318027] nix-daemon[847]: reaped child process 1710, status = succeeded6149worker1 # [ 167.423873] nix-daemon[847]: accepted connection from pid 1712, user nix-grpc-daemon (trusted)6150worker1 # [ 167.445065] nix-daemon[847]: reaped child process 1715, status = succeeded6151lb # [ 167.380494] niks3-server[1098]: 2026/09/22 11:27:45 INFO Received uploads request method=POST path=/api/pending_closures6152worker1 # [ 167.452949] nix-grpc-daemon[1583]: time=2026-09-22T11:27:46.063Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6153worker1 # [ 167.454458] nix-grpc-daemon[1583]: time=2026-09-22T11:27:46.063Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)"6154lb # [ 167.410238] niks3-server[1098]: 2026/09/22 11:27:46 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst6155worker1 # [ 167.488483] nix-grpc-daemon[1583]: time=2026-09-22T11:27:46.099Z level=INFO msg="Uploading 1 narinfos"6156lb # [ 167.425584] niks3-server[1098]: 2026/09/22 11:27:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6157lb # [ 167.436005] niks3-server[1098]: 2026/09/22 11:27:46 INFO Signed narinfos id=51 count=16158lb # [ 167.443950] niks3-server[1098]: 2026/09/22 11:27:46 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls6159lb # [ 167.458215] niks3-server[1098]: 2026/09/22 11:27:46 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6160worker1 # [ 167.532682] nix-grpc-daemon[1583]: time=2026-09-22T11:27:46.143Z level=INFO msg="Upload complete. (375ms)"6161lb # [ 167.475263] niks3-server[1098]: 2026/09/22 11:27:46 INFO Completed upload id=516162worker1 # [ 167.540318] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=0 paths=1 nar_bytes_in=6406163lb # [ 167.479691] niks3-server[1098]: 2026/09/22 11:27:46 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo6164worker1 # [ 167.556235] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946165worker1 # [ 167.563845] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6166worker1 # [ 167.569608] nix-daemon[847]: reaped child process 1705, status = succeeded6167client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051...6168client # farm-b> node-a: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv6169worker1 # [ 167.584192] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6170client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6171client # farm-b> LOG-farm-b-during-drain6172worker1 # [ 168.428690] nix-daemon[847]: reaped child process 1719, status = succeeded6173worker1 # [ 168.541213] nix-daemon[847]: accepted connection from pid 1735, user nix-grpc-daemon (trusted)6174worker1 # [ 168.563002] nix-daemon[847]: reaped child process 1738, status = succeeded6175lb # [ 168.499104] niks3-server[1098]: 2026/09/22 11:27:47 INFO Received uploads request method=POST path=/api/pending_closures6176worker1 # [ 168.573554] nix-grpc-daemon[1583]: time=2026-09-22T11:27:47.183Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6177worker1 # [ 168.575063] nix-grpc-daemon[1583]: time=2026-09-22T11:27:47.183Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)"6178lb # [ 168.537839] niks3-server[1098]: 2026/09/22 11:27:47 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv6179lb # [ 168.553772] niks3-server[1098]: 2026/09/22 11:27:47 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst6180worker1 # [ 168.631201] nix-grpc-daemon[1583]: time=2026-09-22T11:27:47.241Z level=INFO msg="Uploading 1 narinfos"6181lb # [ 168.568884] niks3-server[1098]: 2026/09/22 11:27:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6182lb # [ 168.576350] niks3-server[1098]: 2026/09/22 11:27:47 INFO Signed narinfos id=52 count=16183lb # [ 168.587914] niks3-server[1098]: 2026/09/22 11:27:47 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls6184lb # [ 168.595382] niks3-server[1098]: 2026/09/22 11:27:47 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6185lb # [ 168.604318] niks3-server[1098]: 2026/09/22 11:27:47 INFO Completed upload id=526186worker1 # [ 168.666784] nix-grpc-daemon[1583]: time=2026-09-22T11:27:47.277Z level=INFO msg="Upload complete. (272ms)"6187lb # [ 168.607345] niks3-server[1098]: 2026/09/22 11:27:47 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo6188worker1 # [ 168.689106] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=16189worker1 # [ 168.704379] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:47Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=06190worker1 # [ 168.709353] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:47Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.3:50051 clients=16191worker1 # [ 168.713801] nix-daemon[847]: reaped child process 1717, status = succeeded6192worker1 # [ 168.733804] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946193worker1 # [ 168.750696] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460946194worker1 # [ 168.767435] nix-grpc-daemon[1583]: time=2026-09-22T11:27:47.378Z level=INFO msg="All 2 paths already cached"6195client # copying 1 paths to 'lb:50051'...6196client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'...6197worker1 # [ 168.778519] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:460946198worker1 # [ 168.795028] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6199worker1 # [ 168.951439] nix-daemon[847]: accepted connection from pid 1744, user nix-grpc-daemon (trusted)6200worker1 # [ 168.971445] nix-daemon[847]: reaped child process 1747, status = succeeded6201worker1 # [ 169.077049] nix-daemon[847]: accepted connection from pid 1749, user nix-grpc-daemon (trusted)6202worker1 # [ 169.094138] nix-daemon[847]: reaped child process 1752, status = succeeded6203lb # [ 169.039248] niks3-server[1098]: 2026/09/22 11:27:47 INFO Received uploads request method=POST path=/api/pending_closures6204worker1 # [ 169.111006] nix-grpc-daemon[1583]: time=2026-09-22T11:27:47.721Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6205worker1 # [ 169.112638] nix-grpc-daemon[1583]: time=2026-09-22T11:27:47.721Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)"6206lb # [ 169.072172] niks3-server[1098]: 2026/09/22 11:27:47 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst6207lb # [ 169.087670] niks3-server[1098]: 2026/09/22 11:27:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6208worker1 # [ 169.158282] nix-grpc-daemon[1583]: time=2026-09-22T11:27:47.769Z level=INFO msg="Uploading 1 narinfos"6209lb # [ 169.100905] niks3-server[1098]: 2026/09/22 11:27:47 INFO Signed narinfos id=53 count=16210lb # [ 169.107996] niks3-server[1098]: 2026/09/22 11:27:47 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls6211lb # [ 169.123266] niks3-server[1098]: 2026/09/22 11:27:47 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6212lb # [ 169.133540] niks3-server[1098]: 2026/09/22 11:27:47 INFO Completed upload id=536213worker1 # [ 169.198098] nix-grpc-daemon[1583]: time=2026-09-22T11:27:47.807Z level=INFO msg="Upload complete. (383ms)"6214lb # [ 169.141933] niks3-server[1098]: 2026/09/22 11:27:47 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo6215worker1 # [ 169.209344] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=0 paths=1 nar_bytes_in=7686216worker1 # [ 169.224641] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946217worker1 # [ 169.232474] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6218worker1 # [ 169.237583] nix-daemon[847]: reaped child process 1742, status = succeeded6219client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051...6220worker1 # [ 169.252703] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6221client # farm-top> node-a: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6222client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6223client # farm-top> LOG-farm-top-during-drain6224worker1 # [ 170.102769] nix-daemon[847]: reaped child process 1756, status = succeeded6225worker1 # [ 170.207729] nix-daemon[847]: accepted connection from pid 1772, user nix-grpc-daemon (trusted)6226worker1 # [ 170.224763] nix-daemon[847]: reaped child process 1775, status = succeeded6227lb # [ 170.169239] niks3-server[1098]: 2026/09/22 11:27:48 INFO Received uploads request method=POST path=/api/pending_closures6228worker1 # [ 170.244486] nix-grpc-daemon[1583]: time=2026-09-22T11:27:48.853Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6229worker1 # [ 170.246403] nix-grpc-daemon[1583]: time=2026-09-22T11:27:48.853Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)"6230lb # [ 170.203589] niks3-server[1098]: 2026/09/22 11:27:48 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6231lb # [ 170.218380] niks3-server[1098]: 2026/09/22 11:27:48 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst6232lb # [ 170.236291] niks3-server[1098]: 2026/09/22 11:27:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6233worker1 # [ 170.304556] nix-grpc-daemon[1583]: time=2026-09-22T11:27:48.915Z level=INFO msg="Uploading 1 narinfos"6234lb # [ 170.244848] niks3-server[1098]: 2026/09/22 11:27:48 INFO Signed narinfos id=54 count=16235lb # [ 170.250185] niks3-server[1098]: 2026/09/22 11:27:48 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls6236lb # [ 170.258472] niks3-server[1098]: 2026/09/22 11:27:48 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6237lb # [ 170.264908] niks3-server[1098]: 2026/09/22 11:27:48 INFO Completed upload id=546238worker1 # [ 170.327505] nix-grpc-daemon[1583]: time=2026-09-22T11:27:48.938Z level=INFO msg="Upload complete. (264ms)"6239lb # [ 170.268241] niks3-server[1098]: 2026/09/22 11:27:48 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo6240worker1 # [ 170.341136] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=16241worker1 # [ 170.357691] nix-daemon[847]: reaped child process 1754, status = succeeded6242client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2, in 5.65 seconds)6243client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath6244client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath, in 0.18 seconds)6245worker2: must succeed: pkill -f 'read -t [9]0 x'6246worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)6247client: waiting for success: ! systemctl is-active sw6248worker2 # [ 170.770142] nix-daemon[836]: reaped child process 1794, status = succeeded6249worker2 # [ 170.880125] nix-daemon[836]: accepted connection from pid 1966, user nix-grpc-daemon (trusted)6250worker2 # [ 170.901463] nix-daemon[836]: reaped child process 1969, status = succeeded6251lb # [ 170.900489] niks3-server[1098]: 2026/09/22 11:27:49 INFO Received uploads request method=POST path=/api/pending_closures6252worker2 # [ 170.912576] nix-grpc-daemon[1726]: time=2026-09-22T11:27:49.526Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6253worker2 # [ 170.914171] nix-grpc-daemon[1726]: time=2026-09-22T11:27:49.526Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)"6254lb # [ 170.939947] niks3-server[1098]: 2026/09/22 11:27:49 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6255lb # [ 170.954950] niks3-server[1098]: 2026/09/22 11:27:49 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv6256worker2 # [ 170.966266] nix-grpc-daemon[1726]: time=2026-09-22T11:27:49.579Z level=INFO msg="Uploading 1 narinfos"6257lb # [ 170.966243] niks3-server[1098]: 2026/09/22 11:27:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6258lb # [ 170.974622] niks3-server[1098]: 2026/09/22 11:27:49 INFO Signed narinfos id=55 count=16259lb # [ 170.978868] niks3-server[1098]: 2026/09/22 11:27:49 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls6260lb # [ 170.986042] niks3-server[1098]: 2026/09/22 11:27:49 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6261lb # [ 170.990536] niks3-server[1098]: 2026/09/22 11:27:49 INFO Completed upload id=556262worker2 # [ 170.992133] nix-grpc-daemon[1726]: time=2026-09-22T11:27:49.606Z level=INFO msg="Upload complete. (283ms)"6263lb # [ 170.993164] niks3-server[1098]: 2026/09/22 11:27:49 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo6264worker2 # [ 171.005109] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49346 duration_s=15 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=16265worker2 # [ 171.022715] nix-daemon[836]: reaped child process 1792, status = succeeded6266client # [ 171.179751] systemd[1]: sw.service: Deactivated successfully.6267client # [ 171.184122] systemd[1]: sw.service: Consumed 198ms CPU time over 16.982s wall clock time, 39.4M memory peak, 7.5K incoming IP traffic, 8.7K outgoing IP traffic.6268worker2 # [ 171.510751] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:50Z level=info event=signal_exit6269client: (finished: waiting for success: ! systemctl is-active sw, in 1.09 seconds)6270client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6271client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.05 seconds)6272worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6273worker2 # [ 171.782248] systemd[1]: Starting scheduler-token.service...6274worker2 # [ 171.853492] systemd[1]: scheduler-token.service: Deactivated successfully.6275worker2 # [ 171.858854] systemd[1]: Finished scheduler-token.service.6276worker2 # [ 171.860658] systemd[1]: scheduler-token.service: Consumed 39ms CPU time over 75ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6277worker1 # [ 172.584522] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:51Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:460926278worker2 # [ 172.525531] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:51Z level=info event=scheduler_restarting6279worker1 # [ 172.584900] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:51Z level=info event=worker_gone addr=192.168.1.4:500516280worker2 # [ 172.550376] nix-grpc-daemon[1645]: ts=2026-09-22T11:27:51Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6281worker2 # [ 172.575014] nix-daemon[836]: reaped child process 1697, status = succeeded6282worker2 # [ 172.680769] nix-grpc-daemon[1645]: E0000 00:00:1790076471.295291 1690 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true6283worker2 # [ 172.687795] nix-grpc-daemon[1645]: E0000 00:00:1790076471.295491 1690 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6284worker2 # [ 174.557596] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6285worker2 # [ 174.563739] systemd[1]: nix-grpc-daemon.service: Consumed 897ms CPU time over 32.283s wall clock time, 35.4M memory peak, 21.6K incoming IP traffic, 35.1K outgoing IP traffic.6286worker2 # [ 174.592625] systemd[1]: Starting Nix worker-protocol over gRPC...6287worker2 # [ 174.735811] nix-grpc-daemon[1986]: ts=2026-09-22T11:27:53Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6288worker2 # [ 174.738303] nix-grpc-daemon[1986]: ts=2026-09-22T11:27:53Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6289worker2 # [ 174.742157] nix-grpc-daemon[1986]: ts=2026-09-22T11:27:53Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16290worker2 # [ 174.747422] nix-grpc-daemon[1986]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6291worker2 # [ 174.764244] nix-grpc-daemon[1986]: E0000 00:00:1790076473.361514 1986 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6292worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 3.10 seconds)6293lb: waiting until exactly no endpoints of aarch64-linux are unhealthy6294lb: must succeed: curl -sf localhost:9901/clusters6295worker2 # [ 174.795959] nix-grpc-daemon[1986]: E0000 00:00:1790076473.365107 1986 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6296worker2 # [ 174.798878] nix-grpc-daemon[1986]: ts=2026-09-22T11:27:53Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6297worker2 # [ 174.800933] nix-grpc-daemon[1986]: E0000 00:00:1790076473.373309 2028 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6298worker2 # [ 174.803218] nix-grpc-daemon[1986]: E0000 00:00:1790076473.394136 2028 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6299worker2 # [ 174.805603] nix-grpc-daemon[1986]: E0000 00:00:1790076473.394279 2028 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6300worker2 # [ 174.810647] nix-grpc-daemon[1986]: E0000 00:00:1790076473.409617 2028 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6301worker2 # [ 174.815237] systemd[1]: Started Nix worker-protocol over gRPC.6302worker2 # [ 174.816286] nix-grpc-daemon[1986]: ts=2026-09-22T11:27:53Z level=info event=scheduler_connected addr=lb:500516303worker1 # [ 174.880917] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:53Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:460926304worker1 # [ 174.881290] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:53Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=06305lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6306lb: must succeed: curl -sf localhost:9901/clusters6307lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)6308lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.17 seconds)6309lb: must succeed: curl -sf localhost:9901/clusters6310lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6311worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6312worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6313(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 22.08 seconds)6314subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6315client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16316worker2 # [ 176.299761] nix-grpc-daemon[1986]: ts=2026-09-22T11:27:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:519126317worker2 # [ 176.313878] nix-daemon[836]: accepted connection from pid 1986, user nix-grpc-daemon (trusted)6318worker2 # [ 176.334694] nix-grpc-daemon[1986]: ts=2026-09-22T11:27:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:519126319worker2 # [ 176.350048] nix-grpc-daemon[1986]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 88 ms (attempt 1/5)6320worker2 # [ 176.447403] nix-grpc-daemon[1986]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 141 ms (attempt 2/5)6321worker2 # [ 176.590273] nix-grpc-daemon[1986]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 167 ms (attempt 3/5)6322worker2 # [ 176.797306] nix-grpc-daemon[1986]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 342 ms (attempt 4/5)6323worker2 # [ 177.141746] nix-grpc-daemon[1986]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6324worker2 # [ 177.149082] nix-grpc-daemon[1986]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6325worker1 # [ 177.263858] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:55Z level=debug event=want client=5 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=06326worker1 # [ 177.269489] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:55Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=16327worker1 # [ 177.275501] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:55Z level=debug event=want client=5 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=06328worker1 # [ 177.280915] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:55Z level=debug event=want client=5 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=06329worker1 # [ 177.306997] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946330worker1 # [ 177.323133] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460946331worker1 # [ 177.337959] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:460946332worker1 # [ 177.353121] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6333worker1 # [ 177.515726] nix-daemon[847]: accepted connection from pid 1784, user nix-grpc-daemon (trusted)6334worker1 # [ 177.537339] nix-daemon[847]: reaped child process 1787, status = succeeded6335worker1 # [ 177.632671] nix-daemon[847]: accepted connection from pid 1789, user nix-grpc-daemon (trusted)6336worker1 # [ 177.652296] nix-daemon[847]: reaped child process 1792, status = succeeded6337lb # [ 177.589421] niks3-server[1098]: 2026/09/22 11:27:56 INFO Received uploads request method=POST path=/api/pending_closures6338worker1 # [ 177.661979] nix-grpc-daemon[1583]: time=2026-09-22T11:27:56.272Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6339worker1 # [ 177.663420] nix-grpc-daemon[1583]: time=2026-09-22T11:27:56.272Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)"6340lb # [ 177.617052] niks3-server[1098]: 2026/09/22 11:27:56 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst6341worker1 # [ 177.701774] nix-grpc-daemon[1583]: time=2026-09-22T11:27:56.312Z level=INFO msg="Uploading 1 narinfos"6342lb # [ 177.632983] niks3-server[1098]: 2026/09/22 11:27:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6343lb # [ 177.642145] niks3-server[1098]: 2026/09/22 11:27:56 INFO Signed narinfos id=56 count=16344lb # [ 177.653781] niks3-server[1098]: 2026/09/22 11:27:56 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls6345worker1 # [ 177.732322] nix-grpc-daemon[1583]: time=2026-09-22T11:27:56.343Z level=INFO msg="Upload complete. (362ms)"6346lb # [ 177.664367] niks3-server[1098]: 2026/09/22 11:27:56 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6347lb # [ 177.673294] niks3-server[1098]: 2026/09/22 11:27:56 INFO Completed upload id=566348worker1 # [ 177.740189] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=0 paths=1 nar_bytes_in=5126349lb # [ 177.680222] niks3-server[1098]: 2026/09/22 11:27:56 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo6350worker1 # [ 177.756762] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946351worker1 # [ 177.763974] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6352worker1 # [ 177.769520] nix-daemon[847]: reaped child process 1782, status = succeeded6353worker1 # [ 177.780129] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6354lb # [ 178.559977] envoy[908]: conn peer=[2001:db8:1::1]:37022 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9634 tx=4038 ms=53966355lb # [ 178.560354] envoy[908]: conn peer=[2001:db8:1::1]:55982 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3025 tx=1466 ms=167536356lb # [ 178.560476] envoy[908]: conn peer=[2001:db8:1::4]:41580 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=7565 tx=668 ms=299736357worker1 # [ 178.889749] nix-daemon[847]: reaped child process 1796, status = succeeded6358worker1 # [ 178.962591] nix-daemon[847]: accepted connection from pid 1812, user nix-grpc-daemon (trusted)6359worker1 # [ 178.982311] nix-daemon[847]: reaped child process 1815, status = succeeded6360lb # [ 178.919062] niks3-server[1098]: 2026/09/22 11:27:57 INFO Received uploads request method=POST path=/api/pending_closures6361worker1 # [ 178.992620] nix-grpc-daemon[1583]: time=2026-09-22T11:27:57.602Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6362worker1 # [ 178.994668] nix-grpc-daemon[1583]: time=2026-09-22T11:27:57.602Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)"6363worker1 # [ 179.035457] nix-grpc-daemon[1583]: time=2026-09-22T11:27:57.646Z level=INFO msg="Uploading 1 narinfos"6364lb # [ 178.965635] niks3-server[1098]: 2026/09/22 11:27:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6365lb # [ 178.977123] niks3-server[1098]: 2026/09/22 11:27:57 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6366lb # [ 178.990238] niks3-server[1098]: 2026/09/22 11:27:57 INFO Signed narinfos id=57 count=16367lb # [ 178.995386] niks3-server[1098]: 2026/09/22 11:27:57 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls6368worker1 # [ 179.066842] nix-grpc-daemon[1583]: time=2026-09-22T11:27:57.677Z level=INFO msg="Upload complete. (275ms)"6369lb # [ 179.006523] niks3-server[1098]: 2026/09/22 11:27:57 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6370lb # [ 179.018274] niks3-server[1098]: 2026/09/22 11:27:57 INFO Completed upload id=576371lb # [ 179.023315] niks3-server[1098]: 2026/09/22 11:27:57 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo6372worker1 # [ 179.086457] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=7 outputs=16373lb # [ 179.028758] niks3-server[1098]: 2026/09/22 11:27:57 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv6374worker1 # [ 179.101108] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:57Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=16375worker1 # [ 179.111007] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946376worker1 # [ 179.115724] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:57Z level=debug event=want client=5 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=06377worker1 # [ 179.132374] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460946378worker1 # [ 179.135718] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:460946379worker1 # [ 179.142309] nix-daemon[847]: reaped child process 1794, status = succeeded6380worker1 # [ 179.144185] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6381worker1 # [ 179.267035] nix-daemon[847]: accepted connection from pid 1821, user nix-grpc-daemon (trusted)6382worker1 # [ 179.283440] nix-daemon[847]: reaped child process 1824, status = succeeded6383worker1 # [ 179.383711] nix-daemon[847]: accepted connection from pid 1826, user nix-grpc-daemon (trusted)6384worker1 # [ 179.398568] nix-daemon[847]: reaped child process 1829, status = succeeded6385lb # [ 179.342844] niks3-server[1098]: 2026/09/22 11:27:57 INFO Received uploads request method=POST path=/api/pending_closures6386worker1 # [ 179.418767] nix-grpc-daemon[1583]: time=2026-09-22T11:27:58.029Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6387worker1 # [ 179.420329] nix-grpc-daemon[1583]: time=2026-09-22T11:27:58.029Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)"6388worker1 # [ 179.454535] nix-grpc-daemon[1583]: time=2026-09-22T11:27:58.065Z level=INFO msg="Uploading 1 narinfos"6389lb # [ 179.381233] niks3-server[1098]: 2026/09/22 11:27:57 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst6390lb # [ 179.397041] niks3-server[1098]: 2026/09/22 11:27:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6391lb # [ 179.406400] niks3-server[1098]: 2026/09/22 11:27:57 INFO Signed narinfos id=58 count=16392lb # [ 179.410762] niks3-server[1098]: 2026/09/22 11:27:58 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls6393lb # [ 179.421304] niks3-server[1098]: 2026/09/22 11:27:58 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6394lb # [ 179.426202] niks3-server[1098]: 2026/09/22 11:27:58 INFO Completed upload id=586395worker1 # [ 179.487414] nix-grpc-daemon[1583]: time=2026-09-22T11:27:58.098Z level=INFO msg="Upload complete. (346ms)"6396lb # [ 179.428631] niks3-server[1098]: 2026/09/22 11:27:58 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo6397worker1 # [ 179.497922] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=0 paths=1 nar_bytes_in=5126398worker1 # [ 179.512108] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946399worker1 # [ 179.521154] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6400worker1 # [ 179.524800] nix-daemon[847]: reaped child process 1819, status = succeeded6401worker1 # [ 179.536165] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6402worker1 # [ 180.773788] nix-daemon[847]: reaped child process 1833, status = succeeded6403worker1 # [ 180.868437] nix-daemon[847]: accepted connection from pid 1849, user nix-grpc-daemon (trusted)6404worker1 # [ 180.885613] nix-daemon[847]: reaped child process 1852, status = succeeded6405lb # [ 180.830937] niks3-server[1098]: 2026/09/22 11:27:59 INFO Received uploads request method=POST path=/api/pending_closures6406worker1 # [ 180.904394] nix-grpc-daemon[1583]: time=2026-09-22T11:27:59.513Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6407worker1 # [ 180.905885] nix-grpc-daemon[1583]: time=2026-09-22T11:27:59.513Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)"6408lb # [ 180.873041] niks3-server[1098]: 2026/09/22 11:27:59 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6409worker1 # [ 180.948815] nix-grpc-daemon[1583]: time=2026-09-22T11:27:59.559Z level=INFO msg="Uploading 1 narinfos"6410lb # [ 180.886709] niks3-server[1098]: 2026/09/22 11:27:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6411lb # [ 180.897648] niks3-server[1098]: 2026/09/22 11:27:59 INFO Signed narinfos id=59 count=16412lb # [ 180.903059] niks3-server[1098]: 2026/09/22 11:27:59 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv6413lb # [ 180.911363] niks3-server[1098]: 2026/09/22 11:27:59 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls6414lb # [ 180.920604] niks3-server[1098]: 2026/09/22 11:27:59 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6415lb # [ 180.926040] niks3-server[1098]: 2026/09/22 11:27:59 INFO Completed upload id=596416worker1 # [ 180.988797] nix-grpc-daemon[1583]: time=2026-09-22T11:27:59.599Z level=INFO msg="Upload complete. (241ms)"6417lb # [ 180.929703] niks3-server[1098]: 2026/09/22 11:27:59 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo6418worker1 # [ 181.006485] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=8 outputs=16419worker1 # [ 181.024764] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:59Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=16420worker1 # [ 181.031473] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946421worker1 # [ 181.036514] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:59Z level=debug event=want client=5 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=06422worker1 # [ 181.040944] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460946423worker1 # [ 181.055633] nix-daemon[847]: reaped child process 1831, status = succeeded6424worker1 # [ 181.057674] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6425worker1 # [ 181.059859] nix-grpc-daemon[1487]: ts=2026-09-22T11:27:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:460946426worker1 # [ 181.185597] nix-daemon[847]: accepted connection from pid 1858, user nix-grpc-daemon (trusted)6427worker1 # [ 181.202179] nix-daemon[847]: reaped child process 1861, status = succeeded6428worker1 # [ 181.307630] nix-daemon[847]: accepted connection from pid 1863, user nix-grpc-daemon (trusted)6429worker1 # [ 181.322690] nix-daemon[847]: reaped child process 1866, status = succeeded6430lb # [ 181.267401] niks3-server[1098]: 2026/09/22 11:27:59 INFO Received uploads request method=POST path=/api/pending_closures6431worker1 # [ 181.339732] nix-grpc-daemon[1583]: time=2026-09-22T11:27:59.950Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6432worker1 # [ 181.341408] nix-grpc-daemon[1583]: time=2026-09-22T11:27:59.950Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)"6433lb # [ 181.293924] niks3-server[1098]: 2026/09/22 11:27:59 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst6434lb # [ 181.309945] niks3-server[1098]: 2026/09/22 11:27:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6435worker1 # [ 181.378850] nix-grpc-daemon[1583]: time=2026-09-22T11:27:59.989Z level=INFO msg="Uploading 1 narinfos"6436lb # [ 181.322679] niks3-server[1098]: 2026/09/22 11:27:59 INFO Signed narinfos id=60 count=16437lb # [ 181.329926] niks3-server[1098]: 2026/09/22 11:27:59 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls6438worker1 # [ 181.403643] nix-grpc-daemon[1583]: time=2026-09-22T11:28:00.014Z level=INFO msg="Upload complete. (338ms)"6439lb # [ 181.341885] niks3-server[1098]: 2026/09/22 11:27:59 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6440lb # [ 181.350226] niks3-server[1098]: 2026/09/22 11:27:59 INFO Completed upload id=606441worker1 # [ 181.411202] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=0 paths=1 nar_bytes_in=5126442lb # [ 181.352721] niks3-server[1098]: 2026/09/22 11:27:59 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo6443worker1 # [ 181.424000] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946444worker1 # [ 181.432308] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6445worker1 # [ 181.435020] nix-daemon[847]: reaped child process 1856, status = succeeded6446worker1 # [ 181.448292] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6447worker1 # [ 182.703689] nix-daemon[847]: reaped child process 1870, status = succeeded6448worker1 # [ 182.828841] nix-daemon[847]: accepted connection from pid 1886, user nix-grpc-daemon (trusted)6449worker1 # [ 182.846489] nix-daemon[847]: reaped child process 1889, status = succeeded6450lb # [ 182.789253] niks3-server[1098]: 2026/09/22 11:28:01 INFO Received uploads request method=POST path=/api/pending_closures6451worker1 # [ 182.862607] nix-grpc-daemon[1583]: time=2026-09-22T11:28:01.472Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6452worker1 # [ 182.864338] nix-grpc-daemon[1583]: time=2026-09-22T11:28:01.472Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)"6453lb # [ 182.823553] niks3-server[1098]: 2026/09/22 11:28:01 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv6454lb # [ 182.837131] niks3-server[1098]: 2026/09/22 11:28:01 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6455lb # [ 182.848746] niks3-server[1098]: 2026/09/22 11:28:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6456worker1 # [ 182.916980] nix-grpc-daemon[1583]: time=2026-09-22T11:28:01.527Z level=INFO msg="Uploading 1 narinfos"6457lb # [ 182.858394] niks3-server[1098]: 2026/09/22 11:28:01 INFO Signed narinfos id=61 count=16458lb # [ 182.862838] niks3-server[1098]: 2026/09/22 11:28:01 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls6459lb # [ 182.871755] niks3-server[1098]: 2026/09/22 11:28:01 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6460lb # [ 182.878955] niks3-server[1098]: 2026/09/22 11:28:01 INFO Completed upload id=616461lb # [ 182.882059] niks3-server[1098]: 2026/09/22 11:28:01 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo6462worker1 # [ 182.933005] nix-grpc-daemon[1583]: time=2026-09-22T11:28:01.544Z level=INFO msg="Upload complete. (274ms)"6463worker1 # [ 182.951778] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=9 outputs=16464worker1 # [ 182.969303] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:01Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=16465worker1 # [ 182.974333] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:01Z level=debug event=want client=5 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=06466worker1 # [ 182.978462] nix-daemon[847]: reaped child process 1868, status = succeeded6467worker1 # [ 182.991417] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946468worker1 # [ 183.006189] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460946469worker1 # [ 183.019495] nix-grpc-daemon[1583]: time=2026-09-22T11:28:01.630Z level=INFO msg="All 1 paths already cached"6470worker1 # [ 183.028111] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:460946471worker1 # [ 183.041463] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6472worker1 # [ 183.193412] nix-daemon[847]: accepted connection from pid 1895, user nix-grpc-daemon (trusted)6473worker1 # [ 183.215911] nix-daemon[847]: reaped child process 1898, status = succeeded6474worker1 # [ 183.311303] nix-daemon[847]: accepted connection from pid 1900, user nix-grpc-daemon (trusted)6475worker1 # [ 183.325978] nix-daemon[847]: reaped child process 1903, status = succeeded6476lb # [ 183.270898] niks3-server[1098]: 2026/09/22 11:28:01 INFO Received uploads request method=POST path=/api/pending_closures6477worker1 # [ 183.342007] nix-grpc-daemon[1583]: time=2026-09-22T11:28:01.952Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6478worker1 # [ 183.344525] nix-grpc-daemon[1583]: time=2026-09-22T11:28:01.952Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)"6479lb # [ 183.298618] niks3-server[1098]: 2026/09/22 11:28:01 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst6480lb # [ 183.311806] niks3-server[1098]: 2026/09/22 11:28:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6481worker1 # [ 183.381395] nix-grpc-daemon[1583]: time=2026-09-22T11:28:01.992Z level=INFO msg="Uploading 1 narinfos"6482lb # [ 183.324363] niks3-server[1098]: 2026/09/22 11:28:01 INFO Signed narinfos id=62 count=16483lb # [ 183.327048] niks3-server[1098]: 2026/09/22 11:28:01 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls6484lb # [ 183.339743] niks3-server[1098]: 2026/09/22 11:28:01 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6485lb # [ 183.347849] niks3-server[1098]: 2026/09/22 11:28:01 INFO Completed upload id=626486worker1 # [ 183.410653] nix-grpc-daemon[1583]: time=2026-09-22T11:28:02.021Z level=INFO msg="Upload complete. (354ms)"6487lb # [ 183.351189] niks3-server[1098]: 2026/09/22 11:28:01 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo6488worker1 # [ 183.420344] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=0 paths=1 nar_bytes_in=6246489worker1 # [ 183.433908] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946490worker1 # [ 183.442205] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6491worker1 # [ 183.445640] nix-daemon[847]: reaped child process 1893, status = succeeded6492worker1 # [ 183.455719] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6493worker1 # [ 184.713652] nix-daemon[847]: reaped child process 1907, status = succeeded6494worker1 # [ 184.847861] nix-daemon[847]: accepted connection from pid 1923, user nix-grpc-daemon (trusted)6495worker1 # [ 184.867494] nix-daemon[847]: reaped child process 1926, status = succeeded6496lb # [ 184.811027] niks3-server[1098]: 2026/09/22 11:28:03 INFO Received uploads request method=POST path=/api/pending_closures6497worker1 # [ 184.888458] nix-grpc-daemon[1583]: time=2026-09-22T11:28:03.495Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6498worker1 # [ 184.890276] nix-grpc-daemon[1583]: time=2026-09-22T11:28:03.495Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)"6499lb # [ 184.853595] niks3-server[1098]: 2026/09/22 11:28:03 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst6500lb # [ 184.864788] niks3-server[1098]: 2026/09/22 11:28:03 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv6501worker1 # [ 184.939554] nix-grpc-daemon[1583]: time=2026-09-22T11:28:03.550Z level=INFO msg="Uploading 1 narinfos"6502lb # [ 184.878487] niks3-server[1098]: 2026/09/22 11:28:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6503lb # [ 184.890177] niks3-server[1098]: 2026/09/22 11:28:03 INFO Signed narinfos id=63 count=16504lb # [ 184.895632] niks3-server[1098]: 2026/09/22 11:28:03 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls6505lb # [ 184.904726] niks3-server[1098]: 2026/09/22 11:28:03 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6506lb # [ 184.912890] niks3-server[1098]: 2026/09/22 11:28:03 INFO Completed upload id=636507worker1 # [ 184.975901] nix-grpc-daemon[1583]: time=2026-09-22T11:28:03.586Z level=INFO msg="Upload complete. (307ms)"6508lb # [ 184.916557] niks3-server[1098]: 2026/09/22 11:28:03 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo6509worker1 # [ 184.997211] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=10 outputs=16510worker1 # [ 185.012856] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:03Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=16511worker1 # [ 185.022372] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946512worker1 # [ 185.027904] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:03Z level=debug event=want client=5 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=06513worker1 # [ 185.033236] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460946514worker1 # [ 185.040263] nix-daemon[847]: reaped child process 1905, status = succeeded6515worker1 # [ 185.042421] nix-grpc-daemon[1583]: time=2026-09-22T11:28:03.643Z level=INFO msg="All 1 paths already cached"6516worker1 # [ 185.051628] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:460946517worker1 # [ 185.060183] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6518worker1 # [ 185.185117] nix-daemon[847]: accepted connection from pid 1932, user nix-grpc-daemon (trusted)6519worker1 # [ 185.208514] nix-daemon[847]: reaped child process 1935, status = succeeded6520worker1 # [ 185.302216] nix-daemon[847]: accepted connection from pid 1937, user nix-grpc-daemon (trusted)6521worker1 # [ 185.318044] nix-daemon[847]: reaped child process 1940, status = succeeded6522lb # [ 185.261136] niks3-server[1098]: 2026/09/22 11:28:03 INFO Received uploads request method=POST path=/api/pending_closures6523worker1 # [ 185.332279] nix-grpc-daemon[1583]: time=2026-09-22T11:28:03.943Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6524worker1 # [ 185.333727] nix-grpc-daemon[1583]: time=2026-09-22T11:28:03.943Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)"6525lb # [ 185.289526] niks3-server[1098]: 2026/09/22 11:28:03 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst6526lb # [ 185.303317] niks3-server[1098]: 2026/09/22 11:28:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6527worker1 # [ 185.371145] nix-grpc-daemon[1583]: time=2026-09-22T11:28:03.982Z level=INFO msg="Uploading 1 narinfos"6528lb # [ 185.314720] niks3-server[1098]: 2026/09/22 11:28:03 INFO Signed narinfos id=64 count=16529lb # [ 185.320982] niks3-server[1098]: 2026/09/22 11:28:03 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls6530lb # [ 185.329561] niks3-server[1098]: 2026/09/22 11:28:03 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6531worker1 # [ 185.396271] nix-grpc-daemon[1583]: time=2026-09-22T11:28:04.007Z level=INFO msg="Upload complete. (331ms)"6532lb # [ 185.335847] niks3-server[1098]: 2026/09/22 11:28:03 INFO Completed upload id=646533lb # [ 185.339342] niks3-server[1098]: 2026/09/22 11:28:03 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo6534worker1 # [ 185.404143] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=0 paths=1 nar_bytes_in=6246535worker1 # [ 185.415984] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946536worker1 # [ 185.421679] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6537worker1 # [ 185.426252] nix-daemon[847]: reaped child process 1930, status = succeeded6538worker1 # [ 185.439124] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6539worker1 # [ 186.749881] nix-daemon[847]: reaped child process 1944, status = succeeded6540worker1 # [ 186.817615] nix-daemon[847]: accepted connection from pid 1960, user nix-grpc-daemon (trusted)6541worker1 # [ 186.838102] nix-daemon[847]: reaped child process 1963, status = succeeded6542lb # [ 186.782057] niks3-server[1098]: 2026/09/22 11:28:05 INFO Received uploads request method=POST path=/api/pending_closures6543worker1 # [ 186.852407] nix-grpc-daemon[1583]: time=2026-09-22T11:28:05.462Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6544worker1 # [ 186.854127] nix-grpc-daemon[1583]: time=2026-09-22T11:28:05.462Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)"6545lb # [ 186.818065] niks3-server[1098]: 2026/09/22 11:28:05 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv6546lb # [ 186.833394] niks3-server[1098]: 2026/09/22 11:28:05 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst6547worker1 # [ 186.907686] nix-grpc-daemon[1583]: time=2026-09-22T11:28:05.518Z level=INFO msg="Uploading 1 narinfos"6548lb # [ 186.848306] niks3-server[1098]: 2026/09/22 11:28:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6549lb # [ 186.857573] niks3-server[1098]: 2026/09/22 11:28:05 INFO Signed narinfos id=65 count=16550lb # [ 186.864566] niks3-server[1098]: 2026/09/22 11:28:05 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls6551worker1 # [ 186.939429] nix-grpc-daemon[1583]: time=2026-09-22T11:28:05.550Z level=INFO msg="Upload complete. (289ms)"6552lb # [ 186.875573] niks3-server[1098]: 2026/09/22 11:28:05 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6553lb # [ 186.882098] niks3-server[1098]: 2026/09/22 11:28:05 INFO Completed upload id=656554lb # [ 186.885967] niks3-server[1098]: 2026/09/22 11:28:05 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo6555worker1 # [ 186.958482] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=11 outputs=16556worker1 # [ 186.980303] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:05Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=16557worker1 # [ 186.987951] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946558worker1 # [ 186.995090] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:05Z level=debug event=want client=5 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=06559worker1 # [ 187.001652] nix-daemon[847]: reaped child process 1942, status = succeeded6560worker1 # [ 187.005179] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460946561worker1 # [ 187.018113] nix-grpc-daemon[1583]: time=2026-09-22T11:28:05.615Z level=INFO msg="All 1 paths already cached"6562worker1 # [ 187.025663] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6563worker1 # [ 187.031851] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:460946564worker1 # [ 187.161240] nix-daemon[847]: accepted connection from pid 1969, user nix-grpc-daemon (trusted)6565worker1 # [ 187.177900] nix-daemon[847]: reaped child process 1972, status = succeeded6566worker1 # [ 187.279250] nix-daemon[847]: accepted connection from pid 1974, user nix-grpc-daemon (trusted)6567worker1 # [ 187.296115] nix-daemon[847]: reaped child process 1977, status = succeeded6568lb # [ 187.237615] niks3-server[1098]: 2026/09/22 11:28:05 INFO Received uploads request method=POST path=/api/pending_closures6569worker1 # [ 187.308738] nix-grpc-daemon[1583]: time=2026-09-22T11:28:05.919Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6570worker1 # [ 187.310244] nix-grpc-daemon[1583]: time=2026-09-22T11:28:05.919Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)"6571lb # [ 187.264722] niks3-server[1098]: 2026/09/22 11:28:05 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst6572worker1 # [ 187.341544] nix-grpc-daemon[1583]: time=2026-09-22T11:28:05.952Z level=INFO msg="Uploading 1 narinfos"6573lb # [ 187.279079] niks3-server[1098]: 2026/09/22 11:28:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign6574lb # [ 187.292431] niks3-server[1098]: 2026/09/22 11:28:05 INFO Signed narinfos id=66 count=16575lb # [ 187.300793] niks3-server[1098]: 2026/09/22 11:28:05 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls6576lb # [ 187.312340] niks3-server[1098]: 2026/09/22 11:28:05 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete6577worker1 # [ 187.382721] nix-grpc-daemon[1583]: time=2026-09-22T11:28:05.993Z level=INFO msg="Upload complete. (344ms)"6578lb # [ 187.322051] niks3-server[1098]: 2026/09/22 11:28:05 INFO Completed upload id=666579lb # [ 187.326651] niks3-server[1098]: 2026/09/22 11:28:05 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo6580worker1 # [ 187.391034] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=0 paths=1 nar_bytes_in=6246581worker1 # [ 187.406801] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946582worker1 # [ 187.415248] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6583worker1 # [ 187.420785] nix-daemon[847]: reaped child process 1967, status = succeeded6584worker1 # [ 187.433704] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6585worker1 # [ 188.913009] nix-daemon[847]: reaped child process 1981, status = succeeded6586worker1 # [ 189.020381] nix-daemon[847]: accepted connection from pid 1997, user nix-grpc-daemon (trusted)6587worker1 # [ 189.036801] nix-daemon[847]: reaped child process 2000, status = succeeded6588lb # [ 188.981051] niks3-server[1098]: 2026/09/22 11:28:07 INFO Received uploads request method=POST path=/api/pending_closures6589worker1 # [ 189.055904] nix-grpc-daemon[1583]: time=2026-09-22T11:28:07.664Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6590worker1 # [ 189.057970] nix-grpc-daemon[1583]: time=2026-09-22T11:28:07.664Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)"6591lb # [ 189.019813] niks3-server[1098]: 2026/09/22 11:28:07 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst6592lb # [ 189.033131] niks3-server[1098]: 2026/09/22 11:28:07 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv6593worker1 # [ 189.109599] nix-grpc-daemon[1583]: time=2026-09-22T11:28:07.720Z level=INFO msg="Uploading 1 narinfos"6594lb # [ 189.048530] niks3-server[1098]: 2026/09/22 11:28:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign6595lb # [ 189.059080] niks3-server[1098]: 2026/09/22 11:28:07 INFO Signed narinfos id=67 count=16596lb # [ 189.064767] niks3-server[1098]: 2026/09/22 11:28:07 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls6597lb # [ 189.075207] niks3-server[1098]: 2026/09/22 11:28:07 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete6598worker1 # [ 189.137295] nix-grpc-daemon[1583]: time=2026-09-22T11:28:07.748Z level=INFO msg="Upload complete. (290ms)"6599lb # [ 189.080601] niks3-server[1098]: 2026/09/22 11:28:07 INFO Completed upload id=676600lb # [ 189.083213] niks3-server[1098]: 2026/09/22 11:28:07 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo6601worker1 # [ 189.151629] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=12 outputs=16602worker1 # [ 189.168418] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:07Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=16603worker1 # [ 189.174305] nix-daemon[847]: reaped child process 1979, status = succeeded6604worker1 # [ 189.189464] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946605worker1 # [ 189.196835] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:07Z level=debug event=want client=5 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=06606worker1 # [ 189.207628] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460946607worker1 # [ 189.221808] nix-grpc-daemon[1583]: time=2026-09-22T11:28:07.832Z level=INFO msg="All 2 paths already cached"6608worker1 # [ 189.230551] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:460946609worker1 # [ 189.243977] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6610worker1 # [ 189.390878] nix-daemon[847]: accepted connection from pid 2006, user nix-grpc-daemon (trusted)6611worker1 # [ 189.411833] nix-daemon[847]: reaped child process 2009, status = succeeded6612worker1 # [ 189.510038] nix-daemon[847]: accepted connection from pid 2011, user nix-grpc-daemon (trusted)6613worker1 # [ 189.528135] nix-daemon[847]: reaped child process 2014, status = succeeded6614lb # [ 189.468300] niks3-server[1098]: 2026/09/22 11:28:08 INFO Received uploads request method=POST path=/api/pending_closures6615worker1 # [ 189.539353] nix-grpc-daemon[1583]: time=2026-09-22T11:28:08.150Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6616worker1 # [ 189.540990] nix-grpc-daemon[1583]: time=2026-09-22T11:28:08.150Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)"6617lb # [ 189.493837] niks3-server[1098]: 2026/09/22 11:28:08 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst6618worker1 # [ 189.577943] nix-grpc-daemon[1583]: time=2026-09-22T11:28:08.188Z level=INFO msg="Uploading 1 narinfos"6619lb # [ 189.508270] niks3-server[1098]: 2026/09/22 11:28:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign6620lb # [ 189.521420] niks3-server[1098]: 2026/09/22 11:28:08 INFO Signed narinfos id=68 count=16621lb # [ 189.527543] niks3-server[1098]: 2026/09/22 11:28:08 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls6622lb # [ 189.536929] niks3-server[1098]: 2026/09/22 11:28:08 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete6623lb # [ 189.543802] niks3-server[1098]: 2026/09/22 11:28:08 INFO Completed upload id=686624worker1 # [ 189.605478] nix-grpc-daemon[1583]: time=2026-09-22T11:28:08.216Z level=INFO msg="Upload complete. (347ms)"6625lb # [ 189.546080] niks3-server[1098]: 2026/09/22 11:28:08 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo6626worker1 # [ 189.615481] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=0 paths=1 nar_bytes_in=7526627worker1 # [ 189.626493] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946628worker1 # [ 189.635019] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6629worker1 # [ 189.637479] nix-daemon[847]: reaped child process 2004, status = succeeded6630worker1 # [ 189.651213] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6631worker1 # [ 190.520779] nix-daemon[847]: reaped child process 2018, status = succeeded6632worker1 # [ 190.625852] nix-daemon[847]: accepted connection from pid 2034, user nix-grpc-daemon (trusted)6633worker1 # [ 190.645974] nix-daemon[847]: reaped child process 2037, status = succeeded6634lb # [ 190.585251] niks3-server[1098]: 2026/09/22 11:28:09 INFO Received uploads request method=POST path=/api/pending_closures6635worker1 # [ 190.658305] nix-grpc-daemon[1583]: time=2026-09-22T11:28:09.268Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6636worker1 # [ 190.659974] nix-grpc-daemon[1583]: time=2026-09-22T11:28:09.268Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)"6637lb # [ 190.621379] niks3-server[1098]: 2026/09/22 11:28:09 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv6638lb # [ 190.633414] niks3-server[1098]: 2026/09/22 11:28:09 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst6639worker1 # [ 190.708674] nix-grpc-daemon[1583]: time=2026-09-22T11:28:09.319Z level=INFO msg="Uploading 1 narinfos"6640lb # [ 190.648177] niks3-server[1098]: 2026/09/22 11:28:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign6641lb # [ 190.657756] niks3-server[1098]: 2026/09/22 11:28:09 INFO Signed narinfos id=69 count=16642lb # [ 190.664748] niks3-server[1098]: 2026/09/22 11:28:09 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls6643worker1 # [ 190.730585] nix-grpc-daemon[1583]: time=2026-09-22T11:28:09.341Z level=INFO msg="Upload complete. (269ms)"6644lb # [ 190.678021] niks3-server[1098]: 2026/09/22 11:28:09 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete6645lb # [ 190.682984] niks3-server[1098]: 2026/09/22 11:28:09 INFO Completed upload id=696646lb # [ 190.686023] niks3-server[1098]: 2026/09/22 11:28:09 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo6647worker1 # [ 190.745127] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=13 outputs=16648worker1 # [ 190.760775] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:09Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=16649worker1 # [ 190.765588] nix-daemon[847]: reaped child process 2016, status = succeeded6650worker1 # [ 190.786560] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946651worker1 # [ 190.801281] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460946652worker1 # [ 190.815039] nix-grpc-daemon[1583]: time=2026-09-22T11:28:09.426Z level=INFO msg="All 2 paths already cached"6653worker1 # [ 190.823985] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:460946654worker1 # [ 190.837492] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6655worker1 # [ 190.995654] nix-daemon[847]: accepted connection from pid 2041, user nix-grpc-daemon (trusted)6656worker1 # [ 191.011828] nix-daemon[847]: reaped child process 2044, status = succeeded6657worker1 # [ 191.113601] nix-daemon[847]: accepted connection from pid 2046, user nix-grpc-daemon (trusted)6658worker1 # [ 191.131854] nix-daemon[847]: reaped child process 2049, status = succeeded6659lb # [ 191.070439] niks3-server[1098]: 2026/09/22 11:28:09 INFO Received uploads request method=POST path=/api/pending_closures6660worker1 # [ 191.141513] nix-grpc-daemon[1583]: time=2026-09-22T11:28:09.752Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6661worker1 # [ 191.143028] nix-grpc-daemon[1583]: time=2026-09-22T11:28:09.752Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)"6662lb # [ 191.097421] niks3-server[1098]: 2026/09/22 11:28:09 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst6663worker1 # [ 191.175365] nix-grpc-daemon[1583]: time=2026-09-22T11:28:09.786Z level=INFO msg="Uploading 1 narinfos"6664lb # [ 191.112584] niks3-server[1098]: 2026/09/22 11:28:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign6665lb # [ 191.122268] niks3-server[1098]: 2026/09/22 11:28:09 INFO Signed narinfos id=70 count=16666lb # [ 191.129855] niks3-server[1098]: 2026/09/22 11:28:09 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls6667lb # [ 191.143140] niks3-server[1098]: 2026/09/22 11:28:09 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete6668worker1 # [ 191.212873] nix-grpc-daemon[1583]: time=2026-09-22T11:28:09.823Z level=INFO msg="Upload complete. (359ms)"6669lb # [ 191.151539] niks3-server[1098]: 2026/09/22 11:28:09 INFO Completed upload id=706670worker1 # [ 191.219044] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=0 paths=1 nar_bytes_in=7526671lb # [ 191.157112] niks3-server[1098]: 2026/09/22 11:28:09 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo6672worker1 # [ 191.228852] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946673worker1 # [ 191.239207] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6674worker1 # [ 191.243987] nix-daemon[847]: reaped child process 2039, status = succeeded6675worker1 # [ 191.257546] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6676worker1 # [ 192.359652] nix-daemon[847]: reaped child process 2053, status = succeeded6677worker1 # [ 192.431341] nix-daemon[847]: accepted connection from pid 2069, user nix-grpc-daemon (trusted)6678worker1 # [ 192.448350] nix-daemon[847]: reaped child process 2072, status = succeeded6679lb # [ 192.394638] niks3-server[1098]: 2026/09/22 11:28:10 INFO Received uploads request method=POST path=/api/pending_closures6680worker1 # [ 192.467999] nix-grpc-daemon[1583]: time=2026-09-22T11:28:11.077Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6681worker1 # [ 192.469844] nix-grpc-daemon[1583]: time=2026-09-22T11:28:11.077Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)"6682lb # [ 192.425458] niks3-server[1098]: 2026/09/22 11:28:11 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv6683lb # [ 192.440578] niks3-server[1098]: 2026/09/22 11:28:11 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst6684lb # [ 192.452814] niks3-server[1098]: 2026/09/22 11:28:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign6685worker1 # [ 192.520624] nix-grpc-daemon[1583]: time=2026-09-22T11:28:11.131Z level=INFO msg="Uploading 1 narinfos"6686lb # [ 192.460805] niks3-server[1098]: 2026/09/22 11:28:11 INFO Signed narinfos id=71 count=16687lb # [ 192.466562] niks3-server[1098]: 2026/09/22 11:28:11 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls6688lb # [ 192.475568] niks3-server[1098]: 2026/09/22 11:28:11 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete6689lb # [ 192.482738] niks3-server[1098]: 2026/09/22 11:28:11 INFO Completed upload id=716690worker1 # [ 192.544352] nix-grpc-daemon[1583]: time=2026-09-22T11:28:11.155Z level=INFO msg="Upload complete. (276ms)"6691lb # [ 192.485023] niks3-server[1098]: 2026/09/22 11:28:11 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo6692worker1 # [ 192.557118] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=14 outputs=16693worker1 # [ 192.574180] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:11Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=16694worker1 # [ 192.579171] nix-daemon[847]: reaped child process 2051, status = succeeded6695worker1 # [ 192.598483] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946696worker1 # [ 192.612847] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460946697worker1 # [ 192.625827] nix-grpc-daemon[1583]: time=2026-09-22T11:28:11.236Z level=INFO msg="All 2 paths already cached"6698worker1 # [ 192.634175] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:460946699worker1 # [ 192.646406] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6700worker1 # [ 192.795032] nix-daemon[847]: accepted connection from pid 2076, user nix-grpc-daemon (trusted)6701worker1 # [ 192.811808] nix-daemon[847]: reaped child process 2079, status = succeeded6702worker1 # [ 192.913239] nix-daemon[847]: accepted connection from pid 2081, user nix-grpc-daemon (trusted)6703worker1 # [ 192.929937] nix-daemon[847]: reaped child process 2084, status = succeeded6704lb # [ 192.874242] niks3-server[1098]: 2026/09/22 11:28:11 INFO Received uploads request method=POST path=/api/pending_closures6705worker1 # [ 192.945510] nix-grpc-daemon[1583]: time=2026-09-22T11:28:11.556Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6706worker1 # [ 192.946985] nix-grpc-daemon[1583]: time=2026-09-22T11:28:11.556Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)"6707worker1 # [ 192.977515] nix-grpc-daemon[1583]: time=2026-09-22T11:28:11.588Z level=INFO msg="Uploading 1 narinfos"6708lb # [ 192.904676] niks3-server[1098]: 2026/09/22 11:28:11 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst6709lb # [ 192.919707] niks3-server[1098]: 2026/09/22 11:28:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign6710lb # [ 192.930416] niks3-server[1098]: 2026/09/22 11:28:11 INFO Signed narinfos id=72 count=16711lb # [ 192.936806] niks3-server[1098]: 2026/09/22 11:28:11 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls6712lb # [ 192.947494] niks3-server[1098]: 2026/09/22 11:28:11 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete6713lb # [ 192.953730] niks3-server[1098]: 2026/09/22 11:28:11 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo6714lb # [ 192.958877] niks3-server[1098]: 2026/09/22 11:28:11 INFO Completed upload id=726715worker1 # [ 193.021560] nix-grpc-daemon[1583]: time=2026-09-22T11:28:11.632Z level=INFO msg="Upload complete. (360ms)"6716worker1 # [ 193.028276] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=0 paths=1 nar_bytes_in=7526717worker1 # [ 193.044056] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:460946718worker1 # [ 193.052315] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6719worker1 # [ 193.055202] nix-daemon[847]: reaped child process 2074, status = succeeded6720worker1 # [ 193.067840] nix-daemon[847]: accepted connection from pid 1487, user nix-grpc-daemon (trusted)6721worker1 # [ 194.132711] nix-daemon[847]: reaped child process 2088, status = succeeded6722worker1 # [ 194.231785] nix-daemon[847]: accepted connection from pid 2104, user nix-grpc-daemon (trusted)6723worker1 # [ 194.252311] nix-daemon[847]: reaped child process 2107, status = succeeded6724lb # [ 194.189733] niks3-server[1098]: 2026/09/22 11:28:12 INFO Received uploads request method=POST path=/api/pending_closures6725worker1 # [ 194.262638] nix-grpc-daemon[1583]: time=2026-09-22T11:28:12.872Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6726worker1 # [ 194.264199] nix-grpc-daemon[1583]: time=2026-09-22T11:28:12.872Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)"6727lb # [ 194.231199] niks3-server[1098]: 2026/09/22 11:28:12 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst6728worker1 # [ 194.307684] nix-grpc-daemon[1583]: time=2026-09-22T11:28:12.918Z level=INFO msg="Uploading 1 narinfos"6729lb # [ 194.246189] niks3-server[1098]: 2026/09/22 11:28:12 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv6730lb # [ 194.256588] niks3-server[1098]: 2026/09/22 11:28:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign6731lb # [ 194.264253] niks3-server[1098]: 2026/09/22 11:28:12 INFO Signed narinfos id=73 count=16732lb # [ 194.266730] niks3-server[1098]: 2026/09/22 11:28:12 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls6733lb # [ 194.276122] niks3-server[1098]: 2026/09/22 11:28:12 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete6734lb # [ 194.280812] niks3-server[1098]: 2026/09/22 11:28:12 INFO Completed upload id=736735lb # [ 194.283427] niks3-server[1098]: 2026/09/22 11:28:12 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo6736worker1 # [ 194.343196] nix-grpc-daemon[1583]: time=2026-09-22T11:28:12.954Z level=INFO msg="Upload complete. (265ms)"6737worker1 # [ 194.355516] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46094 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=15 outputs=16738worker1 # [ 194.375043] nix-daemon[847]: reaped child process 2086, status = succeeded6739client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 18.33 seconds)6740worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6741worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6742client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]'6743client # [ 194.670580] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6744client # Running as unit: gpu.service; invocation ID: 8701938f9c2f4fa888d5d2d8758060db6745client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds)6746lb: must succeed: curl -sf localhost:9901/clusters6747lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6748lb: must succeed: curl -sf localhost:9901/clusters6749lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6750worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6751client # [ 194.882591] nix[1533]: don't know how to build these paths:6752worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6753client # [ 194.883939] nix[1533]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv6754worker2 # [ 194.738581] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:519126755worker1 # [ 194.805193] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460946756client # [ 194.910020] nix[1533]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, waiting up to 30s for a worker6757worker1 # [ 194.826558] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:13Z level=debug event=want client=6 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=06758lb: must succeed: curl -sf localhost:9901/clusters6759lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6760lb: must succeed: curl -sf localhost:9901/clusters6761lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6762worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6763worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6764client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6765lb # [ 198.560880] envoy[908]: conn peer=[2001:db8:1::1]:50080 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27101 tx=11348 ms=180566766client # [ 224.915667] nix[1533]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}6767client # [ 224.932155] nix[1533]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed6768client # [ 224.968202] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6769client # [ 224.976334] systemd[1]: gpu.service: Failed with result 'exit-code'.6770client # [ 224.979894] systemd[1]: gpu.service: Consumed 209ms CPU time over 30.303s wall clock time, 37.4M memory peak, 5.1K incoming IP traffic, 5K outgoing IP traffic.6771client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.12 seconds)6772client: must succeed: journalctl -u gpu -o cat6773client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)6774lb: must succeed: curl -sf localhost:9901/clusters6775lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6776lb: must succeed: curl -sf localhost:9901/clusters6777lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6778worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6779worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6780(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 49.19 seconds)6781subtest: low disk drains a worker and builds go to the other6782worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6783worker1 # [ 226.465513] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:45Z level=info event=unhealthy reason=min_free6784worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.18 seconds)6785worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6786worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 0.03 seconds)6787client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&26788worker1 # [ 226.764594] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:45Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:460946789client # don't know how to build these paths:6790client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6791worker1 # [ 226.776935] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:460946792worker1 # [ 226.804112] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:45Z level=debug event=want client=7 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=06793worker1 # [ 226.810917] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:45Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=16794worker2 # [ 226.762150] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:519126795worker2 # [ 226.781856] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:519126796client # copying 1 paths to 'lb:50051'...6797client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'...6798worker2 # [ 226.799056] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:519126799worker2 # [ 226.818240] nix-daemon[836]: accepted connection from pid 1986, user nix-grpc-daemon (trusted)6800worker2 # [ 226.844751] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:45Z level=info event=niks3_push_started pid=20466801worker2 # [ 226.993154] nix-grpc-daemon[2046]: time=2026-09-22T11:28:45.607Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""6802worker2 # [ 227.098700] nix-daemon[836]: accepted connection from pid 2058, user nix-grpc-daemon (trusted)6803worker2 # [ 227.121341] nix-daemon[836]: reaped child process 2061, status = succeeded6804worker2 # [ 227.215373] nix-daemon[836]: accepted connection from pid 2063, user nix-grpc-daemon (trusted)6805worker2 # [ 227.232170] nix-daemon[836]: reaped child process 2066, status = succeeded6806lb # [ 227.238999] niks3-server[1098]: 2026/09/22 11:28:45 INFO Received uploads request method=POST path=/api/pending_closures6807worker2 # [ 227.252229] nix-grpc-daemon[2046]: time=2026-09-22T11:28:45.867Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6808worker2 # [ 227.255442] nix-grpc-daemon[2046]: time=2026-09-22T11:28:45.868Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)"6809lb # [ 227.282157] niks3-server[1098]: 2026/09/22 11:28:45 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst6810worker2 # [ 227.308451] nix-grpc-daemon[2046]: time=2026-09-22T11:28:45.922Z level=INFO msg="Uploading 1 narinfos"6811lb # [ 227.306375] niks3-server[1098]: 2026/09/22 11:28:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign6812lb # [ 227.316159] niks3-server[1098]: 2026/09/22 11:28:45 INFO Signed narinfos id=74 count=16813lb # [ 227.318533] niks3-server[1098]: 2026/09/22 11:28:45 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls6814lb # [ 227.340240] niks3-server[1098]: 2026/09/22 11:28:45 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete6815lb # [ 227.347893] niks3-server[1098]: 2026/09/22 11:28:45 INFO Completed upload id=746816worker2 # [ 227.349645] nix-grpc-daemon[2046]: time=2026-09-22T11:28:45.964Z level=INFO msg="Upload complete. (352ms)"6817lb # [ 227.352528] niks3-server[1098]: 2026/09/22 11:28:45 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo6818worker2 # [ 227.360410] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51912 duration_s=0 paths=1 nar_bytes_in=5126819worker2 # [ 227.379840] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:519126820worker2 # [ 227.392412] nix-daemon[836]: accepted connection from pid 1986, user nix-grpc-daemon (trusted)6821worker2 # [ 227.395885] nix-daemon[836]: reaped child process 2044, status = succeeded6822client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051...6823client # farm-a> worker2-next: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6824worker2 # [ 227.411534] nix-daemon[836]: accepted connection from pid 1986, user nix-grpc-daemon (trusted)6825client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6826client # farm-a> LOG-farm-a-drain6827worker2 # [ 228.236479] nix-daemon[836]: reaped child process 2070, status = succeeded6828worker2 # [ 228.333715] nix-daemon[836]: accepted connection from pid 2086, user nix-grpc-daemon (trusted)6829worker2 # [ 228.353783] nix-daemon[836]: reaped child process 2089, status = succeeded6830lb # [ 228.351768] niks3-server[1098]: 2026/09/22 11:28:46 INFO Received uploads request method=POST path=/api/pending_closures6831worker2 # [ 228.368328] nix-grpc-daemon[2046]: time=2026-09-22T11:28:46.980Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6832worker2 # [ 228.369752] nix-grpc-daemon[2046]: time=2026-09-22T11:28:46.980Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)"6833lb # [ 228.387633] niks3-server[1098]: 2026/09/22 11:28:46 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6834lb # [ 228.405960] niks3-server[1098]: 2026/09/22 11:28:47 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6835lb # [ 228.419483] niks3-server[1098]: 2026/09/22 11:28:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign6836worker2 # [ 228.425857] nix-grpc-daemon[2046]: time=2026-09-22T11:28:47.039Z level=INFO msg="Uploading 1 narinfos"6837lb # [ 228.427899] niks3-server[1098]: 2026/09/22 11:28:47 INFO Signed narinfos id=75 count=16838lb # [ 228.433440] niks3-server[1098]: 2026/09/22 11:28:47 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls6839lb # [ 228.444797] niks3-server[1098]: 2026/09/22 11:28:47 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete6840lb # [ 228.450623] niks3-server[1098]: 2026/09/22 11:28:47 INFO Completed upload id=756841worker2 # [ 228.452196] nix-grpc-daemon[2046]: time=2026-09-22T11:28:47.066Z level=INFO msg="Upload complete. (231ms)"6842lb # [ 228.454309] niks3-server[1098]: 2026/09/22 11:28:47 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo6843worker2 # [ 228.464876] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51912 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=16 outputs=16844worker1 # [ 228.530745] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:47Z level=debug event=want client=7 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=06845worker1 # [ 228.536709] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:47Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=16846worker2 # [ 228.479918] nix-daemon[836]: reaped child process 2068, status = succeeded6847worker2 # [ 228.514564] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:519126848worker2 # [ 228.530875] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:519126849worker2 # [ 228.545230] nix-grpc-daemon[2046]: time=2026-09-22T11:28:47.159Z level=INFO msg="All 1 paths already cached"6850client # copying 1 paths to 'lb:50051'...6851client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'...6852worker2 # [ 228.555461] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:519126853lb # [ 228.559164] envoy[908]: conn peer=[2001:db8:1::1]:54284 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=897 tx=499 ms=300596854worker2 # [ 228.568670] nix-daemon[836]: accepted connection from pid 1986, user nix-grpc-daemon (trusted)6855worker2 # [ 228.724507] nix-daemon[836]: accepted connection from pid 2093, user nix-grpc-daemon (trusted)6856worker2 # [ 228.741369] nix-daemon[836]: reaped child process 2096, status = succeeded6857worker2 # [ 228.839108] nix-daemon[836]: accepted connection from pid 2098, user nix-grpc-daemon (trusted)6858worker2 # [ 228.853303] nix-daemon[836]: reaped child process 2101, status = succeeded6859lb # [ 228.859439] niks3-server[1098]: 2026/09/22 11:28:47 INFO Received uploads request method=POST path=/api/pending_closures6860worker2 # [ 228.869452] nix-grpc-daemon[2046]: time=2026-09-22T11:28:47.483Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6861worker2 # [ 228.870981] nix-grpc-daemon[2046]: time=2026-09-22T11:28:47.483Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)"6862lb # [ 228.886350] niks3-server[1098]: 2026/09/22 11:28:47 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst6863worker2 # [ 228.903314] nix-grpc-daemon[2046]: time=2026-09-22T11:28:47.517Z level=INFO msg="Uploading 1 narinfos"6864lb # [ 228.901988] niks3-server[1098]: 2026/09/22 11:28:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign6865lb # [ 228.911784] niks3-server[1098]: 2026/09/22 11:28:47 INFO Signed narinfos id=76 count=16866lb # [ 228.917090] niks3-server[1098]: 2026/09/22 11:28:47 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls6867lb # [ 228.930129] niks3-server[1098]: 2026/09/22 11:28:47 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete6868worker2 # [ 228.936346] nix-grpc-daemon[2046]: time=2026-09-22T11:28:47.551Z level=INFO msg="Upload complete. (351ms)"6869lb # [ 228.938122] niks3-server[1098]: 2026/09/22 11:28:47 INFO Completed upload id=766870lb # [ 228.943519] niks3-server[1098]: 2026/09/22 11:28:47 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo6871worker2 # [ 228.944353] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51912 duration_s=0 paths=1 nar_bytes_in=6326872worker2 # [ 228.957982] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:519126873worker2 # [ 228.964250] nix-daemon[836]: accepted connection from pid 1986, user nix-grpc-daemon (trusted)6874worker2 # [ 228.969111] nix-daemon[836]: reaped child process 2091, status = succeeded6875client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051...6876worker2 # [ 228.981452] nix-daemon[836]: accepted connection from pid 1986, user nix-grpc-daemon (trusted)6877client # farm-b> worker2-next: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6878client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6879client # farm-b> LOG-farm-b-drain6880worker2 # [ 230.034669] nix-daemon[836]: reaped child process 2105, status = succeeded6881worker2 # [ 230.145781] nix-daemon[836]: accepted connection from pid 2121, user nix-grpc-daemon (trusted)6882worker2 # [ 230.165313] nix-daemon[836]: reaped child process 2124, status = succeeded6883lb # [ 230.170637] niks3-server[1098]: 2026/09/22 11:28:48 INFO Received uploads request method=POST path=/api/pending_closures6884worker2 # [ 230.183578] nix-grpc-daemon[2046]: time=2026-09-22T11:28:48.797Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6885worker2 # [ 230.185544] nix-grpc-daemon[2046]: time=2026-09-22T11:28:48.797Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)"6886lb # [ 230.203207] niks3-server[1098]: 2026/09/22 11:28:48 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6887lb # [ 230.217959] niks3-server[1098]: 2026/09/22 11:28:48 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst6888lb # [ 230.232114] niks3-server[1098]: 2026/09/22 11:28:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign6889worker2 # [ 230.240384] nix-grpc-daemon[2046]: time=2026-09-22T11:28:48.854Z level=INFO msg="Uploading 1 narinfos"6890lb # [ 230.242269] niks3-server[1098]: 2026/09/22 11:28:48 INFO Signed narinfos id=77 count=16891lb # [ 230.249600] niks3-server[1098]: 2026/09/22 11:28:48 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls6892lb # [ 230.259547] niks3-server[1098]: 2026/09/22 11:28:48 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete6893lb # [ 230.267533] niks3-server[1098]: 2026/09/22 11:28:48 INFO Completed upload id=776894worker2 # [ 230.269192] nix-grpc-daemon[2046]: time=2026-09-22T11:28:48.883Z level=INFO msg="Upload complete. (277ms)"6895lb # [ 230.274668] niks3-server[1098]: 2026/09/22 11:28:48 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo6896worker2 # [ 230.281285] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51912 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=17 outputs=16897worker1 # [ 230.351110] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:48Z level=debug event=want client=7 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=06898worker1 # [ 230.357211] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:48Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=16899worker2 # [ 230.302010] nix-daemon[836]: reaped child process 2103, status = succeeded6900worker2 # [ 230.334469] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:519126901worker2 # [ 230.350047] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:519126902worker2 # [ 230.364558] nix-grpc-daemon[2046]: time=2026-09-22T11:28:48.979Z level=INFO msg="All 2 paths already cached"6903client # copying 1 paths to 'lb:50051'...6904client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'...6905worker2 # [ 230.375731] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:519126906worker2 # [ 230.389423] nix-daemon[836]: accepted connection from pid 1986, user nix-grpc-daemon (trusted)6907worker2 # [ 230.541857] nix-daemon[836]: accepted connection from pid 2128, user nix-grpc-daemon (trusted)6908worker2 # [ 230.564303] nix-daemon[836]: reaped child process 2131, status = succeeded6909worker2 # [ 230.660630] nix-daemon[836]: accepted connection from pid 2133, user nix-grpc-daemon (trusted)6910worker2 # [ 230.681293] nix-daemon[836]: reaped child process 2136, status = succeeded6911lb # [ 230.677771] niks3-server[1098]: 2026/09/22 11:28:49 INFO Received uploads request method=POST path=/api/pending_closures6912worker2 # [ 230.687626] nix-grpc-daemon[2046]: time=2026-09-22T11:28:49.302Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6913worker2 # [ 230.689761] nix-grpc-daemon[2046]: time=2026-09-22T11:28:49.304Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)"6914lb # [ 230.706038] niks3-server[1098]: 2026/09/22 11:28:49 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst6915worker2 # [ 230.721293] nix-grpc-daemon[2046]: time=2026-09-22T11:28:49.335Z level=INFO msg="Uploading 1 narinfos"6916lb # [ 230.720750] niks3-server[1098]: 2026/09/22 11:28:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign6917lb # [ 230.732570] niks3-server[1098]: 2026/09/22 11:28:49 INFO Signed narinfos id=78 count=16918lb # [ 230.738566] niks3-server[1098]: 2026/09/22 11:28:49 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls6919lb # [ 230.750193] niks3-server[1098]: 2026/09/22 11:28:49 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete6920worker2 # [ 230.757311] nix-grpc-daemon[2046]: time=2026-09-22T11:28:49.372Z level=INFO msg="Upload complete. (351ms)"6921lb # [ 230.756144] niks3-server[1098]: 2026/09/22 11:28:49 INFO Completed upload id=786922worker2 # [ 230.765104] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51912 duration_s=0 paths=1 nar_bytes_in=7606923lb # [ 230.762142] niks3-server[1098]: 2026/09/22 11:28:49 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo6924worker2 # [ 230.776904] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:519126925worker2 # [ 230.786119] nix-daemon[836]: accepted connection from pid 1986, user nix-grpc-daemon (trusted)6926worker2 # [ 230.791321] nix-daemon[836]: reaped child process 2126, status = succeeded6927client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051...6928worker2 # [ 230.798445] nix-daemon[836]: accepted connection from pid 1986, user nix-grpc-daemon (trusted)6929client # farm-top> worker2-next: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6930client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6931client # farm-top> LOG-farm-top-drain6932worker2 # [ 231.635101] nix-daemon[836]: reaped child process 2140, status = succeeded6933worker2 # [ 231.752101] nix-daemon[836]: accepted connection from pid 2156, user nix-grpc-daemon (trusted)6934worker2 # [ 231.769858] nix-daemon[836]: reaped child process 2159, status = succeeded6935lb # [ 231.775611] niks3-server[1098]: 2026/09/22 11:28:50 INFO Received uploads request method=POST path=/api/pending_closures6936worker2 # [ 231.789484] nix-grpc-daemon[2046]: time=2026-09-22T11:28:50.402Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6937worker2 # [ 231.791057] nix-grpc-daemon[2046]: time=2026-09-22T11:28:50.402Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)"6938lb # [ 231.811015] niks3-server[1098]: 2026/09/22 11:28:50 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst6939worker2 # [ 231.831402] nix-grpc-daemon[2046]: time=2026-09-22T11:28:50.445Z level=INFO msg="Uploading 1 narinfos"6940lb # [ 231.829442] niks3-server[1098]: 2026/09/22 11:28:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign6941lb # [ 231.843055] niks3-server[1098]: 2026/09/22 11:28:50 INFO Signed narinfos id=79 count=16942lb # [ 231.849055] niks3-server[1098]: 2026/09/22 11:28:50 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6943lb # [ 231.858205] niks3-server[1098]: 2026/09/22 11:28:50 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls6944lb # [ 231.869283] niks3-server[1098]: 2026/09/22 11:28:50 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete6945worker2 # [ 231.870314] nix-grpc-daemon[2046]: time=2026-09-22T11:28:50.485Z level=INFO msg="Upload complete. (261ms)"6946lb # [ 231.874812] niks3-server[1098]: 2026/09/22 11:28:50 INFO Completed upload id=796947lb # [ 231.878246] niks3-server[1098]: 2026/09/22 11:28:50 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo6948worker2 # [ 231.885734] nix-grpc-daemon[1986]: ts=2026-09-22T11:28:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51912 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=18 outputs=16949worker2 # [ 231.902021] nix-daemon[836]: reaped child process 2138, status = succeeded6950client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2, in 5.46 seconds)6951client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath6952client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath, in 0.18 seconds)6953worker1: must succeed: rm /nix/.rw-store/fill6954worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.15 seconds)6955worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'6956worker1 # [ 232.467401] nix-grpc-daemon[1487]: ts=2026-09-22T11:28:51Z level=info event=healthy reason=min_free6957worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.10 seconds)6958(finished: subtest: low disk drains a worker and builds go to the other, in 8.20 seconds)6959(finished: run the VM test script, in 234.36 seconds)6960test script finished in 234.61s6961cleanup6962kill QemuMachine (pid 49)6963client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6964client # [2026-09-22T11:28:52Z INFO virtiofsd] Client disconnected, shutting down6965client # [2026-09-22T11:28:52Z INFO virtiofsd] Client disconnected, shutting down6966client # [2026-09-22T11:28:52Z INFO virtiofsd] Client disconnected, shutting down6967kill QemuMachine (pid 50)6968lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6969lb # [2026-09-22T11:28:53Z INFO virtiofsd] Client disconnected, shutting down6970lb # [2026-09-22T11:28:53Z INFO virtiofsd] Client disconnected, shutting down6971lb # [2026-09-22T11:28:53Z INFO virtiofsd] Client disconnected, shutting down6972kill QemuMachine (pid 51)6973worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6974worker1 # [2026-09-22T11:28:53Z INFO virtiofsd] Client disconnected, shutting down6975worker1 # [2026-09-22T11:28:53Z INFO virtiofsd] Client disconnected, shutting down6976worker1 # [2026-09-22T11:28:53Z INFO virtiofsd] Client disconnected, shutting down6977kill QemuMachine (pid 52)6978worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6979worker2 # [2026-09-22T11:28:53Z INFO virtiofsd] Client disconnected, shutting down6980worker2 # [2026-09-22T11:28:53Z INFO virtiofsd] Client disconnected, shutting down6981worker2 # [2026-09-22T11:28:53Z INFO virtiofsd] Client disconnected, shutting down6982(finished: cleanup, in 1.41 seconds)6983additionally exposed symbols:6984 client, lb, worker1, worker2,6985 vlan1,6986 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh