vm-test-run-nix-grpc-farm
checks.aarch64-linux.farm
· build #314
· raw
1tribuchet: building on eliza2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.01 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12client # Disk image does not exist, creating the virtualisation disk image...13client: QEMU running (pid 47)14client # Formatting '/build/vm-state-client/tmp.2JMM0S3H3y', fmt=raw size=107374182415client # mke2fs 1.47.4 (6-Mar-2025)16client # Discarding device blocks: 0/262144 done17client # Creating filesystem with 262144 4k blocks and 65536 inodes18client # Filesystem UUID: 32f2a617-6fbd-4a16-9ba5-91b9449d1caa19client # Superblock backups stored on blocks:20client # 32768, 98304, 163840, 22937621client # 22client # Allocating group tables: 0/8 done23client # Writing inode tables: 0/8 done24client # Creating journal (8192 blocks): done25client # Writing superblocks and filesystem accounting information: 0/8 done26client # 27client # Virtualisation disk image created.28lb # Disk image does not exist, creating the virtualisation disk image...29lb # Formatting '/build/vm-state-lb/tmp.MoTCHbbL5O', fmt=raw size=107374182430lb # mke2fs 1.47.4 (6-Mar-2025)31lb # Discarding device blocks: 0/262144 done32lb # Creating filesystem with 262144 4k blocks and 65536 inodes33lb # Filesystem UUID: fa3e8856-a0fc-4d3d-b184-d1a9aa0fd7fb34lb # Superblock backups stored on blocks:35lb # 32768, 98304, 163840, 22937636lb # 37lb # Allocating group tables: 0/8 done38lb # Writing inode tables: 0/8 done39lb # Creating journal (8192 blocks): done40lb # Writing superblocks and filesystem accounting information: 0/8 done41lb # 42lb # Virtualisation disk image created.43lb: QEMU running (pid 48)44worker1 # Disk image does not exist, creating the virtualisation disk image...45worker1 # Formatting '/build/vm-state-worker1/tmp.HF8Hg2fLQB', fmt=raw size=107374182446worker1 # mke2fs 1.47.4 (6-Mar-2025)47worker1 # Discarding device blocks: 0/262144 done48worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes49worker1: QEMU running (pid 51)50worker1 # Filesystem UUID: 69078a81-8b9f-412b-ab7e-fd25acca65d451worker1 # Superblock backups stored on blocks:52worker1 # 32768, 98304, 163840, 22937653worker1 # 54worker1 # Allocating group tables: 0/8 done55worker1 # Writing inode tables: 0/8 done56worker1 # Creating journal (8192 blocks): done57worker1 # Writing superblocks and filesystem accounting information: 0/8 done58worker1 # 59worker1 # Virtualisation disk image created.60worker2 # Disk image does not exist, creating the virtualisation disk image...61worker2 # Formatting '/build/vm-state-worker2/tmp.eR6CEbyeyf', fmt=raw size=107374182462worker2 # mke2fs 1.47.4 (6-Mar-2025)63worker2 # Discarding device blocks: 0/262144 done64worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes65worker2 # Filesystem UUID: a2eb04b8-245c-4270-8a4f-96b8e59e516166worker2 # Superblock backups stored on blocks:67worker2 # 32768, 98304, 163840, 22937668worker2 # 69worker2 # Allocating group tables: 0/8 done70worker2 # Writing inode tables: 0/8 done71worker2 # Creating journal (8192 blocks): done72worker2 # Writing superblocks and filesystem accounting information: 0/8 done73worker2 # 74worker2 # Virtualisation disk image created.75worker2: QEMU running (pid 53)76(finished: start all VMs, in 3.23 seconds)77client: waiting for unit network-addresses-eth1.service78client: waiting for the VM to finish booting79client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]80client # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 202681client # [ 0.000000] KASLR enabled82client # [ 0.000000] random: crng init done83client # [ 0.000000] Machine model: linux,dummy-virt84client # [ 0.000000] efi: UEFI not found.85client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT86client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]87client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]88client # [ 0.000000] Zone ranges:89client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]90client # [ 0.000000] DMA32 empty91client # [ 0.000000] Normal empty92client # [ 0.000000] Device empty93client # [ 0.000000] Movable zone start for each node94client # [ 0.000000] Early memory node ranges95client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]96client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]97client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c40000098client # [ 0.000000] psci: probing for conduit method from DT.99client # [ 0.000000] psci: PSCIv1.3 detected in firmware.100client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs101client # [ 0.000000] psci: Trusted OS migration not required102client # [ 0.000000] psci: SMC Calling Convention v1.1103client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)104client # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296105client # [ 0.000000] Detected PIPT I-cache on CPU0106client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)107client # [ 0.000000] CPU features: detected: GICv3 CPU interface108client # [ 0.000000] CPU features: detected: Spectre-v4109client # [ 0.000000] CPU features: detected: Spectre-BHB110client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38111client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23112client # [ 0.000000] alternatives: applying boot alternatives113client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/a8gc22jhjd6rmf0wand1fsy50rgzy95i-nixos-system-client-test/init regInfo=/nix/store/wd1ka8jlsww7njwm410558vz4bsfdhn5-closure-info/registration console=ttyAMA0,115200n8 console=tty0114client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/wd1ka8jlsww7njwm410558vz4bsfdhn5-closure-info/registration", will be passed to user space.115client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes116client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)117client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)118client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB119client # [ 0.000000] software IO TLB: area num 1.120client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB121client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)122client # [ 0.000000] Fallback order for Node 0: 0123client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216124client # [ 0.000000] Policy zone: DMA125client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off126client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1127client # [ 0.000000] allocated 3145728 bytes of page_ext128client # [ 0.000000] ftrace: allocating 74885 entries in 294 pages129client # [ 0.000000] ftrace: allocated 294 pages with 4 groups130client # [ 0.000000] rcu: Hierarchical RCU implementation.131client # [ 0.000000] rcu: RCU event tracing is enabled.132client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.133client # [ 0.000000] Trampoline variant of Tasks RCU enabled.134client # [ 0.000000] Rude variant of Tasks RCU enabled.135client # [ 0.000000] Tracing variant of Tasks RCU enabled.136client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.137client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1138client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.139client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.140client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.141client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0142client # [ 0.000000] GICv3: 256 SPIs implemented143client # [ 0.000000] GICv3: 0 Extended SPIs implemented144client # [ 0.000000] Root IRQ handler: gic_handle_irq145client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI146client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0147client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000148client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]149client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)150client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)151client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000152client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000153client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.154client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns155client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).156client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns157client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns158client # [ 0.000050] arm-pv: using stolen time PV159client # [ 0.001015] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)160client # [ 0.001224] Console: colour dummy device 80x25161client # [ 0.001235] printk: legacy console [tty0] enabled162client # [ 0.001446] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)163client # [ 0.001453] pid_max: default: 32768 minimum: 301164client # [ 0.001537] LSM: initializing lsm=capability,landlock,yama,bpf,ima165client # [ 0.001777] landlock: Up and running.166client # [ 0.001780] Yama: becoming mindful.167client # [ 0.002766] LSM support for eBPF active168client # [ 0.003040] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)169client # [ 0.003073] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)170client # [ 0.005109] cacheinfo: Unable to detect cache hierarchy for CPU 0171client # [ 0.005900] rcu: Hierarchical SRCU implementation.172client # [ 0.005904] rcu: Max phase no-delay instances is 1000.173client # [ 0.007182] fsl-mc MSI: its@8080000 domain created174client # [ 0.007283] EFI services will not be available.175client # [ 0.007418] smp: Bringing up secondary CPUs ...176client # [ 0.007428] smp: Brought up 1 node, 1 CPU177client # [ 0.007432] SMP: Total of 1 processors activated.178client # [ 0.007437] CPU: All CPU(s) started at EL1179client # [ 0.007453] CPU features: detected: Branch Target Identification180client # [ 0.007458] CPU features: detected: ARMv8.4 Translation Table Level181client # [ 0.007461] CPU features: detected: Instruction cache invalidation not required for I/D coherence182client # [ 0.007465] CPU features: detected: Data cache clean to the PoU not required for I/D coherence183client # [ 0.007469] CPU features: detected: Common not Private translations184client # [ 0.007472] CPU features: detected: CRC32 instructions185client # [ 0.007474] CPU features: detected: Data cache clean to Point of Deep Persistence186client # [ 0.007478] CPU features: detected: Data cache clean to Point of Persistence187client # [ 0.007482] CPU features: detected: Data independent timing control (DIT)188client # [ 0.007485] CPU features: detected: E0PD189client # [ 0.007487] CPU features: detected: Enhanced Counter Virtualization190client # [ 0.007490] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)191client # [ 0.007493] CPU features: detected: Enhanced Virtualization Traps192client # [ 0.007496] CPU features: detected: Fine Grained Traps193client # [ 0.007500] CPU features: detected: Generic authentication (architected QARMA5 algorithm)194client # [ 0.007506] CPU features: detected: RCpc load-acquire (LDAPR)195client # [ 0.007509] CPU features: detected: LSE atomic instructions196client # [ 0.007512] CPU features: detected: Privileged Access Never197client # [ 0.007514] CPU features: detected: PMUv3198client # [ 0.007517] CPU features: detected: RAS Extension Support199client # [ 0.007520] CPU features: detected: RASv1p1 Extension Support200client # [ 0.007522] CPU features: detected: Random Number Generator201client # [ 0.007525] CPU features: detected: Speculation barrier (SB)202client # [ 0.007528] CPU features: detected: Stage-2 Force Write-Back203client # [ 0.007531] CPU features: detected: TLB range maintenance instructions204client # [ 0.007536] CPU features: detected: Speculative Store Bypassing Safe (SSBS)205client # [ 0.007578] alternatives: applying system-wide alternatives206client # [ 0.010734] CPU features: detected: BBM Level 2 without TLB conflict abort207client # [ 0.010910] Memory: 1406512K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123784K reserved, 32768K cma-reserved)208client # [ 0.011678] devtmpfs: initialized209client # [ 0.013556] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)210client # [ 0.013586] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).211client # [ 0.013826] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL212client # [ 0.013831] 0 pages in range for non-PLT usage213client # [ 0.013832] 508288 pages in range for PLT usage214client # [ 0.013942] pinctrl core: initialized pinctrl subsystem215client # [ 0.014850] DMI not present or invalid.216client # [ 0.018970] NET: Registered PF_NETLINK/PF_ROUTE protocol family217client # [ 0.021434] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations218client # [ 0.021656] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations219client # [ 0.021871] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations220client # [ 0.021908] audit: initializing netlink subsys (disabled)221client # [ 0.022620] thermal_sys: Registered thermal governor 'fair_share'222client # [ 0.022623] thermal_sys: Registered thermal governor 'bang_bang'223client # [ 0.022627] thermal_sys: Registered thermal governor 'step_wise'224client # [ 0.022632] thermal_sys: Registered thermal governor 'user_space'225client # [ 0.022635] thermal_sys: Registered thermal governor 'power_allocator'226client # [ 0.022673] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1227client # [ 0.022684] cpuidle: using governor ladder228client # [ 0.022689] cpuidle: using governor menu229client # [ 0.022915] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.230client # [ 0.022955] ASID allocator initialised with 65536 entries231client # [ 0.024393] Serial: AMBA PL011 UART driver232client # [ 0.029988] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1233client # [ 0.030174] printk: console [ttyAMA0] enabled234client # [ 0.159705] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages235client # [ 0.159733] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page236client # [ 0.159738] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages237client # [ 0.159743] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page238client # [ 0.159747] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages239client # [ 0.159751] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page240client # [ 0.159755] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages241client # [ 0.159759] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page242client # [ 0.237551] fbcon: Taking over console243client # [ 0.237590] ACPI: Interpreter disabled.244client # [ 0.239655] iommu: Default domain type: Translated245client # [ 0.239669] iommu: DMA domain TLB invalidation policy: strict mode246client # [ 0.241477] SCSI subsystem initialized247client # [ 0.268576] usbcore: registered new interface driver usbfs248client # [ 0.268638] usbcore: registered new interface driver hub249client # [ 0.268666] usbcore: registered new device driver usb250client # [ 0.269065] pps_core: LinuxPPS API ver. 1 registered251client # [ 0.269072] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>252client # [ 0.269083] PTP clock support registered253client # [ 0.269134] EDAC MC: Ver: 3.0.0254client # [ 0.282238] scmi_core: SCMI protocol bus registered255client # [ 0.283515] FPGA manager framework256client # [ 0.284609] vgaarb: loaded257client # [ 0.285328] clocksource: Switched to clocksource arch_sys_counter258client # [ 0.286075] VFS: Disk quotas dquot_6.6.0259client # [ 0.286112] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)260client # [ 0.291028] netfs: FS-Cache loaded261client # [ 0.291196] pnp: PnP ACPI: disabled262client # [ 0.295590] NET: Registered PF_INET protocol family263client # [ 0.295908] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)264lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]265lb # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026266client # [ 0.330984] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)267lb # [ 0.000000] KASLR enabled268lb # [ 0.000000] random: crng init done269client # [ 0.331053] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)270lb # [ 0.000000] Machine model: linux,dummy-virt271lb # [ 0.000000] efi: UEFI not found.272client # [ 0.331087] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)273lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT274client # [ 0.331181] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)275lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]276client # [ 0.331325] TCP: Hash tables configured (established 16384 bind 16384)277lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]278lb # [ 0.000000] Zone ranges:279client # [ 0.331430] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)280lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]281client # [ 0.331472] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)282lb # [ 0.000000] DMA32 empty283lb # [ 0.000000] Normal empty284lb # [ 0.000000] Device empty285client # [ 0.331513] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)286lb # [ 0.000000] Movable zone start for each node287client # [ 0.331616] NET: Registered PF_UNIX/PF_LOCAL protocol family288lb # [ 0.000000] Early memory node ranges289client # [ 0.331643] NET: Registered PF_XDP protocol family290lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]291client # [ 0.331662] PCI: CLS 0 bytes, default 64292client # [ 0.331931] Trying to unpack rootfs image as initramfs...293lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]294lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000295lb # [ 0.000000] psci: probing for conduit method from DT.296lb # [ 0.000000] psci: PSCIv1.3 detected in firmware.297lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs298lb # [ 0.000000] psci: Trusted OS migration not required299lb # [ 0.000000] psci: SMC Calling Convention v1.1300lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)301lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296302lb # [ 0.000000] Detected PIPT I-cache on CPU0303client # [ 0.347946] kvm [1]: HYP mode not available304lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)305lb # [ 0.000000] CPU features: detected: GICv3 CPU interface306lb # [ 0.000000] CPU features: detected: Spectre-v4307lb # [ 0.000000] CPU features: detected: Spectre-BHB308lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38309lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23310lb # [ 0.000000] alternatives: applying boot alternatives311lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/lai4n7chb0cnvnygfis27sipihspygia-nixos-system-lb-test/init regInfo=/nix/store/qdg5w64gnxykmz82pw25fgmx46q7lp99-closure-info/registration console=ttyAMA0,115200n8 console=tty0312lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/qdg5w64gnxykmz82pw25fgmx46q7lp99-closure-info/registration", will be passed to user space.313lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes314lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)315lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)316lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB317lb # [ 0.000000] software IO TLB: area num 1.318lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB319lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)320lb # [ 0.000000] Fallback order for Node 0: 0321lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216322lb # [ 0.000000] Policy zone: DMA323lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off324lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1325lb # [ 0.000000] allocated 3145728 bytes of page_ext326lb # [ 0.000000] ftrace: allocating 74885 entries in 294 pages327lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups328lb # [ 0.000000] rcu: Hierarchical RCU implementation.329lb # [ 0.000000] rcu: RCU event tracing is enabled.330lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.331lb # [ 0.000000] Trampoline variant of Tasks RCU enabled.332lb # [ 0.000000] Rude variant of Tasks RCU enabled.333lb # [ 0.000000] Tracing variant of Tasks RCU enabled.334lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.335lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1336lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.337lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.338lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.339lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0340lb # [ 0.000000] GICv3: 256 SPIs implemented341lb # [ 0.000000] GICv3: 0 Extended SPIs implemented342lb # [ 0.000000] Root IRQ handler: gic_handle_irq343lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI344lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0345lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000346lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]347lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)348lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)349lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000350lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000351lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.352lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns353lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).354lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns355lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns356lb # [ 0.000050] arm-pv: using stolen time PV357lb # [ 0.001260] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)358lb # [ 0.001489] Console: colour dummy device 80x25359lb # [ 0.001497] printk: legacy console [tty0] enabled360lb # [ 0.001700] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)361lb # [ 0.001708] pid_max: default: 32768 minimum: 301362lb # [ 0.001790] LSM: initializing lsm=capability,landlock,yama,bpf,ima363lb # [ 0.001996] landlock: Up and running.364lb # [ 0.001999] Yama: becoming mindful.365lb # [ 0.002914] LSM support for eBPF active366lb # [ 0.003182] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)367lb # [ 0.003219] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)368lb # [ 0.004569] cacheinfo: Unable to detect cache hierarchy for CPU 0369lb # [ 0.005440] rcu: Hierarchical SRCU implementation.370lb # [ 0.005446] rcu: Max phase no-delay instances is 1000.371lb # [ 0.006996] fsl-mc MSI: its@8080000 domain created372lb # [ 0.007103] EFI services will not be available.373lb # [ 0.007206] smp: Bringing up secondary CPUs ...374lb # [ 0.007216] smp: Brought up 1 node, 1 CPU375lb # [ 0.007219] SMP: Total of 1 processors activated.376lb # [ 0.007226] CPU: All CPU(s) started at EL1377lb # [ 0.007239] CPU features: detected: Branch Target Identification378lb # [ 0.007244] CPU features: detected: ARMv8.4 Translation Table Level379lb # [ 0.007247] CPU features: detected: Instruction cache invalidation not required for I/D coherence380lb # [ 0.007251] CPU features: detected: Data cache clean to the PoU not required for I/D coherence381lb # [ 0.007254] CPU features: detected: Common not Private translations382lb # [ 0.007258] CPU features: detected: CRC32 instructions383lb # [ 0.007261] CPU features: detected: Data cache clean to Point of Deep Persistence384lb # [ 0.007265] CPU features: detected: Data cache clean to Point of Persistence385lb # [ 0.007269] CPU features: detected: Data independent timing control (DIT)386lb # [ 0.007272] CPU features: detected: E0PD387lb # [ 0.007275] CPU features: detected: Enhanced Counter Virtualization388lb # [ 0.007278] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)389lb # [ 0.007281] CPU features: detected: Enhanced Virtualization Traps390lb # [ 0.007284] CPU features: detected: Fine Grained Traps391lb # [ 0.007288] CPU features: detected: Generic authentication (architected QARMA5 algorithm)392lb # [ 0.007294] CPU features: detected: RCpc load-acquire (LDAPR)393lb # [ 0.007297] CPU features: detected: LSE atomic instructions394lb # [ 0.007300] CPU features: detected: Privileged Access Never395lb # [ 0.007303] CPU features: detected: PMUv3396lb # [ 0.007306] CPU features: detected: RAS Extension Support397lb # [ 0.007309] CPU features: detected: RASv1p1 Extension Support398lb # [ 0.007311] CPU features: detected: Random Number Generator399lb # [ 0.007314] CPU features: detected: Speculation barrier (SB)400lb # [ 0.007317] CPU features: detected: Stage-2 Force Write-Back401lb # [ 0.007320] CPU features: detected: TLB range maintenance instructions402worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]403lb # [ 0.007325] CPU features: detected: Speculative Store Bypassing Safe (SSBS)404lb # [ 0.007372] alternatives: applying system-wide alternatives405worker1 # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026406worker1 # [ 0.000000] KASLR enabled407lb # [ 0.010818] CPU features: detected: BBM Level 2 without TLB conflict abort408worker1 # [ 0.000000] random: crng init done409worker1 # [ 0.000000] Machine model: linux,dummy-virt410worker1 # [ 0.000000] efi: UEFI not found.411lb # [ 0.015295] Memory: 1406516K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123764K reserved, 32768K cma-reserved)412lb # [ 0.017447] devtmpfs: initialized413worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT414worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]415lb # [ 0.019490] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)416worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]417worker1 # [ 0.000000] Zone ranges:418lb # [ 0.019516] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).419worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]420lb # [ 0.019903] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL421worker1 # [ 0.000000] DMA32 empty422worker1 # [ 0.000000] Normal empty423lb # [ 0.019909] 0 pages in range for non-PLT usage424worker1 # [ 0.000000] Device empty425lb # [ 0.019910] 508288 pages in range for PLT usage426worker1 # [ 0.000000] Movable zone start for each node427lb # [ 0.020051] pinctrl core: initialized pinctrl subsystem428worker1 # [ 0.000000] Early memory node ranges429lb # [ 0.020902] DMI not present or invalid.430worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]431lb # [ 0.026095] NET: Registered PF_NETLINK/PF_ROUTE protocol family432worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]433worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000434worker1 # [ 0.000000] psci: probing for conduit method from DT.435lb # [ 0.028494] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations436lb # [ 0.029098] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations437worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware.438lb # [ 0.030625] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations439worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs440lb # [ 0.030670] audit: initializing netlink subsys (disabled)441worker1 # [ 0.000000] psci: Trusted OS migration not required442worker1 # [ 0.000000] psci: SMC Calling Convention v1.1443worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)444worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296445lb # [ 0.031679] audit: type=2000 audit(0.024:1): state=initialized audit_enabled=0 res=1446worker1 # [ 0.000000] Detected PIPT I-cache on CPU0447lb # [ 0.031917] thermal_sys: Registered thermal governor 'fair_share'448worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)449lb # [ 0.031918] thermal_sys: Registered thermal governor 'bang_bang'450worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface451lb # [ 0.031925] thermal_sys: Registered thermal governor 'step_wise'452worker1 # [ 0.000000] CPU features: detected: Spectre-v4453worker1 # [ 0.000000] CPU features: detected: Spectre-BHB454lb # [ 0.031927] thermal_sys: Registered thermal governor 'user_space'455worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38456lb # [ 0.031930] thermal_sys: Registered thermal governor 'power_allocator'457lb # [ 0.031957] cpuidle: using governor ladder458worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23459lb # [ 0.031965] cpuidle: using governor menu460worker1 # [ 0.000000] alternatives: applying boot alternatives461lb # [ 0.032172] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.462lb # [ 0.032192] ASID allocator initialised with 65536 entries463lb # [ 0.033526] Serial: AMBA PL011 UART driver464lb # [ 0.039158] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1465lb # [ 0.039353] printk: console [ttyAMA0] enabled466worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/zwgirfamq9cjz73mhvx6r4x90a72kbbz-nixos-system-worker1-test/init regInfo=/nix/store/8vxdl5gm74j84gimba2hrvfgq4533bsm-closure-info/registration console=ttyAMA0,115200n8 console=tty0467worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/8vxdl5gm74j84gimba2hrvfgq4533bsm-closure-info/registration", will be passed to user space.468worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes469worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)470worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)471worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB472worker1 # [ 0.000000] software IO TLB: area num 1.473worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB474lb # [ 0.160756] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages475lb # [ 0.160783] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page476worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)477worker1 # [ 0.000000] Fallback order for Node 0: 0478lb # [ 0.160789] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages479worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216480lb # [ 0.160793] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page481worker1 # [ 0.000000] Policy zone: DMA482lb # [ 0.160797] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages483worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off484lb # [ 0.160802] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page485worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1486lb # [ 0.160806] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages487worker1 # [ 0.000000] allocated 3145728 bytes of page_ext488lb # [ 0.160810] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page489worker1 # [ 0.000000] ftrace: allocating 74885 entries in 294 pages490worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups491worker1 # [ 0.000000] rcu: Hierarchical RCU implementation.492worker1 # [ 0.000000] rcu: RCU event tracing is enabled.493worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.494worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled.495worker1 # [ 0.000000] Rude variant of Tasks RCU enabled.496worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled.497worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.498worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1499worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.500worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.501worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.502worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0503worker1 # [ 0.000000] GICv3: 256 SPIs implemented504worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented505worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq506lb # [ 0.182141] fbcon: Taking over console507worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI508lb # [ 0.182178] ACPI: Interpreter disabled.509worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0510lb # [ 0.184220] iommu: Default domain type: Translated511worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000512lb # [ 0.184233] iommu: DMA domain TLB invalidation policy: strict mode513worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]514lb # [ 0.186040] SCSI subsystem initialized515lb # [ 0.186778] usbcore: registered new interface driver usbfs516lb # [ 0.186820] usbcore: registered new interface driver hub517lb # [ 0.186850] usbcore: registered new device driver usb518lb # [ 0.187168] pps_core: LinuxPPS API ver. 1 registered519lb # [ 0.187174] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>520worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)521lb # [ 0.187185] PTP clock support registered522lb # [ 0.187241] EDAC MC: Ver: 3.0.0523worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)524lb # [ 0.192084] scmi_core: SCMI protocol bus registered525worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000526worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000527lb # [ 0.193127] FPGA manager framework528lb # [ 0.194248] vgaarb: loaded529worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.530worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns531worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).532worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns533lb # [ 0.197954] clocksource: Switched to clocksource arch_sys_counter534worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns535worker1 # [ 0.000036] arm-pv: using stolen time PV536lb # [ 0.203894] VFS: Disk quotas dquot_6.6.0537worker1 # [ 0.000762] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)538worker1 # [ 0.000945] Console: colour dummy device 80x25539worker1 # [ 0.000953] printk: legacy console [tty0] enabled540lb # [ 0.203967] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)541worker1 # [ 0.001154] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)542worker1 # [ 0.001161] pid_max: default: 32768 minimum: 301543worker1 # [ 0.001252] LSM: initializing lsm=capability,landlock,yama,bpf,ima544worker1 # [ 0.001461] landlock: Up and running.545worker1 # [ 0.001464] Yama: becoming mindful.546worker1 # [ 0.002137] LSM support for eBPF active547lb # [ 0.209950] netfs: FS-Cache loaded548worker1 # [ 0.002295] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)549lb # [ 0.210161] pnp: PnP ACPI: disabled550worker1 # [ 0.002330] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)551worker1 # [ 0.003497] cacheinfo: Unable to detect cache hierarchy for CPU 0552worker1 # [ 0.005220] rcu: Hierarchical SRCU implementation.553worker1 # [ 0.005227] rcu: Max phase no-delay instances is 1000.554worker1 # [ 0.006526] fsl-mc MSI: its@8080000 domain created555worker1 # [ 0.006696] EFI services will not be available.556worker1 # [ 0.006777] smp: Bringing up secondary CPUs ...557worker1 # [ 0.006788] smp: Brought up 1 node, 1 CPU558lb # [ 0.215115] NET: Registered PF_INET protocol family559lb # [ 0.215399] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)560worker1 # [ 0.006792] SMP: Total of 1 processors activated.561worker1 # [ 0.006798] CPU: All CPU(s) started at EL1562worker1 # [ 0.006811] CPU features: detected: Branch Target Identification563worker1 # [ 0.006816] CPU features: detected: ARMv8.4 Translation Table Level564worker1 # [ 0.006819] CPU features: detected: Instruction cache invalidation not required for I/D coherence565worker1 # [ 0.006823] CPU features: detected: Data cache clean to the PoU not required for I/D coherence566worker1 # [ 0.006826] CPU features: detected: Common not Private translations567worker1 # [ 0.006829] CPU features: detected: CRC32 instructions568worker1 # [ 0.006832] CPU features: detected: Data cache clean to Point of Deep Persistence569worker1 # [ 0.006836] CPU features: detected: Data cache clean to Point of Persistence570worker1 # [ 0.006839] CPU features: detected: Data independent timing control (DIT)571worker1 # [ 0.006842] CPU features: detected: E0PD572worker1 # [ 0.006845] CPU features: detected: Enhanced Counter Virtualization573worker1 # [ 0.006848] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)574worker1 # [ 0.006851] CPU features: detected: Enhanced Virtualization Traps575worker1 # [ 0.006854] CPU features: detected: Fine Grained Traps576worker1 # [ 0.006858] CPU features: detected: Generic authentication (architected QARMA5 algorithm)577worker1 # [ 0.006863] CPU features: detected: RCpc load-acquire (LDAPR)578worker1 # [ 0.006867] CPU features: detected: LSE atomic instructions579worker1 # [ 0.006870] CPU features: detected: Privileged Access Never580worker1 # [ 0.006872] CPU features: detected: PMUv3581worker1 # [ 0.006875] CPU features: detected: RAS Extension Support582worker1 # [ 0.006878] CPU features: detected: RASv1p1 Extension Support583worker1 # [ 0.006880] CPU features: detected: Random Number Generator584worker1 # [ 0.006883] CPU features: detected: Speculation barrier (SB)585worker1 # [ 0.006886] CPU features: detected: Stage-2 Force Write-Back586worker1 # [ 0.006889] CPU features: detected: TLB range maintenance instructions587worker1 # [ 0.006894] CPU features: detected: Speculative Store Bypassing Safe (SSBS)588worker1 # [ 0.006942] alternatives: applying system-wide alternatives589worker1 # [ 0.010282] CPU features: detected: BBM Level 2 without TLB conflict abort590worker1 # [ 0.010530] Memory: 1406544K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123772K reserved, 32768K cma-reserved)591worker1 # [ 0.011395] devtmpfs: initialized592worker1 # [ 0.013370] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)593worker1 # [ 0.013396] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).594worker1 # [ 0.013632] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL595worker1 # [ 0.013637] 0 pages in range for non-PLT usage596worker1 # [ 0.013638] 508288 pages in range for PLT usage597worker1 # [ 0.013770] pinctrl core: initialized pinctrl subsystem598worker1 # [ 0.014731] DMI not present or invalid.599worker1 # [ 0.018429] NET: Registered PF_NETLINK/PF_ROUTE protocol family600worker1 # [ 0.021447] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations601worker1 # [ 0.021671] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations602worker1 # [ 0.021903] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations603worker1 # [ 0.021942] audit: initializing netlink subsys (disabled)604worker1 # [ 0.022345] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1605worker1 # [ 0.022708] thermal_sys: Registered thermal governor 'fair_share'606worker1 # [ 0.022710] thermal_sys: Registered thermal governor 'bang_bang'607worker1 # [ 0.022717] thermal_sys: Registered thermal governor 'step_wise'608worker1 # [ 0.022720] thermal_sys: Registered thermal governor 'user_space'609worker1 # [ 0.022723] thermal_sys: Registered thermal governor 'power_allocator'610worker1 # [ 0.022746] cpuidle: using governor ladder611worker1 # [ 0.022753] cpuidle: using governor menu612lb # [ 0.250513] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)613worker1 # [ 0.022962] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.614worker1 # [ 0.022981] ASID allocator initialised with 65536 entries615lb # [ 0.250584] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)616worker1 # [ 0.024297] Serial: AMBA PL011 UART driver617lb # [ 0.250625] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)618worker1 # [ 0.029809] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1619worker1 # [ 0.029973] printk: console [ttyAMA0] enabled620lb # [ 0.250726] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)621lb # [ 0.250874] TCP: Hash tables configured (established 16384 bind 16384)622lb # [ 0.250987] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)623lb # [ 0.251196] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)624lb # [ 0.251245] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)625lb # [ 0.251547] NET: Registered PF_UNIX/PF_LOCAL protocol family626lb # [ 0.251690] NET: Registered PF_XDP protocol family627lb # [ 0.251716] PCI: CLS 0 bytes, default 64628lb # [ 0.251990] Trying to unpack rootfs image as initramfs...629worker1 # [ 0.182813] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages630worker1 # [ 0.182840] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page631worker1 # [ 0.182846] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages632worker1 # [ 0.182851] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page633lb # [ 0.278021] kvm [1]: HYP mode not available634worker1 # [ 0.182855] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages635worker1 # [ 0.182859] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page636worker1 # [ 0.182864] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages637worker1 # [ 0.182868] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page638worker1 # [ 0.192084] fbcon: Taking over console639worker1 # [ 0.192108] ACPI: Interpreter disabled.640worker1 # [ 0.194045] iommu: Default domain type: Translated641worker1 # [ 0.194058] iommu: DMA domain TLB invalidation policy: strict mode642worker1 # [ 0.197099] SCSI subsystem initialized643worker1 # [ 0.197858] usbcore: registered new interface driver usbfs644worker1 # [ 0.197894] usbcore: registered new interface driver hub645worker1 # [ 0.197914] usbcore: registered new device driver usb646worker1 # [ 0.198229] pps_core: LinuxPPS API ver. 1 registered647worker1 # [ 0.198237] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>648worker1 # [ 0.198247] PTP clock support registered649worker1 # [ 0.198302] EDAC MC: Ver: 3.0.0650client # [ 0.597958] Initialise system trusted keyrings651worker1 # [ 0.203125] scmi_core: SCMI protocol bus registered652worker1 # [ 0.204190] FPGA manager framework653worker1 # [ 0.205247] vgaarb: loaded654worker1 # [ 0.205968] clocksource: Switched to clocksource arch_sys_counter655worker1 # [ 0.206762] VFS: Disk quotas dquot_6.6.0656worker1 # [ 0.206800] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)657client # [ 0.598757] workingset: timestamp_bits=42 max_order=19 bucket_order=0658worker1 # [ 0.211674] netfs: FS-Cache loaded659worker1 # [ 0.211814] pnp: PnP ACPI: disabled660client # [ 0.609681] squashfs: version 4.0 (2009/01/31) Phillip Lougher661client # [ 0.610494] 9p: Installing v9fs 9p2000 file system support662worker1 # [ 0.216051] NET: Registered PF_INET protocol family663worker1 # [ 0.216341] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)664worker1 # [ 0.251147] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)665client # [ 0.649552] Key type asymmetric registered666client # [ 0.649589] Asymmetric key parser 'x509' registered667client # [ 0.649687] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)668client # [ 0.651854] io scheduler mq-deadline registered669client # [ 0.651870] io scheduler kyber registered670worker1 # [ 0.251215] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)671worker1 # [ 0.251250] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)672worker1 # [ 0.251347] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)673worker1 # [ 0.251492] TCP: Hash tables configured (established 16384 bind 16384)674worker1 # [ 0.251595] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)675worker1 # [ 0.251637] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)676worker1 # [ 0.251676] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)677worker1 # [ 0.251776] NET: Registered PF_UNIX/PF_LOCAL protocol family678worker1 # [ 0.251799] NET: Registered PF_XDP protocol family679worker1 # [ 0.251820] PCI: CLS 0 bytes, default 64680worker1 # [ 0.252127] Trying to unpack rootfs image as initramfs...681worker1 # [ 0.267843] kvm [1]: HYP mode not available682client # [ 0.665566] pl061_gpio 9030000.pl061: PL061 GPIO chip registered683client # [ 0.675107] ledtrig-cpu: registered to indicate activity on CPUs684client # [ 0.675873] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:685client # [ 0.675901] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000686client # [ 0.675916] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000687client # [ 0.675924] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000688client # [ 0.675947] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits689client # [ 0.675979] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]690client # [ 0.676074] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00691client # [ 0.676083] pci_bus 0000:00: root bus resource [bus 00-ff]692client # [ 0.676090] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]693client # [ 0.676095] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]694client # [ 0.676100] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]695client # [ 0.676220] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint696client # [ 0.676680] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint697client # [ 0.676872] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]698client # [ 0.676889] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]699client # [ 0.676919] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]700client # [ 0.676935] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]701client # [ 0.697667] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint702client # [ 0.697878] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]703client # [ 0.697897] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]704client # [ 0.697927] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]705client # [ 0.698504] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint706client # [ 0.698691] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]707client # [ 0.698707] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]708client # [ 0.698737] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]709client # [ 0.699226] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint710client # [ 0.699405] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]711client # [ 0.699422] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]712client # [ 0.699454] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]713client # [ 0.699913] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint714client # [ 0.700093] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]715client # [ 0.700109] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]716client # [ 0.700138] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]717client # [ 0.700593] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint718client # [ 0.700778] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]719client # [ 0.700794] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]720client # [ 0.700824] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]721client # [ 0.701276] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint722client # [ 0.737589] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]723client # [ 0.737635] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]724client # [ 0.737665] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]725client # [ 0.737683] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]726client # [ 0.738282] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint727client # [ 0.738481] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]728client # [ 0.738511] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]729client # [ 0.743174] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint730client # [ 0.743400] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]731client # [ 0.743432] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]732client # [ 0.743933] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint733client # [ 0.744120] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]734client # [ 0.744384] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint735client # [ 0.744634] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]736client # [ 0.744651] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]737client # [ 0.744681] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]738client # [ 0.745151] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint739client # [ 0.759625] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]740client # [ 0.759651] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]741client # [ 0.759680] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]742client # [ 0.760325] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned743client # [ 0.760338] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned744client # [ 0.760344] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned745client # [ 0.760391] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned746client # [ 0.760439] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned747client # [ 0.760487] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned748client # [ 0.760535] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned749client # [ 0.760583] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned750client # [ 0.760632] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned751client # [ 0.760680] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned752client # [ 0.760729] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned753client # [ 0.760775] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned754client # [ 0.760855] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned755client # [ 0.760901] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned756client # [ 0.760923] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned757client # [ 0.760945] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned758client # [ 0.760966] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned759client # [ 0.760991] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned760client # [ 0.761013] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned761client # [ 0.761036] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned762client # [ 0.761058] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned763client # [ 0.761081] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned764client # [ 0.761103] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned765client # [ 0.761127] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned766client # [ 0.761156] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned767client # [ 0.761178] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned768client # [ 0.761200] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned769lb # [ 0.494820] Initialise system trusted keyrings770client # [ 0.761221] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned771client # [ 0.761242] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned772lb # [ 0.495687] workingset: timestamp_bits=42 max_order=19 bucket_order=0773client # [ 0.761264] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned774client # [ 0.761286] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned775client # [ 0.761308] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned776client # [ 0.761352] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned777client # [ 0.761373] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned778client # [ 0.761400] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]779client # [ 0.761410] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]780client # [ 0.761415] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]781lb # [ 0.502431] squashfs: version 4.0 (2009/01/31) Phillip Lougher782client # [ 0.762577] pci 0000:00:0a.0: enabling device (0000 -> 0002)783lb # [ 0.503351] 9p: Installing v9fs 9p2000 file system support784lb # [ 0.536348] Key type asymmetric registered785lb # [ 0.536386] Asymmetric key parser 'x509' registered786lb # [ 0.536490] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)787lb # [ 0.538991] io scheduler mq-deadline registered788lb # [ 0.539013] io scheduler kyber registered789client # [ 0.842369] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)790lb # [ 0.545116] pl061_gpio 9030000.pl061: PL061 GPIO chip registered791worker1 # [ 0.488943] Initialise system trusted keyrings792worker1 # [ 0.489790] workingset: timestamp_bits=42 max_order=19 bucket_order=0793worker1 # [ 0.491308] squashfs: version 4.0 (2009/01/31) Phillip Lougher794worker1 # [ 0.491372] 9p: Installing v9fs 9p2000 file system support795worker1 # [ 0.511442] Key type asymmetric registered796worker1 # [ 0.511482] Asymmetric key parser 'x509' registered797worker1 # [ 0.511578] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)798worker1 # [ 0.513693] io scheduler mq-deadline registered799client # [ 0.894715] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)800worker1 # [ 0.513707] io scheduler kyber registered801client # [ 0.898720] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)802client # [ 0.901900] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)803client # [ 0.904230] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)804client # [ 0.906869] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)805lb # [ 0.614037] ledtrig-cpu: registered to indicate activity on CPUs806lb # [ 0.614800] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:807lb # [ 0.614828] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000808lb # [ 0.614842] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000809lb # [ 0.614851] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000810lb # [ 0.614878] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits811lb # [ 0.614914] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]812lb # [ 0.615016] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00813lb # [ 0.615026] pci_bus 0000:00: root bus resource [bus 00-ff]814lb # [ 0.615032] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]815lb # [ 0.615037] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]816worker1 # [ 0.528416] pl061_gpio 9030000.pl061: PL061 GPIO chip registered817lb # [ 0.615043] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]818worker1 # [ 0.529813] ledtrig-cpu: registered to indicate activity on CPUs819lb # [ 0.615143] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint820lb # [ 0.615622] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint821client # [ 0.914713] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)822worker1 # [ 0.531222] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:823worker1 # [ 0.531247] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000824lb # [ 0.615831] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]825lb # [ 0.615850] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]826worker1 # [ 0.531260] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000827lb # [ 0.615882] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]828worker1 # [ 0.531269] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000829lb # [ 0.615899] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]830worker1 # [ 0.531294] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits831lb # [ 0.616377] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint832lb # [ 0.616564] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]833worker1 # [ 0.531320] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]834lb # [ 0.616580] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]835worker1 # [ 0.531432] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00836lb # [ 0.616613] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]837worker1 # [ 0.531443] pci_bus 0000:00: root bus resource [bus 00-ff]838lb # [ 0.617126] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint839lb # [ 0.617328] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]840worker1 # [ 0.531449] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]841worker1 # [ 0.531454] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]842lb # [ 0.617346] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]843worker1 # [ 0.531460] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]844lb # [ 0.617381] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]845worker1 # [ 0.531580] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint846client # [ 0.932741] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)847lb # [ 0.617877] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint848worker1 # [ 0.532069] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint849client # [ 0.939434] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)850worker1 # [ 0.532279] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]851worker1 # [ 0.532297] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]852worker1 # [ 0.532328] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]853worker1 # [ 0.532345] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]854client # [ 0.943432] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)855worker1 # [ 0.532816] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint856worker1 # [ 0.533001] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]857worker1 # [ 0.533018] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]858worker1 # [ 0.533048] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]859worker1 # [ 0.533510] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint860client # [ 0.947169] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)861worker1 # [ 0.533693] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]862worker1 # [ 0.533709] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]863worker1 # [ 0.533739] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]864lb # [ 0.654185] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]865lb # [ 0.654279] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]866lb # [ 0.654399] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]867lb # [ 0.655029] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint868lb # [ 0.655228] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]869lb # [ 0.655246] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]870worker1 # [ 0.562401] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint871client # [ 0.961851] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled872worker1 # [ 0.562659] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]873worker1 # [ 0.562679] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]874worker1 # [ 0.562714] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]875worker1 # [ 0.563219] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint876worker1 # [ 0.563405] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]877lb # [ 0.655276] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]878lb # [ 0.655753] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint879lb # [ 0.656040] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]880lb # [ 0.656059] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]881lb # [ 0.656089] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]882lb # [ 0.656592] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint883lb # [ 0.656777] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]884lb # [ 0.656793] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]885lb # [ 0.656823] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]886worker1 # [ 0.563423] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]887worker1 # [ 0.563453] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]888worker1 # [ 0.563913] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint889worker1 # [ 0.564107] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]890worker1 # [ 0.564126] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]891lb # [ 0.656839] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]892worker1 # [ 0.564158] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]893lb # [ 0.657391] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint894worker1 # [ 0.564649] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint895worker1 # [ 0.564846] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]896lb # [ 0.657581] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]897client # [ 0.971103] msm_serial: driver initialized898worker1 # [ 0.564863] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]899lb # [ 0.657611] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]900client # [ 0.971295] SuperH (H)SCI(F) driver initialized901client # [ 0.971350] STM32 USART driver initialized902worker1 # [ 0.564894] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]903lb # [ 0.658107] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint904worker1 # [ 0.564910] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]905lb # [ 0.658309] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]906worker1 # [ 0.565431] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint907lb # [ 0.658340] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]908worker1 # [ 0.565623] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]909worker1 # [ 0.565655] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]910lb # [ 0.658741] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint911lb # [ 0.659076] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]912lb # [ 0.659341] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint913lb # [ 0.659597] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]914lb # [ 0.659615] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]915lb # [ 0.659645] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]916lb # [ 0.660230] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint917lb # [ 0.660422] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]918lb # [ 0.660438] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]919lb # [ 0.660468] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]920lb # [ 0.661186] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned921worker1 # [ 0.586300] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint922lb # [ 0.661197] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned923worker1 # [ 0.586555] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]924lb # [ 0.661203] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned925worker1 # [ 0.586588] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]926lb # [ 0.661254] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned927worker1 # [ 0.587024] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint928worker1 # [ 0.587212] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]929lb # [ 0.661305] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned930worker1 # [ 0.587482] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint931lb # [ 0.661355] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned932worker1 # [ 0.587744] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]933worker1 # [ 0.587763] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]934lb # [ 0.661405] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned935worker1 # [ 0.587793] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]936lb # [ 0.661452] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned937worker1 # [ 0.588275] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint938lb # [ 0.661501] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned939worker1 # [ 0.588613] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]940worker1 # [ 0.588632] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]941worker1 # [ 0.588663] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]942worker1 # [ 0.589302] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned943worker1 # [ 0.589316] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned944lb # [ 0.661550] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned945lb # [ 0.661603] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned946lb # [ 0.661720] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned947lb # [ 0.661815] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned948worker1 # [ 0.589322] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned949lb # [ 0.661863] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned950worker1 # [ 0.589371] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned951lb # [ 0.661886] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned952worker1 # [ 0.589420] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned953lb # [ 0.661908] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned954worker1 # [ 0.589470] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned955lb # [ 0.661931] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned956worker1 # [ 0.589519] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned957worker1 # [ 0.589568] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned958worker1 # [ 0.589618] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned959worker1 # [ 0.589671] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned960worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]961worker1 # [ 0.589720] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned962lb # [ 0.706706] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned963lb # [ 0.706744] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned964worker1 # [ 0.589770] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned965worker2 # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026966worker2 # [ 0.000000] KASLR enabled967lb # [ 0.706767] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned968worker1 # [ 0.589865] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned969worker2 # [ 0.000000] random: crng init done970worker2 # [ 0.000000] Machine model: linux,dummy-virt971worker1 # [ 0.589915] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned972worker2 # [ 0.000000] efi: UEFI not found.973worker1 # [ 0.589939] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned974worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT975worker1 # [ 0.589961] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned976worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]977worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]978worker2 # [ 0.000000] Zone ranges:979worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]980lb # [ 0.706791] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned981worker2 # [ 0.000000] DMA32 empty982worker2 # [ 0.000000] Normal empty983lb # [ 0.706814] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned984worker2 # [ 0.000000] Device empty985worker2 # [ 0.000000] Movable zone start for each node986lb # [ 0.706837] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned987worker2 # [ 0.000000] Early memory node ranges988worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]989lb # [ 0.706861] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned990lb # [ 0.706885] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned991worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]992client # [ 1.014215] loop: module loaded993client # [ 1.014493] virtio_blk virtio5: 1/0/0 default/read/poll queues994client # [ 1.015379] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)995worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000996lb # [ 0.706907] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned997worker2 # [ 0.000000] psci: probing for conduit method from DT.998lb # [ 0.706929] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned999worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware.1000lb # [ 0.706952] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1001worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs1002lb # [ 0.706974] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1003worker2 # [ 0.000000] psci: Trusted OS migration not required1004worker2 # [ 0.000000] psci: SMC Calling Convention v1.11005lb # [ 0.706997] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1006lb # [ 0.707020] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1007worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)1008lb # [ 0.707043] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1009worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u3112961010worker2 # [ 0.000000] Detected PIPT I-cache on CPU01011lb # [ 0.707066] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1012worker1 # [ 0.622107] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1013lb # [ 0.707088] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1014worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)1015worker1 # [ 0.622141] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1016lb # [ 0.707124] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1017worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface1018worker1 # [ 0.622166] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1019worker2 # [ 0.000000] CPU features: detected: Spectre-v41020worker2 # [ 0.000000] CPU features: detected: Spectre-BHB1021worker1 # [ 0.622191] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1022worker1 # [ 0.622215] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1023worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_381024worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_231025worker2 # [ 0.000000] alternatives: applying boot alternatives1026worker1 # [ 0.622241] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1027worker1 # [ 0.622266] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1028worker1 # [ 0.622291] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1029client # [ 1.022257] megasas: 07.734.00.00-rc11030worker1 # [ 0.622317] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1031lb # [ 0.707133] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1032lb # [ 0.707138] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1033worker1 # [ 0.622341] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1034lb # [ 0.708528] pci 0000:00:0a.0: enabling device (0000 -> 0002)1035worker1 # [ 0.622364] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1036worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/0r31wjlygcn47dyyzl916b35v9f3ng6k-nixos-system-worker2-test/init regInfo=/nix/store/ik3fad33q44if18fhw4ncajdi342n3lv-closure-info/registration console=ttyAMA0,115200n8 console=tty01037worker1 # [ 0.622386] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1038worker1 # [ 0.622408] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1039client # [ 1.028173] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1040worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/ik3fad33q44if18fhw4ncajdi342n3lv-closure-info/registration", will be passed to user space.1041worker1 # [ 0.622431] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1042worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes1043worker1 # [ 0.622455] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1044worker1 # [ 0.622498] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1045worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)1046worker1 # [ 0.622521] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1047worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)1048worker1 # [ 0.622544] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1049worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB1050worker1 # [ 0.622583] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1051worker2 # [ 0.000000] software IO TLB: area num 1.1052worker1 # [ 0.622594] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1053worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB1054worker1 # [ 0.622599] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1055worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)1056worker2 # [ 0.000000] Fallback order for Node 0: 01057worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 3932161058worker2 # [ 0.000000] Policy zone: DMA1059worker1 # [ 0.623815] pci 0000:00:0a.0: enabling device (0000 -> 0002)1060worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off1061worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=11062worker2 # [ 0.000000] allocated 3145728 bytes of page_ext1063worker2 # [ 0.000000] ftrace: allocating 74885 entries in 294 pages1064worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups1065worker2 # [ 0.000000] rcu: Hierarchical RCU implementation.1066worker2 # [ 0.000000] rcu: RCU event tracing is enabled.1067worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.1068worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled.1069worker2 # [ 0.000000] Rude variant of Tasks RCU enabled.1070worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled.1071worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.1072client # [ 1.045705] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001073worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=11074client # [ 1.045818] Intel/Sharp Extended Query Table at 0x00311075worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.1076client # [ 1.049069] Using buffer write method1077worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.1078worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.1079worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 01080client # [ 1.049191] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1081worker2 # [ 0.000000] GICv3: 256 SPIs implemented1082worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented1083worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq1084worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI1085worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=01086worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a00001087worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]1088worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1)1089worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1)1090worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a000001091worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a100001092worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.1093worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns1094worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).1095worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns1096worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns1097worker2 # [ 0.000039] arm-pv: using stolen time PV1098worker2 # [ 0.000866] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)1099worker2 # [ 0.001085] Console: colour dummy device 80x251100worker2 # [ 0.001093] printk: legacy console [tty0] enabled1101worker2 # [ 0.001288] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)1102worker2 # [ 0.001295] pid_max: default: 32768 minimum: 3011103worker2 # [ 0.001384] LSM: initializing lsm=capability,landlock,yama,bpf,ima1104worker2 # [ 0.001605] landlock: Up and running.1105client # [ 1.070938] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001106worker2 # [ 0.001608] Yama: becoming mindful.1107worker2 # [ 0.002547] LSM support for eBPF active1108client # [ 1.071022] Intel/Sharp Extended Query Table at 0x00311109client # [ 1.075508] Using buffer write method1110worker1 # [ 0.674325] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1111worker2 # [ 0.002787] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)1112client # [ 1.075548] Concatenating MTD devices:1113client # [ 1.075552] (0): "0.flash"1114worker1 # [ 0.677665] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1115worker2 # [ 0.002823] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)1116client # [ 1.075556] (1): "0.flash"1117client # [ 1.075559] into device "0.flash"1118worker2 # [ 0.004160] cacheinfo: Unable to detect cache hierarchy for CPU 01119worker2 # [ 0.004971] rcu: Hierarchical SRCU implementation.1120worker2 # [ 0.004977] rcu: Max phase no-delay instances is 1000.1121worker2 # [ 0.006278] fsl-mc MSI: its@8080000 domain created1122worker2 # [ 0.006398] EFI services will not be available.1123worker2 # [ 0.006503] smp: Bringing up secondary CPUs ...1124worker2 # [ 0.006514] smp: Brought up 1 node, 1 CPU1125worker2 # [ 0.006517] SMP: Total of 1 processors activated.1126worker2 # [ 0.006523] CPU: All CPU(s) started at EL11127worker2 # [ 0.006537] CPU features: detected: Branch Target Identification1128worker2 # [ 0.006542] CPU features: detected: ARMv8.4 Translation Table Level1129worker2 # [ 0.006546] CPU features: detected: Instruction cache invalidation not required for I/D coherence1130worker2 # [ 0.006549] CPU features: detected: Data cache clean to the PoU not required for I/D coherence1131worker1 # [ 0.688048] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1132worker2 # [ 0.006553] CPU features: detected: Common not Private translations1133lb # [ 0.788317] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1134worker2 # [ 0.006556] CPU features: detected: CRC32 instructions1135worker2 # [ 0.006559] CPU features: detected: Data cache clean to Point of Deep Persistence1136worker2 # [ 0.006563] CPU features: detected: Data cache clean to Point of Persistence1137worker2 # [ 0.006566] CPU features: detected: Data independent timing control (DIT)1138worker2 # [ 0.006569] CPU features: detected: E0PD1139lb # [ 0.792948] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1140worker1 # [ 0.691236] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1141worker2 # [ 0.006572] CPU features: detected: Enhanced Counter Virtualization1142worker1 # [ 0.693601] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1143worker2 # [ 0.006575] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)1144worker2 # [ 0.006578] CPU features: detected: Enhanced Virtualization Traps1145worker2 # [ 0.006581] CPU features: detected: Fine Grained Traps1146worker2 # [ 0.006586] CPU features: detected: Generic authentication (architected QARMA5 algorithm)1147worker2 # [ 0.006591] CPU features: detected: RCpc load-acquire (LDAPR)1148worker2 # [ 0.006594] CPU features: detected: LSE atomic instructions1149worker2 # [ 0.006597] CPU features: detected: Privileged Access Never1150worker2 # [ 0.006600] CPU features: detected: PMUv31151worker2 # [ 0.006603] CPU features: detected: RAS Extension Support1152worker2 # [ 0.006605] CPU features: detected: RASv1p1 Extension Support1153worker2 # [ 0.006608] CPU features: detected: Random Number Generator1154worker2 # [ 0.006611] CPU features: detected: Speculation barrier (SB)1155worker2 # [ 0.006614] CPU features: detected: Stage-2 Force Write-Back1156worker2 # [ 0.006617] CPU features: detected: TLB range maintenance instructions1157worker2 # [ 0.006623] CPU features: detected: Speculative Store Bypassing Safe (SSBS)1158worker2 # [ 0.006667] alternatives: applying system-wide alternatives1159worker2 # [ 0.010281] CPU features: detected: BBM Level 2 without TLB conflict abort1160worker2 # [ 0.010531] Memory: 1406768K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123772K reserved, 32768K cma-reserved)1161worker2 # [ 0.011809] devtmpfs: initialized1162worker2 # [ 0.013764] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)1163worker2 # [ 0.013789] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).1164worker2 # [ 0.014033] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL1165worker2 # [ 0.014039] 0 pages in range for non-PLT usage1166worker2 # [ 0.014040] 508288 pages in range for PLT usage1167worker2 # [ 0.014174] pinctrl core: initialized pinctrl subsystem1168lb # [ 0.807803] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1169worker2 # [ 0.015134] DMI not present or invalid.1170worker1 # [ 0.711790] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1171worker2 # [ 0.019704] NET: Registered PF_NETLINK/PF_ROUTE protocol family1172worker2 # [ 0.022278] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations1173worker2 # [ 0.022505] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations1174worker2 # [ 0.022735] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations1175worker2 # [ 0.022773] audit: initializing netlink subsys (disabled)1176worker2 # [ 0.024610] thermal_sys: Registered thermal governor 'fair_share'1177worker2 # [ 0.024613] thermal_sys: Registered thermal governor 'bang_bang'1178worker2 # [ 0.024618] thermal_sys: Registered thermal governor 'step_wise'1179worker2 # [ 0.024625] thermal_sys: Registered thermal governor 'user_space'1180worker2 # [ 0.024628] thermal_sys: Registered thermal governor 'power_allocator'1181worker2 # [ 0.024667] audit: type=2000 audit(0.020:1): state=initialized audit_enabled=0 res=11182worker2 # [ 0.024680] cpuidle: using governor ladder1183worker2 # [ 0.024686] cpuidle: using governor menu1184worker2 # [ 0.024936] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.1185worker2 # [ 0.024958] ASID allocator initialised with 65536 entries1186worker2 # [ 0.026273] Serial: AMBA PL011 UART driver1187lb # [ 0.822670] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1188worker2 # [ 0.031960] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev11189worker2 # [ 0.032162] printk: console [ttyAMA0] enabled1190worker1 # [ 0.715430] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1191worker1 # [ 0.718068] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1192worker1 # [ 0.720582] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1193lb # [ 0.837326] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1194worker2 # [ 0.160149] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages1195worker2 # [ 0.160177] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page1196worker2 # [ 0.160183] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages1197worker2 # [ 0.160188] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page1198worker2 # [ 0.160192] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages1199worker2 # [ 0.160197] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page1200worker2 # [ 0.160201] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages1201worker2 # [ 0.160206] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page1202worker2 # [ 0.167892] fbcon: Taking over console1203worker2 # [ 0.167913] ACPI: Interpreter disabled.1204lb # [ 0.849395] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1205worker1 # [ 0.748152] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1206worker2 # [ 0.176512] iommu: Default domain type: Translated1207worker2 # [ 0.176529] iommu: DMA domain TLB invalidation policy: strict mode1208worker2 # [ 0.178381] SCSI subsystem initialized1209worker2 # [ 0.179133] usbcore: registered new interface driver usbfs1210worker2 # [ 0.179167] usbcore: registered new interface driver hub1211worker2 # [ 0.179186] usbcore: registered new device driver usb1212worker2 # [ 0.179548] pps_core: LinuxPPS API ver. 1 registered1213worker2 # [ 0.179556] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>1214worker2 # [ 0.179567] PTP clock support registered1215worker2 # [ 0.179635] EDAC MC: Ver: 3.0.01216worker2 # [ 0.184450] scmi_core: SCMI protocol bus registered1217worker2 # [ 0.185514] FPGA manager framework1218worker2 # [ 0.186527] vgaarb: loaded1219worker2 # [ 0.187281] clocksource: Switched to clocksource arch_sys_counter1220lb # [ 0.852711] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1221worker2 # [ 0.192111] VFS: Disk quotas dquot_6.6.01222worker2 # [ 0.192184] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)1223worker2 # [ 0.193896] netfs: FS-Cache loaded1224worker2 # [ 0.194055] pnp: PnP ACPI: disabled1225worker1 # [ 0.767668] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1226worker2 # [ 0.198146] NET: Registered PF_INET protocol family1227worker2 # [ 0.198442] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)1228lb # [ 0.887821] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1229worker1 # [ 0.801742] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1230worker1 # [ 0.804743] msm_serial: driver initialized1231worker1 # [ 0.804897] SuperH (H)SCI(F) driver initialized1232worker1 # [ 0.804955] STM32 USART driver initialized1233lb # [ 0.912563] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1234worker2 # [ 0.238423] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)1235lb # [ 0.923475] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1236worker2 # [ 0.238491] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)1237worker2 # [ 0.238530] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)1238worker2 # [ 0.238631] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)1239worker2 # [ 0.238775] TCP: Hash tables configured (established 16384 bind 16384)1240worker2 # [ 0.238891] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)1241worker2 # [ 0.238936] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)1242worker2 # [ 0.238977] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)1243worker2 # [ 0.239091] NET: Registered PF_UNIX/PF_LOCAL protocol family1244worker2 # [ 0.239117] NET: Registered PF_XDP protocol family1245worker2 # [ 0.239139] PCI: CLS 0 bytes, default 641246worker2 # [ 0.239406] Trying to unpack rootfs image as initramfs...1247lb # [ 0.934143] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1248worker2 # [ 0.258668] kvm [1]: HYP mode not available1249worker1 # [ 0.848364] loop: module loaded1250worker1 # [ 0.848644] virtio_blk virtio5: 1/0/0 default/read/poll queues1251worker1 # [ 0.849622] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1252lb # [ 0.953135] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1253lb # [ 0.956699] msm_serial: driver initialized1254lb # [ 0.956901] SuperH (H)SCI(F) driver initialized1255lb # [ 0.956959] STM32 USART driver initialized1256worker1 # [ 0.858990] megasas: 07.734.00.00-rc11257worker1 # [ 0.859995] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1258lb # [ 1.005482] loop: module loaded1259lb # [ 1.005736] virtio_blk virtio5: 1/0/0 default/read/poll queues1260worker1 # [ 0.907699] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001261worker1 # [ 0.907770] Intel/Sharp Extended Query Table at 0x00311262lb # [ 1.015058] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1263worker1 # [ 0.918925] Using buffer write method1264worker1 # [ 0.919058] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1265lb # [ 1.018841] megasas: 07.734.00.00-rc11266lb # [ 1.019690] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1267worker1 # [ 0.921169] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001268worker1 # [ 0.921286] Intel/Sharp Extended Query Table at 0x00311269worker1 # [ 0.955715] Using buffer write method1270worker1 # [ 0.955779] Concatenating MTD devices:1271worker1 # [ 0.955784] (0): "0.flash"1272worker1 # [ 0.955789] (1): "0.flash"1273worker1 # [ 0.955792] into device "0.flash"1274lb # [ 1.057884] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001275lb # [ 1.059202] Intel/Sharp Extended Query Table at 0x00311276lb # [ 1.083030] Using buffer write method1277lb # [ 1.083190] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1278lb # [ 1.104137] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001279lb # [ 1.104256] Intel/Sharp Extended Query Table at 0x00311280lb # [ 1.128384] Using buffer write method1281lb # [ 1.128442] Concatenating MTD devices:1282lb # [ 1.128447] (0): "0.flash"1283lb # [ 1.128452] (1): "0.flash"1284lb # [ 1.128455] into device "0.flash"1285worker2 # [ 0.540757] Initialise system trusted keyrings1286worker2 # [ 0.541550] workingset: timestamp_bits=42 max_order=19 bucket_order=01287worker2 # [ 0.542884] squashfs: version 4.0 (2009/01/31) Phillip Lougher1288worker2 # [ 0.543655] 9p: Installing v9fs 9p2000 file system support1289worker2 # [ 0.581325] Key type asymmetric registered1290worker2 # [ 0.581364] Asymmetric key parser 'x509' registered1291worker2 # [ 0.581515] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)1292worker2 # [ 0.583968] io scheduler mq-deadline registered1293worker2 # [ 0.583982] io scheduler kyber registered1294worker2 # [ 0.599509] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1295worker2 # [ 0.601335] ledtrig-cpu: registered to indicate activity on CPUs1296client # [ 1.603932] Freeing initrd memory: 26160K1297client # [ 1.612312] tun: Universal TUN/TAP device driver, 1.61298worker2 # [ 0.627859] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1299worker2 # [ 0.627914] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001300client # [ 1.617477] thunder_xcv, ver 1.01301client # [ 1.617536] thunder_bgx, ver 1.01302client # [ 1.617559] nicpf, ver 1.01303client # [ 1.618148] e1000: Intel(R) PRO/1000 Network Driver1304client # [ 1.618156] e1000: Copyright (c) 1999-2006 Intel Corporation.1305client # [ 1.618183] e1000e: Intel(R) PRO/1000 Network Driver1306client # [ 1.618194] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1307client # [ 1.618225] igb: Intel(R) Gigabit Ethernet Network Driver1308client # [ 1.618231] igb: Copyright (c) 2007-2014 Intel Corporation.1309worker2 # [ 0.643314] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001310client # [ 1.618256] igbvf: Intel(R) Gigabit Virtual Function Network Driver1311worker2 # [ 0.643328] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001312client # [ 1.618262] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1313client # [ 1.618401] sky2: driver version 1.301314worker2 # [ 0.643396] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1315client # [ 1.620319] usbcore: registered new interface driver usb-storage1316client # [ 1.620454] usbcore: registered new interface driver usbserial_generic1317client # [ 1.620471] usbserial: USB Serial support registered for generic1318client # [ 1.621108] hv_vmbus: registering driver hyperv_keyboard1319client # [ 1.622278] ehci-pci 0000:00:0a.0: EHCI Host Controller1320client # [ 1.622323] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11321worker2 # [ 0.643432] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1322client # [ 1.622585] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001323worker2 # [ 0.643611] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001324client # [ 1.633668] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001325worker2 # [ 0.643623] pci_bus 0000:00: root bus resource [bus 00-ff]1326client # [ 1.634161] hub 1-0:1.0: USB hub found1327client # [ 1.634224] hub 1-0:1.0: 6 ports detected1328worker2 # [ 0.643630] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1329client # [ 1.639411] rtc-pl031 9010000.pl031: registered as rtc01330client # [ 1.639459] rtc-pl031 9010000.pl031: setting system clock to 2026-09-20T12:20:54 UTC (1789906854)1331client # [ 1.639820] i2c_dev: i2c /dev entries driver1332worker2 # [ 0.643635] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1333worker2 # [ 0.643641] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1334worker2 # [ 0.643741] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1335client # [ 1.645066] sdhci: Secure Digital Host Controller Interface driver1336client # [ 1.645083] sdhci: Copyright(c) Pierre Ossman1337client # [ 1.646735] Synopsys Designware Multimedia Card Interface Driver1338client # [ 1.650288] sdhci-pltfm: SDHCI platform and OF driver helper1339client # [ 1.652576] hid: raw HID events driver (C) Jiri Kosina1340worker2 # [ 0.679250] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1341client # [ 1.652900] usbcore: registered new interface driver usbhid1342client # [ 1.652909] usbhid: USB HID core driver1343client # [ 1.667979] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1344worker2 # [ 0.696279] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1345worker2 # [ 0.696327] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1346worker2 # [ 0.696357] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1347worker2 # [ 0.696374] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1348client # [ 1.677953] drop_monitor: Initializing network drop monitor service1349client # [ 1.678271] NET: Registered PF_INET6 protocol family1350worker2 # [ 0.696992] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1351worker2 # [ 0.697179] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1352client # [ 1.684576] Segment Routing with IPv61353worker2 # [ 0.697196] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1354client # [ 1.684615] In-situ OAM (IOAM) with IPv61355worker2 # [ 0.697225] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1356client # [ 1.684677] NET: Registered PF_PACKET protocol family1357worker2 # [ 0.697695] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1358worker2 # [ 0.697879] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f]1359worker2 # [ 0.697895] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1360worker2 # [ 0.697924] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1361worker2 # [ 0.698380] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1362worker2 # [ 0.698563] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1363worker2 # [ 0.698579] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1364worker2 # [ 0.698616] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1365worker2 # [ 0.699084] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint1366client # [ 1.686460] 9pnet: Installing 9P2000 support1367worker2 # [ 0.699265] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f]1368client # [ 1.696468] Key type dns_resolver registered1369worker2 # [ 0.699296] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1370worker2 # [ 0.699324] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1371worker2 # [ 0.699788] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1372client # [ 1.708351] registered taskstats version 11373client # [ 1.708569] Loading compiled-in X.509 certificates1374worker2 # [ 0.734597] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f]1375worker2 # [ 0.734647] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1376worker2 # [ 0.734677] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1377worker2 # [ 0.735333] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1378worker2 # [ 0.735528] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f]1379worker2 # [ 0.735545] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff]1380worker2 # [ 0.735574] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1381worker2 # [ 0.735590] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref]1382worker2 # [ 0.737417] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1383worker2 # [ 0.737613] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1384client # [ 1.723142] Demotion targets for Node 0: null1385client # [ 1.723334] Key type .fscrypt registered1386client # [ 1.723344] Key type fscrypt-provisioning registered1387client # [ 1.723464] ima: No TPM chip found, activating TPM-bypass!1388worker2 # [ 0.737643] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1389client # [ 1.723487] ima: Allocated hash algorithm: sha11390client # [ 1.723517] ima: No architecture policies found1391worker2 # [ 0.738129] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1392worker2 # [ 0.738317] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1393client # [ 1.729437] input: gpio-keys as /devices/platform/gpio-keys/input/input01394worker2 # [ 0.738347] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1395worker2 # [ 0.738737] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1396worker2 # [ 0.738918] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff]1397client # [ 1.756755] clk: Disabling unused clocks1398client # [ 1.756802] PM: genpd: Disabling unused power domains1399client # [ 1.761747] Freeing unused kernel memory: 4736K1400client # [ 1.762172] Run /init as init process1401client # [ 1.780822] systemd[1]: Successfully made /usr/ read-only.1402lb # [ 1.511203] Freeing initrd memory: 26140K1403lb # [ 1.518082] tun: Universal TUN/TAP device driver, 1.61404lb # [ 1.522688] thunder_xcv, ver 1.01405lb # [ 1.522747] thunder_bgx, ver 1.01406lb # [ 1.522770] nicpf, ver 1.01407lb # [ 1.523359] e1000: Intel(R) PRO/1000 Network Driver1408lb # [ 1.523367] e1000: Copyright (c) 1999-2006 Intel Corporation.1409lb # [ 1.523393] e1000e: Intel(R) PRO/1000 Network Driver1410lb # [ 1.523403] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1411lb # [ 1.523432] igb: Intel(R) Gigabit Ethernet Network Driver1412lb # [ 1.523439] igb: Copyright (c) 2007-2014 Intel Corporation.1413lb # [ 1.523462] igbvf: Intel(R) Gigabit Virtual Function Network Driver1414lb # [ 1.523468] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1415lb # [ 1.523610] sky2: driver version 1.301416lb # [ 1.525301] usbcore: registered new interface driver usb-storage1417lb # [ 1.525411] usbcore: registered new interface driver usbserial_generic1418lb # [ 1.525426] usbserial: USB Serial support registered for generic1419lb # [ 1.527336] ehci-pci 0000:00:0a.0: EHCI Host Controller1420lb # [ 1.527406] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11421lb # [ 1.527647] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001422lb # [ 1.538492] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001423lb # [ 1.544434] hub 1-0:1.0: USB hub found1424lb # [ 1.544468] hub 1-0:1.0: 6 ports detected1425lb # [ 1.546365] hv_vmbus: registering driver hyperv_keyboard1426lb # [ 1.548161] rtc-pl031 9010000.pl031: registered as rtc01427lb # [ 1.548203] rtc-pl031 9010000.pl031: setting system clock to 2026-09-20T12:20:54 UTC (1789906854)1428lb # [ 1.548567] i2c_dev: i2c /dev entries driver1429lb # [ 1.553808] sdhci: Secure Digital Host Controller Interface driver1430lb # [ 1.553825] sdhci: Copyright(c) Pierre Ossman1431lb # [ 1.555463] Synopsys Designware Multimedia Card Interface Driver1432lb # [ 1.555863] sdhci-pltfm: SDHCI platform and OF driver helper1433lb # [ 1.557779] hid: raw HID events driver (C) Jiri Kosina1434lb # [ 1.560238] usbcore: registered new interface driver usbhid1435lb # [ 1.560250] usbhid: USB HID core driver1436lb # [ 1.562675] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1437worker1 # [ 1.463982] Freeing initrd memory: 26148K1438lb # [ 1.564324] drop_monitor: Initializing network drop monitor service1439lb # [ 1.564534] NET: Registered PF_INET6 protocol family1440lb # [ 1.568213] Segment Routing with IPv61441lb # [ 1.568246] In-situ OAM (IOAM) with IPv61442lb # [ 1.568284] NET: Registered PF_PACKET protocol family1443worker1 # [ 1.471668] tun: Universal TUN/TAP device driver, 1.61444lb # [ 1.570079] 9pnet: Installing 9P2000 support1445lb # [ 1.572711] Key type dns_resolver registered1446worker1 # [ 1.476206] thunder_xcv, ver 1.01447worker1 # [ 1.476273] thunder_bgx, ver 1.01448worker1 # [ 1.476299] nicpf, ver 1.01449worker1 # [ 1.477225] e1000: Intel(R) PRO/1000 Network Driver1450worker1 # [ 1.477240] e1000: Copyright (c) 1999-2006 Intel Corporation.1451worker1 # [ 1.477267] e1000e: Intel(R) PRO/1000 Network Driver1452worker1 # [ 1.477280] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1453worker1 # [ 1.477311] igb: Intel(R) Gigabit Ethernet Network Driver1454lb # [ 1.580228] registered taskstats version 11455lb # [ 1.580412] Loading compiled-in X.509 certificates1456worker1 # [ 1.477317] igb: Copyright (c) 2007-2014 Intel Corporation.1457worker1 # [ 1.477341] igbvf: Intel(R) Gigabit Virtual Function Network Driver1458worker1 # [ 1.477347] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1459worker1 # [ 1.477485] sky2: driver version 1.301460client # [ 1.885767] usb 1-1: new high-speed USB device number 2 using ehci-pci1461worker1 # [ 1.487904] usbcore: registered new interface driver usb-storage1462lb # [ 1.593170] Demotion targets for Node 0: null1463worker1 # [ 1.488071] usbcore: registered new interface driver usbserial_generic1464lb # [ 1.593367] Key type .fscrypt registered1465worker1 # [ 1.488087] usbserial: USB Serial support registered for generic1466lb # [ 1.593376] Key type fscrypt-provisioning registered1467worker1 # [ 1.488900] hv_vmbus: registering driver hyperv_keyboard1468lb # [ 1.593502] ima: No TPM chip found, activating TPM-bypass!1469lb # [ 1.593525] ima: Allocated hash algorithm: sha11470worker1 # [ 1.490743] ehci-pci 0000:00:0a.0: EHCI Host Controller1471worker1 # [ 1.490784] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11472lb # [ 1.593551] ima: No architecture policies found1473worker1 # [ 1.491171] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001474lb # [ 1.598996] input: gpio-keys as /devices/platform/gpio-keys/input/input01475worker1 # [ 1.499407] rtc-pl031 9010000.pl031: registered as rtc01476worker1 # [ 1.499462] rtc-pl031 9010000.pl031: setting system clock to 2026-09-20T12:20:54 UTC (1789906854)1477worker1 # [ 1.499972] i2c_dev: i2c /dev entries driver1478worker1 # [ 1.503611] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001479worker1 # [ 1.504082] hub 1-0:1.0: USB hub found1480worker1 # [ 1.504107] hub 1-0:1.0: 6 ports detected1481worker1 # [ 1.509044] sdhci: Secure Digital Host Controller Interface driver1482worker1 # [ 1.509071] sdhci: Copyright(c) Pierre Ossman1483worker1 # [ 1.509342] Synopsys Designware Multimedia Card Interface Driver1484worker1 # [ 1.509721] sdhci-pltfm: SDHCI platform and OF driver helper1485worker1 # [ 1.515169] hid: raw HID events driver (C) Jiri Kosina1486worker1 # [ 1.515445] usbcore: registered new interface driver usbhid1487worker1 # [ 1.515454] usbhid: USB HID core driver1488worker2 # [ 0.928179] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1489worker1 # [ 1.518877] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1490worker1 # [ 1.522050] drop_monitor: Initializing network drop monitor service1491worker1 # [ 1.522352] NET: Registered PF_INET6 protocol family1492worker2 # [ 0.928481] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1493worker2 # [ 0.928504] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1494worker2 # [ 0.928533] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1495worker1 # [ 1.526431] Segment Routing with IPv61496worker1 # [ 1.526494] In-situ OAM (IOAM) with IPv61497worker1 # [ 1.526537] NET: Registered PF_PACKET protocol family1498worker1 # [ 1.528296] 9pnet: Installing 9P2000 support1499lb # [ 1.629295] clk: Disabling unused clocks1500lb # [ 1.629349] PM: genpd: Disabling unused power domains1501worker2 # [ 0.929150] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1502worker2 # [ 0.929336] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1503lb # [ 1.634320] Freeing unused kernel memory: 4736K1504lb # [ 1.634600] Run /init as init process1505worker2 # [ 0.929352] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1506worker2 # [ 0.929381] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1507worker2 # [ 0.930000] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1508worker2 # [ 0.930013] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1509worker2 # [ 0.930019] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1510worker2 # [ 0.930068] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1511worker1 # [ 1.552129] Key type dns_resolver registered1512lb # [ 1.654735] systemd[1]: Successfully made /usr/ read-only.1513worker1 # [ 1.559754] registered taskstats version 11514worker2 # [ 0.930118] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1515worker1 # [ 1.559945] Loading compiled-in X.509 certificates1516worker2 # [ 0.930167] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1517worker2 # [ 0.930217] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1518worker1 # [ 1.570804] Demotion targets for Node 0: null1519worker1 # [ 1.570968] Key type .fscrypt registered1520worker1 # [ 1.570976] Key type fscrypt-provisioning registered1521worker1 # [ 1.571102] ima: No TPM chip found, activating TPM-bypass!1522worker2 # [ 0.930267] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1523worker1 # [ 1.571123] ima: Allocated hash algorithm: sha11524worker2 # [ 0.930319] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1525worker1 # [ 1.571153] ima: No architecture policies found1526worker2 # [ 0.930370] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1527worker2 # [ 0.930418] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1528worker2 # [ 0.930467] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1529worker2 # [ 0.930529] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1530worker2 # [ 0.930574] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1531worker2 # [ 0.930596] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1532worker2 # [ 0.930619] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1533worker2 # [ 0.930641] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1534worker2 # [ 0.930665] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1535worker2 # [ 0.930688] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1536worker2 # [ 0.930710] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned1537worker2 # [ 0.930732] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1538worker1 # [ 1.591126] input: gpio-keys as /devices/platform/gpio-keys/input/input01539worker2 # [ 0.930754] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1540worker2 # [ 0.930776] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned1541worker2 # [ 0.930800] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1542worker2 # [ 0.930822] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1543worker2 # [ 0.930843] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned1544worker2 # [ 0.930864] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned1545worker2 # [ 0.930885] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned1546worker2 # [ 0.930907] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned1547worker2 # [ 0.930928] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned1548worker2 # [ 0.930949] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned1549worker2 # [ 0.930971] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned1550worker2 # [ 0.930993] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned1551client # [ 2.035618] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11552worker2 # [ 0.931014] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned1553worker1 # [ 1.643544] clk: Disabling unused clocks1554worker2 # [ 0.931040] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1555worker2 # [ 0.931050] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1556worker1 # [ 1.643609] PM: genpd: Disabling unused power domains1557worker2 # [ 0.931055] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1558worker2 # [ 0.937499] pci 0000:00:0a.0: enabling device (0000 -> 0002)1559worker1 # [ 1.650208] Freeing unused kernel memory: 4736K1560worker1 # [ 1.650558] Run /init as init process1561worker1 # [ 1.673339] systemd[1]: Successfully made /usr/ read-only.1562worker2 # [ 1.091163] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1563lb # [ 1.790114] usb 1-1: new high-speed USB device number 2 using ehci-pci1564worker2 # [ 1.109256] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1565worker2 # [ 1.116887] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1566client # [ 2.105654] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01567worker2 # [ 1.131545] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1568worker2 # [ 1.134134] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003)1569client # [ 2.115934] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1570client # [ 2.120556] systemd[1]: Detected virtualization qemu.1571client # [ 2.121309] systemd[1]: Detected architecture arm64.1572client # [ 2.122004] systemd[1]: Running in initrd.1573client # [ 2.123019] systemd[1]: Initializing machine ID from random generator.1574client # [ 2.124008] systemd[1]: Hostname set to <client>.1575worker2 # [ 1.154748] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003)1576worker1 # [ 1.750601] usb 1-1: new high-speed USB device number 2 using ehci-pci1577worker2 # [ 1.164988] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)1578worker2 # [ 1.185114] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1579worker2 # [ 1.191634] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1580worker2 # [ 1.209744] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1581worker2 # [ 1.237981] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1582client # [ 2.229557] usb 1-2: new high-speed USB device number 3 using ehci-pci1583lb # [ 1.940046] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11584worker2 # [ 1.268766] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1585worker2 # [ 1.271905] msm_serial: driver initialized1586worker2 # [ 1.277179] SuperH (H)SCI(F) driver initialized1587worker2 # [ 1.277262] STM32 USART driver initialized1588lb # [ 1.989890] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1589lb # [ 1.994349] systemd[1]: Detected virtualization qemu.1590lb # [ 1.995018] systemd[1]: Detected architecture arm64.1591lb # [ 1.995628] systemd[1]: Running in initrd.1592lb # [ 1.996585] systemd[1]: Initializing machine ID from random generator.1593lb # [ 1.997508] systemd[1]: Hostname set to <lb>.1594worker1 # [ 1.901644] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11595worker2 # [ 1.331096] loop: module loaded1596worker2 # [ 1.332130] virtio_blk virtio5: 1/0/0 default/read/poll queues1597worker2 # [ 1.333067] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1598worker2 # [ 1.346433] megasas: 07.734.00.00-rc11599lb # [ 2.030381] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01600worker2 # [ 1.356222] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1601worker2 # [ 1.361581] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001602worker2 # [ 1.361639] Intel/Sharp Extended Query Table at 0x00311603worker2 # [ 1.367356] Using buffer write method1604worker2 # [ 1.367455] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1605worker2 # [ 1.384191] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001606worker2 # [ 1.384261] Intel/Sharp Extended Query Table at 0x00311607worker2 # [ 1.386205] Using buffer write method1608worker2 # [ 1.386256] Concatenating MTD devices:1609worker2 # [ 1.386260] (0): "0.flash"1610worker2 # [ 1.386264] (1): "0.flash"1611worker2 # [ 1.386267] into device "0.flash"1612worker1 # [ 1.974376] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01613client # [ 2.392612] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21614client # [ 2.394687] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01615client # [ 2.416443] systemd[1]: bpf-restrict-fs: LSM BPF program attached1616worker1 # [ 2.013011] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1617worker1 # [ 2.023577] systemd[1]: Detected virtualization qemu.1618worker1 # [ 2.024315] systemd[1]: Detected architecture arm64.1619worker1 # [ 2.024966] systemd[1]: Running in initrd.1620worker1 # [ 2.026025] systemd[1]: Initializing machine ID from random generator.1621worker1 # [ 2.029647] systemd[1]: Hostname set to <worker1>.1622lb # [ 2.150110] usb 1-2: new high-speed USB device number 3 using ehci-pci1623client # [ 2.488392] systemd[1]: Queued start job for default target Initrd Default Target.1624worker1 # [ 2.098060] usb 1-2: new high-speed USB device number 3 using ehci-pci1625client # [ 2.503613] systemd[1]: Created slice Slice /system/modprobe.1626client # [ 2.504730] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1627client # [ 2.505915] systemd[1]: Expecting device /dev/disk/by-label/nixos...1628client # [ 2.505956] systemd[1]: Reached target Path Units.1629client # [ 2.505973] systemd[1]: Reached target Slice Units.1630client # [ 2.505991] systemd[1]: Reached target Swaps.1631client # [ 2.506007] systemd[1]: Reached target Timer Units.1632client # [ 2.506250] systemd[1]: Listening on D-Bus System Message Bus Socket.1633client # [ 2.506383] systemd[1]: Listening on Journal Socket (/dev/log).1634client # [ 2.506506] systemd[1]: Listening on Journal Sockets.1635client # [ 2.506617] systemd[1]: Listening on udev Control Socket.1636client # [ 2.506716] systemd[1]: Listening on udev Kernel Socket.1637client # [ 2.506731] systemd[1]: Reached target Socket Units.1638client # [ 2.516468] systemd[1]: Starting Create List of Static Device Nodes...1639client # [ 2.529710] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1640client # [ 2.530739] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1641client # [ 2.537111] systemd[1]: Mounting Kernel Configuration File System...1642lb # [ 2.259095] systemd[1]: bpf-restrict-fs: LSM BPF program attached1643client # [ 2.565992] systemd[1]: Starting Journal Service...1644client # [ 2.577739] systemd[1]: Starting Load Kernel Modules...1645client # [ 2.577961] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1646lb # [ 2.308477] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21647client # [ 2.608256] systemd-journald[73]: Collecting audit messages is disabled.1648client # [ 2.609545] systemd[1]: Starting Coldplug All udev Devices...1649lb # [ 2.311176] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01650client # [ 2.612066] systemd[1]: Finished Create List of Static Device Nodes.1651client # [ 2.614555] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1652client # [ 2.632076] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1653worker1 # [ 2.251475] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21654client # [ 2.651155] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1655client # [ 2.652581] systemd[1]: Mounted Kernel Configuration File System.1656worker1 # [ 2.255719] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01657lb # [ 2.356101] systemd[1]: Queued start job for default target Initrd Default Target.1658lb # [ 2.368769] systemd[1]: Created slice Slice /system/modprobe.1659lb # [ 2.369893] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1660lb # [ 2.371190] systemd[1]: Expecting device /dev/disk/by-label/nixos...1661lb # [ 2.371235] systemd[1]: Reached target Path Units.1662lb # [ 2.371254] systemd[1]: Reached target Slice Units.1663lb # [ 2.371275] systemd[1]: Reached target Swaps.1664lb # [ 2.371295] systemd[1]: Reached target Timer Units.1665lb # [ 2.371506] systemd[1]: Listening on D-Bus System Message Bus Socket.1666lb # [ 2.371653] systemd[1]: Listening on Journal Socket (/dev/log).1667lb # [ 2.371776] systemd[1]: Listening on Journal Sockets.1668lb # [ 2.371897] systemd[1]: Listening on udev Control Socket.1669lb # [ 2.372005] systemd[1]: Listening on udev Kernel Socket.1670lb # [ 2.372027] systemd[1]: Reached target Socket Units.1671lb # [ 2.385886] systemd[1]: Starting Create List of Static Device Nodes...1672lb # [ 2.399720] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1673lb # [ 2.399849] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1674client # [ 2.703656] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1675lb # [ 2.414435] systemd[1]: Mounting Kernel Configuration File System...1676worker1 # [ 2.321534] systemd[1]: bpf-restrict-fs: LSM BPF program attached1677lb # [ 2.434868] systemd[1]: Starting Journal Service...1678client # [ 2.729492] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1679client # [ 2.736056] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01680client # [ 2.736335] [drm] features: -virgl +edid -resource_blob -host_visible1681client # [ 2.736348] [drm] features: -context_init1682client # [ 2.737181] [drm] number of scanouts: 11683client # [ 2.737203] [drm] number of cap sets: 01684lb # [ 2.454387] systemd[1]: Starting Load Kernel Modules...1685lb # [ 2.455385] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1686client # [ 2.754254] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1687client # [ 2.758217] systemd[1]: Starting Create Static Device Nodes in /dev...1688lb # [ 2.473128] systemd[1]: Starting Coldplug All udev Devices...1689client # [ 2.769575] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1690client # [ 2.769611] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01691worker1 # [ 2.393410] systemd[1]: Queued start job for default target Initrd Default Target.1692worker1 # [ 2.404761] systemd[1]: Created slice Slice /system/modprobe.1693lb # [ 2.510367] systemd[1]: Finished Create List of Static Device Nodes.1694lb # [ 2.511880] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1695lb # [ 2.512278] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1696lb # [ 2.512666] systemd[1]: Mounted Kernel Configuration File System.1697worker1 # [ 2.415703] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1698worker1 # [ 2.415767] systemd[1]: Expecting device /dev/disk/by-label/nixos...1699client # [ 2.702359] systemd-modules-load[75]: Inserted module 'dm_mod'1700worker1 # [ 2.415807] systemd[1]: Reached target Path Units.1701worker1 # [ 2.415824] systemd[1]: Reached target Slice Units.1702client # [ 2.817215] systemd[1]: Started Journal Service.1703worker1 # [ 2.415840] systemd[1]: Reached target Swaps.1704worker1 # [ 2.415857] systemd[1]: Reached target Timer Units.1705worker1 # [ 2.417037] systemd[1]: Listening on D-Bus System Message Bus Socket.1706worker1 # [ 2.425118] systemd[1]: Listening on Journal Socket (/dev/log).1707worker1 # [ 2.426186] systemd[1]: Listening on Journal Sockets.1708worker1 # [ 2.427436] systemd[1]: Listening on udev Control Socket.1709worker1 # [ 2.428302] systemd[1]: Listening on udev Kernel Socket.1710worker1 # [ 2.429020] systemd[1]: Reached target Socket Units.1711lb # [ 2.530715] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1712worker1 # [ 2.431871] systemd[1]: Starting Create List of Static Device Nodes...1713client # [ 2.823161] Console: switching to colour frame buffer device 160x501714client # [ 2.825078] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1715client # [ 2.714132] systemd-modules-load[75]: Module 'virtio_balloon' is built in1716client # [ 2.724563] systemd-modules-load[75]: Module 'virtio_console' is built in1717worker1 # [ 2.440200] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1718worker1 # [ 2.447726] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1719client # [ 2.731733] systemd-modules-load[75]: Inserted module 'virtio_gpu'1720client # [ 2.741531] systemd-modules-load[75]: Module 'virtio_rng' is built in1721client # [ 2.746669] systemd[1]: Finished Load Kernel Modules.1722client # [ 2.756997] systemd[1]: Starting Apply Kernel Variables...1723client # [ 2.759333] systemd[1]: Finished Create Static Device Nodes in /dev.1724worker1 # [ 2.470780] systemd[1]: Mounting Kernel Configuration File System...1725lb # [ 2.579469] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1726lb # [ 2.583743] systemd-journald[73]: Collecting audit messages is disabled.1727client # [ 2.775675] systemd[1]: Reached target Preparation for Local File Systems.1728client # [ 2.784594] systemd[1]: Reached target Local File Systems.1729worker1 # [ 2.504370] systemd[1]: Starting Journal Service...1730lb # [ 2.602076] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1731lb # [ 2.607961] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01732lb # [ 2.608244] [drm] features: -virgl +edid -resource_blob -host_visible1733lb # [ 2.608257] [drm] features: -context_init1734lb # [ 2.609065] [drm] number of scanouts: 11735lb # [ 2.609087] [drm] number of cap sets: 01736client # [ 2.796226] systemd[1]: Starting Create System Files and Directories...1737lb # [ 2.618899] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1738worker1 # [ 2.522866] systemd[1]: Starting Load Kernel Modules...1739worker1 # [ 2.524045] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1740client # [ 2.810523] systemd[1]: Starting Rule-based Manager for Device Events and Files...1741lb # [ 2.626536] systemd[1]: Starting Create Static Device Nodes in /dev...1742lb # [ 2.634740] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1743lb # [ 2.634774] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01744client # [ 2.834226] systemd[1]: Finished Apply Kernel Variables.1745worker2 # [ 1.974595] Freeing initrd memory: 26148K1746worker2 # [ 1.981534] tun: Universal TUN/TAP device driver, 1.61747worker2 # [ 1.986874] thunder_xcv, ver 1.01748worker2 # [ 1.986940] thunder_bgx, ver 1.01749worker2 # [ 1.986965] nicpf, ver 1.01750worker2 # [ 1.988857] e1000: Intel(R) PRO/1000 Network Driver1751worker2 # [ 1.988871] e1000: Copyright (c) 1999-2006 Intel Corporation.1752worker2 # [ 1.988897] e1000e: Intel(R) PRO/1000 Network Driver1753worker2 # [ 1.988906] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1754worker2 # [ 1.988939] igb: Intel(R) Gigabit Ethernet Network Driver1755worker2 # [ 1.988945] igb: Copyright (c) 2007-2014 Intel Corporation.1756worker2 # [ 1.988968] igbvf: Intel(R) Gigabit Virtual Function Network Driver1757worker2 # [ 1.988975] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1758worker2 # [ 1.989111] sky2: driver version 1.301759worker2 # [ 1.990812] usbcore: registered new interface driver usb-storage1760worker2 # [ 1.990951] usbcore: registered new interface driver usbserial_generic1761worker2 # [ 1.990967] usbserial: USB Serial support registered for generic1762worker2 # [ 2.000096] hv_vmbus: registering driver hyperv_keyboard1763lb # [ 2.673663] Console: switching to colour frame buffer device 160x501764worker2 # [ 2.002282] ehci-pci 0000:00:0a.0: EHCI Host Controller1765worker2 # [ 2.002326] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 11766worker2 # [ 2.002574] ehci-pci 0000:00:0a.0: irq 15, io mem 0x100890001767worker1 # [ 2.582320] systemd[1]: Starting Coldplug All udev Devices...1768worker1 # [ 2.584997] systemd[1]: Finished Create List of Static Device Nodes.1769worker2 # [ 2.006014] rtc-pl031 9010000.pl031: registered as rtc01770worker2 # [ 2.006104] rtc-pl031 9010000.pl031: setting system clock to 2026-09-20T12:20:55 UTC (1789906855)1771worker1 # [ 2.587383] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1772worker2 # [ 2.006847] i2c_dev: i2c /dev entries driver1773client # [ 2.873360] systemd[1]: Finished Create System Files and Directories.1774worker2 # [ 2.013551] sdhci: Secure Digital Host Controller Interface driver1775worker2 # [ 2.013576] sdhci: Copyright(c) Pierre Ossman1776worker2 # [ 2.013861] Synopsys Designware Multimedia Card Interface Driver1777worker2 # [ 2.014233] sdhci-pltfm: SDHCI platform and OF driver helper1778worker2 # [ 2.019221] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.001779worker2 # [ 2.020763] hub 1-0:1.0: USB hub found1780lb # [ 2.699203] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1781worker2 # [ 2.021503] hub 1-0:1.0: 6 ports detected1782worker1 # [ 2.602806] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1783worker2 # [ 2.024024] hid: raw HID events driver (C) Jiri Kosina1784worker2 # [ 2.024325] usbcore: registered new interface driver usbhid1785lb # [ 2.704937] systemd[1]: Finished Create Static Device Nodes in /dev.1786worker2 # [ 2.024333] usbhid: USB HID core driver1787lb # [ 2.706457] systemd[1]: Reached target Preparation for Local File Systems.1788worker2 # [ 2.028102] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1789worker1 # [ 2.607166] systemd[1]: Mounted Kernel Configuration File System.1790worker2 # [ 2.029844] drop_monitor: Initializing network drop monitor service1791worker2 # [ 2.030063] NET: Registered PF_INET6 protocol family1792lb # [ 2.708342] systemd[1]: Reached target Local File Systems.1793worker2 # [ 2.033354] Segment Routing with IPv61794worker2 # [ 2.033381] In-situ OAM (IOAM) with IPv61795worker2 # [ 2.033416] NET: Registered PF_PACKET protocol family1796worker1 # [ 2.612437] systemd-journald[73]: Collecting audit messages is disabled.1797worker2 # [ 2.035428] 9pnet: Installing 9P2000 support1798worker2 # [ 2.038107] Key type dns_resolver registered1799client # [ 2.907483] systemd-udevd[88]: Using default interface naming scheme 'v261'.1800worker2 # [ 2.045414] registered taskstats version 11801worker2 # [ 2.045624] Loading compiled-in X.509 certificates1802worker1 # [ 2.627969] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1803worker2 # [ 2.054853] Demotion targets for Node 0: null1804worker2 # [ 2.055034] Key type .fscrypt registered1805worker2 # [ 2.055044] Key type fscrypt-provisioning registered1806worker2 # [ 2.055166] ima: No TPM chip found, activating TPM-bypass!1807worker2 # [ 2.055191] ima: Allocated hash algorithm: sha11808worker2 # [ 2.055221] ima: No architecture policies found1809lb # [ 2.730931] systemd[1]: Starting Rule-based Manager for Device Events and Files...1810worker2 # [ 2.060383] input: gpio-keys as /devices/platform/gpio-keys/input/input01811worker1 # [ 2.640673] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1812worker1 # [ 2.645689] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1813client # [ 2.943450] systemd[1]: Started Rule-based Manager for Device Events and Files.1814worker1 # [ 2.659543] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01815lb # [ 2.759173] systemd[1]: Finished Load Kernel Modules.1816worker1 # [ 2.659835] [drm] features: -virgl +edid -resource_blob -host_visible1817worker1 # [ 2.659847] [drm] features: -context_init1818worker1 # [ 2.660688] [drm] number of scanouts: 11819worker1 # [ 2.660710] [drm] number of cap sets: 01820worker2 # [ 2.093290] clk: Disabling unused clocks1821worker2 # [ 2.093339] PM: genpd: Disabling unused power domains1822worker2 # [ 2.099632] Freeing unused kernel memory: 4736K1823worker2 # [ 2.099867] Run /init as init process1824lb # [ 2.799080] systemd[1]: Starting Apply Kernel Variables...1825worker1 # [ 2.695317] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1826worker2 # [ 2.119470] systemd[1]: Successfully made /usr/ read-only.1827worker1 # [ 2.695357] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01828worker1 # [ 2.746321] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1829client # [ 3.036256] systemd[1]: Starting Virtual Console Setup...1830worker1 # [ 2.753255] Console: switching to colour frame buffer device 160x501831worker1 # [ 2.761081] systemd[1]: Starting Create Static Device Nodes in /dev...1832worker1 # [ 2.791989] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1833worker1 # [ 2.803091] systemd[1]: Finished Load Kernel Modules.1834client # [ 3.096719] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1835worker1 # [ 2.820594] systemd[1]: Starting Apply Kernel Variables...1836client # [ 3.116955] systemd[1]: Finished Virtual Console Setup.1837worker2 # [ 2.267413] usb 1-1: new high-speed USB device number 2 using ehci-pci1838worker1 # [ 2.876312] systemd[1]: Finished Create Static Device Nodes in /dev.1839worker1 # [ 2.881679] systemd[1]: Reached target Preparation for Local File Systems.1840lb # [ 2.984436] systemd[1]: Started Journal Service.1841lb # [ 2.972606] systemd-modules-load[74]: Inserted module 'dm_mod'1842lb # [ 2.981090] systemd-modules-load[74]: Module 'virtio_balloon' is built in1843lb # [ 2.983019] systemd-modules-load[74]: Module 'virtio_console' is built in1844worker1 # [ 2.900223] systemd[1]: Reached target Local File Systems.1845worker1 # [ 2.903687] systemd[1]: Starting Rule-based Manager for Device Events and Files...1846worker1 # [ 2.905997] systemd-modules-load[74]: Inserted module 'dm_mod'1847lb # [ 3.018096] systemd-modules-load[74]: Inserted module 'virtio_gpu'1848worker1 # [ 2.911897] systemd-modules-load[74]: Module 'virtio_balloon' is built in1849worker1 # [ 2.944731] systemd[1]: Started Journal Service.1850lb # [ 3.028305] systemd-modules-load[74]: Module 'virtio_rng' is built in1851worker1 # [ 2.932539] systemd-modules-load[74]: Module 'virtio_console' is built in1852worker1 # [ 2.935606] systemd-modules-load[74]: Inserted module 'virtio_gpu'1853lb # [ 3.044525] systemd[1]: Starting Create System Files and Directories...1854worker1 # [ 2.944818] systemd-modules-load[74]: Module 'virtio_rng' is built in1855worker1 # [ 2.946550] systemd[1]: Finished Apply Kernel Variables.1856worker1 # [ 2.947869] systemd[1]: Starting Create System Files and Directories...1857lb # [ 3.056875] systemd[1]: Finished Apply Kernel Variables.1858lb # [ 3.096613] systemd-udevd[79]: Using default interface naming scheme 'v261'.1859lb # [ 3.099681] systemd[1]: Finished Create System Files and Directories.1860worker2 # [ 2.438716] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11861worker1 # [ 3.005354] systemd-udevd[81]: Using default interface naming scheme 'v261'.1862lb # [ 3.126627] systemd[1]: Started Rule-based Manager for Device Events and Files.1863worker1 # [ 3.021381] systemd[1]: Finished Create System Files and Directories.1864worker2 # [ 2.456609] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1865worker1 # [ 3.045969] systemd[1]: Started Rule-based Manager for Device Events and Files.1866worker2 # [ 2.491365] systemd[1]: Detected virtualization qemu.1867worker2 # [ 2.500303] systemd[1]: Detected architecture arm64.1868worker2 # [ 2.500934] systemd[1]: Running in initrd.1869worker2 # [ 2.507854] systemd[1]: Initializing machine ID from random generator.1870worker2 # [ 2.509071] systemd[1]: Hostname set to <worker2>.1871lb # [ 3.200836] systemd[1]: Starting Virtual Console Setup...1872worker2 # [ 2.538805] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input01873worker1 # [ 3.136985] systemd[1]: Starting Virtual Console Setup...1874lb # [ 3.264808] systemd-vconsole-setup[102]: Configuration of first virtual console was skipped, ignoring remaining ones.1875lb # [ 3.270042] systemd[1]: Finished Virtual Console Setup.1876worker2 # [ 2.667398] usb 1-2: new high-speed USB device number 3 using ehci-pci1877worker1 # [ 3.224710] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones.1878worker1 # [ 3.229966] systemd[1]: Finished Virtual Console Setup.1879worker2 # [ 2.815938] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21880worker2 # [ 2.821993] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input01881worker2 # [ 2.946021] systemd[1]: bpf-restrict-fs: LSM BPF program attached1882worker2 # [ 3.014253] systemd[1]: Queued start job for default target Initrd Default Target.1883worker2 # [ 3.026121] systemd[1]: Created slice Slice /system/modprobe.1884worker2 # [ 3.027703] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1885worker2 # [ 3.029261] systemd[1]: Expecting device /dev/disk/by-label/nixos...1886worker2 # [ 3.030387] systemd[1]: Reached target Path Units.1887worker2 # [ 3.031339] systemd[1]: Reached target Slice Units.1888worker2 # [ 3.032247] systemd[1]: Reached target Swaps.1889worker2 # [ 3.033031] systemd[1]: Reached target Timer Units.1890worker2 # [ 3.034136] systemd[1]: Listening on D-Bus System Message Bus Socket.1891worker2 # [ 3.035628] systemd[1]: Listening on Journal Socket (/dev/log).1892worker2 # [ 3.035757] systemd[1]: Listening on Journal Sockets.1893worker2 # [ 3.035873] systemd[1]: Listening on udev Control Socket.1894worker2 # [ 3.036027] systemd[1]: Listening on udev Kernel Socket.1895worker2 # [ 3.036047] systemd[1]: Reached target Socket Units.1896worker2 # [ 3.042672] systemd[1]: Starting Create List of Static Device Nodes...1897worker2 # [ 3.051533] systemd[1]: Starting Load Kernel Module 9pnet_virtio...1898worker2 # [ 3.051637] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1899worker2 # [ 3.063551] systemd[1]: Mounting Kernel Configuration File System...1900worker2 # [ 3.090120] systemd[1]: Starting Journal Service...1901worker2 # [ 3.096890] systemd[1]: Starting Load Kernel Modules...1902worker2 # [ 3.098290] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1903client # [ 3.964418] systemd[1]: Finished Coldplug All udev Devices.1904client # [ 3.966543] systemd[1]: Reached target System Initialization.1905client # [ 3.967916] systemd[1]: Reached target Basic System.1906worker2 # [ 3.110374] systemd[1]: Starting Coldplug All udev Devices...1907client # [ 4.144353] (udev-worker)[95]: Network interface NamePolicy= disabled on kernel command line.1908client # [ 4.199540] (udev-worker)[99]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1909client # [ 4.209360] (udev-worker)[99]: Network interface NamePolicy= disabled on kernel command line.1910client # [ 4.303630] systemd[1]: Found device /dev/disk/by-label/nixos.1911client # [ 4.305990] systemd[1]: Reached target Initrd Root Device.1912worker1 # [ 3.991885] systemd[1]: Finished Coldplug All udev Devices.1913worker1 # [ 4.001353] systemd[1]: Reached target System Initialization.1914worker1 # [ 4.002386] systemd[1]: Reached target Basic System.1915client # [ 4.313485] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1916client # [ 4.384451] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks1917client # [ 4.398242] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1918client # [ 4.416679] systemd[1]: Mounting /sysroot...1919client # [ 4.562494] EXT4-fs (vda): mounted filesystem 32f2a617-6fbd-4a16-9ba5-91b9449d1caa r/w with ordered data mode. Quota mode: none.1920client # [ 4.461188] systemd[1]: Mounted /sysroot.1921client # [ 4.463110] systemd[1]: Reached target Initrd Root File System.1922worker2 # [ 3.514928] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1923worker2 # [ 3.521046] systemd[1]: Finished Create List of Static Device Nodes.1924worker2 # [ 3.522173] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully.1925worker2 # [ 3.522536] systemd[1]: Finished Load Kernel Module 9pnet_virtio.1926worker2 # [ 3.522915] systemd[1]: Mounted Kernel Configuration File System.1927worker2 # [ 3.531792] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1928worker2 # [ 3.612212] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1929client # [ 4.492343] systemd[1]: Starting Mountpoints Configured in the Real Root...1930client # [ 4.528191] systemd-sysroot-fstab-check[120]: /sysroot should be mounted in the initrd, will request daemon-reload.1931client # [ 4.535202] systemd[1]: Mounting /sysroot/nix/.ro-store...1932client # [ 4.552369] systemd[1]: Mounting /sysroot/nix/.rw-store...1933worker2 # [ 3.628542] [drm] pci: virtio-gpu-pci detected at 0000:00:08.01934worker2 # [ 3.628919] [drm] features: -virgl +edid -resource_blob -host_visible1935worker2 # [ 3.628929] [drm] features: -context_init1936worker2 # [ 3.629723] [drm] number of scanouts: 11937worker2 # [ 3.629741] [drm] number of cap sets: 01938worker2 # [ 3.665300] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic1939worker2 # [ 3.665339] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 01940worker2 # [ 3.676051] Console: switching to colour frame buffer device 160x501941client # [ 4.568816] systemd[1]: Mounting /sysroot/run...1942worker2 # [ 3.676712] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device1943lb # [ 4.364499] systemd[1]: Finished Coldplug All udev Devices.1944lb # [ 4.365543] systemd[1]: Reached target System Initialization.1945lb # [ 4.366330] systemd[1]: Reached target Basic System.1946worker2 # [ 3.756285] systemd[1]: Finished Load Kernel Modules.1947worker2 # [ 3.796494] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1948worker2 # [ 3.801502] systemd[1]: Starting Apply Kernel Variables...1949client # [ 4.676188] systemd[1]: Mounting /sysroot/tmp/shared...1950client # [ 4.701955] systemd[1]: Mounting /sysroot/tmp/xchg...1951client # [ 4.712481] systemd[1]: Reload requested from client PID 120 ('systemd-sysroot') (unit initrd-parse-etc.service)...1952client # [ 4.715107] systemd[1]: Reloading...1953worker2 # [ 3.823522] systemd[1]: Starting Create Static Device Nodes in /dev...1954worker1 # [ 4.410295] (udev-worker)[98]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1955worker1 # [ 4.465318] (udev-worker)[106]: Network interface NamePolicy= disabled on kernel command line.1956worker1 # [ 4.520417] systemd[1]: Found device /dev/disk/by-label/nixos.1957worker1 # [ 4.524759] systemd[1]: Reached target Initrd Root Device.1958worker1 # [ 4.529231] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1959worker2 # [ 3.980211] systemd[1]: Finished Apply Kernel Variables.1960lb # [ 4.647335] (udev-worker)[97]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1961worker2 # [ 3.992266] systemd[1]: Finished Create Static Device Nodes in /dev.1962worker2 # [ 3.992561] systemd[1]: Reached target Preparation for Local File Systems.1963lb # [ 4.654945] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line.1964worker2 # [ 3.992604] systemd[1]: Reached target Local File Systems.1965lb # [ 4.666777] (udev-worker)[95]: Network interface NamePolicy= disabled on kernel command line.1966worker2 # [ 4.011604] systemd[1]: Starting Rule-based Manager for Device Events and Files...1967worker1 # [ 4.588719] systemd-fsck[111]: nixos: clean, 12/65536 files, 13019/262144 blocks1968worker1 # [ 4.614540] (udev-worker)[98]: Network interface NamePolicy= disabled on kernel command line.1969worker2 # [ 4.056384] systemd-journald[73]: Collecting audit messages is disabled.1970worker1 # [ 4.623501] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1971worker1 # [ 4.637364] systemd[1]: Mounting /sysroot...1972lb # [ 4.752408] systemd[1]: Found device /dev/disk/by-label/nixos.1973lb # [ 4.755037] systemd[1]: Reached target Initrd Root Device.1974lb # [ 4.760155] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1975worker1 # [ 4.741888] EXT4-fs (vda): mounted filesystem 69078a81-8b9f-412b-ab7e-fd25acca65d4 r/w with ordered data mode. Quota mode: none.1976lb # [ 4.830204] systemd-fsck[111]: nixos: clean, 12/65536 files, 13019/262144 blocks1977worker2 # [ 4.177265] systemd[1]: Started Rule-based Manager for Device Events and Files.1978worker1 # [ 4.727858] systemd[1]: Mounted /sysroot.1979worker1 # [ 4.731559] systemd[1]: Reached target Initrd Root File System.1980lb # [ 4.844130] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1981worker1 # [ 4.742209] systemd[1]: Mounting /sysroot/nix/.ro-store...1982lb # [ 4.853129] systemd[1]: Mounting /sysroot...1983worker2 # [ 4.199569] systemd[1]: Started Journal Service.1984worker2 # [ 4.184468] systemd-modules-load[74]: Inserted module 'dm_mod'1985worker2 # [ 4.190051] systemd-modules-load[74]: Module 'virtio_balloon' is built in1986worker1 # [ 4.760896] systemd[1]: Mounting /sysroot/nix/.rw-store...1987worker2 # [ 4.204773] systemd-modules-load[74]: Module 'virtio_console' is built in1988worker1 # [ 4.775797] systemd[1]: Mounting /sysroot/run...1989worker2 # [ 4.206667] systemd-modules-load[74]: Inserted module 'virtio_gpu'1990worker2 # [ 4.221301] systemd-modules-load[74]: Module 'virtio_rng' is built in1991worker2 # [ 4.222902] systemd-udevd[80]: Using default interface naming scheme 'v261'.1992lb # [ 4.923027] EXT4-fs (vda): mounted filesystem fa3e8856-a0fc-4d3d-b184-d1a9aa0fd7fb r/w with ordered data mode. Quota mode: none.1993lb # [ 4.909724] systemd[1]: Mounted /sysroot.1994worker1 # [ 4.802717] systemd[1]: Mounting /sysroot/tmp/shared...1995worker2 # [ 4.234697] systemd[1]: Starting Create System Files and Directories...1996lb # [ 4.913764] systemd[1]: Reached target Initrd Root File System.1997lb # [ 4.924285] systemd[1]: Starting Mountpoints Configured in the Real Root...1998lb # [ 4.953186] systemd-sysroot-fstab-check[119]: /sysroot should be mounted in the initrd, will request daemon-reload.1999lb # [ 4.960132] systemd[1]: Reload requested from client PID 119 ('systemd-sysroot') (unit initrd-parse-etc.service)...2000lb # [ 4.966527] systemd[1]: Reloading...
loading 1,772 hidden lines…
3773worker2 # [ 44.100365] nix-grpc-daemon[862]: ts=2026-09-20T12:21:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3774worker1 # [ 44.849234] nix-grpc-daemon[855]: ts=2026-09-20T12:21:37Z level=info event=scheduler_connected addr=lb:500513775worker1 # [ 44.858174] nix-grpc-daemon[855]: ts=2026-09-20T12:21:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3776worker2 # [ 44.297050] nix-grpc-daemon[862]: ts=2026-09-20T12:21:37Z level=info event=scheduler_connected addr=lb:500513777worker2 # [ 44.303684] nix-grpc-daemon[862]: ts=2026-09-20T12:21:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3778worker1 # [ 45.059185] nix-grpc-daemon[855]: ts=2026-09-20T12:21:38Z level=info event=scheduler_connected addr=lb:500513779worker1 # [ 45.067975] nix-grpc-daemon[855]: ts=2026-09-20T12:21:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3780worker2 # [ 44.505639] nix-grpc-daemon[862]: ts=2026-09-20T12:21:38Z level=info event=scheduler_connected addr=lb:500513781worker2 # [ 44.513387] nix-grpc-daemon[862]: ts=2026-09-20T12:21:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3782worker1 # [ 45.269511] nix-grpc-daemon[855]: ts=2026-09-20T12:21:38Z level=info event=scheduler_connected addr=lb:500513783worker1 # [ 45.276522] nix-grpc-daemon[855]: ts=2026-09-20T12:21:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3784worker2 # [ 44.713905] nix-grpc-daemon[862]: ts=2026-09-20T12:21:38Z level=info event=scheduler_connected addr=lb:500513785worker2 # [ 44.717083] nix-grpc-daemon[862]: ts=2026-09-20T12:21:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3786worker1 # [ 45.479321] nix-grpc-daemon[855]: ts=2026-09-20T12:21:38Z level=info event=scheduler_connected addr=lb:500513787worker1 # [ 45.483933] nix-grpc-daemon[855]: ts=2026-09-20T12:21:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3788worker2 # [ 44.918412] nix-grpc-daemon[862]: ts=2026-09-20T12:21:38Z level=info event=scheduler_connected addr=lb:500513789worker2 # [ 44.923431] nix-grpc-daemon[862]: ts=2026-09-20T12:21:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3790worker1 # [ 45.680301] nix-grpc-daemon[855]: ts=2026-09-20T12:21:38Z level=info event=scheduler_connected addr=lb:500513791worker1 # [ 45.685980] nix-grpc-daemon[855]: ts=2026-09-20T12:21:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3792worker2 # [ 45.125393] nix-grpc-daemon[862]: ts=2026-09-20T12:21:38Z level=info event=scheduler_connected addr=lb:500513793worker2 # [ 45.127826] nix-grpc-daemon[862]: ts=2026-09-20T12:21:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3794worker1 # [ 45.887519] nix-grpc-daemon[855]: ts=2026-09-20T12:21:38Z level=info event=scheduler_connected addr=lb:500513795worker2 # [ 45.327796] nix-grpc-daemon[862]: ts=2026-09-20T12:21:38Z level=info event=scheduler_connected addr=lb:500513796worker1 # [ 45.895049] nix-grpc-daemon[855]: ts=2026-09-20T12:21:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3797worker2 # [ 45.335085] nix-grpc-daemon[862]: ts=2026-09-20T12:21:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3798worker1 # [ 46.095610] nix-grpc-daemon[855]: ts=2026-09-20T12:21:39Z level=info event=scheduler_connected addr=lb:500513799worker1 # [ 46.099652] nix-grpc-daemon[855]: ts=2026-09-20T12:21:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3800worker2 # [ 45.536419] nix-grpc-daemon[862]: ts=2026-09-20T12:21:39Z level=info event=scheduler_connected addr=lb:500513801worker2 # [ 45.542554] nix-grpc-daemon[862]: ts=2026-09-20T12:21:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3802worker1 # [ 46.300898] nix-grpc-daemon[855]: ts=2026-09-20T12:21:39Z level=info event=scheduler_connected addr=lb:500513803worker1 # [ 46.307047] nix-grpc-daemon[855]: ts=2026-09-20T12:21:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3804worker2 # [ 45.743934] nix-grpc-daemon[862]: ts=2026-09-20T12:21:39Z level=info event=scheduler_connected addr=lb:500513805worker2 # [ 45.750420] nix-grpc-daemon[862]: ts=2026-09-20T12:21:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3806worker1 # [ 46.509646] nix-grpc-daemon[855]: ts=2026-09-20T12:21:39Z level=info event=scheduler_connected addr=lb:500513807worker1 # [ 46.515530] nix-grpc-daemon[855]: ts=2026-09-20T12:21:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3808worker2 # [ 45.952060] nix-grpc-daemon[862]: ts=2026-09-20T12:21:39Z level=info event=scheduler_connected addr=lb:500513809worker2 # [ 45.958047] nix-grpc-daemon[862]: ts=2026-09-20T12:21:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3810worker1 # [ 46.716600] nix-grpc-daemon[855]: ts=2026-09-20T12:21:39Z level=info event=scheduler_connected addr=lb:500513811worker1 # [ 46.722221] nix-grpc-daemon[855]: ts=2026-09-20T12:21:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3812worker2 # [ 46.161268] nix-grpc-daemon[862]: ts=2026-09-20T12:21:39Z level=info event=scheduler_connected addr=lb:500513813worker2 # [ 46.166394] nix-grpc-daemon[862]: ts=2026-09-20T12:21:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3814worker1 # [ 46.923085] nix-grpc-daemon[855]: ts=2026-09-20T12:21:39Z level=info event=scheduler_connected addr=lb:500513815worker1 # [ 46.928803] nix-grpc-daemon[855]: ts=2026-09-20T12:21:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3816worker2 # [ 46.367227] nix-grpc-daemon[862]: ts=2026-09-20T12:21:39Z level=info event=scheduler_connected addr=lb:500513817worker2 # [ 46.369648] nix-grpc-daemon[862]: ts=2026-09-20T12:21:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3818worker1 # [ 47.129620] nix-grpc-daemon[855]: ts=2026-09-20T12:21:40Z level=info event=scheduler_connected addr=lb:500513819worker1 # [ 47.137898] nix-grpc-daemon[855]: ts=2026-09-20T12:21:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3820worker2 # [ 46.570536] nix-grpc-daemon[862]: ts=2026-09-20T12:21:40Z level=info event=scheduler_connected addr=lb:500513821worker2 # [ 46.581401] nix-grpc-daemon[862]: ts=2026-09-20T12:21:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3822worker1 # [ 47.335080] nix-grpc-daemon[855]: ts=2026-09-20T12:21:40Z level=info event=scheduler_connected addr=lb:500513823worker1 # [ 47.340948] nix-grpc-daemon[855]: ts=2026-09-20T12:21:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3824worker2 # [ 46.783019] nix-grpc-daemon[862]: ts=2026-09-20T12:21:40Z level=info event=scheduler_connected addr=lb:500513825worker2 # [ 46.789033] nix-grpc-daemon[862]: ts=2026-09-20T12:21:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3826worker1 # [ 47.542379] nix-grpc-daemon[855]: ts=2026-09-20T12:21:40Z level=info event=scheduler_connected addr=lb:500513827worker1 # [ 47.545340] nix-grpc-daemon[855]: ts=2026-09-20T12:21:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3828worker2 # [ 46.990540] nix-grpc-daemon[862]: ts=2026-09-20T12:21:40Z level=info event=scheduler_connected addr=lb:500513829worker2 # [ 46.996575] nix-grpc-daemon[862]: ts=2026-09-20T12:21:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3830worker1 # [ 47.746474] nix-grpc-daemon[855]: ts=2026-09-20T12:21:40Z level=info event=scheduler_connected addr=lb:500513831worker1 # [ 47.753334] nix-grpc-daemon[855]: ts=2026-09-20T12:21:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3832worker2 # [ 47.200713] nix-grpc-daemon[862]: ts=2026-09-20T12:21:40Z level=info event=scheduler_connected addr=lb:500513833worker2 # [ 47.205863] nix-grpc-daemon[862]: ts=2026-09-20T12:21:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3834worker1 # [ 47.954282] nix-grpc-daemon[855]: ts=2026-09-20T12:21:40Z level=info event=scheduler_connected addr=lb:500513835worker1 # [ 47.960936] nix-grpc-daemon[855]: ts=2026-09-20T12:21:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3836worker2 # [ 47.400904] nix-grpc-daemon[862]: ts=2026-09-20T12:21:40Z level=info event=scheduler_connected addr=lb:500513837worker2 # [ 47.404378] nix-grpc-daemon[862]: ts=2026-09-20T12:21:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3838worker1 # [ 48.161227] nix-grpc-daemon[855]: ts=2026-09-20T12:21:41Z level=info event=scheduler_connected addr=lb:500513839worker1 # [ 48.164464] nix-grpc-daemon[855]: ts=2026-09-20T12:21:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3840worker2 # [ 47.606143] nix-grpc-daemon[862]: ts=2026-09-20T12:21:41Z level=info event=scheduler_connected addr=lb:500513841worker2 # [ 47.612300] nix-grpc-daemon[862]: ts=2026-09-20T12:21:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3842worker1 # [ 48.365856] nix-grpc-daemon[855]: ts=2026-09-20T12:21:41Z level=info event=scheduler_connected addr=lb:500513843worker1 # [ 48.371969] nix-grpc-daemon[855]: ts=2026-09-20T12:21:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3844worker2 # [ 47.812975] nix-grpc-daemon[862]: ts=2026-09-20T12:21:41Z level=info event=scheduler_connected addr=lb:500513845worker2 # [ 47.821479] nix-grpc-daemon[862]: ts=2026-09-20T12:21:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3846worker1 # [ 48.575348] nix-grpc-daemon[855]: ts=2026-09-20T12:21:41Z level=info event=scheduler_connected addr=lb:500513847worker1 # [ 48.580580] nix-grpc-daemon[855]: ts=2026-09-20T12:21:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3848worker2 # [ 48.023527] nix-grpc-daemon[862]: ts=2026-09-20T12:21:41Z level=info event=scheduler_connected addr=lb:500513849worker2 # [ 48.029795] nix-grpc-daemon[862]: ts=2026-09-20T12:21:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3850worker1 # [ 48.776444] nix-grpc-daemon[855]: ts=2026-09-20T12:21:41Z level=info event=scheduler_connected addr=lb:500513851worker1 # [ 48.779613] nix-grpc-daemon[855]: ts=2026-09-20T12:21:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3852worker2 # [ 48.233991] nix-grpc-daemon[862]: ts=2026-09-20T12:21:41Z level=info event=scheduler_connected addr=lb:500513853worker2 # [ 48.238712] nix-grpc-daemon[862]: ts=2026-09-20T12:21:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3854worker1 # [ 48.980415] nix-grpc-daemon[855]: ts=2026-09-20T12:21:42Z level=info event=scheduler_connected addr=lb:500513855worker1 # [ 48.983458] nix-grpc-daemon[855]: ts=2026-09-20T12:21:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3856worker2 # [ 48.435136] nix-grpc-daemon[862]: ts=2026-09-20T12:21:41Z level=info event=scheduler_connected addr=lb:500513857worker2 # [ 48.437698] nix-grpc-daemon[862]: ts=2026-09-20T12:21:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3858worker1 # [ 49.184815] nix-grpc-daemon[855]: ts=2026-09-20T12:21:42Z level=info event=scheduler_connected addr=lb:500513859worker1 # [ 49.191096] nix-grpc-daemon[855]: ts=2026-09-20T12:21:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3860worker2 # [ 48.640933] nix-grpc-daemon[862]: ts=2026-09-20T12:21:42Z level=info event=scheduler_connected addr=lb:500513861worker2 # [ 48.643142] nix-grpc-daemon[862]: ts=2026-09-20T12:21:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3862worker1 # [ 49.391591] nix-grpc-daemon[855]: ts=2026-09-20T12:21:42Z level=info event=scheduler_connected addr=lb:500513863worker1 # [ 49.395055] nix-grpc-daemon[855]: ts=2026-09-20T12:21:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3864worker2 # [ 48.842792] nix-grpc-daemon[862]: ts=2026-09-20T12:21:42Z level=info event=scheduler_connected addr=lb:500513865worker2 # [ 48.848864] nix-grpc-daemon[862]: ts=2026-09-20T12:21:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3866worker1 # [ 49.597726] nix-grpc-daemon[855]: ts=2026-09-20T12:21:42Z level=info event=scheduler_connected addr=lb:500513867worker1 # [ 49.604375] nix-grpc-daemon[855]: ts=2026-09-20T12:21:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3868worker2 # [ 49.049820] nix-grpc-daemon[862]: ts=2026-09-20T12:21:42Z level=info event=scheduler_connected addr=lb:500513869worker2 # [ 49.053201] nix-grpc-daemon[862]: ts=2026-09-20T12:21:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3870worker1 # [ 49.805547] nix-grpc-daemon[855]: ts=2026-09-20T12:21:42Z level=info event=scheduler_connected addr=lb:500513871worker1 # [ 49.813631] nix-grpc-daemon[855]: ts=2026-09-20T12:21:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3872worker2 # [ 49.257849] nix-grpc-daemon[862]: ts=2026-09-20T12:21:42Z level=info event=scheduler_connected addr=lb:500513873worker2 # [ 49.263257] nix-grpc-daemon[862]: ts=2026-09-20T12:21:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3874worker1 # [ 50.015289] nix-grpc-daemon[855]: ts=2026-09-20T12:21:43Z level=info event=scheduler_connected addr=lb:500513875worker1 # [ 50.023001] nix-grpc-daemon[855]: ts=2026-09-20T12:21:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3876worker2 # [ 49.459194] nix-grpc-daemon[862]: ts=2026-09-20T12:21:42Z level=info event=scheduler_connected addr=lb:500513877worker2 # [ 49.465837] nix-grpc-daemon[862]: ts=2026-09-20T12:21:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3878worker1 # [ 50.224641] nix-grpc-daemon[855]: ts=2026-09-20T12:21:43Z level=info event=scheduler_connected addr=lb:500513879worker1 # [ 50.231413] nix-grpc-daemon[855]: ts=2026-09-20T12:21:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3880worker2 # [ 49.668355] nix-grpc-daemon[862]: ts=2026-09-20T12:21:43Z level=info event=scheduler_connected addr=lb:500513881worker2 # [ 49.673838] nix-grpc-daemon[862]: ts=2026-09-20T12:21:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3882worker1 # [ 50.439173] nix-grpc-daemon[855]: ts=2026-09-20T12:21:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3883worker2 # [ 49.876071] nix-grpc-daemon[862]: ts=2026-09-20T12:21:43Z level=info event=scheduler_connected addr=lb:500513884worker2 # [ 49.882441] nix-grpc-daemon[862]: ts=2026-09-20T12:21:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3885worker2 # [ 50.084701] nix-grpc-daemon[862]: ts=2026-09-20T12:21:43Z level=info event=scheduler_connected addr=lb:500513886worker2 # [ 50.099815] nix-grpc-daemon[862]: ts=2026-09-20T12:21:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3887worker1 # [ 50.846562] nix-grpc-daemon[855]: ts=2026-09-20T12:21:43Z level=info event=scheduler_connected addr=lb:500513888worker1 # [ 50.851549] nix-grpc-daemon[855]: ts=2026-09-20T12:21:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3889worker2 # [ 50.317318] nix-grpc-daemon[862]: ts=2026-09-20T12:21:43Z level=info event=scheduler_connected addr=lb:500513890worker2 # [ 50.329881] nix-grpc-daemon[862]: ts=2026-09-20T12:21:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3891worker1 # [ 51.046675] nix-grpc-daemon[855]: ts=2026-09-20T12:21:44Z level=info event=scheduler_connected addr=lb:500513892worker1 # [ 51.056202] nix-grpc-daemon[855]: ts=2026-09-20T12:21:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3893worker2 # [ 50.519780] nix-grpc-daemon[862]: ts=2026-09-20T12:21:44Z level=info event=scheduler_connected addr=lb:500513894worker2 # [ 50.535740] nix-grpc-daemon[862]: ts=2026-09-20T12:21:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3895worker1 # [ 51.265919] nix-grpc-daemon[855]: ts=2026-09-20T12:21:44Z level=info event=scheduler_connected addr=lb:500513896worker1 # [ 51.275574] nix-grpc-daemon[855]: ts=2026-09-20T12:21:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3897worker2 # [ 50.743071] nix-grpc-daemon[862]: ts=2026-09-20T12:21:44Z level=info event=scheduler_connected addr=lb:500513898worker2 # [ 50.749807] nix-grpc-daemon[862]: ts=2026-09-20T12:21:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3899worker1 # [ 51.478119] nix-grpc-daemon[855]: ts=2026-09-20T12:21:44Z level=info event=scheduler_connected addr=lb:500513900worker1 # [ 51.489169] nix-grpc-daemon[855]: ts=2026-09-20T12:21:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3901worker2 # [ 50.952514] nix-grpc-daemon[862]: ts=2026-09-20T12:21:44Z level=info event=scheduler_connected addr=lb:500513902worker2 # [ 50.959414] nix-grpc-daemon[862]: ts=2026-09-20T12:21:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3903worker1 # [ 51.691785] nix-grpc-daemon[855]: ts=2026-09-20T12:21:44Z level=info event=scheduler_connected addr=lb:500513904worker1 # [ 51.705265] nix-grpc-daemon[855]: ts=2026-09-20T12:21:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3905worker2 # [ 51.162059] nix-grpc-daemon[862]: ts=2026-09-20T12:21:44Z level=info event=scheduler_connected addr=lb:500513906worker2 # [ 51.174169] nix-grpc-daemon[862]: ts=2026-09-20T12:21:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3907worker1 # [ 51.910871] nix-grpc-daemon[855]: ts=2026-09-20T12:21:44Z level=info event=scheduler_connected addr=lb:500513908worker1 # [ 51.915949] nix-grpc-daemon[855]: ts=2026-09-20T12:21:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3909worker2 # [ 51.388697] nix-grpc-daemon[862]: ts=2026-09-20T12:21:44Z level=info event=scheduler_connected addr=lb:500513910worker2 # [ 51.394472] nix-grpc-daemon[862]: ts=2026-09-20T12:21:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3911worker1 # [ 52.115847] nix-grpc-daemon[855]: ts=2026-09-20T12:21:45Z level=info event=scheduler_connected addr=lb:500513912worker1 # [ 52.121857] nix-grpc-daemon[855]: ts=2026-09-20T12:21:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3913worker2 # [ 51.585648] nix-grpc-daemon[862]: ts=2026-09-20T12:21:45Z level=info event=scheduler_connected addr=lb:500513914worker2 # [ 51.593993] nix-grpc-daemon[862]: ts=2026-09-20T12:21:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3915worker1 # [ 52.318827] nix-grpc-daemon[855]: ts=2026-09-20T12:21:45Z level=info event=scheduler_connected addr=lb:500513916worker1 # [ 52.327796] nix-grpc-daemon[855]: ts=2026-09-20T12:21:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3917worker2 # [ 51.796146] nix-grpc-daemon[862]: ts=2026-09-20T12:21:45Z level=info event=scheduler_connected addr=lb:500513918worker2 # [ 51.803512] nix-grpc-daemon[862]: ts=2026-09-20T12:21:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3919worker1 # [ 52.530493] nix-grpc-daemon[855]: ts=2026-09-20T12:21:45Z level=info event=scheduler_connected addr=lb:500513920worker1 # [ 52.537765] nix-grpc-daemon[855]: ts=2026-09-20T12:21:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3921worker2 # [ 52.006268] nix-grpc-daemon[862]: ts=2026-09-20T12:21:45Z level=info event=scheduler_connected addr=lb:500513922worker2 # [ 52.016269] nix-grpc-daemon[862]: ts=2026-09-20T12:21:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3923worker1 # [ 52.739207] nix-grpc-daemon[855]: ts=2026-09-20T12:21:45Z level=info event=scheduler_connected addr=lb:500513924worker1 # [ 52.746151] nix-grpc-daemon[855]: ts=2026-09-20T12:21:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3925worker2 # [ 52.215554] nix-grpc-daemon[862]: ts=2026-09-20T12:21:45Z level=info event=scheduler_connected addr=lb:500513926worker2 # [ 52.224235] nix-grpc-daemon[862]: ts=2026-09-20T12:21:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3927worker1 # [ 52.949294] nix-grpc-daemon[855]: ts=2026-09-20T12:21:45Z level=info event=scheduler_connected addr=lb:500513928worker1 # [ 52.955127] nix-grpc-daemon[855]: ts=2026-09-20T12:21:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3929worker1 # [ 52.994922] nix-grpc-daemon[855]: ts=2026-09-20T12:21:46Z level=info event=scheduler_lock_lost error="error: http://lb:5751/api/farm/lead: Timeout was reached"3930worker2 # [ 52.433761] nix-grpc-daemon[862]: ts=2026-09-20T12:21:45Z level=info event=scheduler_connected addr=lb:500513931worker2 # [ 52.440662] nix-grpc-daemon[862]: ts=2026-09-20T12:21:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3932worker1 # [ 53.159457] nix-grpc-daemon[855]: ts=2026-09-20T12:21:46Z level=info event=scheduler_connected addr=lb:500513933worker1 # [ 53.165942] nix-grpc-daemon[855]: ts=2026-09-20T12:21:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3934worker2 # [ 52.642216] nix-grpc-daemon[862]: ts=2026-09-20T12:21:46Z level=info event=scheduler_connected addr=lb:500513935worker2 # [ 52.656311] nix-grpc-daemon[862]: ts=2026-09-20T12:21:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3936worker1 # [ 53.363159] nix-grpc-daemon[855]: ts=2026-09-20T12:21:46Z level=info event=scheduler_connected addr=lb:500513937worker1 # [ 53.369789] nix-grpc-daemon[855]: ts=2026-09-20T12:21:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3938worker2 # [ 52.858530] nix-grpc-daemon[862]: ts=2026-09-20T12:21:46Z level=info event=scheduler_connected addr=lb:500513939worker2 # [ 52.866083] nix-grpc-daemon[862]: ts=2026-09-20T12:21:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3940worker2 # [ 52.974629] nix-grpc-daemon[862]: ts=2026-09-20T12:21:46Z level=info event=scheduler_lock_lost error="error: http://lb:5751/api/farm/lead: Timeout was reached"3941worker1 # [ 53.571295] nix-grpc-daemon[855]: ts=2026-09-20T12:21:46Z level=info event=scheduler_connected addr=lb:500513942worker1 # [ 53.580515] nix-grpc-daemon[855]: ts=2026-09-20T12:21:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3943worker2 # [ 53.068748] nix-grpc-daemon[862]: ts=2026-09-20T12:21:46Z level=info event=scheduler_connected addr=lb:500513944worker2 # [ 53.077706] nix-grpc-daemon[862]: ts=2026-09-20T12:21:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3945worker1 # [ 53.782313] nix-grpc-daemon[855]: ts=2026-09-20T12:21:46Z level=info event=scheduler_connected addr=lb:500513946worker1 # [ 53.791948] nix-grpc-daemon[855]: ts=2026-09-20T12:21:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3947worker2 # [ 53.278235] nix-grpc-daemon[862]: ts=2026-09-20T12:21:46Z level=info event=scheduler_connected addr=lb:500513948worker2 # [ 53.285717] nix-grpc-daemon[862]: ts=2026-09-20T12:21:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3949worker1 # [ 53.998535] nix-grpc-daemon[855]: ts=2026-09-20T12:21:47Z level=info event=scheduler_connected addr=lb:500513950worker1 # [ 54.001325] nix-grpc-daemon[855]: ts=2026-09-20T12:21:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3951worker2 # [ 53.492940] nix-grpc-daemon[862]: ts=2026-09-20T12:21:47Z level=info event=scheduler_connected addr=lb:500513952worker2 # [ 53.497933] nix-grpc-daemon[862]: ts=2026-09-20T12:21:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3953worker1 # [ 54.197424] nix-grpc-daemon[855]: ts=2026-09-20T12:21:47Z level=info event=scheduler_connected addr=lb:500513954worker1 # [ 54.207641] nix-grpc-daemon[855]: ts=2026-09-20T12:21:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3955worker2 # [ 53.693509] nix-grpc-daemon[862]: ts=2026-09-20T12:21:47Z level=info event=scheduler_connected addr=lb:500513956worker2 # [ 53.699197] nix-grpc-daemon[862]: ts=2026-09-20T12:21:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3957worker1 # [ 54.408516] nix-grpc-daemon[855]: ts=2026-09-20T12:21:47Z level=info event=scheduler_connected addr=lb:500513958worker1 # [ 54.411302] nix-grpc-daemon[855]: ts=2026-09-20T12:21:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3959worker2 # [ 53.900625] nix-grpc-daemon[862]: ts=2026-09-20T12:21:47Z level=info event=scheduler_connected addr=lb:500513960worker2 # [ 53.906462] nix-grpc-daemon[862]: ts=2026-09-20T12:21:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3961worker1 # [ 54.612340] nix-grpc-daemon[855]: ts=2026-09-20T12:21:47Z level=info event=scheduler_connected addr=lb:500513962worker1 # [ 54.617853] nix-grpc-daemon[855]: ts=2026-09-20T12:21:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3963worker2 # [ 54.108122] nix-grpc-daemon[862]: ts=2026-09-20T12:21:47Z level=info event=scheduler_connected addr=lb:500513964worker2 # [ 54.113960] nix-grpc-daemon[862]: ts=2026-09-20T12:21:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3965worker1 # [ 54.819074] nix-grpc-daemon[855]: ts=2026-09-20T12:21:47Z level=info event=scheduler_connected addr=lb:500513966worker1 # [ 54.824968] nix-grpc-daemon[855]: ts=2026-09-20T12:21:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3967worker2 # [ 54.315350] nix-grpc-daemon[862]: ts=2026-09-20T12:21:47Z level=info event=scheduler_connected addr=lb:500513968worker2 # [ 54.321401] nix-grpc-daemon[862]: ts=2026-09-20T12:21:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3969worker1 # [ 55.028587] nix-grpc-daemon[855]: ts=2026-09-20T12:21:48Z level=info event=scheduler_connected addr=lb:500513970worker1 # [ 55.033071] nix-grpc-daemon[855]: ts=2026-09-20T12:21:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3971worker2 # [ 54.524302] nix-grpc-daemon[862]: ts=2026-09-20T12:21:48Z level=info event=scheduler_connected addr=lb:500513972worker2 # [ 54.527759] nix-grpc-daemon[862]: ts=2026-09-20T12:21:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3973worker1 # [ 55.229106] nix-grpc-daemon[855]: ts=2026-09-20T12:21:48Z level=info event=scheduler_connected addr=lb:500513974worker1 # [ 55.232543] nix-grpc-daemon[855]: ts=2026-09-20T12:21:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3975worker2 # [ 54.725610] nix-grpc-daemon[862]: ts=2026-09-20T12:21:48Z level=info event=scheduler_connected addr=lb:500513976worker2 # [ 54.728813] nix-grpc-daemon[862]: ts=2026-09-20T12:21:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3977worker1 # [ 55.433684] nix-grpc-daemon[855]: ts=2026-09-20T12:21:48Z level=info event=scheduler_connected addr=lb:500513978worker1 # [ 55.439260] nix-grpc-daemon[855]: ts=2026-09-20T12:21:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3979worker2 # [ 54.929838] nix-grpc-daemon[862]: ts=2026-09-20T12:21:48Z level=info event=scheduler_connected addr=lb:500513980worker2 # [ 54.934799] nix-grpc-daemon[862]: ts=2026-09-20T12:21:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3981worker1 # [ 55.639979] nix-grpc-daemon[855]: ts=2026-09-20T12:21:48Z level=info event=scheduler_connected addr=lb:500513982worker1 # [ 55.643609] nix-grpc-daemon[855]: ts=2026-09-20T12:21:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3983worker2 # [ 55.135720] nix-grpc-daemon[862]: ts=2026-09-20T12:21:48Z level=info event=scheduler_connected addr=lb:500513984worker2 # [ 55.139261] nix-grpc-daemon[862]: ts=2026-09-20T12:21:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3985worker1 # [ 55.844952] nix-grpc-daemon[855]: ts=2026-09-20T12:21:48Z level=info event=scheduler_connected addr=lb:500513986worker1 # [ 55.850090] nix-grpc-daemon[855]: ts=2026-09-20T12:21:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3987worker2 # [ 55.340567] nix-grpc-daemon[862]: ts=2026-09-20T12:21:48Z level=info event=scheduler_connected addr=lb:500513988worker2 # [ 55.346869] nix-grpc-daemon[862]: ts=2026-09-20T12:21:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3989worker1 # [ 56.051377] nix-grpc-daemon[855]: ts=2026-09-20T12:21:49Z level=info event=scheduler_connected addr=lb:500513990worker1 # [ 56.053748] nix-grpc-daemon[855]: ts=2026-09-20T12:21:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3991worker2 # [ 55.548086] nix-grpc-daemon[862]: ts=2026-09-20T12:21:49Z level=info event=scheduler_connected addr=lb:500513992worker2 # [ 55.552544] nix-grpc-daemon[862]: ts=2026-09-20T12:21:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3993worker1 # [ 56.254678] nix-grpc-daemon[855]: ts=2026-09-20T12:21:49Z level=info event=scheduler_connected addr=lb:500513994worker1 # [ 56.259613] nix-grpc-daemon[855]: ts=2026-09-20T12:21:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3995worker2 # [ 55.749490] nix-grpc-daemon[862]: ts=2026-09-20T12:21:49Z level=info event=scheduler_connected addr=lb:500513996worker2 # [ 55.754762] nix-grpc-daemon[862]: ts=2026-09-20T12:21:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3997worker1 # [ 56.460405] nix-grpc-daemon[855]: ts=2026-09-20T12:21:49Z level=info event=scheduler_connected addr=lb:500513998worker1 # [ 56.463064] nix-grpc-daemon[855]: ts=2026-09-20T12:21:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3999worker2 # [ 55.955805] nix-grpc-daemon[862]: ts=2026-09-20T12:21:49Z level=info event=scheduler_connected addr=lb:500514000worker2 # [ 55.959777] nix-grpc-daemon[862]: ts=2026-09-20T12:21:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4001worker1 # [ 56.664544] nix-grpc-daemon[855]: ts=2026-09-20T12:21:49Z level=info event=scheduler_connected addr=lb:500514002worker1 # [ 56.670293] nix-grpc-daemon[855]: ts=2026-09-20T12:21:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4003worker2 # [ 56.161536] nix-grpc-daemon[862]: ts=2026-09-20T12:21:49Z level=info event=scheduler_connected addr=lb:500514004worker2 # [ 56.167248] nix-grpc-daemon[862]: ts=2026-09-20T12:21:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4005worker1 # [ 56.871276] nix-grpc-daemon[855]: ts=2026-09-20T12:21:49Z level=info event=scheduler_connected addr=lb:500514006worker1 # [ 56.876912] nix-grpc-daemon[855]: ts=2026-09-20T12:21:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4007worker2 # [ 56.368202] nix-grpc-daemon[862]: ts=2026-09-20T12:21:49Z level=info event=scheduler_connected addr=lb:500514008worker2 # [ 56.371320] nix-grpc-daemon[862]: ts=2026-09-20T12:21:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4009worker1 # [ 57.080744] nix-grpc-daemon[855]: ts=2026-09-20T12:21:50Z level=info event=scheduler_connected addr=lb:500514010worker1 # [ 57.092506] nix-grpc-daemon[855]: ts=2026-09-20T12:21:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4011worker2 # [ 56.573975] nix-grpc-daemon[862]: ts=2026-09-20T12:21:50Z level=info event=scheduler_connected addr=lb:500514012worker2 # [ 56.579226] nix-grpc-daemon[862]: ts=2026-09-20T12:21:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4013worker1 # [ 57.280716] nix-grpc-daemon[855]: ts=2026-09-20T12:21:50Z level=info event=scheduler_connected addr=lb:500514014worker1 # [ 57.286365] nix-grpc-daemon[855]: ts=2026-09-20T12:21:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4015worker2 # [ 56.776982] nix-grpc-daemon[862]: ts=2026-09-20T12:21:50Z level=info event=scheduler_connected addr=lb:500514016worker2 # [ 56.782766] nix-grpc-daemon[862]: ts=2026-09-20T12:21:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4017worker1 # [ 57.487441] nix-grpc-daemon[855]: ts=2026-09-20T12:21:50Z level=info event=scheduler_connected addr=lb:500514018worker1 # [ 57.493599] nix-grpc-daemon[855]: ts=2026-09-20T12:21:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4019lb # [ 57.617735] rustfs-bucket-start[1123]: mb s3://farm4020lb # [ 57.631168] systemd[1]: Finished rustfs-bucket.service.4021lb # [ 57.645216] systemd[1]: Starting niks3 server...4022worker2 # [ 56.984301] nix-grpc-daemon[862]: ts=2026-09-20T12:21:50Z level=info event=scheduler_connected addr=lb:500514023worker2 # [ 56.990228] nix-grpc-daemon[862]: ts=2026-09-20T12:21:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4024worker1 # [ 57.694991] nix-grpc-daemon[855]: ts=2026-09-20T12:21:50Z level=info event=scheduler_connected addr=lb:500514025worker1 # [ 57.700851] nix-grpc-daemon[855]: ts=2026-09-20T12:21:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4026worker2 # [ 57.191729] nix-grpc-daemon[862]: ts=2026-09-20T12:21:50Z level=info event=scheduler_connected addr=lb:500514027worker2 # [ 57.197765] nix-grpc-daemon[862]: ts=2026-09-20T12:21:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4028lb # [ 57.954675] postgres[1142]: [1142] ERROR: relation "goose_db_version" does not exist at character 364029lb # [ 57.957020] postgres[1142]: [1142] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC4030lb # [ 58.006609] niks3-server[1132]: 2026/09/20 12:21:50 OK 20241026095416_initial_model.sql (28.32ms)4031lb # [ 58.012704] niks3-server[1132]: 2026/09/20 12:21:50 OK 20251210153512_drop_unused_gin_index.sql (4.24ms)4032worker1 # [ 57.901746] nix-grpc-daemon[855]: ts=2026-09-20T12:21:50Z level=info event=scheduler_connected addr=lb:500514033worker1 # [ 57.907495] nix-grpc-daemon[855]: ts=2026-09-20T12:21:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4034lb # [ 58.020527] niks3-server[1132]: 2026/09/20 12:21:50 OK 20251218171726_add_pins.sql (7.72ms)4035lb # [ 58.027719] niks3-server[1132]: 2026/09/20 12:21:51 OK 20260628120000_add_object_size_and_stats.sql (7.12ms)4036lb # [ 58.033995] niks3-server[1132]: 2026/09/20 12:21:51 OK 20260905000000_add_claims.sql (6.23ms)4037lb # [ 58.039985] niks3-server[1132]: 2026/09/20 12:21:51 OK 20260920000000_drop_claims.sql (5.76ms)4038lb # [ 58.041410] niks3-server[1132]: 2026/09/20 12:21:51 goose: successfully migrated database to version: 202609200000004039lb # [ 58.048116] niks3-server[1132]: 2026/09/20 12:21:51 OK 1_commit_pending_closure.sql (8.05ms)4040lb # [ 58.051621] niks3-server[1132]: 2026/09/20 12:21:51 OK 2_object_stats_trigger.sql (3.39ms)4041lb # [ 58.053444] niks3-server[1132]: 2026/09/20 12:21:51 goose: up to current file version: 24042lb # [ 58.057875] niks3-server[1132]: 2026/09/20 12:21:51 INFO Loaded signing key name=farm-test-1 path=/nix/store/8lzz3wcf17v05vfdp1zcfaa9hipw3f70-key4043worker2 # [ 57.399344] nix-grpc-daemon[862]: ts=2026-09-20T12:21:50Z level=info event=scheduler_connected addr=lb:500514044worker2 # [ 57.405178] nix-grpc-daemon[862]: ts=2026-09-20T12:21:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4045lb # [ 58.117244] niks3-server[1132]: 2026/09/20 12:21:51 INFO Read proxy enabled — serving cache objects from S34046lb # [ 58.120244] niks3-server[1132]: 2026/09/20 12:21:51 INFO Using socket-activated listener address=0.0.0.0:57514047lb # [ 58.124242] niks3-server[1132]: 2026/09/20 12:21:51 INFO systemd watchdog enabled interval=15s4048lb # [ 58.125485] niks3-server[1132]: 2026/09/20 12:21:51 INFO Starting HTTP server address=0.0.0.0:57514049lb # [ 58.132984] systemd[1]: Started niks3 server.4050lb # [ 58.138077] niks3-server[1132]: 2026/09/20 12:21:51 INFO lead: acquired remote=192.168.1.4:409804051lb # [ 58.139360] systemd[1]: Reached target Multi-User System.4052worker2 # [ 57.463907] nix-grpc-daemon[862]: ts=2026-09-20T12:21:50Z level=info event=scheduler_take_over4053lb # [ 58.148382] systemd[1]: Startup finished in 1.618s (kernel) + 8.475s (initrd) + 48.036s (userspace) = 58.129s.4054worker1 # [ 58.110490] nix-grpc-daemon[855]: ts=2026-09-20T12:21:51Z level=info event=scheduler_connected addr=lb:500514055worker1 # [ 58.114941] nix-grpc-daemon[855]: ts=2026-09-20T12:21:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4056worker2 # [ 57.608694] nix-grpc-daemon[862]: ts=2026-09-20T12:21:51Z level=info event=scheduler_connected addr=lb:500514057worker2 # [ 57.613303] nix-grpc-daemon[862]: ts=2026-09-20T12:21:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4058worker1 # [ 58.310797] nix-grpc-daemon[855]: ts=2026-09-20T12:21:51Z level=info event=scheduler_connected addr=lb:500514059worker1 # [ 58.314048] nix-grpc-daemon[855]: ts=2026-09-20T12:21:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4060worker2 # [ 57.809718] nix-grpc-daemon[862]: ts=2026-09-20T12:21:51Z level=info event=scheduler_connected addr=lb:500514061worker2 # [ 57.815442] nix-grpc-daemon[862]: ts=2026-09-20T12:21:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4062worker1 # [ 58.514842] nix-grpc-daemon[855]: ts=2026-09-20T12:21:51Z level=info event=scheduler_connected addr=lb:500514063worker2 # [ 57.956704] nix-grpc-daemon[862]: ts=2026-09-20T12:21:51Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:428204064worker2 # [ 57.963730] nix-grpc-daemon[862]: ts=2026-09-20T12:21:51Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=04065worker2 # [ 58.016692] nix-grpc-daemon[862]: ts=2026-09-20T12:21:51Z level=info event=scheduler_connected addr=lb:500514066worker2 # [ 58.035664] nix-grpc-daemon[862]: ts=2026-09-20T12:21:51Z level=info event=oidc_jwks provider=mock keys=14067worker2 # [ 58.038351] nix-grpc-daemon[862]: ts=2026-09-20T12:21:51Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:428204068worker2 # [ 58.041222] nix-grpc-daemon[862]: ts=2026-09-20T12:21:51Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=04069lb: (finished: waiting for unit niks3.service, in 28.96 seconds)4070lb: waiting for TCP port 5751 on localhost4071lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded!4072lb: (finished: waiting for TCP port 5751 on localhost, in 0.11 seconds)4073lb: waiting for unit envoy.service4074lb: (finished: waiting for unit envoy.service, in 0.07 seconds)4075lb: waiting for TCP port 50051 on localhost4076lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded!4077lb # [ 59.146528] envoy[882]: conn peer=[::1]:33884 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=44078lb: (finished: waiting for TCP port 50051 on localhost, in 0.05 seconds)4079worker2 # [ 58.652988] systemd[1]: Starting scheduler-token.service...4080worker2 # [ 58.768920] systemd[1]: scheduler-token.service: Deactivated successfully.4081worker2 # [ 58.771652] systemd[1]: Finished scheduler-token.service.4082worker2 # [ 58.773896] systemd[1]: scheduler-token.service: Consumed 35ms CPU time over 117ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4083lb: waiting until exactly no endpoints of aarch64-linux are unhealthy4084lb: must succeed: curl -sf localhost:9901/clusters4085lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.11 seconds)4086lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.11 seconds)4087subtest: one node schedules and both builders hold a WorkerSession on it4088lb: must succeed: curl -sf localhost:9901/clusters4089lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4090worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4091worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)4092lb: must succeed: curl -sf localhost:9901/clusters4093lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4094lb: must succeed: curl -sf localhost:9901/clusters4095lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4096worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'4097worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.07 seconds)4098(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 0.37 seconds)4099subtest: balancer auth: client cert, bearer token, nothing, unknown CN4100worker2 # [ 59.633704] nix-grpc-daemon[862]: ts=2026-09-20T12:21:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:428324101worker2 # [ 59.655168] systemd[1]: Started Nix Daemon.4102worker2 # [ 59.974273] nix-daemon[936]: accepted connection from pid 862, user nix-grpc-daemon (trusted)4103worker2 # [ 60.014492] nix-grpc-daemon[862]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 62 ms (attempt 1/5)4104worker2 # [ 60.079894] nix-grpc-daemon[862]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 176 ms (attempt 2/5)4105worker2 # [ 60.259696] nix-grpc-daemon[862]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 176 ms (attempt 3/5)4106worker2 # [ 60.444302] nix-grpc-daemon[862]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 565 ms (attempt 4/5)4107worker2 # [ 61.017608] nix-grpc-daemon[862]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4108worker1 # [ 61.625838] nix-grpc-daemon[855]: ts=2026-09-20T12:21:54Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:416524109worker1 # [ 61.646616] systemd[1]: Started Nix Daemon.4110worker1 # [ 61.973118] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4111worker1 # [ 62.004845] nix-grpc-daemon[855]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 8 ms (attempt 1/5)4112worker1 # [ 62.012956] nix-grpc-daemon[855]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 164 ms (attempt 2/5)4113worker1 # [ 62.180643] nix-grpc-daemon[855]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 21 ms (attempt 3/5)4114worker1 # [ 62.209571] nix-grpc-daemon[855]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 413 ms (attempt 4/5)4115worker1 # [ 62.634402] nix-grpc-daemon[855]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4116worker1 # [ 62.651598] nix-grpc-daemon[855]: ts=2026-09-20T12:21:55Z level=debug event=path_info_miss path=a8gc22jhjd6rmf0wand1fsy50rgzy95i-nixos-system-client-test error="error: path '/nix/store/a8gc22jhjd6rmf0wand1fsy50rgzy95i-nixos-system-client-test' is required, but there is no substituter that can build it"4117worker1 # [ 62.665950] nix-grpc-daemon[855]: ts=2026-09-20T12:21:55Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:416524118worker1 # [ 62.679948] nix-grpc-daemon[855]: ts=2026-09-20T12:21:55Z level=debug event=path_info_miss path=a8gc22jhjd6rmf0wand1fsy50rgzy95i-nixos-system-client-test error="error: path '/nix/store/a8gc22jhjd6rmf0wand1fsy50rgzy95i-nixos-system-client-test' is required, but there is no substituter that can build it"4119client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt4120client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.12 seconds)4121client: must succeed: install -D /nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key4122client: (finished: must succeed: install -D /nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.06 seconds)4123worker1 # [ 63.166709] nix-grpc-daemon[855]: ts=2026-09-20T12:21:56Z level=info event=oidc_jwks provider=mock keys=14124worker1 # [ 63.171413] nix-grpc-daemon[855]: ts=2026-09-20T12:21:56Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:416524125worker1 # [ 63.183691] nix-grpc-daemon[855]: ts=2026-09-20T12:21:56Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:416524126worker1 # [ 63.192822] nix-grpc-daemon[855]: ts=2026-09-20T12:21:56Z level=debug event=path_info_miss path=a8gc22jhjd6rmf0wand1fsy50rgzy95i-nixos-system-client-test error="error: path '/nix/store/a8gc22jhjd6rmf0wand1fsy50rgzy95i-nixos-system-client-test' is required, but there is no substituter that can build it"4127worker2 # [ 62.635262] nix-grpc-daemon[862]: ts=2026-09-20T12:21:56Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:428324128worker2 # [ 62.650041] nix-grpc-daemon[862]: ts=2026-09-20T12:21:56Z level=debug event=path_info_miss path=a8gc22jhjd6rmf0wand1fsy50rgzy95i-nixos-system-client-test error="error: path '/nix/store/a8gc22jhjd6rmf0wand1fsy50rgzy95i-nixos-system-client-test' is required, but there is no substituter that can build it"4129client: must succeed: rm -r /var/lib/nix-grpc-store4130client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.05 seconds)4131worker2 # [ 62.976805] nix-grpc-daemon[862]: ts=2026-09-20T12:21:56Z level=info event=denied method=QueryMissing cn=- role=none4132worker2 # [ 63.247570] nix-grpc-daemon[862]: ts=2026-09-20T12:21:56Z level=info event=denied method=QueryMissing cn=stranger role=none4133(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 4.02 seconds)4134subtest: DAG build is scheduled across workers and lands in the cache4135client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&24136worker1 # [ 64.091663] nix-grpc-daemon[855]: ts=2026-09-20T12:21:57Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:416524137client # don't know how to build these paths:4138client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4139worker1 # [ 64.107665] nix-grpc-daemon[855]: ts=2026-09-20T12:21:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:416524140worker2 # [ 63.582845] nix-grpc-daemon[862]: ts=2026-09-20T12:21:57Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=04141worker2 # [ 63.589790] nix-grpc-daemon[862]: ts=2026-09-20T12:21:57Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.3:50051 clients=14142worker1 # [ 64.200352] nix-grpc-daemon[855]: ts=2026-09-20T12:21:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524143worker1 # [ 64.218852] nix-grpc-daemon[855]: ts=2026-09-20T12:21:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:416524144client # copying 1 paths to 'lb:50051'...4145worker1 # [ 64.240572] nix-grpc-daemon[855]: ts=2026-09-20T12:21:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:416524146client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'...4147worker1 # [ 64.270042] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4148worker1 # [ 64.308999] nix-grpc-daemon[855]: ts=2026-09-20T12:21:57Z level=info event=niks3_push_started pid=9284149worker1 # [ 64.627319] nix-grpc-daemon[928]: time=2026-09-20T12:21:57.655Z level=INFO msg="Configuring client TLS" cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/worker1.pem key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/worker1.key ca=""4150worker1 # [ 64.839452] nix-daemon[919]: accepted connection from pid 941, user nix-grpc-daemon (trusted)4151worker1 # [ 64.863392] nix-daemon[919]: reaped child process 944, status = succeeded4152worker1 # [ 65.059986] nix-daemon[919]: accepted connection from pid 946, user nix-grpc-daemon (trusted)4153worker1 # [ 65.080267] nix-daemon[919]: reaped child process 949, status = succeeded4154lb # [ 65.201573] niks3-server[1132]: 2026/09/20 12:21:58 INFO Received uploads request method=POST path=/api/pending_closures4155worker1 # [ 65.120575] nix-grpc-daemon[928]: time=2026-09-20T12:21:58.150Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4156worker1 # [ 65.124320] nix-grpc-daemon[928]: time=2026-09-20T12:21:58.152Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)"4157lb # [ 65.263079] niks3-server[1132]: 2026/09/20 12:21:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4158lb # [ 65.281606] niks3-server[1132]: 2026/09/20 12:21:58 INFO Signed narinfos id=1 count=14159worker1 # [ 65.178159] nix-grpc-daemon[928]: time=2026-09-20T12:21:58.207Z level=INFO msg="Uploading 1 narinfos"4160lb # [ 65.288361] niks3-server[1132]: 2026/09/20 12:21:58 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst4161lb # [ 65.306994] niks3-server[1132]: 2026/09/20 12:21:58 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls4162lb # [ 65.318674] niks3-server[1132]: 2026/09/20 12:21:58 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4163lb # [ 65.325780] niks3-server[1132]: 2026/09/20 12:21:58 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo4164lb # [ 65.330559] niks3-server[1132]: 2026/09/20 12:21:58 INFO Completed upload id=14165worker1 # [ 65.223421] nix-grpc-daemon[928]: time=2026-09-20T12:21:58.253Z level=INFO msg="Upload complete. (587ms)"4166worker1 # [ 65.230568] nix-grpc-daemon[855]: ts=2026-09-20T12:21:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=0 paths=1 nar_bytes_in=5044167worker1 # [ 65.248738] nix-grpc-daemon[855]: ts=2026-09-20T12:21:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524168worker1 # [ 65.258144] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4169worker1 # [ 65.264608] nix-daemon[919]: reaped child process 926, status = succeeded4170client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051...4171client # farm-a> node-a: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4172worker1 # [ 65.283556] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4173lb # [ 65.543648] envoy[882]: conn peer=[2001:db8:1::1]:40546 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=407 ms=25244174lb # [ 65.548525] envoy[882]: conn peer=[2001:db8:1::1]:44536 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=774175lb # [ 65.553404] envoy[882]: conn peer=[2001:db8:1::1]:44552 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=134176lb # [ 65.557156] envoy[882]: conn peer=[2001:db8:1::1]:44560 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=214177client # farm-a> LOG-farm-a-t14178worker1 # [ 66.513151] nix-daemon[919]: reaped child process 953, status = succeeded4179worker1 # [ 66.713178] nix-daemon[919]: accepted connection from pid 968, user nix-grpc-daemon (trusted)4180worker1 # [ 66.739328] nix-daemon[919]: reaped child process 971, status = succeeded4181lb # [ 66.858852] niks3-server[1132]: 2026/09/20 12:21:59 INFO Received uploads request method=POST path=/api/pending_closures4182worker1 # [ 66.769876] nix-grpc-daemon[928]: time=2026-09-20T12:21:59.797Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4183worker1 # [ 66.772154] nix-grpc-daemon[928]: time=2026-09-20T12:21:59.797Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)"4184lb # [ 66.908624] niks3-server[1132]: 2026/09/20 12:21:59 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4185lb # [ 66.919408] niks3-server[1132]: 2026/09/20 12:21:59 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4186lb # [ 66.932661] niks3-server[1132]: 2026/09/20 12:21:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4187worker1 # [ 66.831468] nix-grpc-daemon[928]: time=2026-09-20T12:21:59.861Z level=INFO msg="Uploading 1 narinfos"4188lb # [ 66.940284] niks3-server[1132]: 2026/09/20 12:21:59 INFO Signed narinfos id=2 count=14189lb # [ 66.944449] niks3-server[1132]: 2026/09/20 12:21:59 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls4190lb # [ 66.957133] niks3-server[1132]: 2026/09/20 12:21:59 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4191worker1 # [ 66.858523] nix-grpc-daemon[928]: time=2026-09-20T12:21:59.888Z level=INFO msg="Upload complete. (362ms)"4192lb # [ 66.965918] niks3-server[1132]: 2026/09/20 12:21:59 INFO Completed upload id=24193lb # [ 66.971485] niks3-server[1132]: 2026/09/20 12:21:59 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo4194worker1 # [ 66.864720] nix-grpc-daemon[855]: ts=2026-09-20T12:21:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=14195worker1 # [ 66.874570] nix-daemon[919]: reaped child process 951, status = succeeded4196worker2 # [ 66.309127] nix-grpc-daemon[862]: ts=2026-09-20T12:21:59Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=04197worker2 # [ 66.313901] nix-grpc-daemon[862]: ts=2026-09-20T12:21:59Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.3:50051 clients=14198worker1 # [ 66.929439] nix-grpc-daemon[855]: ts=2026-09-20T12:21:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524199worker1 # [ 66.951063] nix-grpc-daemon[855]: ts=2026-09-20T12:21:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:416524200worker1 # [ 66.967641] nix-grpc-daemon[928]: time=2026-09-20T12:21:59.997Z level=INFO msg="All 1 paths already cached"4201client # copying 1 paths to 'lb:50051'...4202worker1 # [ 66.982681] nix-grpc-daemon[855]: ts=2026-09-20T12:22:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:416524203client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'...4204worker1 # [ 67.005214] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4205worker1 # [ 67.255218] nix-daemon[919]: accepted connection from pid 975, user nix-grpc-daemon (trusted)4206worker1 # [ 67.281373] nix-daemon[919]: reaped child process 978, status = succeeded4207worker1 # [ 67.466847] nix-daemon[919]: accepted connection from pid 980, user nix-grpc-daemon (trusted)4208worker1 # [ 67.495212] nix-daemon[919]: reaped child process 983, status = succeeded4209lb # [ 67.617485] niks3-server[1132]: 2026/09/20 12:22:00 INFO Received uploads request method=POST path=/api/pending_closures4210worker1 # [ 67.519289] nix-grpc-daemon[928]: time=2026-09-20T12:22:00.548Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4211worker1 # [ 67.521183] nix-grpc-daemon[928]: time=2026-09-20T12:22:00.549Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)"4212lb # [ 67.658197] niks3-server[1132]: 2026/09/20 12:22:00 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst4213lb # [ 67.672082] niks3-server[1132]: 2026/09/20 12:22:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4214worker1 # [ 67.571813] nix-grpc-daemon[928]: time=2026-09-20T12:22:00.601Z level=INFO msg="Uploading 1 narinfos"4215lb # [ 67.683347] niks3-server[1132]: 2026/09/20 12:22:00 INFO Signed narinfos id=3 count=14216lb # [ 67.690769] niks3-server[1132]: 2026/09/20 12:22:00 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls4217lb # [ 67.704157] niks3-server[1132]: 2026/09/20 12:22:00 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4218lb # [ 67.711401] niks3-server[1132]: 2026/09/20 12:22:00 INFO Completed upload id=34219lb # [ 67.714388] niks3-server[1132]: 2026/09/20 12:22:00 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo4220worker1 # [ 67.604280] nix-grpc-daemon[928]: time=2026-09-20T12:22:00.634Z level=INFO msg="Upload complete. (574ms)"4221worker1 # [ 67.616972] nix-grpc-daemon[855]: ts=2026-09-20T12:22:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=0 paths=1 nar_bytes_in=6244222worker1 # [ 67.628554] nix-grpc-daemon[855]: ts=2026-09-20T12:22:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524223worker1 # [ 67.642825] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4224worker1 # [ 67.649490] nix-daemon[919]: reaped child process 973, status = succeeded4225client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051...4226worker1 # [ 67.665326] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4227client # farm-b> node-a: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4228client # farm-b> LOG-farm-b-t14229worker1 # [ 68.507125] nix-daemon[919]: reaped child process 987, status = succeeded4230worker1 # [ 68.694914] nix-daemon[919]: accepted connection from pid 1002, user nix-grpc-daemon (trusted)4231worker1 # [ 68.720428] nix-daemon[919]: reaped child process 1005, status = succeeded4232lb # [ 68.841748] niks3-server[1132]: 2026/09/20 12:22:01 INFO Received uploads request method=POST path=/api/pending_closures4233worker1 # [ 68.746905] nix-grpc-daemon[928]: time=2026-09-20T12:22:01.776Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4234worker1 # [ 68.748727] nix-grpc-daemon[928]: time=2026-09-20T12:22:01.776Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)"4235lb # [ 68.894965] niks3-server[1132]: 2026/09/20 12:22:01 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4236lb # [ 68.906268] niks3-server[1132]: 2026/09/20 12:22:01 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst4237lb # [ 68.918242] niks3-server[1132]: 2026/09/20 12:22:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4238lb # [ 68.927253] niks3-server[1132]: 2026/09/20 12:22:01 INFO Signed narinfos id=4 count=14239worker1 # [ 68.820043] nix-grpc-daemon[928]: time=2026-09-20T12:22:01.849Z level=INFO msg="Uploading 1 narinfos"4240lb # [ 68.932330] niks3-server[1132]: 2026/09/20 12:22:01 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls4241lb # [ 68.947751] niks3-server[1132]: 2026/09/20 12:22:01 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4242lb # [ 68.957626] niks3-server[1132]: 2026/09/20 12:22:01 INFO Completed upload id=44243worker1 # [ 68.850344] nix-grpc-daemon[928]: time=2026-09-20T12:22:01.880Z level=INFO msg="Upload complete. (369ms)"4244lb # [ 68.960653] niks3-server[1132]: 2026/09/20 12:22:01 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo4245worker1 # [ 68.866291] nix-grpc-daemon[855]: ts=2026-09-20T12:22:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=14246worker2 # [ 68.301509] nix-grpc-daemon[862]: ts=2026-09-20T12:22:01Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=04247worker2 # [ 68.308209] nix-grpc-daemon[862]: ts=2026-09-20T12:22:01Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.3:50051 clients=14248worker1 # [ 68.887027] nix-daemon[919]: reaped child process 985, status = succeeded4249worker1 # [ 68.914315] nix-grpc-daemon[855]: ts=2026-09-20T12:22:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524250worker1 # [ 68.939886] nix-grpc-daemon[855]: ts=2026-09-20T12:22:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:416524251worker1 # [ 68.956493] nix-grpc-daemon[928]: time=2026-09-20T12:22:01.986Z level=INFO msg="All 2 paths already cached"4252client # copying 1 paths to 'lb:50051'...4253worker1 # [ 68.969795] nix-grpc-daemon[855]: ts=2026-09-20T12:22:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:416524254client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'...4255worker1 # [ 68.991351] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4256worker1 # [ 69.198121] nix-daemon[919]: accepted connection from pid 1009, user nix-grpc-daemon (trusted)4257worker1 # [ 69.227183] nix-daemon[919]: reaped child process 1012, status = succeeded4258worker1 # [ 69.433635] nix-daemon[919]: accepted connection from pid 1014, user nix-grpc-daemon (trusted)4259worker1 # [ 69.457881] nix-daemon[919]: reaped child process 1017, status = succeeded4260lb # [ 69.573942] niks3-server[1132]: 2026/09/20 12:22:02 INFO Received uploads request method=POST path=/api/pending_closures4261worker1 # [ 69.480187] nix-grpc-daemon[928]: time=2026-09-20T12:22:02.510Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4262worker1 # [ 69.481799] nix-grpc-daemon[928]: time=2026-09-20T12:22:02.510Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)"4263lb # [ 69.613698] niks3-server[1132]: 2026/09/20 12:22:02 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst4264lb # [ 69.627192] niks3-server[1132]: 2026/09/20 12:22:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4265worker1 # [ 69.527112] nix-grpc-daemon[928]: time=2026-09-20T12:22:02.557Z level=INFO msg="Uploading 1 narinfos"4266lb # [ 69.636420] niks3-server[1132]: 2026/09/20 12:22:02 INFO Signed narinfos id=5 count=14267lb # [ 69.638893] niks3-server[1132]: 2026/09/20 12:22:02 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls4268lb # [ 69.651401] niks3-server[1132]: 2026/09/20 12:22:02 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4269worker1 # [ 69.549436] nix-grpc-daemon[928]: time=2026-09-20T12:22:02.579Z level=INFO msg="Upload complete. (548ms)"4270lb # [ 69.658605] niks3-server[1132]: 2026/09/20 12:22:02 INFO Completed upload id=54271lb # [ 69.662412] niks3-server[1132]: 2026/09/20 12:22:02 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo4272worker1 # [ 69.556231] nix-grpc-daemon[855]: ts=2026-09-20T12:22:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=0 paths=1 nar_bytes_in=7524273worker1 # [ 69.572003] nix-grpc-daemon[855]: ts=2026-09-20T12:22:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524274worker1 # [ 69.579620] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4275worker1 # [ 69.585999] nix-daemon[919]: reaped child process 1007, status = succeeded4276client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051...4277client # farm-top> node-a: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4278worker1 # [ 69.602585] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4279client # farm-top> LOG-farm-top-t14280worker1 # [ 70.445789] nix-daemon[919]: reaped child process 1021, status = succeeded4281worker1 # [ 70.650926] nix-daemon[919]: accepted connection from pid 1036, user nix-grpc-daemon (trusted)4282worker1 # [ 70.674727] nix-daemon[919]: reaped child process 1039, status = succeeded4283lb # [ 70.800990] niks3-server[1132]: 2026/09/20 12:22:03 INFO Received uploads request method=POST path=/api/pending_closures4284worker1 # [ 70.704374] nix-grpc-daemon[928]: time=2026-09-20T12:22:03.732Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4285worker1 # [ 70.705960] nix-grpc-daemon[928]: time=2026-09-20T12:22:03.732Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)"4286lb # [ 70.856781] niks3-server[1132]: 2026/09/20 12:22:03 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4287lb # [ 70.868616] niks3-server[1132]: 2026/09/20 12:22:03 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst4288lb # [ 70.895318] niks3-server[1132]: 2026/09/20 12:22:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4289worker1 # [ 70.792375] nix-grpc-daemon[928]: time=2026-09-20T12:22:03.822Z level=INFO msg="Uploading 1 narinfos"4290lb # [ 70.908514] niks3-server[1132]: 2026/09/20 12:22:03 INFO Signed narinfos id=6 count=14291lb # [ 70.917149] niks3-server[1132]: 2026/09/20 12:22:03 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls4292lb # [ 70.930150] niks3-server[1132]: 2026/09/20 12:22:03 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4293lb # [ 70.936918] niks3-server[1132]: 2026/09/20 12:22:03 INFO Completed upload id=64294worker1 # [ 70.829696] nix-grpc-daemon[928]: time=2026-09-20T12:22:03.859Z level=INFO msg="Upload complete. (415ms)"4295lb # [ 70.940507] niks3-server[1132]: 2026/09/20 12:22:03 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo4296worker1 # [ 70.841393] nix-grpc-daemon[855]: ts=2026-09-20T12:22:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=14297worker1 # [ 70.861525] nix-daemon[919]: reaped child process 1019, status = succeeded4298client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2, in 7.06 seconds)4299client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath4300client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath, in 0.27 seconds)4301client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4302client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.03 seconds)4303client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4304client # copying 3 paths...4305client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'...4306client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'...4307client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'...4308client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.35 seconds)4309(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.84 seconds)4310subtest: repeat is answered Cached by the scheduler without building4311worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4312worker1 # finding garbage collector roots...4313worker1 # deleting specified paths...4314worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4315worker1 # deleting unused links...4316worker1 # note: hard linking is currently saving 0.0 KiB4317worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.31 seconds)4318worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4319worker2 # finding garbage collector roots...4320worker2 # deleting specified paths...4321worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4322worker2 # deleting unused links...4323worker2 # note: hard linking is currently saving 0.0 KiB4324worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.29 seconds)4325client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&24326worker2 # [ 72.010956] nix-grpc-daemon[862]: ts=2026-09-20T12:22:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:428324327client # don't know how to build these paths:4328client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4329worker1 # [ 72.593158] nix-grpc-daemon[855]: ts=2026-09-20T12:22:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:416524330worker1 # [ 72.609248] nix-grpc-daemon[928]: time=2026-09-20T12:22:05.637Z level=INFO msg="All 2 paths already cached"4331worker2 # [ 72.057848] nix-grpc-daemon[862]: ts=2026-09-20T12:22:05Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=14332client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2, in 0.37 seconds)4333client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath4334client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath, in 0.23 seconds)4335lb: must succeed: curl -sf localhost:9901/clusters4336lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.11 seconds)4337lb: must succeed: curl -sf localhost:9901/clusters4338lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4339worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4340worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)4341(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.61 seconds)4342subtest: inputs already in the cache are not sent to the scheduler4343lb: must succeed: curl -sf localhost:9901/clusters4344lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)4345lb: must succeed: curl -sf localhost:9901/clusters4346lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4347worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4348worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.08 seconds)4349client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b >&24350worker2 # [ 73.237848] nix-grpc-daemon[862]: ts=2026-09-20T12:22:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:428324351client # don't know how to build these paths:4352client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4353worker2 # [ 73.250921] nix-grpc-daemon[862]: ts=2026-09-20T12:22:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:428324354worker2 # [ 73.290420] nix-grpc-daemon[862]: ts=2026-09-20T12:22:06Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=04355worker2 # [ 73.298585] nix-grpc-daemon[862]: ts=2026-09-20T12:22:06Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.3:50051 clients=14356worker1 # [ 73.909493] nix-grpc-daemon[855]: ts=2026-09-20T12:22:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524357worker1 # [ 73.933055] nix-grpc-daemon[855]: ts=2026-09-20T12:22:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:416524358worker1 # [ 73.950105] nix-grpc-daemon[928]: time=2026-09-20T12:22:06.979Z level=INFO msg="All 2 paths already cached"4359client # copying 1 paths to 'lb:50051'...4360worker1 # [ 73.963168] nix-grpc-daemon[855]: ts=2026-09-20T12:22:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:416524361client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'...4362worker1 # [ 73.977570] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4363worker1 # [ 74.200290] nix-daemon[919]: accepted connection from pid 1056, user nix-grpc-daemon (trusted)4364worker1 # [ 74.226705] nix-daemon[919]: reaped child process 1059, status = succeeded4365worker1 # [ 74.438419] nix-daemon[919]: accepted connection from pid 1061, user nix-grpc-daemon (trusted)4366worker1 # [ 74.464068] nix-daemon[919]: reaped child process 1064, status = succeeded4367lb # [ 74.584488] niks3-server[1132]: 2026/09/20 12:22:07 INFO Received uploads request method=POST path=/api/pending_closures4368worker1 # [ 74.484116] nix-grpc-daemon[928]: time=2026-09-20T12:22:07.514Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4369worker1 # [ 74.485814] nix-grpc-daemon[928]: time=2026-09-20T12:22:07.514Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)"4370lb # [ 74.626311] niks3-server[1132]: 2026/09/20 12:22:07 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst4371worker1 # [ 74.534729] nix-grpc-daemon[928]: time=2026-09-20T12:22:07.564Z level=INFO msg="Uploading 1 narinfos"4372lb # [ 74.645005] niks3-server[1132]: 2026/09/20 12:22:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4373lb # [ 74.653495] niks3-server[1132]: 2026/09/20 12:22:07 INFO Signed narinfos id=7 count=14374lb # [ 74.663792] niks3-server[1132]: 2026/09/20 12:22:07 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4375lb # [ 74.667375] niks3-server[1132]: 2026/09/20 12:22:07 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls4376lb # [ 74.684544] niks3-server[1132]: 2026/09/20 12:22:07 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo4377lb # [ 74.695449] niks3-server[1132]: 2026/09/20 12:22:07 INFO Completed upload id=74378worker1 # [ 74.588474] nix-grpc-daemon[928]: time=2026-09-20T12:22:07.618Z level=INFO msg="Upload complete. (587ms)"4379worker1 # [ 74.604624] nix-grpc-daemon[855]: ts=2026-09-20T12:22:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=0 paths=1 nar_bytes_in=7524380worker1 # [ 74.616633] nix-grpc-daemon[855]: ts=2026-09-20T12:22:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524381worker1 # [ 74.631480] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4382worker1 # [ 74.640680] nix-daemon[919]: reaped child process 1054, status = succeeded4383client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051...4384worker1 # [ 74.657117] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4385client # farm-top> node-a: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4386lb # [ 75.542104] envoy[882]: conn peer=[2001:db8:1::1]:44576 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9497 tx=3857 ms=67914387lb # [ 75.544498] envoy[882]: conn peer=[2001:db8:1::1]:59764 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=481 ms=774388client # farm-top> LOG-farm-top-t1b4389worker1 # [ 75.698601] nix-daemon[919]: reaped child process 1068, status = succeeded4390worker1 # [ 75.891521] nix-daemon[919]: accepted connection from pid 1083, user nix-grpc-daemon (trusted)4391worker1 # [ 75.918024] nix-daemon[919]: reaped child process 1086, status = succeeded4392lb # [ 76.031268] niks3-server[1132]: 2026/09/20 12:22:09 INFO Received uploads request method=POST path=/api/pending_closures4393worker1 # [ 76.099343] nix-grpc-daemon[928]: time=2026-09-20T12:22:09.128Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4394worker1 # [ 76.114753] nix-grpc-daemon[928]: time=2026-09-20T12:22:09.134Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)"4395worker1 # [ 76.173038] nix-grpc-daemon[928]: time=2026-09-20T12:22:09.202Z level=INFO msg="Uploading 1 narinfos"4396lb # [ 76.272984] niks3-server[1132]: 2026/09/20 12:22:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4397lb # [ 76.290966] niks3-server[1132]: 2026/09/20 12:22:09 INFO Signed narinfos id=8 count=14398lb # [ 76.298898] niks3-server[1132]: 2026/09/20 12:22:09 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4399lb # [ 76.308761] niks3-server[1132]: 2026/09/20 12:22:09 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst4400lb # [ 76.321978] niks3-server[1132]: 2026/09/20 12:22:09 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls4401worker1 # [ 76.221092] nix-grpc-daemon[928]: time=2026-09-20T12:22:09.251Z level=INFO msg="Upload complete. (546ms)"4402lb # [ 76.329697] niks3-server[1132]: 2026/09/20 12:22:09 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4403lb # [ 76.336739] niks3-server[1132]: 2026/09/20 12:22:09 INFO Completed upload id=84404lb # [ 76.339372] niks3-server[1132]: 2026/09/20 12:22:09 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo4405worker1 # [ 76.233402] nix-grpc-daemon[855]: ts=2026-09-20T12:22:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=14406worker1 # [ 76.256988] nix-daemon[919]: reaped child process 1066, status = succeeded4407client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.74 seconds)4408client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b outPath4409client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.24 seconds)4410lb: must succeed: curl -sf localhost:9901/clusters4411lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4412lb: must succeed: curl -sf localhost:9901/clusters4413lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4414worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4415worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.08 seconds)4416(finished: subtest: inputs already in the cache are not sent to the scheduler, in 3.42 seconds)4417subtest: two clients wanting the same drv build it once4418client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup4419client # [ 77.086072] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup.4420client # Running as unit: dedup1.service; invocation ID: ff269ff42f854546983b7bdd95b24f524421client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup, in 0.07 seconds)4422client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup4423client # [ 77.350723] nix[967]: don't know how to build these paths:4424worker2 # [ 76.468147] nix-grpc-daemon[862]: ts=2026-09-20T12:22:09Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:428324425client # [ 77.351775] nix[967]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4426worker2 # [ 76.480920] nix-grpc-daemon[862]: ts=2026-09-20T12:22:09Z level=debug event=want client=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04427worker2 # [ 76.486374] nix-grpc-daemon[862]: ts=2026-09-20T12:22:09Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.4:50051 clients=14428worker2 # [ 76.492559] nix-grpc-daemon[862]: ts=2026-09-20T12:22:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:428324429worker2 # [ 76.513868] nix-grpc-daemon[862]: ts=2026-09-20T12:22:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:428324430client # [ 77.408234] nix[967]: copying 1 paths to 'lb:50051'...4431client # [ 77.411997] nix[967]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'...4432worker2 # [ 76.532184] nix-grpc-daemon[862]: ts=2026-09-20T12:22:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:428324433worker2 # [ 76.544950] nix-daemon[936]: accepted connection from pid 862, user nix-grpc-daemon (trusted)4434worker2 # [ 76.568343] nix-grpc-daemon[862]: ts=2026-09-20T12:22:10Z level=info event=niks3_push_started pid=9844435worker2 # [ 76.852562] nix-grpc-daemon[984]: time=2026-09-20T12:22:10.363Z level=INFO msg="Configuring client TLS" cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/worker2.pem key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/worker2.key ca=""4436worker2 # [ 77.064951] nix-daemon[936]: accepted connection from pid 997, user nix-grpc-daemon (trusted)4437worker2 # [ 77.087769] nix-daemon[936]: reaped child process 1000, status = succeeded4438worker2 # [ 77.288141] nix-daemon[936]: accepted connection from pid 1002, user nix-grpc-daemon (trusted)4439worker2 # [ 77.310134] nix-daemon[936]: reaped child process 1005, status = succeeded4440lb # [ 78.001492] niks3-server[1132]: 2026/09/20 12:22:10 INFO Received uploads request method=POST path=/api/pending_closures4441client # [ 78.199247] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup.4442worker2 # [ 77.339700] nix-grpc-daemon[984]: time=2026-09-20T12:22:10.852Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4443worker2 # [ 77.343368] nix-grpc-daemon[984]: time=2026-09-20T12:22:10.854Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)"4444client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup, in 1.13 seconds)4445lb # [ 78.057868] niks3-server[1132]: 2026/09/20 12:22:11 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst4446lb # [ 78.069137] niks3-server[1132]: 2026/09/20 12:22:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4447worker2 # [ 77.400532] nix-grpc-daemon[984]: time=2026-09-20T12:22:10.912Z level=INFO msg="Uploading 1 narinfos"4448lb # [ 78.082163] niks3-server[1132]: 2026/09/20 12:22:11 INFO Signed narinfos id=9 count=14449lb # [ 78.091461] niks3-server[1132]: 2026/09/20 12:22:11 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls4450lb # [ 78.106080] niks3-server[1132]: 2026/09/20 12:22:11 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4451lb # [ 78.118823] niks3-server[1132]: 2026/09/20 12:22:11 INFO Completed upload id=94452worker2 # [ 77.448593] nix-grpc-daemon[984]: time=2026-09-20T12:22:10.957Z level=INFO msg="Upload complete. (582ms)"4453lb # [ 78.122944] niks3-server[1132]: 2026/09/20 12:22:11 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo4454worker2 # [ 77.451256] nix-grpc-daemon[862]: ts=2026-09-20T12:22:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42832 duration_s=0 paths=1 nar_bytes_in=4324455worker2 # [ 77.462579] nix-grpc-daemon[862]: ts=2026-09-20T12:22:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:428324456worker2 # [ 77.478406] nix-daemon[936]: accepted connection from pid 862, user nix-grpc-daemon (trusted)4457worker2 # [ 77.482597] nix-daemon[936]: reaped child process 982, status = succeeded4458client # Running as unit: dedup2.service; invocation ID: ab04d405e20544929a091fa6118c1cce4459client # [ 78.375790] nix[967]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4460worker2 # [ 77.498127] nix-daemon[936]: accepted connection from pid 862, user nix-grpc-daemon (trusted)4461worker2 # [ 77.694014] nix-grpc-daemon[862]: ts=2026-09-20T12:22:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:428324462client # [ 78.589828] nix[983]: this derivation will be built:4463worker1 # [ 78.282668] nix-grpc-daemon[855]: ts=2026-09-20T12:22:11Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:416524464worker1 # [ 78.304164] nix-grpc-daemon[855]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' from 'http://lb:5751'...4465client # [ 78.639185] nix[983]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4466worker2 # [ 77.774692] nix-grpc-daemon[862]: ts=2026-09-20T12:22:11Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04467worker2 # [ 77.785242] nix-grpc-daemon[862]: ts=2026-09-20T12:22:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:428324468client # [ 78.670299] nix[983]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4469client: must succeed: sleep 24470client: (finished: must succeed: sleep 2, in 2.07 seconds)4471client: waiting for success: ! systemctl is-active dedup1 dedup24472worker2 # [ 82.135027] nix-daemon[936]: reaped child process 1013, status = succeeded4473worker2 # [ 82.326355] nix-daemon[936]: accepted connection from pid 1044, user nix-grpc-daemon (trusted)4474worker2 # [ 82.348915] nix-daemon[936]: reaped child process 1047, status = succeeded4475lb # [ 83.037935] niks3-server[1132]: 2026/09/20 12:22:16 INFO Received uploads request method=POST path=/api/pending_closures4476worker2 # [ 82.376527] nix-grpc-daemon[984]: time=2026-09-20T12:22:15.888Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4477worker2 # [ 82.378177] nix-grpc-daemon[984]: time=2026-09-20T12:22:15.888Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)"4478lb # [ 83.104197] niks3-server[1132]: 2026/09/20 12:22:16 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4479lb # [ 83.113939] niks3-server[1132]: 2026/09/20 12:22:16 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4480lb # [ 83.124166] niks3-server[1132]: 2026/09/20 12:22:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4481worker2 # [ 82.454326] nix-grpc-daemon[984]: time=2026-09-20T12:22:15.966Z level=INFO msg="Uploading 1 narinfos"4482lb # [ 83.133536] niks3-server[1132]: 2026/09/20 12:22:16 INFO Signed narinfos id=10 count=14483lb # [ 83.139571] niks3-server[1132]: 2026/09/20 12:22:16 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls4484lb # [ 83.152959] niks3-server[1132]: 2026/09/20 12:22:16 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4485lb # [ 83.161168] niks3-server[1132]: 2026/09/20 12:22:16 INFO Completed upload id=104486worker2 # [ 82.483484] nix-grpc-daemon[984]: time=2026-09-20T12:22:15.996Z level=INFO msg="Upload complete. (366ms)"4487lb # [ 83.164523] niks3-server[1132]: 2026/09/20 12:22:16 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo4488worker2 # [ 82.505466] nix-grpc-daemon[862]: ts=2026-09-20T12:22:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42832 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14489worker2 # [ 82.520003] nix-grpc-daemon[862]: ts=2026-09-20T12:22:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42832 duration_s=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14490worker2 # [ 82.525941] nix-daemon[936]: reaped child process 1011, status = succeeded4491client # [ 83.430308] systemd[1]: dedup1.service: Deactivated successfully.4492client # [ 83.432832] systemd[1]: dedup1.service: Consumed 167ms CPU time over 6.347s wall clock time, 37.2M memory peak, 8.6K incoming IP traffic, 9.1K outgoing IP traffic.4493client # [ 83.438228] systemd[1]: dedup2.service: Deactivated successfully.4494client # [ 83.441025] systemd[1]: dedup2.service: Consumed 210ms CPU time over 5.239s wall clock time, 31M memory peak, 7.9K incoming IP traffic, 6.4K outgoing IP traffic.4495client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.17 seconds)4496client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4497client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.06 seconds)4498worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4499worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.08 seconds)4500worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4501worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.08 seconds)4502worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4503worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.11 seconds)4504worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4505worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.10 seconds)4506(finished: subtest: two clients wanting the same drv build it once, in 7.51 seconds)4507subtest: build hook: nix-daemon with builders = grpc://lb4508client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag hook 2>/tmp/hook.log4509client # [ 84.813490] systemd[1]: Started Nix Daemon.4510client # [ 85.053191] nix-daemon[1025]: accepted connection from pid 1022, user root (trusted)4511worker1 # [ 84.983447] nix-grpc-daemon[855]: ts=2026-09-20T12:22:18Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:416524512worker1 # [ 84.993559] nix-grpc-daemon[855]: ts=2026-09-20T12:22:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:416524513worker2 # [ 84.442302] nix-grpc-daemon[862]: ts=2026-09-20T12:22:17Z level=debug event=want client=6 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=04514worker2 # [ 84.442525] nix-grpc-daemon[862]: ts=2026-09-20T12:22:17Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.3:50051 clients=14515worker1 # [ 85.058957] nix-grpc-daemon[855]: ts=2026-09-20T12:22:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524516worker1 # [ 85.080454] nix-grpc-daemon[855]: ts=2026-09-20T12:22:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:416524517worker1 # [ 85.099981] nix-grpc-daemon[855]: ts=2026-09-20T12:22:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:416524518worker1 # [ 85.127156] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4519worker1 # [ 85.368985] nix-daemon[919]: accepted connection from pid 1116, user nix-grpc-daemon (trusted)4520worker1 # [ 85.393813] nix-daemon[919]: reaped child process 1119, status = succeeded4521lb # [ 85.542880] envoy[882]: conn peer=[2001:db8:1::1]:59774 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4228 tx=1746 ms=24514522lb # [ 85.548590] envoy[882]: conn peer=[2001:db8:1::1]:59776 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=1397 ms=60484523lb # [ 85.553977] envoy[882]: conn peer=[2001:db8:1::1]:59778 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1579 tx=1487 ms=48284524worker1 # [ 85.573147] nix-daemon[919]: accepted connection from pid 1121, user nix-grpc-daemon (trusted)4525worker1 # [ 85.594864] nix-daemon[919]: reaped child process 1124, status = succeeded4526lb # [ 85.719128] niks3-server[1132]: 2026/09/20 12:22:18 INFO Received uploads request method=POST path=/api/pending_closures4527worker1 # [ 85.624402] nix-grpc-daemon[928]: time=2026-09-20T12:22:18.653Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4528worker1 # [ 85.625987] nix-grpc-daemon[928]: time=2026-09-20T12:22:18.653Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)"4529lb # [ 85.766060] niks3-server[1132]: 2026/09/20 12:22:18 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst4530worker1 # [ 85.675233] nix-grpc-daemon[928]: time=2026-09-20T12:22:18.705Z level=INFO msg="Uploading 1 narinfos"4531lb # [ 85.784905] niks3-server[1132]: 2026/09/20 12:22:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4532lb # [ 85.794232] niks3-server[1132]: 2026/09/20 12:22:18 INFO Signed narinfos id=11 count=14533lb # [ 85.801126] niks3-server[1132]: 2026/09/20 12:22:18 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls4534lb # [ 85.811235] niks3-server[1132]: 2026/09/20 12:22:18 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4535lb # [ 85.817760] niks3-server[1132]: 2026/09/20 12:22:18 INFO Completed upload id=114536worker1 # [ 85.708790] nix-grpc-daemon[928]: time=2026-09-20T12:22:18.739Z level=INFO msg="Upload complete. (557ms)"4537lb # [ 85.820589] niks3-server[1132]: 2026/09/20 12:22:18 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo4538worker1 # [ 85.724771] nix-grpc-daemon[855]: ts=2026-09-20T12:22:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=0 paths=1 nar_bytes_in=5124539worker1 # [ 85.740841] nix-grpc-daemon[855]: ts=2026-09-20T12:22:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524540worker1 # [ 85.749484] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4541worker1 # [ 85.755402] nix-daemon[919]: reaped child process 1114, status = succeeded4542worker1 # [ 85.769033] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4543worker1 # [ 87.009826] nix-daemon[919]: reaped child process 1128, status = succeeded4544worker1 # [ 87.208969] nix-daemon[919]: accepted connection from pid 1143, user nix-grpc-daemon (trusted)4545worker1 # [ 87.232487] nix-daemon[919]: reaped child process 1146, status = succeeded4546lb # [ 87.358562] niks3-server[1132]: 2026/09/20 12:22:20 INFO Received uploads request method=POST path=/api/pending_closures4547worker1 # [ 87.264645] nix-grpc-daemon[928]: time=2026-09-20T12:22:20.292Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4548worker1 # [ 87.266259] nix-grpc-daemon[928]: time=2026-09-20T12:22:20.292Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)"4549lb # [ 87.412877] niks3-server[1132]: 2026/09/20 12:22:20 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv4550lb # [ 87.429821] niks3-server[1132]: 2026/09/20 12:22:20 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4551lb # [ 87.442649] niks3-server[1132]: 2026/09/20 12:22:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4552worker1 # [ 87.334932] nix-grpc-daemon[928]: time=2026-09-20T12:22:20.364Z level=INFO msg="Uploading 1 narinfos"4553lb # [ 87.447853] niks3-server[1132]: 2026/09/20 12:22:20 INFO Signed narinfos id=12 count=14554lb # [ 87.455818] niks3-server[1132]: 2026/09/20 12:22:20 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls4555lb # [ 87.468708] niks3-server[1132]: 2026/09/20 12:22:20 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4556lb # [ 87.475892] niks3-server[1132]: 2026/09/20 12:22:20 INFO Completed upload id=124557worker1 # [ 87.367077] nix-grpc-daemon[928]: time=2026-09-20T12:22:20.397Z level=INFO msg="Upload complete. (381ms)"4558lb # [ 87.478374] niks3-server[1132]: 2026/09/20 12:22:20 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo4559worker1 # [ 87.381812] nix-grpc-daemon[855]: ts=2026-09-20T12:22:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=6 outputs=14560worker2 # [ 86.819417] nix-grpc-daemon[862]: ts=2026-09-20T12:22:20Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:428324561worker1 # [ 87.405051] nix-daemon[919]: reaped child process 1126, status = succeeded4562worker2 # [ 86.845349] nix-grpc-daemon[862]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'...4563worker1 # [ 87.471565] nix-grpc-daemon[855]: ts=2026-09-20T12:22:20Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:416524564worker1 # [ 87.479057] nix-grpc-daemon[855]: ts=2026-09-20T12:22:20Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=0 paths=1 nar_bytes_out=1284565worker1 # [ 87.778816] nix-grpc-daemon[855]: ts=2026-09-20T12:22:20Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:416524566worker2 # [ 87.217274] nix-grpc-daemon[862]: ts=2026-09-20T12:22:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:428324567worker2 # [ 87.234916] nix-grpc-daemon[984]: time=2026-09-20T12:22:20.746Z level=INFO msg="All 1 paths already cached"4568worker2 # [ 87.258219] nix-grpc-daemon[862]: ts=2026-09-20T12:22:20Z level=debug event=want client=7 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=04569worker2 # [ 87.265693] nix-grpc-daemon[862]: ts=2026-09-20T12:22:20Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.3:50051 clients=14570worker1 # [ 87.838430] nix-grpc-daemon[855]: ts=2026-09-20T12:22:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524571worker1 # [ 87.859132] nix-grpc-daemon[855]: ts=2026-09-20T12:22:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:416524572worker1 # [ 87.869617] nix-grpc-daemon[928]: time=2026-09-20T12:22:20.899Z level=INFO msg="All 1 paths already cached"4573worker1 # [ 87.879412] nix-grpc-daemon[855]: ts=2026-09-20T12:22:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:416524574worker1 # [ 87.897452] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4575worker1 # [ 88.150652] nix-daemon[919]: accepted connection from pid 1150, user nix-grpc-daemon (trusted)4576worker1 # [ 88.177453] nix-daemon[919]: reaped child process 1153, status = succeeded4577worker1 # [ 88.381410] nix-daemon[919]: accepted connection from pid 1155, user nix-grpc-daemon (trusted)4578worker1 # [ 88.407127] nix-daemon[919]: reaped child process 1158, status = succeeded4579lb # [ 88.529830] niks3-server[1132]: 2026/09/20 12:22:21 INFO Received uploads request method=POST path=/api/pending_closures4580worker1 # [ 88.556539] nix-grpc-daemon[928]: time=2026-09-20T12:22:21.584Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4581worker1 # [ 88.561514] nix-grpc-daemon[928]: time=2026-09-20T12:22:21.584Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)"4582worker1 # [ 88.598961] nix-grpc-daemon[928]: time=2026-09-20T12:22:21.628Z level=INFO msg="Uploading 1 narinfos"4583lb # [ 88.696728] niks3-server[1132]: 2026/09/20 12:22:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4584lb # [ 88.713274] niks3-server[1132]: 2026/09/20 12:22:21 INFO Signed narinfos id=13 count=14585lb # [ 88.722749] niks3-server[1132]: 2026/09/20 12:22:21 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst4586lb # [ 88.734725] niks3-server[1132]: 2026/09/20 12:22:21 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls4587worker1 # [ 88.639421] nix-grpc-daemon[928]: time=2026-09-20T12:22:21.669Z level=INFO msg="Upload complete. (719ms)"4588lb # [ 88.748848] niks3-server[1132]: 2026/09/20 12:22:21 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4589lb # [ 88.754963] niks3-server[1132]: 2026/09/20 12:22:21 INFO Completed upload id=134590lb # [ 88.758245] niks3-server[1132]: 2026/09/20 12:22:21 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo4591worker1 # [ 88.653598] nix-grpc-daemon[855]: ts=2026-09-20T12:22:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=0 paths=1 nar_bytes_in=6244592worker1 # [ 88.670068] nix-grpc-daemon[855]: ts=2026-09-20T12:22:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524593worker1 # [ 88.689514] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4594worker1 # [ 88.698333] nix-daemon[919]: reaped child process 1148, status = succeeded4595worker1 # [ 88.720319] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4596worker1 # [ 89.956527] nix-daemon[919]: reaped child process 1162, status = succeeded4597worker1 # [ 90.159954] nix-daemon[919]: accepted connection from pid 1177, user nix-grpc-daemon (trusted)4598worker1 # [ 90.185490] nix-daemon[919]: reaped child process 1180, status = succeeded4599lb # [ 90.311194] niks3-server[1132]: 2026/09/20 12:22:23 INFO Received uploads request method=POST path=/api/pending_closures4600worker1 # [ 90.218729] nix-grpc-daemon[928]: time=2026-09-20T12:22:23.248Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4601worker1 # [ 90.222444] nix-grpc-daemon[928]: time=2026-09-20T12:22:23.250Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)"4602lb # [ 90.343192] niks3-server[1132]: 2026/09/20 12:22:23 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv4603lb # [ 90.355224] niks3-server[1132]: 2026/09/20 12:22:23 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst4604lb # [ 90.362492] niks3-server[1132]: 2026/09/20 12:22:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4605worker1 # [ 90.258566] nix-grpc-daemon[928]: time=2026-09-20T12:22:23.288Z level=INFO msg="Uploading 1 narinfos"4606lb # [ 90.369355] niks3-server[1132]: 2026/09/20 12:22:23 INFO Signed narinfos id=14 count=14607lb # [ 90.372121] niks3-server[1132]: 2026/09/20 12:22:23 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls4608lb # [ 90.379661] niks3-server[1132]: 2026/09/20 12:22:23 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4609lb # [ 90.387151] niks3-server[1132]: 2026/09/20 12:22:23 INFO Completed upload id=144610worker1 # [ 90.279998] nix-grpc-daemon[928]: time=2026-09-20T12:22:23.310Z level=INFO msg="Upload complete. (340ms)"4611lb # [ 90.389270] niks3-server[1132]: 2026/09/20 12:22:23 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo4612worker1 # [ 90.286944] nix-grpc-daemon[855]: ts=2026-09-20T12:22:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=7 outputs=14613worker1 # [ 90.296529] nix-daemon[919]: reaped child process 1160, status = succeeded4614worker2 # [ 89.729291] nix-grpc-daemon[862]: ts=2026-09-20T12:22:23Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:428324615worker2 # [ 89.747593] nix-grpc-daemon[862]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'...4616worker1 # [ 90.367205] nix-grpc-daemon[855]: ts=2026-09-20T12:22:23Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:416524617worker1 # [ 90.370633] nix-grpc-daemon[855]: ts=2026-09-20T12:22:23Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=0 paths=1 nar_bytes_out=1764618worker2 # [ 90.079350] nix-grpc-daemon[862]: ts=2026-09-20T12:22:23Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:428324619worker2 # [ 90.092591] nix-grpc-daemon[862]: ts=2026-09-20T12:22:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:428324620worker2 # [ 90.107041] nix-grpc-daemon[984]: time=2026-09-20T12:22:23.618Z level=INFO msg="All 2 paths already cached"4621worker2 # [ 90.134231] nix-grpc-daemon[862]: ts=2026-09-20T12:22:23Z level=debug event=want client=8 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=04622worker2 # [ 90.141698] nix-grpc-daemon[862]: ts=2026-09-20T12:22:23Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.3:50051 clients=14623worker1 # [ 90.715181] nix-grpc-daemon[855]: ts=2026-09-20T12:22:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524624worker1 # [ 90.739646] nix-grpc-daemon[855]: ts=2026-09-20T12:22:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:416524625worker1 # [ 90.759468] nix-grpc-daemon[928]: time=2026-09-20T12:22:23.789Z level=INFO msg="All 2 paths already cached"4626worker1 # [ 90.773454] nix-grpc-daemon[855]: ts=2026-09-20T12:22:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:416524627worker1 # [ 90.792513] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4628worker1 # [ 91.035456] nix-daemon[919]: accepted connection from pid 1184, user nix-grpc-daemon (trusted)4629worker1 # [ 91.057803] nix-daemon[919]: reaped child process 1187, status = succeeded4630worker1 # [ 91.248001] nix-daemon[919]: accepted connection from pid 1189, user nix-grpc-daemon (trusted)4631worker1 # [ 91.271508] nix-daemon[919]: reaped child process 1192, status = succeeded4632lb # [ 91.393820] niks3-server[1132]: 2026/09/20 12:22:24 INFO Received uploads request method=POST path=/api/pending_closures4633worker1 # [ 91.301950] nix-grpc-daemon[928]: time=2026-09-20T12:22:24.331Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4634worker1 # [ 91.303612] nix-grpc-daemon[928]: time=2026-09-20T12:22:24.331Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)"4635lb # [ 91.420179] niks3-server[1132]: 2026/09/20 12:22:24 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst4636lb # [ 91.427984] niks3-server[1132]: 2026/09/20 12:22:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4637lb # [ 91.430265] niks3-server[1132]: 2026/09/20 12:22:24 INFO Signed narinfos id=15 count=14638worker1 # [ 91.324542] nix-grpc-daemon[928]: time=2026-09-20T12:22:24.354Z level=INFO msg="Uploading 1 narinfos"4639lb # [ 91.437323] niks3-server[1132]: 2026/09/20 12:22:24 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls4640lb # [ 91.444761] niks3-server[1132]: 2026/09/20 12:22:24 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4641lb # [ 91.449208] niks3-server[1132]: 2026/09/20 12:22:24 INFO Completed upload id=154642worker1 # [ 91.341688] nix-grpc-daemon[928]: time=2026-09-20T12:22:24.372Z level=INFO msg="Upload complete. (527ms)"4643lb # [ 91.451061] niks3-server[1132]: 2026/09/20 12:22:24 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo4644worker1 # [ 91.353121] nix-grpc-daemon[855]: ts=2026-09-20T12:22:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=0 paths=1 nar_bytes_in=7524645worker1 # [ 91.363149] nix-grpc-daemon[855]: ts=2026-09-20T12:22:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524646worker1 # [ 91.370033] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4647worker1 # [ 91.374865] nix-daemon[919]: reaped child process 1182, status = succeeded4648worker1 # [ 91.387794] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4649worker1 # [ 92.416996] nix-daemon[919]: reaped child process 1196, status = succeeded4650worker1 # [ 92.605303] nix-daemon[919]: accepted connection from pid 1211, user nix-grpc-daemon (trusted)4651worker1 # [ 92.628779] nix-daemon[919]: reaped child process 1214, status = succeeded4652lb # [ 92.750049] niks3-server[1132]: 2026/09/20 12:22:25 INFO Received uploads request method=POST path=/api/pending_closures4653worker1 # [ 92.656563] nix-grpc-daemon[928]: time=2026-09-20T12:22:25.684Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4654worker1 # [ 92.658164] nix-grpc-daemon[928]: time=2026-09-20T12:22:25.685Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)"4655lb # [ 92.812641] niks3-server[1132]: 2026/09/20 12:22:25 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst4656worker1 # [ 92.716648] nix-grpc-daemon[928]: time=2026-09-20T12:22:25.746Z level=INFO msg="Uploading 1 narinfos"4657lb # [ 92.828003] niks3-server[1132]: 2026/09/20 12:22:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4658lb # [ 92.831327] niks3-server[1132]: 2026/09/20 12:22:25 INFO Signed narinfos id=16 count=14659lb # [ 92.841591] niks3-server[1132]: 2026/09/20 12:22:25 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv4660lb # [ 92.846240] niks3-server[1132]: 2026/09/20 12:22:25 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls4661lb # [ 92.853798] niks3-server[1132]: 2026/09/20 12:22:25 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4662lb # [ 92.857816] niks3-server[1132]: 2026/09/20 12:22:25 INFO Completed upload id=164663lb # [ 92.859865] niks3-server[1132]: 2026/09/20 12:22:25 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo4664worker1 # [ 92.750658] nix-grpc-daemon[928]: time=2026-09-20T12:22:25.780Z level=INFO msg="Upload complete. (349ms)"4665worker2 # [ 92.200129] nix-grpc-daemon[862]: ts=2026-09-20T12:22:25Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:428324666worker1 # [ 92.765579] nix-grpc-daemon[855]: ts=2026-09-20T12:22:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=8 outputs=14667worker2 # [ 92.209029] nix-grpc-daemon[862]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'...4668worker1 # [ 92.784849] nix-daemon[919]: reaped child process 1194, status = succeeded4669worker2 # [ 92.256231] nix-grpc-daemon[862]: ts=2026-09-20T12:22:25Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:428324670worker2 # [ 92.265978] nix-grpc-daemon[862]: ts=2026-09-20T12:22:25Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:42832 duration_s=0 paths=1 nar_bytes_out=2324671client # [ 93.307320] nix-daemon[1025]: reaped child process 1027, status = succeeded4672client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag hook 2>/tmp/hook.log, in 8.77 seconds)4673client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top4674client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.06 seconds)4675client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4676client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.06 seconds)4677client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4678client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.04 seconds)4679(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 8.93 seconds)4680subtest: an input only one worker has still reaches the builder4681client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag w1only4682client # this derivation will be built:4683client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv4684client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'...4685client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag w1only, in 0.54 seconds)4686client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure4687client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.23 seconds)4688worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4689worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.31 seconds)4690worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only4691worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.05 seconds)4692client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt a >&24693client # [ 94.849956] nix-daemon[1025]: accepted connection from pid 1107, user root (trusted)4694client # this derivation will be built:4695client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4696worker1 # [ 94.785727] nix-grpc-daemon[855]: ts=2026-09-20T12:22:27Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:416524697client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key&system=aarch64-linux'...4698worker2 # [ 94.231175] nix-grpc-daemon[862]: ts=2026-09-20T12:22:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:428324699client # copying 1 paths...4700client # copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' to 'grpc://lb:50051'...4701worker1 # [ 94.818104] nix-grpc-daemon[855]: ts=2026-09-20T12:22:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:416524702worker1 # [ 94.837091] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4703worker1 # [ 95.088705] nix-daemon[919]: accepted connection from pid 1223, user nix-grpc-daemon (trusted)4704worker1 # [ 95.112254] nix-daemon[919]: reaped child process 1226, status = succeeded4705lb # [ 95.238734] niks3-server[1132]: 2026/09/20 12:22:28 INFO Received uploads request method=POST path=/api/pending_closures4706worker1 # [ 95.142953] nix-grpc-daemon[928]: time=2026-09-20T12:22:28.172Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4707worker1 # [ 95.145177] nix-grpc-daemon[928]: time=2026-09-20T12:22:28.175Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)"4708lb # [ 95.276569] niks3-server[1132]: 2026/09/20 12:22:28 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4709lb # [ 95.297104] niks3-server[1132]: 2026/09/20 12:22:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4710worker1 # [ 95.195323] nix-grpc-daemon[928]: time=2026-09-20T12:22:28.225Z level=INFO msg="Uploading 1 narinfos"4711lb # [ 95.310417] niks3-server[1132]: 2026/09/20 12:22:28 INFO Signed narinfos id=17 count=14712lb # [ 95.317612] niks3-server[1132]: 2026/09/20 12:22:28 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls4713lb # [ 95.327960] niks3-server[1132]: 2026/09/20 12:22:28 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4714worker1 # [ 95.227781] nix-grpc-daemon[928]: time=2026-09-20T12:22:28.258Z level=INFO msg="Upload complete. (371ms)"4715lb # [ 95.337424] niks3-server[1132]: 2026/09/20 12:22:28 INFO Completed upload id=174716lb # [ 95.340653] niks3-server[1132]: 2026/09/20 12:22:28 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo4717worker1 # [ 95.238533] nix-grpc-daemon[855]: ts=2026-09-20T12:22:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=0 paths=1 nar_bytes_in=1204718worker1 # [ 95.259324] nix-daemon[919]: reaped child process 1221, status = succeeded4719worker2 # [ 94.697753] nix-grpc-daemon[862]: ts=2026-09-20T12:22:28Z level=debug event=want client=9 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=04720worker2 # [ 94.707110] nix-grpc-daemon[862]: ts=2026-09-20T12:22:28Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=14721worker2 # [ 94.746307] nix-grpc-daemon[862]: ts=2026-09-20T12:22:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:428324722worker2 # [ 94.763560] nix-grpc-daemon[862]: ts=2026-09-20T12:22:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:428324723client # copying 2 paths to 'lb:50051'...4724worker2 # [ 94.791390] nix-grpc-daemon[862]: ts=2026-09-20T12:22:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:428324725client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'...4726client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'...4727worker2 # [ 94.817928] nix-daemon[936]: accepted connection from pid 862, user nix-grpc-daemon (trusted)4728lb # [ 95.544173] envoy[882]: conn peer=[2001:db8:1::1]:34150 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=333 ms=414729lb # [ 95.549410] envoy[882]: conn peer=[2001:db8:1::1]:34140 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3437 tx=1792 ms=25304730lb # [ 95.554346] envoy[882]: conn peer=[2001:db8:1::1]:34154 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3888 tx=1989 ms=26114731lb # [ 95.559043] envoy[882]: conn peer=[2001:db8:1::1]:34168 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=375 ms=274732lb # [ 95.563000] envoy[882]: conn peer=[2001:db8:1::1]:35224 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4314 tx=2223 ms=22524733lb # [ 95.567401] envoy[882]: conn peer=[2001:db8:1::1]:35228 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=438 tx=407 ms=834734worker2 # [ 95.069902] nix-daemon[936]: accepted connection from pid 1074, user nix-grpc-daemon (trusted)4735worker2 # [ 95.096468] nix-daemon[936]: reaped child process 1077, status = succeeded4736worker2 # [ 95.292588] nix-daemon[936]: accepted connection from pid 1079, user nix-grpc-daemon (trusted)4737worker2 # [ 95.316571] nix-daemon[936]: reaped child process 1082, status = succeeded4738lb # [ 95.997582] niks3-server[1132]: 2026/09/20 12:22:28 INFO Received uploads request method=POST path=/api/pending_closures4739lb # [ 96.010781] niks3-server[1132]: 2026/09/20 12:22:28 INFO Received uploads request method=POST path=/api/pending_closures4740worker2 # [ 95.348100] nix-grpc-daemon[984]: time=2026-09-20T12:22:28.859Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4741worker2 # [ 95.349655] nix-grpc-daemon[984]: time=2026-09-20T12:22:28.859Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)"4742worker2 # [ 95.351478] nix-grpc-daemon[984]: time=2026-09-20T12:22:28.859Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)"4743lb # [ 96.049718] niks3-server[1132]: 2026/09/20 12:22:29 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst4744lb # [ 96.074257] niks3-server[1132]: 2026/09/20 12:22:29 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst4745lb # [ 96.091215] niks3-server[1132]: 2026/09/20 12:22:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign4746lb # [ 96.101107] niks3-server[1132]: 2026/09/20 12:22:29 INFO Signed narinfos id=18 count=14747worker2 # [ 95.427178] nix-grpc-daemon[984]: time=2026-09-20T12:22:28.938Z level=INFO msg="Uploading 3 narinfos"4748lb # [ 96.107116] niks3-server[1132]: 2026/09/20 12:22:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign4749lb # [ 96.114118] niks3-server[1132]: 2026/09/20 12:22:29 INFO Signed narinfos id=19 count=24750lb # [ 96.119088] niks3-server[1132]: 2026/09/20 12:22:29 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls4751lb # [ 96.125605] niks3-server[1132]: 2026/09/20 12:22:29 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls4752lb # [ 96.143551] niks3-server[1132]: 2026/09/20 12:22:29 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo4753lb # [ 96.152874] niks3-server[1132]: 2026/09/20 12:22:29 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete4754lb # [ 96.160183] niks3-server[1132]: 2026/09/20 12:22:29 INFO Completed upload id=184755lb # [ 96.161709] niks3-server[1132]: 2026/09/20 12:22:29 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete4756lb # [ 96.163947] niks3-server[1132]: 2026/09/20 12:22:29 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4757lb # [ 96.167372] niks3-server[1132]: 2026/09/20 12:22:29 INFO Completed upload id=194758lb # [ 96.169464] niks3-server[1132]: 2026/09/20 12:22:29 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4759worker2 # [ 95.492317] nix-grpc-daemon[984]: time=2026-09-20T12:22:29.005Z level=INFO msg="Upload complete. (649ms)"4760worker2 # [ 95.506568] nix-grpc-daemon[862]: ts=2026-09-20T12:22:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42832 duration_s=0 paths=2 nar_bytes_in=9844761worker2 # [ 95.525134] nix-grpc-daemon[862]: ts=2026-09-20T12:22:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:428324762worker2 # [ 95.538279] nix-daemon[936]: accepted connection from pid 862, user nix-grpc-daemon (trusted)4763worker2 # [ 95.546507] nix-daemon[936]: reaped child process 1072, status = succeeded4764worker2 # [ 95.566142] nix-grpc-daemon[862]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'...4765client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051...4766worker2 # [ 95.589262] nix-daemon[936]: accepted connection from pid 862, user nix-grpc-daemon (trusted)4767client # uses-input-w1onlya> worker2: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4768worker2 # [ 96.830462] nix-daemon[936]: reaped child process 1087, status = succeeded4769worker2 # [ 97.027469] nix-daemon[936]: accepted connection from pid 1102, user nix-grpc-daemon (trusted)4770worker2 # [ 97.051634] nix-daemon[936]: reaped child process 1105, status = succeeded4771lb # [ 97.743028] niks3-server[1132]: 2026/09/20 12:22:30 INFO Received uploads request method=POST path=/api/pending_closures4772worker2 # [ 97.076162] nix-grpc-daemon[984]: time=2026-09-20T12:22:30.588Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4773lb # [ 97.782769] niks3-server[1132]: 2026/09/20 12:22:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign4774worker2 # [ 97.120109] nix-grpc-daemon[984]: time=2026-09-20T12:22:30.632Z level=INFO msg="Uploading 1 narinfos"4775lb # [ 97.801270] niks3-server[1132]: 2026/09/20 12:22:30 INFO Signed narinfos id=20 count=14776lb # [ 97.807818] niks3-server[1132]: 2026/09/20 12:22:30 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4777lb # [ 97.817161] niks3-server[1132]: 2026/09/20 12:22:30 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls4778lb # [ 97.828557] niks3-server[1132]: 2026/09/20 12:22:30 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete4779worker2 # [ 97.157847] nix-grpc-daemon[984]: time=2026-09-20T12:22:30.670Z level=INFO msg="Upload complete. (357ms)"4780lb # [ 97.834234] niks3-server[1132]: 2026/09/20 12:22:30 INFO Completed upload id=204781lb # [ 97.838906] niks3-server[1132]: 2026/09/20 12:22:30 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo4782worker2 # [ 97.164574] nix-grpc-daemon[862]: ts=2026-09-20T12:22:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42832 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=9 outputs=14783worker2 # [ 97.176776] nix-daemon[936]: reaped child process 1084, status = succeeded4784client # copying 1 paths...4785worker1 # [ 97.791212] nix-grpc-daemon[855]: ts=2026-09-20T12:22:30Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:416524786worker1 # [ 97.806081] nix-grpc-daemon[855]: copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'http://lb:5751'...4787client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'...4788worker2 # [ 97.296196] nix-grpc-daemon[862]: ts=2026-09-20T12:22:30Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:428324789worker2 # [ 97.306285] nix-grpc-daemon[862]: ts=2026-09-20T12:22:30Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:42832 duration_s=0 paths=1 nar_bytes_out=1204790client # [ 98.275807] nix-daemon[1025]: reaped child process 1110, status = succeeded4791client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt a >&2, in 3.70 seconds)4792client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt b >&24793client # [ 98.530329] nix-daemon[1025]: accepted connection from pid 1128, user root (trusted)4794client # this derivation will be built:4795client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4796worker2 # [ 97.892692] nix-grpc-daemon[862]: ts=2026-09-20T12:22:31Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:428324797client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key&system=aarch64-linux'...4798worker2 # [ 97.903761] nix-grpc-daemon[862]: ts=2026-09-20T12:22:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:428324799worker2 # [ 97.918204] nix-grpc-daemon[984]: time=2026-09-20T12:22:31.430Z level=INFO msg="All 1 paths already cached"4800worker2 # [ 97.942087] nix-grpc-daemon[862]: ts=2026-09-20T12:22:31Z level=debug event=want client=10 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=04801worker2 # [ 97.951735] nix-grpc-daemon[862]: ts=2026-09-20T12:22:31Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=14802worker1 # [ 98.522980] nix-grpc-daemon[855]: ts=2026-09-20T12:22:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524803worker1 # [ 98.543160] nix-grpc-daemon[855]: ts=2026-09-20T12:22:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:416524804client # copying 1 paths to 'lb:50051'...4805client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'...4806worker1 # [ 98.568816] nix-grpc-daemon[855]: ts=2026-09-20T12:22:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:416524807worker1 # [ 98.589850] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4808worker1 # [ 98.621364] nix-grpc-daemon[855]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 45 ms (attempt 1/5)4809worker1 # [ 98.673596] nix-grpc-daemon[855]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 185 ms (attempt 2/5)4810worker1 # [ 98.868703] nix-grpc-daemon[855]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 74 ms (attempt 3/5)4811worker1 # [ 98.942780] nix-grpc-daemon[855]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 258 ms (attempt 4/5)4812worker1 # [ 99.210392] nix-grpc-daemon[855]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4813worker1 # [ 99.221716] nix-grpc-daemon[855]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'...4814worker1 # [ 99.509090] nix-daemon[919]: accepted connection from pid 1235, user nix-grpc-daemon (trusted)4815worker1 # [ 99.532680] nix-daemon[919]: reaped child process 1238, status = succeeded4816worker1 # [ 99.742225] nix-daemon[919]: accepted connection from pid 1240, user nix-grpc-daemon (trusted)4817worker1 # [ 99.763188] nix-daemon[919]: reaped child process 1243, status = succeeded4818lb # [ 99.886135] niks3-server[1132]: 2026/09/20 12:22:32 INFO Received uploads request method=POST path=/api/pending_closures4819worker1 # [ 99.784757] nix-grpc-daemon[928]: time=2026-09-20T12:22:32.814Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4820worker1 # [ 99.786379] nix-grpc-daemon[928]: time=2026-09-20T12:22:32.814Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)"4821lb # [ 99.907309] niks3-server[1132]: 2026/09/20 12:22:32 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst4822worker1 # [ 99.814984] nix-grpc-daemon[928]: time=2026-09-20T12:22:32.845Z level=INFO msg="Uploading 1 narinfos"4823lb # [ 99.918575] niks3-server[1132]: 2026/09/20 12:22:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign4824lb # [ 99.924796] niks3-server[1132]: 2026/09/20 12:22:32 INFO Signed narinfos id=21 count=14825lb # [ 99.928671] niks3-server[1132]: 2026/09/20 12:22:32 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls4826lb # [ 99.938703] niks3-server[1132]: 2026/09/20 12:22:32 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete4827lb # [ 99.944109] niks3-server[1132]: 2026/09/20 12:22:32 INFO Completed upload id=214828worker1 # [ 99.836586] nix-grpc-daemon[928]: time=2026-09-20T12:22:32.866Z level=INFO msg="Upload complete. (585ms)"4829lb # [ 99.946127] niks3-server[1132]: 2026/09/20 12:22:32 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo4830worker1 # [ 99.844302] nix-grpc-daemon[855]: ts=2026-09-20T12:22:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=1 paths=1 nar_bytes_in=5684831worker1 # [ 99.852653] nix-grpc-daemon[855]: ts=2026-09-20T12:22:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416524832worker1 # [ 99.861637] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4833worker1 # [ 99.872284] nix-daemon[919]: reaped child process 1229, status = succeeded4834client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051...4835worker1 # [ 99.880254] nix-daemon[919]: accepted connection from pid 855, user nix-grpc-daemon (trusted)4836client # uses-input-w1onlyb> node-a: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4837worker1 # [ 100.912680] nix-daemon[919]: reaped child process 1247, status = succeeded4838worker1 # [ 101.113232] nix-daemon[919]: accepted connection from pid 1262, user nix-grpc-daemon (trusted)4839worker1 # [ 101.135803] nix-daemon[919]: reaped child process 1265, status = succeeded4840lb # [ 101.259401] niks3-server[1132]: 2026/09/20 12:22:34 INFO Received uploads request method=POST path=/api/pending_closures4841worker1 # [ 101.165453] nix-grpc-daemon[928]: time=2026-09-20T12:22:34.194Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4842lb # [ 101.302748] niks3-server[1132]: 2026/09/20 12:22:34 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls4843lb # [ 101.315101] niks3-server[1132]: 2026/09/20 12:22:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign4844worker1 # [ 101.215768] nix-grpc-daemon[928]: time=2026-09-20T12:22:34.245Z level=INFO msg="Uploading 1 narinfos"4845lb # [ 101.329329] niks3-server[1132]: 2026/09/20 12:22:34 INFO Signed narinfos id=22 count=14846lb # [ 101.336199] niks3-server[1132]: 2026/09/20 12:22:34 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4847lb # [ 101.350389] niks3-server[1132]: 2026/09/20 12:22:34 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete4848lb # [ 101.359705] niks3-server[1132]: 2026/09/20 12:22:34 INFO Completed upload id=224849worker1 # [ 101.252743] nix-grpc-daemon[928]: time=2026-09-20T12:22:34.283Z level=INFO msg="Upload complete. (360ms)"4850lb # [ 101.363916] niks3-server[1132]: 2026/09/20 12:22:34 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo4851worker1 # [ 101.265592] nix-grpc-daemon[855]: ts=2026-09-20T12:22:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41652 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=10 outputs=14852worker1 # [ 101.288365] nix-daemon[919]: reaped child process 1245, status = succeeded4853client # copying 1 paths...4854worker2 # [ 100.745270] nix-grpc-daemon[862]: ts=2026-09-20T12:22:34Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:428324855worker2 # [ 100.762591] nix-grpc-daemon[862]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'...4856client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'...4857worker2 # [ 100.822186] nix-grpc-daemon[862]: ts=2026-09-20T12:22:34Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:428324858worker2 # [ 100.828128] nix-grpc-daemon[862]: ts=2026-09-20T12:22:34Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:42832 duration_s=0 paths=1 nar_bytes_out=1204859client # [ 101.838996] nix-daemon[1025]: reaped child process 1131, status = succeeded4860client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.56 seconds)4861client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt c >&24862client # [ 102.093323] nix-daemon[1025]: accepted connection from pid 1149, user root (trusted)4863client # this derivation will be built:4864client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4865worker2 # [ 101.420609] nix-grpc-daemon[862]: ts=2026-09-20T12:22:34Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:428324866client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key&system=aarch64-linux'...4867worker2 # [ 101.427350] nix-grpc-daemon[862]: ts=2026-09-20T12:22:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:428324868worker2 # [ 101.433637] nix-grpc-daemon[984]: time=2026-09-20T12:22:34.945Z level=INFO msg="All 1 paths already cached"4869worker2 # [ 101.446909] nix-grpc-daemon[862]: ts=2026-09-20T12:22:34Z level=debug event=want client=11 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=04870worker2 # [ 101.454373] nix-grpc-daemon[862]: ts=2026-09-20T12:22:34Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=14871worker2 # [ 101.460823] nix-grpc-daemon[862]: ts=2026-09-20T12:22:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:428324872worker2 # [ 101.466016] nix-grpc-daemon[862]: ts=2026-09-20T12:22:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:428324873worker2 # [ 101.475587] nix-grpc-daemon[984]: time=2026-09-20T12:22:34.988Z level=INFO msg="All 1 paths already cached"4874client # copying 1 paths to 'lb:50051'...4875client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'...4876worker2 # [ 101.486487] nix-grpc-daemon[862]: ts=2026-09-20T12:22:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:428324877worker2 # [ 101.501134] nix-daemon[936]: accepted connection from pid 862, user nix-grpc-daemon (trusted)4878worker2 # [ 101.706566] nix-daemon[936]: accepted connection from pid 1114, user nix-grpc-daemon (trusted)4879worker2 # [ 101.731036] nix-daemon[936]: reaped child process 1117, status = succeeded4880worker2 # [ 101.918359] nix-daemon[936]: accepted connection from pid 1119, user nix-grpc-daemon (trusted)4881worker2 # [ 101.938567] nix-daemon[936]: reaped child process 1122, status = succeeded4882lb # [ 102.627338] niks3-server[1132]: 2026/09/20 12:22:35 INFO Received uploads request method=POST path=/api/pending_closures4883worker2 # [ 102.015229] nix-grpc-daemon[984]: time=2026-09-20T12:22:35.527Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4884worker2 # [ 102.018119] nix-grpc-daemon[984]: time=2026-09-20T12:22:35.527Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)"4885lb # [ 102.712853] niks3-server[1132]: 2026/09/20 12:22:35 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst4886lb # [ 102.725729] niks3-server[1132]: 2026/09/20 12:22:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign4887worker2 # [ 102.057740] nix-grpc-daemon[984]: time=2026-09-20T12:22:35.570Z level=INFO msg="Uploading 1 narinfos"4888lb # [ 102.736927] niks3-server[1132]: 2026/09/20 12:22:35 INFO Signed narinfos id=23 count=14889lb # [ 102.745462] niks3-server[1132]: 2026/09/20 12:22:35 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls4890lb # [ 102.756443] niks3-server[1132]: 2026/09/20 12:22:35 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete4891lb # [ 102.766994] niks3-server[1132]: 2026/09/20 12:22:35 INFO Completed upload id=234892worker2 # [ 102.091983] nix-grpc-daemon[984]: time=2026-09-20T12:22:35.604Z level=INFO msg="Upload complete. (575ms)"4893lb # [ 102.770775] niks3-server[1132]: 2026/09/20 12:22:35 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo4894worker2 # [ 102.106341] nix-grpc-daemon[862]: ts=2026-09-20T12:22:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42832 duration_s=0 paths=1 nar_bytes_in=5684895worker2 # [ 102.121071] nix-grpc-daemon[862]: ts=2026-09-20T12:22:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:428324896worker2 # [ 102.130932] nix-daemon[936]: accepted connection from pid 862, user nix-grpc-daemon (trusted)4897worker2 # [ 102.138642] nix-daemon[936]: reaped child process 1112, status = succeeded4898client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051...4899worker2 # [ 102.154023] nix-daemon[936]: accepted connection from pid 862, user nix-grpc-daemon (trusted)4900client # uses-input-w1onlyc> worker2: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4901worker2 # [ 103.190351] nix-daemon[936]: reaped child process 1126, status = succeeded4902worker2 # [ 103.380278] nix-daemon[936]: accepted connection from pid 1141, user nix-grpc-daemon (trusted)4903worker2 # [ 103.404255] nix-daemon[936]: reaped child process 1144, status = succeeded4904lb # [ 104.094897] niks3-server[1132]: 2026/09/20 12:22:37 INFO Received uploads request method=POST path=/api/pending_closures4905worker2 # [ 103.435904] nix-grpc-daemon[984]: time=2026-09-20T12:22:36.946Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4906lb # [ 104.136382] niks3-server[1132]: 2026/09/20 12:22:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign4907worker2 # [ 103.473053] nix-grpc-daemon[984]: time=2026-09-20T12:22:36.985Z level=INFO msg="Uploading 1 narinfos"4908lb # [ 104.152711] niks3-server[1132]: 2026/09/20 12:22:37 INFO Signed narinfos id=24 count=14909lb # [ 104.166410] niks3-server[1132]: 2026/09/20 12:22:37 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete4910lb # [ 104.174933] niks3-server[1132]: 2026/09/20 12:22:37 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4911lb # [ 104.184299] niks3-server[1132]: 2026/09/20 12:22:37 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls4912lb # [ 104.189216] niks3-server[1132]: 2026/09/20 12:22:37 INFO Completed upload id=244913worker2 # [ 103.514092] nix-grpc-daemon[984]: time=2026-09-20T12:22:37.027Z level=INFO msg="Upload complete. (343ms)"4914lb # [ 104.191888] niks3-server[1132]: 2026/09/20 12:22:37 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo4915worker2 # [ 103.529156] nix-grpc-daemon[862]: ts=2026-09-20T12:22:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42832 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=11 outputs=14916worker2 # [ 103.545768] nix-daemon[936]: reaped child process 1124, status = succeeded4917client # copying 1 paths...4918worker2 # [ 103.572460] nix-grpc-daemon[862]: ts=2026-09-20T12:22:37Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:428324919client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'...4920worker2 # [ 103.604145] nix-grpc-daemon[862]: ts=2026-09-20T12:22:37Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:428324921worker2 # [ 103.611456] nix-grpc-daemon[862]: ts=2026-09-20T12:22:37Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:42832 duration_s=0 paths=1 nar_bytes_out=1204922client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.83 seconds)4923client # [ 104.682161] nix-daemon[1025]: reaped child process 1152, status = succeeded4924(finished: subtest: an input only one worker has still reaches the builder, in 11.21 seconds)4925subtest: upload whose reference the worker lacks is completed from the cache4926client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag viacache4927client # this derivation will be built:4928client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv4929client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'...4930client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag viacache, in 0.55 seconds)4931client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A referrer --argstr tag viacache4932client # this derivation will be built:4933client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv4934client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'...4935client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A referrer --argstr tag viacache, in 0.51 seconds)4936client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure4937lb # [ 105.544990] envoy[882]: conn peer=[2001:db8:1::1]:35238 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4222 tx=1909 ms=31104938client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.24 seconds)4939worker1: must succeed: nix-store --import < /tmp/shared/ref.closure4940worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.32 seconds)4941lb # [ 105.550712] envoy[882]: conn peer=[2001:db8:1::1]:35248 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=329 ms=364942lb # [ 105.551026] envoy[882]: conn peer=[2001:db8:1::1]:35250 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3698 tx=1917 ms=29954943lb # [ 105.551231] envoy[882]: conn peer=[2001:db8:1::1]:40386 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=714944lb # [ 105.551402] envoy[882]: conn peer=[2001:db8:1::1]:40392 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3712 tx=1803 ms=22524945lb # [ 105.551509] envoy[882]: conn peer=[2001:db8:1::1]:40406 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=734946worker2 # [ 105.519263] systemd[1]: Stopping Nix worker-protocol over gRPC...4947worker2 # [ 105.802403] nix-grpc-daemon[862]: ts=2026-09-20T12:22:39Z level=info event=signal_exit4948worker2 # [ 106.255835] nix-grpc-daemon[862]: ts=2026-09-20T12:22:39Z level=info event=scheduler_yield4949lb # [ 106.937977] niks3-server[1132]: 2026/09/20 12:22:39 INFO lead: released remote=192.168.1.4:409804950worker2 # [ 106.264761] nix-grpc-daemon[862]: ts=2026-09-20T12:22:39Z level=info event=scheduler_restarting addr=lb:500514951worker2 # [ 106.270912] nix-grpc-daemon[862]: ts=2026-09-20T12:22:39Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:428204952worker2 # [ 106.276592] nix-grpc-daemon[862]: ts=2026-09-20T12:22:39Z level=info event=worker_gone addr=192.168.1.4:500514953worker1 # [ 106.845106] nix-grpc-daemon[855]: ts=2026-09-20T12:22:39Z level=info event=scheduler_restarting addr=lb:500514954worker2 # [ 106.283881] nix-grpc-daemon[862]: ts=2026-09-20T12:22:39Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:428204955worker2 # [ 106.288742] nix-grpc-daemon[862]: ts=2026-09-20T12:22:39Z level=info event=worker_gone addr=192.168.1.3:500514956worker2 # [ 106.466443] nix-grpc-daemon[862]: ts=2026-09-20T12:22:39Z level=info event=scheduler_connected addr=lb:500514957worker2 # [ 106.472488] nix-grpc-daemon[862]: ts=2026-09-20T12:22:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4958worker1 # [ 107.045148] nix-grpc-daemon[855]: ts=2026-09-20T12:22:40Z level=info event=scheduler_connected addr=lb:500514959worker1 # [ 107.052134] nix-grpc-daemon[855]: ts=2026-09-20T12:22:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4960worker2 # [ 106.674952] nix-grpc-daemon[862]: ts=2026-09-20T12:22:40Z level=info event=scheduler_connected addr=lb:500514961worker2 # [ 106.682026] nix-grpc-daemon[862]: ts=2026-09-20T12:22:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4962worker1 # [ 107.253290] nix-grpc-daemon[855]: ts=2026-09-20T12:22:40Z level=info event=scheduler_connected addr=lb:500514963worker1 # [ 107.260369] nix-grpc-daemon[855]: ts=2026-09-20T12:22:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4964worker2 # [ 106.887200] nix-grpc-daemon[862]: ts=2026-09-20T12:22:40Z level=info event=scheduler_connected addr=lb:500514965worker2 # [ 106.892485] nix-grpc-daemon[862]: ts=2026-09-20T12:22:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4966worker1 # [ 107.461810] nix-grpc-daemon[855]: ts=2026-09-20T12:22:40Z level=info event=scheduler_connected addr=lb:500514967worker1 # [ 107.468381] nix-grpc-daemon[855]: ts=2026-09-20T12:22:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4968worker2 # [ 107.087867] nix-grpc-daemon[862]: ts=2026-09-20T12:22:40Z level=info event=scheduler_connected addr=lb:500514969worker2 # [ 107.094152] nix-grpc-daemon[862]: ts=2026-09-20T12:22:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4970worker1 # [ 107.664874] nix-grpc-daemon[855]: ts=2026-09-20T12:22:40Z level=info event=scheduler_connected addr=lb:500514971worker1 # [ 107.670747] nix-grpc-daemon[855]: ts=2026-09-20T12:22:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4972worker2 # [ 107.262897] nix-grpc-daemon[862]: ts=2026-09-20T12:22:40Z level=info event=scheduler_restarting4973worker1 # [ 107.874552] nix-grpc-daemon[855]: ts=2026-09-20T12:22:40Z level=info event=scheduler_connected addr=lb:500514974worker1 # [ 107.879093] nix-grpc-daemon[855]: ts=2026-09-20T12:22:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4975worker2 # [ 107.323836] nix-grpc-daemon[862]: ts=2026-09-20T12:22:40Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4976worker2 # [ 107.358508] nix-daemon[936]: reaped child process 938, status = succeeded4977lb # [ 108.130990] niks3-server[1132]: 2026/09/20 12:22:41 INFO lead: acquired remote=192.168.1.3:339764978worker1 # [ 108.024337] nix-grpc-daemon[855]: ts=2026-09-20T12:22:41Z level=info event=scheduler_take_over4979worker1 # [ 108.075065] nix-grpc-daemon[855]: ts=2026-09-20T12:22:41Z level=info event=scheduler_connected addr=lb:500514980worker1 # [ 108.080003] nix-grpc-daemon[855]: ts=2026-09-20T12:22:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4981worker1 # [ 108.281525] nix-grpc-daemon[855]: ts=2026-09-20T12:22:41Z level=info event=scheduler_connected addr=lb:500514982worker1 # [ 108.288147] nix-grpc-daemon[855]: ts=2026-09-20T12:22:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4983worker2 # [ 107.737918] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4984worker2 # [ 107.749450] systemd[1]: Stopped Nix worker-protocol over gRPC.4985worker2 # [ 107.751791] systemd[1]: nix-grpc-daemon.service: Consumed 2.452s CPU time over 1min 12.296s wall clock time, 52.4M memory peak, 66.5K incoming IP traffic, 189.7K outgoing IP traffic.4986worker2 # [ 107.766756] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.4987worker2 # [ 107.772663] systemd[1]: Closed Nix worker-protocol over gRPC.4988worker2 # [ 107.775081] systemd[1]: nix-grpc-daemon.socket: Consumed 952us CPU time over 1min 22.083s wall clock time, 256K memory peak, 80.7K incoming IP traffic, 69.3K outgoing IP traffic.4989lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy4990lb: must succeed: curl -sf localhost:9901/clusters4991lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.11 seconds)4992lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.11 seconds)4993client: must succeed: nix path-info --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&24994worker1 # [ 108.489030] nix-grpc-daemon[855]: ts=2026-09-20T12:22:41Z level=info event=scheduler_connected addr=lb:500514995worker1 # [ 108.495109] nix-grpc-daemon[855]: ts=2026-09-20T12:22:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4996worker1 # [ 108.696602] nix-grpc-daemon[855]: ts=2026-09-20T12:22:41Z level=info event=scheduler_connected addr=lb:500514997worker1 # [ 108.702612] nix-grpc-daemon[855]: ts=2026-09-20T12:22:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4998worker1 # [ 108.782077] nix-grpc-daemon[855]: ts=2026-09-20T12:22:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:416524999worker1 # [ 108.792705] nix-grpc-daemon[855]: ts=2026-09-20T12:22:41Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:416525000client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache5001client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.37 seconds)5002worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache5003worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.04 seconds)5004worker1 # [ 108.905538] nix-grpc-daemon[855]: ts=2026-09-20T12:22:41Z level=info event=scheduler_connected addr=lb:500515005worker1 # [ 108.909866] nix-grpc-daemon[855]: ts=2026-09-20T12:22:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5006worker2 # [ 108.377953] systemd[1]: Listening on Nix worker-protocol over gRPC.5007worker1 # [ 109.030092] systemd[1]: Stopping Nix worker-protocol over gRPC...5008worker1 # [ 109.106393] nix-grpc-daemon[855]: ts=2026-09-20T12:22:42Z level=info event=scheduler_connected addr=lb:500515009worker1 # [ 109.122362] nix-grpc-daemon[855]: ts=2026-09-20T12:22:42Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:530925010worker1 # [ 109.128552] nix-grpc-daemon[855]: ts=2026-09-20T12:22:42Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05011worker2 # [ 108.988458] systemd[1]: Starting scheduler-token.service...5012worker2 # [ 109.075344] systemd[1]: scheduler-token.service: Deactivated successfully.5013worker2 # [ 109.077897] systemd[1]: Finished scheduler-token.service.5014worker2 # [ 109.078760] systemd[1]: scheduler-token.service: Consumed 35ms CPU time over 89ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5015worker2 # [ 109.085100] systemd[1]: Starting Nix worker-protocol over gRPC...5016worker1 # [ 109.806681] nix-grpc-daemon[855]: ts=2026-09-20T12:22:42Z level=info event=signal_exit5017worker2 # [ 109.267867] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:42Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5018worker2 # [ 109.270694] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:42Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5019worker2 # [ 109.275134] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:42Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15020worker2 # [ 109.284064] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:42Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5021worker2 # [ 109.296305] systemd[1]: Started Nix worker-protocol over gRPC.5022worker2 # [ 109.313430] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:42Z level=info event=scheduler_connected addr=lb:500515023worker1 # [ 109.885102] nix-grpc-daemon[855]: ts=2026-09-20T12:22:42Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:530925024worker1 # [ 109.891278] nix-grpc-daemon[855]: ts=2026-09-20T12:22:42Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05025worker1 # [ 110.292402] nix-grpc-daemon[855]: ts=2026-09-20T12:22:43Z level=info event=scheduler_yield5026lb # [ 110.401302] niks3-server[1132]: 2026/09/20 12:22:43 INFO lead: released remote=192.168.1.3:339765027worker2 # [ 109.728732] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:43Z level=info event=scheduler_restarting addr=lb:500515028worker1 # [ 110.301939] nix-grpc-daemon[855]: ts=2026-09-20T12:22:43Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:530925029worker1 # [ 110.305274] nix-grpc-daemon[855]: ts=2026-09-20T12:22:43Z level=info event=worker_gone addr=192.168.1.4:500515030worker1 # [ 110.310537] nix-grpc-daemon[855]: ts=2026-09-20T12:22:43Z level=info event=scheduler_restarting addr=lb:500515031worker1 # [ 110.313289] nix-grpc-daemon[855]: ts=2026-09-20T12:22:43Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:530925032worker1 # [ 110.316095] nix-grpc-daemon[855]: ts=2026-09-20T12:22:43Z level=info event=worker_gone addr=192.168.1.3:500515033worker2 # [ 109.930311] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:43Z level=info event=scheduler_connected addr=lb:500515034worker1 # [ 110.510089] nix-grpc-daemon[855]: ts=2026-09-20T12:22:43Z level=info event=scheduler_connected addr=lb:500515035worker1 # [ 110.520721] nix-grpc-daemon[855]: ts=2026-09-20T12:22:43Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5036worker2 # [ 109.953227] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:43Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5037worker2 # [ 110.154627] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:43Z level=info event=scheduler_connected addr=lb:500515038worker1 # [ 110.722018] nix-grpc-daemon[855]: ts=2026-09-20T12:22:43Z level=info event=scheduler_connected addr=lb:500515039worker1 # [ 110.729944] nix-grpc-daemon[855]: ts=2026-09-20T12:22:43Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5040worker2 # [ 110.162360] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:43Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5041worker1 # [ 110.931117] nix-grpc-daemon[855]: ts=2026-09-20T12:22:43Z level=info event=scheduler_connected addr=lb:500515042worker2 # [ 110.365261] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:43Z level=info event=scheduler_connected addr=lb:500515043worker2 # [ 110.370649] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:43Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5044worker1 # [ 110.939142] nix-grpc-daemon[855]: ts=2026-09-20T12:22:43Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5045worker1 # [ 111.140330] nix-grpc-daemon[855]: ts=2026-09-20T12:22:44Z level=info event=scheduler_connected addr=lb:500515046worker2 # [ 110.572297] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:44Z level=info event=scheduler_connected addr=lb:500515047worker1 # [ 111.143482] nix-grpc-daemon[855]: ts=2026-09-20T12:22:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5048worker2 # [ 110.577999] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5049worker1 # [ 111.293886] nix-grpc-daemon[855]: ts=2026-09-20T12:22:44Z level=info event=scheduler_restarting5050worker1 # [ 111.328165] nix-daemon[919]: reaped child process 921, status = succeeded5051worker2 # [ 110.779269] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:44Z level=info event=scheduler_connected addr=lb:500515052worker1 # [ 111.355120] nix-grpc-daemon[855]: ts=2026-09-20T12:22:44Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5053worker2 # [ 110.784995] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5054worker2 # [ 110.986349] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:44Z level=info event=scheduler_connected addr=lb:500515055worker2 # [ 110.991965] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5056worker2 # [ 111.193435] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:44Z level=info event=scheduler_connected addr=lb:500515057worker2 # [ 111.198946] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5058worker2 # [ 111.402349] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:44Z level=info event=scheduler_connected addr=lb:500515059worker2 # [ 111.406857] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5060worker1 # [ 112.010014] nix-grpc-daemon[855]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5061worker1 # [ 112.010522] nix-grpc-daemon[855]: E0000 00:00:1789906965.040072 898 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 11, started true, shutdown true5062worker1 # [ 112.055103] nix-grpc-daemon[855]: E0000 00:00:1789906965.085358 898 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5063worker2 # [ 111.603324] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:45Z level=info event=scheduler_connected addr=lb:500515064worker2 # [ 111.609360] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5065worker1 # [ 112.200486] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5066worker1 # [ 112.208926] systemd[1]: Stopped Nix worker-protocol over gRPC.5067worker1 # [ 112.211692] systemd[1]: nix-grpc-daemon.service: Consumed 4.533s CPU time over 1min 16.780s wall clock time, 59.5M memory peak, 95.2K incoming IP traffic, 176.1K outgoing IP traffic.5068worker1 # [ 112.231608] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5069worker1 # [ 112.234721] systemd[1]: Closed Nix worker-protocol over gRPC.5070worker1 # [ 112.236511] systemd[1]: nix-grpc-daemon.socket: Consumed 908us CPU time over 1min 27.344s wall clock time, 256K memory peak, 96.1K incoming IP traffic, 77.4K outgoing IP traffic.5071lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy5072lb: must succeed: curl -sf localhost:9901/clusters5073lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5074lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.08 seconds)5075client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&25076worker2 # [ 111.810418] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:45Z level=info event=scheduler_connected addr=lb:500515077worker2 # [ 111.816157] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5078worker2 # [ 112.019019] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:529465079worker2 # [ 112.031167] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:45Z level=info event=scheduler_connected addr=lb:500515080worker2 # [ 112.042117] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5081worker2 # [ 112.057490] nix-daemon[936]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)5082worker2 # [ 112.074458] nix-grpc-daemon[1168]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 23 ms (attempt 1/5)5083worker2 # [ 112.097660] nix-grpc-daemon[1168]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 78 ms (attempt 2/5)5084worker2 # [ 112.183560] nix-grpc-daemon[1168]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 130 ms (attempt 3/5)5085worker2 # [ 112.227410] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:45Z level=info event=scheduler_connected addr=lb:500515086worker2 # [ 112.230423] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5087worker2 # [ 112.318211] nix-grpc-daemon[1168]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 153 ms (attempt 4/5)5088worker2 # [ 112.434119] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:45Z level=info event=scheduler_connected addr=lb:500515089worker2 # [ 112.439511] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5090worker2 # [ 112.480510] nix-grpc-daemon[1168]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5091client # copying 2 paths...5092client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'...5093worker2 # [ 112.509505] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:529465094client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'...5095worker2 # [ 112.525079] nix-daemon[936]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)5096worker2 # [ 112.562502] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:46Z level=info event=niks3_push_started pid=12245097worker2 # [ 112.644988] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:46Z level=info event=scheduler_connected addr=lb:500515098worker2 # [ 112.664831] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5099worker2 # [ 112.798915] nix-grpc-daemon[1224]: time=2026-09-20T12:22:46.311Z level=INFO msg="Configuring client TLS" cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/worker2.pem key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/worker2.key ca=""5100worker2 # [ 112.849430] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:46Z level=info event=scheduler_connected addr=lb:500515101worker2 # [ 112.852165] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5102worker2 # [ 112.988668] nix-daemon[936]: accepted connection from pid 1237, user nix-grpc-daemon (trusted)5103worker2 # [ 113.007153] nix-daemon[936]: reaped child process 1240, status = succeeded5104lb # [ 113.699782] niks3-server[1132]: 2026/09/20 12:22:46 INFO Received uploads request method=POST path=/api/pending_closures5105lb # [ 113.710751] niks3-server[1132]: 2026/09/20 12:22:46 INFO Received uploads request method=POST path=/api/pending_closures5106worker2 # [ 113.047219] nix-grpc-daemon[1224]: time=2026-09-20T12:22:46.558Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5107worker2 # [ 113.049919] nix-grpc-daemon[1224]: time=2026-09-20T12:22:46.558Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)"5108worker2 # [ 113.052712] nix-grpc-daemon[1224]: time=2026-09-20T12:22:46.558Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)"5109worker2 # [ 113.055981] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:46Z level=info event=scheduler_connected addr=lb:500515110worker2 # [ 113.057845] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5111lb # [ 113.769302] niks3-server[1132]: 2026/09/20 12:22:46 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst5112lb # [ 113.786278] niks3-server[1132]: 2026/09/20 12:22:46 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5113lb # [ 113.796498] niks3-server[1132]: 2026/09/20 12:22:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5114lb # [ 113.809234] niks3-server[1132]: 2026/09/20 12:22:46 INFO Signed narinfos id=25 count=15115worker2 # [ 113.135638] nix-grpc-daemon[1224]: time=2026-09-20T12:22:46.646Z level=INFO msg="Uploading 3 narinfos"5116lb # [ 113.818225] niks3-server[1132]: 2026/09/20 12:22:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5117lb # [ 113.832286] niks3-server[1132]: 2026/09/20 12:22:46 INFO Signed narinfos id=26 count=25118lb # [ 113.834547] niks3-server[1132]: 2026/09/20 12:22:46 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls5119lb # [ 113.865638] niks3-server[1132]: 2026/09/20 12:22:46 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5120lb # [ 113.876652] niks3-server[1132]: 2026/09/20 12:22:46 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls5121lb # [ 113.888949] niks3-server[1132]: 2026/09/20 12:22:46 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo5122lb # [ 113.896975] niks3-server[1132]: 2026/09/20 12:22:46 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5123lb # [ 113.907196] niks3-server[1132]: 2026/09/20 12:22:46 INFO Completed upload id=255124lb # [ 113.912389] niks3-server[1132]: 2026/09/20 12:22:46 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5125lb # [ 113.915961] niks3-server[1132]: 2026/09/20 12:22:46 INFO Completed upload id=265126worker2 # [ 113.239923] nix-grpc-daemon[1224]: time=2026-09-20T12:22:46.752Z level=INFO msg="Upload complete. (437ms)"5127lb # [ 113.922492] niks3-server[1132]: 2026/09/20 12:22:46 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5128worker2 # [ 113.252338] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52946 duration_s=0 paths=2 nar_bytes_in=3125129worker2 # [ 113.272761] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:46Z level=info event=scheduler_connected addr=lb:500515130worker2 # [ 113.276853] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5131worker2 # [ 113.282292] nix-daemon[936]: reaped child process 1222, status = succeeded5132client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.54 seconds)5133worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache5134worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.04 seconds)5135worker1 # [ 114.005259] systemd[1]: Listening on Nix worker-protocol over gRPC.5136worker1 # [ 114.011087] systemd[1]: Starting Nix worker-protocol over gRPC...5137worker2 # [ 113.468539] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:46Z level=info event=scheduler_connected addr=lb:500515138worker2 # [ 113.472920] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5139worker1 # [ 114.211434] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:47Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5140worker1 # [ 114.214284] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:47Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5141worker1 # [ 114.218991] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:47Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15142worker1 # [ 114.233260] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:47Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5143worker2 # [ 113.670339] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:47Z level=info event=scheduler_connected addr=lb:500515144worker1 # [ 114.242827] systemd[1]: Started Nix worker-protocol over gRPC.5145worker1 # [ 114.245489] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:47Z level=info event=scheduler_take_over5146worker2 # [ 113.676207] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:47Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5147lb: waiting until exactly no endpoints of aarch64-linux are unhealthy5148lb: must succeed: curl -sf localhost:9901/clusters5149lb # [ 114.353477] niks3-server[1132]: 2026/09/20 12:22:47 INFO lead: acquired remote=192.168.1.3:479145150worker1 # [ 114.275749] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:47Z level=info event=scheduler_connected addr=lb:500515151worker1 # [ 114.282028] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:47Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:466205152worker1 # [ 114.284168] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:47Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05153lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.12 seconds)5154lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.12 seconds)5155(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 10.00 seconds)5156subtest: scheduler down: the other node takes the lock and keeps it5157lb: must succeed: curl -sf localhost:9901/clusters5158lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5159lb: must succeed: curl -sf localhost:9901/clusters5160worker2 # [ 113.877712] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:47Z level=info event=scheduler_connected addr=lb:500515161worker1 # [ 114.452156] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:47Z level=info event=oidc_jwks provider=mock keys=15162worker1 # [ 114.454244] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:47Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:466205163worker1 # [ 114.457353] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:47Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05164lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5165lb: must succeed: curl -sf localhost:9901/clusters5166lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5167lb: must succeed: curl -sf localhost:9901/clusters5168lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5169worker1 # [ 114.642765] systemd[1]: Stopping Nix worker-protocol over gRPC...5170worker1 # [ 115.234432] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:48Z level=info event=signal_exit5171lb # [ 115.543347] envoy[882]: conn peer=[2001:db8:1::4]:58926 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=86213 tx=4171 ms=715225172lb # [ 115.548873] envoy[882]: conn peer=[2001:db8:1::1]:40422 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=523 ms=365173worker1 # [ 115.444223] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:48Z level=info event=scheduler_yield5174worker2 # [ 114.883196] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:48Z level=info event=scheduler_restarting addr=lb:500515175lb # [ 115.558265] envoy[882]: conn peer=192.168.1.3:51928 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=17785 tx=4835 ms=743945176worker1 # [ 115.451174] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:48Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:466205177worker1 # [ 115.456961] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:48Z level=info event=worker_gone addr=192.168.1.4:500515178worker1 # [ 115.463209] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:48Z level=info event=scheduler_restarting addr=lb:500515179lb # [ 115.570730] envoy[882]: conn peer=[2001:db8:1::1]:34340 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1007 tx=305 ms=12485180lb # [ 115.581093] niks3-server[1132]: 2026/09/20 12:22:48 INFO lead: released remote=192.168.1.3:479145181worker1 # [ 115.472918] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:48Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:466205182worker1 # [ 115.478008] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:48Z level=info event=worker_gone addr=192.168.1.3:500515183worker2 # [ 115.083813] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:48Z level=info event=scheduler_connected addr=lb:500515184worker2 # [ 115.089537] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5185worker1 # [ 115.664468] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:48Z level=info event=scheduler_connected addr=lb:500515186worker1 # [ 115.670285] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5187worker2 # [ 115.290761] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:48Z level=info event=scheduler_connected addr=lb:500515188worker2 # [ 115.296635] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5189worker1 # [ 115.871200] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:48Z level=info event=scheduler_connected addr=lb:500515190worker1 # [ 115.876986] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:48Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5191worker2 # [ 115.498014] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:49Z level=info event=scheduler_connected addr=lb:500515192worker2 # [ 115.504167] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5193worker1 # [ 116.078192] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:49Z level=info event=scheduler_connected addr=lb:500515194worker1 # [ 116.083971] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5195worker2 # [ 115.705555] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:49Z level=info event=scheduler_connected addr=lb:500515196worker2 # [ 115.711220] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5197worker1 # [ 116.286653] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:49Z level=info event=scheduler_connected addr=lb:500515198worker1 # [ 116.291170] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5199worker1 # [ 116.449595] nix-grpc-daemon[1288]: ts=2026-09-20T12:22:49Z level=info event=scheduler_restarting5200worker2 # [ 115.912567] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:49Z level=info event=scheduler_connected addr=lb:500515201worker2 # [ 115.915057] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5202worker2 # [ 116.117170] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:49Z level=info event=scheduler_connected addr=lb:500515203worker2 # [ 116.123763] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5204worker2 # [ 116.325413] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:49Z level=info event=scheduler_connected addr=lb:500515205worker2 # [ 116.331813] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:49Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5206worker2 # [ 116.532620] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:50Z level=info event=scheduler_connected addr=lb:500515207worker2 # [ 116.535916] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5208worker1 # [ 117.284265] nix-grpc-daemon[1288]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5209worker1 # [ 117.290258] nix-grpc-daemon[1288]: E0000 00:00:1789906970.314240 1331 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true5210worker1 # [ 117.297418] nix-grpc-daemon[1288]: E0000 00:00:1789906970.314518 1331 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5211worker2 # [ 116.738401] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:50Z level=info event=scheduler_connected addr=lb:500515212worker2 # [ 116.744156] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5213worker2 # [ 116.947129] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:50Z level=info event=scheduler_connected addr=lb:500515214worker2 # [ 116.951592] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5215worker2 # [ 117.148676] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:50Z level=info event=scheduler_connected addr=lb:500515216worker2 # [ 117.154372] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5217worker2 # [ 117.355456] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:50Z level=info event=scheduler_connected addr=lb:500515218worker2 # [ 117.361235] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:50Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5219worker2 # [ 117.562731] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:51Z level=info event=scheduler_connected addr=lb:500515220worker2 # [ 117.568496] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5221worker1 # [ 118.263423] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5222worker1 # [ 118.268972] systemd[1]: Stopped Nix worker-protocol over gRPC.5223worker1 # [ 118.271354] systemd[1]: nix-grpc-daemon.service: Consumed 174ms CPU time over 4.248s wall clock time, 10.8M memory peak, 9.2K incoming IP traffic, 7.9K outgoing IP traffic.5224worker1 # [ 118.288634] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5225worker1 # [ 118.290703] systemd[1]: Closed Nix worker-protocol over gRPC.5226worker1 # [ 118.294529] systemd[1]: nix-grpc-daemon.socket: Consumed 1ms CPU time over 4.302s wall clock time, 256K memory peak, 13.4K incoming IP traffic, 10.5K outgoing IP traffic.5227worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5228worker2 # [ 117.770060] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:51Z level=info event=scheduler_connected addr=lb:500515229worker2 # [ 117.777646] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5230worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.11 seconds)5231worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5232worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5233worker2 # [ 117.980709] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:51Z level=info event=scheduler_connected addr=lb:500515234worker2 # [ 117.985345] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5235worker2 # [ 118.186801] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:51Z level=info event=scheduler_connected addr=lb:500515236worker2 # [ 118.192430] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5237worker2 # [ 118.393775] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:51Z level=info event=scheduler_connected addr=lb:500515238worker2 # [ 118.399355] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:51Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5239worker2 # [ 118.600844] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:52Z level=info event=scheduler_connected addr=lb:500515240worker2 # [ 118.606558] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5241worker2 # [ 118.807847] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:52Z level=info event=scheduler_connected addr=lb:500515242worker2 # [ 118.813463] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5243worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5244worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5245worker2 # [ 119.017056] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:52Z level=info event=scheduler_connected addr=lb:500515246worker2 # [ 119.021435] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5247worker2 # [ 119.218703] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:52Z level=info event=scheduler_connected addr=lb:500515248worker2 # [ 119.224541] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:52Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5249worker2 # [ 119.298366] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:52Z level=info event=scheduler_take_over5250lb # [ 119.974377] niks3-server[1132]: 2026/09/20 12:22:52 INFO lead: acquired remote=192.168.1.4:472485251worker2 # [ 119.425781] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:52Z level=info event=scheduler_connected addr=lb:500515252worker2 # [ 119.456174] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:52Z level=info event=oidc_jwks provider=mock keys=15253worker2 # [ 119.461135] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:52Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:529545254worker2 # [ 119.466607] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:52Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05255worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5256worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5257client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby >&25258client # don't know how to build these paths:5259client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5260worker2 # [ 120.327947] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:529465261worker2 # [ 120.336834] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:529465262worker2 # [ 120.370409] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:53Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=05263worker2 # [ 120.378426] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:53Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.4:50051 clients=15264worker2 # [ 120.418412] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529465265worker2 # [ 120.436326] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:529465266client # copying 1 paths to 'lb:50051'...5267client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'...5268worker2 # [ 120.453336] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:529465269worker2 # [ 120.469149] nix-daemon[936]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)5270worker2 # [ 120.680129] nix-daemon[936]: accepted connection from pid 1264, user nix-grpc-daemon (trusted)5271worker2 # [ 120.703227] nix-daemon[936]: reaped child process 1267, status = succeeded5272worker2 # [ 120.889832] nix-daemon[936]: accepted connection from pid 1269, user nix-grpc-daemon (trusted)5273worker2 # [ 120.910196] nix-daemon[936]: reaped child process 1272, status = succeeded5274lb # [ 121.598224] niks3-server[1132]: 2026/09/20 12:22:54 INFO Received uploads request method=POST path=/api/pending_closures5275worker2 # [ 120.934808] nix-grpc-daemon[1224]: time=2026-09-20T12:22:54.447Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5276worker2 # [ 120.936739] nix-grpc-daemon[1224]: time=2026-09-20T12:22:54.447Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)"5277lb # [ 121.633244] niks3-server[1132]: 2026/09/20 12:22:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5278lb # [ 121.653125] niks3-server[1132]: 2026/09/20 12:22:54 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst5279lb # [ 121.667074] niks3-server[1132]: 2026/09/20 12:22:54 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls5280lb # [ 121.674663] niks3-server[1132]: 2026/09/20 12:22:54 INFO Signed narinfos id=27 count=15281worker2 # [ 121.000077] nix-grpc-daemon[1224]: time=2026-09-20T12:22:54.512Z level=INFO msg="Uploading 1 narinfos"5282lb # [ 121.684298] niks3-server[1132]: 2026/09/20 12:22:54 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5283lb # [ 121.697294] niks3-server[1132]: 2026/09/20 12:22:54 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo5284lb # [ 121.706963] niks3-server[1132]: 2026/09/20 12:22:54 INFO Completed upload id=275285worker2 # [ 121.031864] nix-grpc-daemon[1224]: time=2026-09-20T12:22:54.544Z level=INFO msg="Upload complete. (545ms)"5286worker2 # [ 121.044123] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52946 duration_s=0 paths=1 nar_bytes_in=5205287worker2 # [ 121.060834] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529465288worker2 # [ 121.074025] nix-daemon[936]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)5289worker2 # [ 121.080168] nix-daemon[936]: reaped child process 1262, status = succeeded5290client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051...5291client # farm-a> worker2: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5292worker2 # [ 121.096210] nix-daemon[936]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)5293client # farm-a> LOG-farm-a-on-standby5294worker2 # [ 122.343989] nix-daemon[936]: reaped child process 1276, status = succeeded5295worker2 # [ 122.544064] nix-daemon[936]: accepted connection from pid 1291, user nix-grpc-daemon (trusted)5296worker2 # [ 122.563758] nix-daemon[936]: reaped child process 1294, status = succeeded5297lb # [ 123.253812] niks3-server[1132]: 2026/09/20 12:22:56 INFO Received uploads request method=POST path=/api/pending_closures5298worker2 # [ 122.593160] nix-grpc-daemon[1224]: time=2026-09-20T12:22:56.102Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5299worker2 # [ 122.595180] nix-grpc-daemon[1224]: time=2026-09-20T12:22:56.102Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)"5300lb # [ 123.290561] niks3-server[1132]: 2026/09/20 12:22:56 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5301lb # [ 123.305649] niks3-server[1132]: 2026/09/20 12:22:56 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5302lb # [ 123.324229] niks3-server[1132]: 2026/09/20 12:22:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5303worker2 # [ 122.656137] nix-grpc-daemon[1224]: time=2026-09-20T12:22:56.168Z level=INFO msg="Uploading 1 narinfos"5304lb # [ 123.334236] niks3-server[1132]: 2026/09/20 12:22:56 INFO Signed narinfos id=28 count=15305lb # [ 123.339582] niks3-server[1132]: 2026/09/20 12:22:56 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls5306lb # [ 123.354925] niks3-server[1132]: 2026/09/20 12:22:56 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5307worker2 # [ 122.686066] nix-grpc-daemon[1224]: time=2026-09-20T12:22:56.199Z level=INFO msg="Upload complete. (376ms)"5308lb # [ 123.363497] niks3-server[1132]: 2026/09/20 12:22:56 INFO Completed upload id=285309lb # [ 123.367939] niks3-server[1132]: 2026/09/20 12:22:56 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo5310worker2 # [ 122.707911] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52946 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=15311worker2 # [ 122.724434] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:56Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=05312worker2 # [ 122.731356] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:56Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.4:50051 clients=15313worker2 # [ 122.736168] nix-daemon[936]: reaped child process 1274, status = succeeded5314worker2 # [ 122.754831] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529465315worker2 # [ 122.772089] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:529465316worker2 # [ 122.788597] nix-grpc-daemon[1224]: time=2026-09-20T12:22:56.301Z level=INFO msg="All 1 paths already cached"5317client # copying 1 paths to 'lb:50051'...5318client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'...5319worker2 # [ 122.801034] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:529465320worker2 # [ 122.816222] nix-daemon[936]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)5321worker2 # [ 123.034484] nix-daemon[936]: accepted connection from pid 1300, user nix-grpc-daemon (trusted)5322worker2 # [ 123.054128] nix-daemon[936]: reaped child process 1303, status = succeeded5323worker2 # [ 123.235197] nix-daemon[936]: accepted connection from pid 1305, user nix-grpc-daemon (trusted)5324worker2 # [ 123.254247] nix-daemon[936]: reaped child process 1308, status = succeeded5325lb # [ 123.942462] niks3-server[1132]: 2026/09/20 12:22:56 INFO Received uploads request method=POST path=/api/pending_closures5326worker2 # [ 123.273391] nix-grpc-daemon[1224]: time=2026-09-20T12:22:56.786Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5327worker2 # [ 123.274940] nix-grpc-daemon[1224]: time=2026-09-20T12:22:56.786Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)"5328lb # [ 123.963002] niks3-server[1132]: 2026/09/20 12:22:56 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst5329worker2 # [ 123.294424] nix-grpc-daemon[1224]: time=2026-09-20T12:22:56.807Z level=INFO msg="Uploading 1 narinfos"5330lb # [ 123.969483] niks3-server[1132]: 2026/09/20 12:22:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5331lb # [ 123.975394] niks3-server[1132]: 2026/09/20 12:22:56 INFO Signed narinfos id=29 count=15332lb # [ 123.979228] niks3-server[1132]: 2026/09/20 12:22:56 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls5333lb # [ 123.984701] niks3-server[1132]: 2026/09/20 12:22:56 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5334lb # [ 123.991629] niks3-server[1132]: 2026/09/20 12:22:56 INFO Completed upload id=295335worker2 # [ 123.316243] nix-grpc-daemon[1224]: time=2026-09-20T12:22:56.829Z level=INFO msg="Upload complete. (482ms)"5336lb # [ 123.994026] niks3-server[1132]: 2026/09/20 12:22:56 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo5337worker2 # [ 123.323748] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52946 duration_s=0 paths=1 nar_bytes_in=6405338worker2 # [ 123.327324] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529465339worker2 # [ 123.337609] nix-daemon[936]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)5340worker2 # [ 123.341616] nix-daemon[936]: reaped child process 1298, status = succeeded5341client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051...5342client # farm-b> worker2: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5343worker2 # [ 123.353739] nix-daemon[936]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)5344client # farm-b> LOG-farm-b-on-standby5345worker2 # [ 124.385488] nix-daemon[936]: reaped child process 1312, status = succeeded5346worker2 # [ 124.558108] nix-daemon[936]: accepted connection from pid 1327, user nix-grpc-daemon (trusted)5347worker2 # [ 124.580689] nix-daemon[936]: reaped child process 1330, status = succeeded5348lb # [ 125.269275] niks3-server[1132]: 2026/09/20 12:22:58 INFO Received uploads request method=POST path=/api/pending_closures5349worker2 # [ 124.606367] nix-grpc-daemon[1224]: time=2026-09-20T12:22:58.118Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5350worker2 # [ 124.608070] nix-grpc-daemon[1224]: time=2026-09-20T12:22:58.118Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)"5351lb # [ 125.304485] niks3-server[1132]: 2026/09/20 12:22:58 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5352lb # [ 125.317858] niks3-server[1132]: 2026/09/20 12:22:58 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst5353lb # [ 125.331084] niks3-server[1132]: 2026/09/20 12:22:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5354worker2 # [ 124.660726] nix-grpc-daemon[1224]: time=2026-09-20T12:22:58.173Z level=INFO msg="Uploading 1 narinfos"5355lb # [ 125.341431] niks3-server[1132]: 2026/09/20 12:22:58 INFO Signed narinfos id=30 count=15356lb # [ 125.346828] niks3-server[1132]: 2026/09/20 12:22:58 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls5357worker2 # [ 124.682561] nix-grpc-daemon[1224]: time=2026-09-20T12:22:58.195Z level=INFO msg="Upload complete. (319ms)"5358lb # [ 125.358143] niks3-server[1132]: 2026/09/20 12:22:58 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5359lb # [ 125.362167] niks3-server[1132]: 2026/09/20 12:22:58 INFO Completed upload id=305360lb # [ 125.364572] niks3-server[1132]: 2026/09/20 12:22:58 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo5361worker2 # [ 124.688705] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52946 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=15362worker2 # [ 124.704384] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:58Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=05363worker2 # [ 124.707367] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:58Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.4:50051 clients=15364worker2 # [ 124.710642] nix-daemon[936]: reaped child process 1310, status = succeeded5365worker2 # [ 124.750048] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529465366worker2 # [ 124.766590] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:529465367worker2 # [ 124.782779] nix-grpc-daemon[1224]: time=2026-09-20T12:22:58.295Z level=INFO msg="All 2 paths already cached"5368client # copying 1 paths to 'lb:50051'...5369client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'...5370worker2 # [ 124.794318] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:529465371worker2 # [ 124.809121] nix-daemon[936]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)5372lb # [ 125.542624] envoy[882]: conn peer=[2001:db8:1::3]:50054 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1028 tx=349 ms=22355373worker2 # [ 125.018309] nix-daemon[936]: accepted connection from pid 1336, user nix-grpc-daemon (trusted)5374worker2 # [ 125.040349] nix-daemon[936]: reaped child process 1339, status = succeeded5375worker2 # [ 125.223591] nix-daemon[936]: accepted connection from pid 1341, user nix-grpc-daemon (trusted)5376worker2 # [ 125.245521] nix-daemon[936]: reaped child process 1344, status = succeeded5377lb # [ 125.934286] niks3-server[1132]: 2026/09/20 12:22:58 INFO Received uploads request method=POST path=/api/pending_closures5378worker2 # [ 125.268290] nix-grpc-daemon[1224]: time=2026-09-20T12:22:58.781Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5379worker2 # [ 125.270441] nix-grpc-daemon[1224]: time=2026-09-20T12:22:58.783Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)"5380lb # [ 125.965102] niks3-server[1132]: 2026/09/20 12:22:58 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst5381worker2 # [ 125.305049] nix-grpc-daemon[1224]: time=2026-09-20T12:22:58.817Z level=INFO msg="Uploading 1 narinfos"5382lb # [ 125.981499] niks3-server[1132]: 2026/09/20 12:22:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5383lb # [ 125.991458] niks3-server[1132]: 2026/09/20 12:22:58 INFO Signed narinfos id=31 count=15384lb # [ 126.001069] niks3-server[1132]: 2026/09/20 12:22:58 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5385lb # [ 126.011995] niks3-server[1132]: 2026/09/20 12:22:58 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls5386lb # [ 126.017123] niks3-server[1132]: 2026/09/20 12:22:58 INFO Completed upload id=315387lb # [ 126.020207] niks3-server[1132]: 2026/09/20 12:22:58 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo5388worker2 # [ 125.341781] nix-grpc-daemon[1224]: time=2026-09-20T12:22:58.854Z level=INFO msg="Upload complete. (516ms)"5389worker2 # [ 125.352542] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52946 duration_s=0 paths=1 nar_bytes_in=7685390worker2 # [ 125.361257] nix-grpc-daemon[1168]: ts=2026-09-20T12:22:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529465391worker2 # [ 125.373547] nix-daemon[936]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)5392worker2 # [ 125.379474] nix-daemon[936]: reaped child process 1334, status = succeeded5393client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051...5394client # farm-top> worker2: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5395worker2 # [ 125.397384] nix-daemon[936]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)5396client # farm-top> LOG-farm-top-on-standby5397worker2 # [ 126.238592] nix-daemon[936]: reaped child process 1348, status = succeeded5398worker2 # [ 126.437695] nix-daemon[936]: accepted connection from pid 1363, user nix-grpc-daemon (trusted)5399worker2 # [ 126.460145] nix-daemon[936]: reaped child process 1366, status = succeeded5400lb # [ 127.150089] niks3-server[1132]: 2026/09/20 12:23:00 INFO Received uploads request method=POST path=/api/pending_closures5401worker2 # [ 126.487322] nix-grpc-daemon[1224]: time=2026-09-20T12:22:59.998Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5402worker2 # [ 126.489636] nix-grpc-daemon[1224]: time=2026-09-20T12:22:59.998Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)"5403lb # [ 127.190941] niks3-server[1132]: 2026/09/20 12:23:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5404worker2 # [ 126.533332] nix-grpc-daemon[1224]: time=2026-09-20T12:23:00.045Z level=INFO msg="Uploading 1 narinfos"5405lb # [ 127.209532] niks3-server[1132]: 2026/09/20 12:23:00 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst5406lb # [ 127.221879] niks3-server[1132]: 2026/09/20 12:23:00 INFO Signed narinfos id=32 count=15407lb # [ 127.226656] niks3-server[1132]: 2026/09/20 12:23:00 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls5408lb # [ 127.234588] niks3-server[1132]: 2026/09/20 12:23:00 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5409lb # [ 127.238174] niks3-server[1132]: 2026/09/20 12:23:00 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5410lb # [ 127.242145] niks3-server[1132]: 2026/09/20 12:23:00 INFO Completed upload id=325411worker2 # [ 126.566480] nix-grpc-daemon[1224]: time=2026-09-20T12:23:00.079Z level=INFO msg="Upload complete. (357ms)"5412lb # [ 127.244573] niks3-server[1132]: 2026/09/20 12:23:00 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo5413worker2 # [ 126.581407] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52946 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=15414worker2 # [ 126.600720] nix-daemon[936]: reaped child process 1346, status = succeeded5415client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby >&2, in 6.56 seconds)5416client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby outPath5417client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby outPath, in 0.22 seconds)5418worker1 # [ 127.493124] systemd[1]: Listening on Nix worker-protocol over gRPC.5419worker1 # [ 127.501492] systemd[1]: Starting Nix worker-protocol over gRPC...5420worker1 # [ 127.697812] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:00Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5421worker1 # [ 127.702098] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:00Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5422worker1 # [ 127.706504] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:00Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15423worker1 # [ 127.715522] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:00Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5424worker1 # [ 127.724632] systemd[1]: Started Nix worker-protocol over gRPC.5425lb: must succeed: curl -sf localhost:9901/clusters5426worker1 # [ 127.744918] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:00Z level=info event=scheduler_connected addr=lb:500515427worker2 # [ 127.179736] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:00Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:529545428worker2 # [ 127.185087] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:00Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05429lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds)5430worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5431worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5432lb: must succeed: curl -sf localhost:9901/clusters5433lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.11 seconds)5434lb: must succeed: curl -sf localhost:9901/clusters5435lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5436worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5437worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5438(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 13.80 seconds)5439subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5440client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs5441client # [ 128.553408] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs.5442client # Running as unit: rs.service; invocation ID: 43251dcbccfb4595856f072501f22ce55443client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs, in 0.10 seconds)5444client # [ 128.814955] nix[1253]: don't know how to build these paths:5445client # [ 128.816041] nix[1253]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5446worker2 # [ 127.933351] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:01Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:529465447worker2 # [ 127.949697] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:01Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=05448worker2 # [ 127.958067] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:01Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.4:50051 clients=15449worker2 # [ 127.998493] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529465450worker2 # [ 128.016477] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:529465451client # [ 128.912190] nix[1253]: copying 1 paths to 'lb:50051'...5452worker2 # [ 128.033584] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:529465453client # [ 128.916889] nix[1253]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'...5454worker2 # [ 128.051924] nix-daemon[936]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)5455worker2 # [ 128.274277] nix-daemon[936]: accepted connection from pid 1379, user nix-grpc-daemon (trusted)5456worker2 # [ 128.296930] nix-daemon[936]: reaped child process 1382, status = succeeded5457worker2 # [ 128.479242] nix-daemon[936]: accepted connection from pid 1384, user nix-grpc-daemon (trusted)5458worker2 # [ 128.501698] nix-daemon[936]: reaped child process 1387, status = succeeded5459lb # [ 129.191437] niks3-server[1132]: 2026/09/20 12:23:02 INFO Received uploads request method=POST path=/api/pending_closures5460worker2 # [ 128.529325] nix-grpc-daemon[1224]: time=2026-09-20T12:23:02.041Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5461worker2 # [ 128.530909] nix-grpc-daemon[1224]: time=2026-09-20T12:23:02.041Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)"5462lb # [ 129.227267] niks3-server[1132]: 2026/09/20 12:23:02 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst5463lb # [ 129.245992] niks3-server[1132]: 2026/09/20 12:23:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5464worker2 # [ 128.575604] nix-grpc-daemon[1224]: time=2026-09-20T12:23:02.088Z level=INFO msg="Uploading 1 narinfos"5465lb # [ 129.257701] niks3-server[1132]: 2026/09/20 12:23:02 INFO Signed narinfos id=33 count=15466lb # [ 129.263966] niks3-server[1132]: 2026/09/20 12:23:02 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls5467lb # [ 129.273456] niks3-server[1132]: 2026/09/20 12:23:02 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5468worker2 # [ 128.604299] nix-grpc-daemon[1224]: time=2026-09-20T12:23:02.117Z level=INFO msg="Upload complete. (533ms)"5469lb # [ 129.282375] niks3-server[1132]: 2026/09/20 12:23:02 INFO Completed upload id=335470lb # [ 129.287274] niks3-server[1132]: 2026/09/20 12:23:02 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo5471worker2 # [ 128.616992] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:52946 duration_s=0 paths=1 nar_bytes_in=4165472worker2 # [ 128.631183] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:529465473worker2 # [ 128.637805] nix-daemon[936]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)5474worker2 # [ 128.643075] nix-daemon[936]: reaped child process 1377, status = succeeded5475client # [ 129.535244] nix[1253]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051...5476worker2 # [ 128.658600] nix-daemon[936]: accepted connection from pid 1168, user nix-grpc-daemon (trusted)5477lb: must succeed: curl -sf localhost:9901/clusters5478lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5479lb: must succeed: curl -sf localhost:9901/clusters5480lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5481lb: must succeed: curl -sf localhost:9901/clusters5482lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5483lb: must succeed: curl -sf localhost:9901/clusters5484lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5485worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5486worker1 # [ 131.266185] systemd-journald[265]: Received client request to rotate journal, rotating.5487worker1 # [ 131.287861] systemd-journald[265]: Vacuuming done, freed 0B of archived journals from /var/log/journal/efbc8ee5e71d4dc6b275e1e5f21ad766.5488worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.10 seconds)5489worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service5490worker2 # [ 130.765402] systemd[1]: Stopping Nix worker-protocol over gRPC...5491worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.06 seconds)5492client: must succeed: sleep 5.05493worker2 # [ 131.299564] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:04Z level=info event=draining5494lb # [ 135.545329] envoy[882]: conn peer=[2001:db8:1::1]:56006 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9662 tx=3839 ms=62695495client: (finished: must succeed: sleep 5.0, in 5.06 seconds)5496worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5497worker2 # [ 136.313841] nix-daemon[936]: reaped child process 1391, status = succeeded5498worker2 # [ 136.492428] nix-daemon[936]: accepted connection from pid 1431, user nix-grpc-daemon (trusted)5499worker2 # [ 136.511460] nix-daemon[936]: reaped child process 1434, status = succeeded5500lb # [ 137.199971] niks3-server[1132]: 2026/09/20 12:23:10 INFO Received uploads request method=POST path=/api/pending_closures5501worker2 # [ 136.536474] nix-grpc-daemon[1224]: time=2026-09-20T12:23:10.047Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5502worker2 # [ 136.538265] nix-grpc-daemon[1224]: time=2026-09-20T12:23:10.047Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)"5503lb # [ 137.250538] niks3-server[1132]: 2026/09/20 12:23:10 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5504worker2 # [ 136.588680] nix-grpc-daemon[1224]: time=2026-09-20T12:23:10.101Z level=INFO msg="Uploading 1 narinfos"5505lb # [ 137.264440] niks3-server[1132]: 2026/09/20 12:23:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5506lb # [ 137.274220] niks3-server[1132]: 2026/09/20 12:23:10 INFO Signed narinfos id=34 count=15507lb # [ 137.278751] niks3-server[1132]: 2026/09/20 12:23:10 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5508lb # [ 137.287480] niks3-server[1132]: 2026/09/20 12:23:10 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls5509lb # [ 137.290702] niks3-server[1132]: 2026/09/20 12:23:10 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5510worker2 # [ 136.616065] nix-grpc-daemon[1224]: time=2026-09-20T12:23:10.129Z level=INFO msg="Upload complete. (339ms)"5511lb # [ 137.294770] niks3-server[1132]: 2026/09/20 12:23:10 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo5512lb # [ 137.299123] niks3-server[1132]: 2026/09/20 12:23:10 INFO Completed upload id=345513worker2 # [ 136.633076] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:52946 duration_s=8 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=15514worker2 # [ 136.648104] nix-daemon[936]: reaped child process 1389, status = succeeded5515client # [ 137.553756] systemd[1]: rs.service: Deactivated successfully.5516client # [ 137.558145] systemd[1]: rs.service: Consumed 174ms CPU time over 9.004s wall clock time, 37.2M memory peak, 8.5K incoming IP traffic, 9K outgoing IP traffic.5517worker2 # [ 137.301824] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:10Z level=info event=signal_exit5518worker2 # [ 137.527195] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:11Z level=info event=scheduler_yield5519worker1 # [ 138.099940] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:11Z level=info event=scheduler_restarting addr=lb:500515520worker2 # [ 137.533497] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:11Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:529545521worker2 # [ 137.538138] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:11Z level=info event=worker_gone addr=192.168.1.3:500515522lb # [ 138.207901] niks3-server[1132]: 2026/09/20 12:23:11 INFO lead: released remote=192.168.1.4:472485523worker2 # [ 137.543939] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:11Z level=info event=scheduler_restarting addr=lb:500515524worker2 # [ 137.548475] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:11Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:529545525worker2 # [ 137.552904] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:11Z level=info event=worker_gone addr=192.168.1.4:500515526worker1 # [ 138.299951] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:11Z level=info event=scheduler_connected addr=lb:500515527worker2 # [ 137.736401] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:11Z level=info event=scheduler_connected addr=lb:500515528worker1 # [ 138.305361] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5529worker2 # [ 137.740598] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5530worker2 # [ 137.937145] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:11Z level=info event=scheduler_connected addr=lb:500515531worker1 # [ 138.506127] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:11Z level=info event=scheduler_connected addr=lb:500515532worker2 # [ 137.942428] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5533worker1 # [ 138.511379] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5534worker2 # [ 138.143580] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:11Z level=info event=scheduler_connected addr=lb:500515535worker1 # [ 138.712327] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:11Z level=info event=scheduler_connected addr=lb:500515536worker2 # [ 138.149125] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5537worker1 # [ 138.717557] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5538worker2 # [ 138.350110] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:11Z level=info event=scheduler_connected addr=lb:500515539worker1 # [ 138.918436] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:11Z level=info event=scheduler_connected addr=lb:500515540worker2 # [ 138.355375] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5541worker1 # [ 138.924583] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5542worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.24 seconds)5543client: waiting for success: ! systemctl is-active rs5544client: (finished: waiting for success: ! systemctl is-active rs, in 0.07 seconds)5545client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5546client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds)5547client: must fail: journalctl -u rs | grep -q 'reconnecting'5548worker2 # [ 138.533495] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:12Z level=info event=scheduler_restarting5549worker1 # [ 139.126686] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:12Z level=info event=scheduler_connected addr=lb:500515550client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5551lb: must succeed: curl -sf localhost:9901/clusters5552worker1 # [ 139.131267] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5553worker2 # [ 138.577176] nix-grpc-daemon[1168]: ts=2026-09-20T12:23:12Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5554worker2 # [ 138.593727] nix-daemon[936]: reaped child process 1217, status = succeeded5555lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5556worker1 # [ 139.327880] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:12Z level=info event=scheduler_connected addr=lb:500515557worker1 # [ 139.333212] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5558worker1 # [ 139.534222] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:12Z level=info event=scheduler_connected addr=lb:500515559worker1 # [ 139.539778] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5560worker1 # [ 139.740661] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:12Z level=info event=scheduler_connected addr=lb:500515561worker1 # [ 139.746568] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5562worker1 # [ 139.947404] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:12Z level=info event=scheduler_connected addr=lb:500515563worker1 # [ 139.953409] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5564worker2 # [ 139.399760] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5565worker2 # [ 139.403055] systemd[1]: Stopped Nix worker-protocol over gRPC.5566worker2 # [ 139.409280] systemd[1]: nix-grpc-daemon.service: Consumed 2.132s CPU time over 30.311s wall clock time, 34.2M memory peak, 63.5K incoming IP traffic, 159.4K outgoing IP traffic.5567worker2 # [ 139.427268] systemd[1]: Starting scheduler-token.service...5568worker2 # [ 139.522522] systemd[1]: scheduler-token.service: Deactivated successfully.5569worker2 # [ 139.528458] systemd[1]: Finished scheduler-token.service.5570worker2 # [ 139.530547] systemd[1]: scheduler-token.service: Consumed 43ms CPU time over 98ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5571worker2 # [ 139.536134] systemd[1]: Starting Nix worker-protocol over gRPC...5572worker1 # [ 140.157173] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:13Z level=info event=scheduler_connected addr=lb:500515573worker1 # [ 140.161825] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5574lb: must succeed: curl -sf localhost:9901/clusters5575worker2 # [ 139.709577] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:13Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5576worker2 # [ 139.712278] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:13Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5577worker2 # [ 139.716394] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:13Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15578worker2 # [ 139.724076] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:13Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5579worker2 # [ 139.733105] systemd[1]: Started Nix worker-protocol over gRPC.5580lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5581worker2 # [ 139.742230] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:13Z level=info event=scheduler_take_over5582lb # [ 140.418434] niks3-server[1132]: 2026/09/20 12:23:13 INFO lead: acquired remote=192.168.1.4:332225583worker2 # [ 139.752146] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:13Z level=info event=scheduler_connected addr=lb:500515584worker2 # [ 139.754041] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5585worker1 # [ 140.357975] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:13Z level=info event=scheduler_connected addr=lb:500515586worker1 # [ 140.365107] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5587worker2 # [ 139.955704] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:13Z level=info event=scheduler_connected addr=lb:500515588worker2 # [ 139.961293] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5589worker1 # [ 140.565340] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:13Z level=info event=scheduler_connected addr=lb:500515590worker1 # [ 140.571682] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5591worker2 # [ 140.162475] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:13Z level=info event=scheduler_connected addr=lb:500515592worker2 # [ 140.168147] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5593worker1 # [ 140.772534] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:13Z level=info event=scheduler_connected addr=lb:500515594worker1 # [ 140.778011] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5595worker2 # [ 140.369361] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:13Z level=info event=scheduler_connected addr=lb:500515596worker2 # [ 140.374782] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5597worker1 # [ 140.979018] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:14Z level=info event=scheduler_connected addr=lb:500515598worker1 # [ 140.984790] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5599worker2 # [ 140.576133] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:14Z level=info event=scheduler_connected addr=lb:500515600worker2 # [ 140.581663] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5601worker1 # [ 141.187847] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:14Z level=info event=scheduler_connected addr=lb:500515602worker1 # [ 141.192303] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5603lb: must succeed: curl -sf localhost:9901/clusters5604worker2 # [ 140.784593] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:14Z level=info event=scheduler_connected addr=lb:500515605worker2 # [ 140.802608] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:14Z level=info event=oidc_jwks provider=mock keys=15606worker2 # [ 140.807227] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:14Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:404585607worker2 # [ 140.812738] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:14Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05608worker1 # [ 141.388694] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:14Z level=info event=scheduler_connected addr=lb:500515609lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5610worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5611worker2 # [ 140.824849] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:14Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:404585612worker2 # [ 140.833741] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:14Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05613worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)5614(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 13.31 seconds)5615subtest: scheduler restart mid-build: build finishes, no double build5616client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid5617client # [ 141.860697] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid.5618client # Running as unit: mid.service; invocation ID: 368924fb7b45438981c543a91b0178e85619client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid, in 0.10 seconds)5620worker2 # [ 141.236204] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:14Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:404745621worker2 # [ 141.253409] nix-daemon[936]: accepted connection from pid 1444, user nix-grpc-daemon (trusted)5622client # [ 142.152859] nix[1289]: don't know how to build these paths:5623client # [ 142.154532] nix[1289]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5624worker2 # [ 141.287822] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:14Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05625worker2 # [ 141.295351] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:14Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=15626worker2 # [ 141.342442] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:404745627worker2 # [ 141.361871] nix-grpc-daemon[1444]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 3 ms (attempt 1/5)5628worker2 # [ 141.373090] nix-grpc-daemon[1444]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 151 ms (attempt 2/5)5629worker2 # [ 141.532416] nix-grpc-daemon[1444]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 259 ms (attempt 3/5)5630worker2 # [ 141.800860] nix-grpc-daemon[1444]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 545 ms (attempt 4/5)5631worker2 # [ 142.354689] nix-grpc-daemon[1444]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5632worker2 # [ 142.373669] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:404745633worker2 # [ 142.383003] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:404745634client # [ 143.265226] nix[1289]: copying 1 paths to 'lb:50051'...5635client # [ 143.265593] nix[1289]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'...5636worker2 # [ 142.398919] nix-daemon[936]: accepted connection from pid 1444, user nix-grpc-daemon (trusted)5637worker2 # [ 142.417633] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:15Z level=info event=niks3_push_started pid=15135638worker2 # [ 142.626304] nix-grpc-daemon[1513]: time=2026-09-20T12:23:16.138Z level=INFO msg="Configuring client TLS" cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/worker2.pem key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/worker2.key ca=""5639worker2 # [ 142.818847] nix-daemon[936]: accepted connection from pid 1526, user nix-grpc-daemon (trusted)5640worker2 # [ 142.840780] nix-daemon[936]: reaped child process 1529, status = succeeded5641worker2 # [ 143.025988] nix-daemon[936]: accepted connection from pid 1531, user nix-grpc-daemon (trusted)5642worker2 # [ 143.045568] nix-daemon[936]: reaped child process 1534, status = succeeded5643lb # [ 143.733602] niks3-server[1132]: 2026/09/20 12:23:16 INFO Received uploads request method=POST path=/api/pending_closures5644worker2 # [ 143.288075] nix-grpc-daemon[1513]: time=2026-09-20T12:23:16.797Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5645worker2 # [ 143.294244] nix-grpc-daemon[1513]: time=2026-09-20T12:23:16.797Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)"5646lb # [ 144.005807] niks3-server[1132]: 2026/09/20 12:23:16 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst5647worker2 # [ 143.343304] nix-grpc-daemon[1513]: time=2026-09-20T12:23:16.856Z level=INFO msg="Uploading 1 narinfos"5648lb # [ 144.018583] niks3-server[1132]: 2026/09/20 12:23:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5649lb # [ 144.024330] niks3-server[1132]: 2026/09/20 12:23:16 INFO Signed narinfos id=35 count=15650lb # [ 144.049457] niks3-server[1132]: 2026/09/20 12:23:17 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls5651lb # [ 144.062411] niks3-server[1132]: 2026/09/20 12:23:17 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5652worker2 # [ 143.397836] nix-grpc-daemon[1513]: time=2026-09-20T12:23:16.910Z level=INFO msg="Upload complete. (767ms)"5653lb # [ 144.076123] niks3-server[1132]: 2026/09/20 12:23:17 INFO Completed upload id=355654lb # [ 144.082929] niks3-server[1132]: 2026/09/20 12:23:17 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo5655worker2 # [ 143.410769] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40474 duration_s=1 paths=1 nar_bytes_in=4245656worker2 # [ 143.432763] nix-grpc-daemon[1444]: ts=2026-09-20T12:23:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:404745657worker2 # [ 143.447597] nix-daemon[936]: accepted connection from pid 1444, user nix-grpc-daemon (trusted)5658worker2 # [ 143.451109] nix-daemon[936]: reaped child process 1511, status = succeeded5659worker2 # [ 143.464864] nix-daemon[936]: accepted connection from pid 1444, user nix-grpc-daemon (trusted)5660client # [ 144.345470] nix[1289]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5661lb: must succeed: curl -sf localhost:9901/clusters5662lb # [ 145.541674] envoy[882]: conn peer=[2001:db8:1::1]:56020 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2940 tx=1351 ms=87105663lb # [ 145.546891] envoy[882]: conn peer=[2001:db8:1::4]:51174 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=37013 tx=2156 ms=292555664lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5665lb: must succeed: curl -sf localhost:9901/clusters5666lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5667worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5668worker2 # [ 145.072483] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1444 (nix-grpc-daemon) on client request.5669worker2 # [ 145.074188] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5670lb # [ 145.759778] niks3-server[1132]: 2026/09/20 12:23:18 INFO lead: released remote=192.168.1.4:332225671worker2 # [ 145.102552] systemd[1]: Starting scheduler-token.service...5672client # [ 145.983744] nix[1289]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again5673worker1 # [ 145.670545] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:18Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"5674worker2 # [ 145.127415] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5675worker2 # [ 145.129963] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5676worker2 # [ 145.131233] systemd[1]: nix-grpc-daemon.service: Consumed 524ms CPU time over 5.588s wall clock time, 31.6M memory peak, 14.5K incoming IP traffic, 20.9K outgoing IP traffic.5677worker2 # [ 145.154028] nix-daemon[936]: reaped child process 1536, status = succeeded5678worker2 # [ 145.155511] nix-daemon[936]: reaped child process 1500, status = succeeded5679worker2 # [ 145.164646] nix-daemon[1538]: unexpected Nix daemon error: error: interrupted by the user5680worker2 # [ 145.170553] nix-daemon[936]: reaped child process 1538, status = failed with exit code 15681worker2 # [ 145.216653] systemd[1]: scheduler-token.service: Deactivated successfully.5682worker2 # [ 145.217821] systemd[1]: Finished scheduler-token.service.5683worker2 # [ 145.218832] systemd[1]: scheduler-token.service: Consumed 26ms CPU time over 110ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5684worker2 # [ 145.222578] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.5685worker2 # [ 145.224893] systemd[1]: Starting scheduler-token.service...5686worker2 # [ 145.298978] systemd[1]: scheduler-token.service: Deactivated successfully.5687worker2 # [ 145.303703] systemd[1]: Finished scheduler-token.service.5688worker2 # [ 145.305108] systemd[1]: scheduler-token.service: Consumed 25ms CPU time over 78ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5689worker2 # [ 145.308882] systemd[1]: Starting Nix worker-protocol over gRPC...5690worker1 # [ 145.879733] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:18Z level=info event=scheduler_connected addr=lb:500515691worker2 # [ 145.486936] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:18Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5692worker2 # [ 145.489572] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:19Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5693worker2 # [ 145.493790] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:19Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15694worker2 # [ 145.502226] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:19Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5695worker2 # [ 145.511410] systemd[1]: Started Nix worker-protocol over gRPC.5696worker2 # [ 145.521192] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:19Z level=info event=scheduler_take_over5697worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.50 seconds)5698lb # [ 146.195534] niks3-server[1132]: 2026/09/20 12:23:19 INFO lead: acquired remote=192.168.1.4:330305699client: must succeed: sleep 5.05700worker1 # [ 146.086741] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:19Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5701worker2 # [ 145.534909] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:19Z level=info event=scheduler_connected addr=lb:500515702worker2 # [ 145.541560] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:19Z level=info event=oidc_jwks provider=mock keys=15703worker2 # [ 145.542949] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:19Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:404845704worker2 # [ 145.544932] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:19Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05705worker2 # [ 145.615208] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:19Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05706worker2 # [ 145.619671] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:19Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=15707worker2 # [ 145.677437] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:404965708worker2 # [ 145.691897] nix-daemon[936]: accepted connection from pid 1580, user nix-grpc-daemon (trusted)5709worker2 # [ 145.713837] nix-daemon[936]: accepted connection from pid 1580, user nix-grpc-daemon (trusted)5710worker1 # [ 146.287925] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:19Z level=info event=scheduler_connected addr=lb:500515711worker2 # [ 145.722111] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:19Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:404845712worker2 # [ 145.725892] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:19Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05713worker2 # [ 145.737186] nix-daemon[936]: accepted connection from pid 1580, user nix-grpc-daemon (trusted)5714client # [ 145.990384] nix[1289]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5715client # [ 146.621246] nix[1289]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5716client: (finished: must succeed: sleep 5.0, in 5.06 seconds)5717client: waiting for success: ! systemctl is-active mid5718worker2 # [ 150.958695] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:24Z level=info event=niks3_push_started pid=16595719worker2 # [ 150.978897] nix-daemon[936]: reaped child process 1635, status = succeeded5720worker2 # [ 151.160318] nix-grpc-daemon[1659]: time=2026-09-20T12:23:24.672Z level=INFO msg="Configuring client TLS" cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/worker2.pem key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/worker2.key ca=""5721worker2 # [ 151.341387] nix-daemon[936]: accepted connection from pid 1672, user nix-grpc-daemon (trusted)5722worker2 # [ 151.361600] nix-daemon[936]: reaped child process 1675, status = succeeded5723lb # [ 152.050726] niks3-server[1132]: 2026/09/20 12:23:25 INFO Received uploads request method=POST path=/api/pending_closures5724worker2 # [ 151.389947] nix-grpc-daemon[1659]: time=2026-09-20T12:23:24.900Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5725worker2 # [ 151.393039] nix-grpc-daemon[1659]: time=2026-09-20T12:23:24.900Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)"5726lb # [ 152.102362] niks3-server[1132]: 2026/09/20 12:23:25 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst5727lb # [ 152.115415] niks3-server[1132]: 2026/09/20 12:23:25 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5728lb # [ 152.128546] niks3-server[1132]: 2026/09/20 12:23:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5729lb # [ 152.140160] niks3-server[1132]: 2026/09/20 12:23:25 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls5730lb # [ 152.150041] niks3-server[1132]: 2026/09/20 12:23:25 INFO Signed narinfos id=36 count=15731worker2 # [ 151.476126] nix-grpc-daemon[1659]: time=2026-09-20T12:23:24.988Z level=INFO msg="Uploading 1 narinfos"5732lb # [ 152.163420] niks3-server[1132]: 2026/09/20 12:23:25 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5733lb # [ 152.174168] niks3-server[1132]: 2026/09/20 12:23:25 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo5734lb # [ 152.183637] niks3-server[1132]: 2026/09/20 12:23:25 INFO Completed upload id=365735worker2 # [ 151.508597] nix-grpc-daemon[1659]: time=2026-09-20T12:23:25.021Z level=INFO msg="Upload complete. (344ms)"5736worker2 # [ 151.524697] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40496 duration_s=5 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=15737worker2 # [ 151.544140] nix-daemon[936]: reaped child process 1633, status = succeeded5738client # [ 152.447537] systemd[1]: mid.service: Deactivated successfully.5739client # [ 152.452119] systemd[1]: mid.service: Consumed 180ms CPU time over 10.591s wall clock time, 37M memory peak, 9.7K incoming IP traffic, 10.7K outgoing IP traffic.5740client: (finished: waiting for success: ! systemctl is-active mid, in 1.14 seconds)5741client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }5742client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.06 seconds)5743(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 11.20 seconds)5744subtest: interrupting the client stops the build on the worker5745client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr5746client # [ 153.026614] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr.5747client # Running as unit: intr.service; invocation ID: 129ee46e58a948dea05c480a0169f5d55748client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr, in 0.06 seconds)5749worker1 # [ 152.963814] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:25Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:417845750worker1 # [ 152.978634] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)5751client # [ 153.310884] nix[1325]: don't know how to build these paths:5752client # [ 153.313558] nix[1325]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5753worker2 # [ 152.445386] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:25Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=05754worker1 # [ 153.065938] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417845755worker1 # [ 153.085878] nix-grpc-daemon[1352]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 43 ms (attempt 1/5)5756worker1 # [ 153.136691] nix-grpc-daemon[1352]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 61 ms (attempt 2/5)5757worker1 # [ 153.205415] nix-grpc-daemon[1352]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 283 ms (attempt 3/5)5758worker1 # [ 153.488896] nix-grpc-daemon[1352]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 315 ms (attempt 4/5)5759worker2 # [ 152.451778] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:25Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.3:50051 clients=15760worker1 # [ 153.812513] nix-grpc-daemon[1352]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5761worker1 # [ 153.834962] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417845762client # [ 154.164184] nix[1325]: copying 1 paths to 'lb:50051'...5763worker1 # [ 153.851932] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417845764client # [ 154.164554] nix[1325]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'...5765worker1 # [ 153.869831] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)5766worker1 # [ 153.897061] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:26Z level=info event=niks3_push_started pid=14905767worker1 # [ 154.127258] nix-grpc-daemon[1490]: time=2026-09-20T12:23:27.157Z level=INFO msg="Configuring client TLS" cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/worker1.pem key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/worker1.key ca=""5768worker1 # [ 154.311589] nix-daemon[919]: accepted connection from pid 1507, user nix-grpc-daemon (trusted)5769worker1 # [ 154.329637] nix-daemon[919]: reaped child process 1510, status = succeeded5770worker1 # [ 154.510289] nix-daemon[919]: accepted connection from pid 1512, user nix-grpc-daemon (trusted)5771worker1 # [ 154.530571] nix-daemon[919]: reaped child process 1515, status = succeeded5772lb # [ 154.655034] niks3-server[1132]: 2026/09/20 12:23:27 INFO Received uploads request method=POST path=/api/pending_closures5773worker1 # [ 154.560095] nix-grpc-daemon[1490]: time=2026-09-20T12:23:27.589Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5774worker1 # [ 154.561632] nix-grpc-daemon[1490]: time=2026-09-20T12:23:27.589Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)"5775lb # [ 154.703637] niks3-server[1132]: 2026/09/20 12:23:27 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst5776worker1 # [ 154.605942] nix-grpc-daemon[1490]: time=2026-09-20T12:23:27.636Z level=INFO msg="Uploading 1 narinfos"5777lb # [ 154.713439] niks3-server[1132]: 2026/09/20 12:23:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5778lb # [ 154.721738] niks3-server[1132]: 2026/09/20 12:23:27 INFO Signed narinfos id=37 count=15779lb # [ 154.730260] niks3-server[1132]: 2026/09/20 12:23:27 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls5780lb # [ 154.742744] niks3-server[1132]: 2026/09/20 12:23:27 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5781lb # [ 154.748710] niks3-server[1132]: 2026/09/20 12:23:27 INFO Completed upload id=375782worker1 # [ 154.640903] nix-grpc-daemon[1490]: time=2026-09-20T12:23:27.671Z level=INFO msg="Upload complete. (508ms)"5783worker1 # [ 154.653224] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=0 paths=1 nar_bytes_in=4245784worker1 # [ 154.667986] nix-grpc-daemon[1490]: time=2026-09-20T12:23:27.676Z level=WARN msg="Failed to register uploaded object" key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo error="server returned 404: object is not pending upload\n"5785worker1 # [ 154.678634] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417845786worker1 # [ 154.683540] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)5787worker1 # [ 154.686134] nix-daemon[919]: reaped child process 1488, status = succeeded5788client # [ 155.010485] nix[1325]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051...5789worker1 # [ 154.701046] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)5790lb # [ 155.542958] envoy[882]: conn peer=[2001:db8:1::4]:57898 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=4906 tx=426 ms=53545791lb # [ 155.548348] envoy[882]: conn peer=[2001:db8:1::1]:46686 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3841 tx=1661 ms=103035792client: must succeed: systemctl kill -s INT intr5793client # [ 156.800446] systemd[1]: intr.service: Sent signal SIGINT to main process 1325 (nix) on client request.5794worker1 # [ 156.496676] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:29Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5795client # [ 156.810303] nix[1325]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5796client # [ 156.820917] nix[1325]: error: interrupted by the user5797client: (finished: must succeed: systemctl kill -s INT intr, in 0.09 seconds)5798worker1 # [ 156.536524] nix-daemon[919]: reaped child process 1517, status = succeeded5799worker1 # [ 156.541008] nix-daemon[1519]: unexpected Nix daemon error: error: interrupted by the user5800client # [ 156.855904] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE5801client # [ 156.859003] systemd[1]: intr.service: Failed with result 'exit-code'.5802worker1 # [ 156.547812] nix-daemon[919]: reaped child process 1519, status = failed with exit code 15803client # [ 156.863975] systemd[1]: intr.service: Consumed 157ms CPU time over 3.835s wall clock time, 39.2M memory peak, 7.8K incoming IP traffic, 8.8K outgoing IP traffic.5804(finished: subtest: interrupting the client stops the build on the worker, in 4.00 seconds)5805subtest: a deploy mid-build drains: the build finishes, the new generation takes over5806client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw5807client # [ 157.058027] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw.5808client # Running as unit: sw.service; invocation ID: ffeb8f4a3a7845a2841fff1d8fa16baf5809client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw, in 0.10 seconds)5810worker1 # [ 156.982696] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:30Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:417845811client # [ 157.304586] nix[1345]: don't know how to build these paths:5812client # [ 157.306176] nix[1345]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv5813worker2 # [ 156.437139] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:29Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=05814worker2 # [ 156.442698] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:29Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.4:50051 clients=15815worker2 # [ 156.481786] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:404965816worker2 # [ 156.499099] nix-grpc-daemon[1580]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 75 ms (attempt 1/5)5817worker2 # [ 156.582377] nix-grpc-daemon[1580]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 61 ms (attempt 2/5)5818worker2 # [ 156.651174] nix-grpc-daemon[1580]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 374 ms (attempt 3/5)5819worker2 # [ 157.033141] nix-grpc-daemon[1580]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 493 ms (attempt 4/5)5820worker2 # [ 157.535596] nix-grpc-daemon[1580]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5821worker2 # [ 157.558202] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:404965822client # [ 158.456739] nix[1345]: copying 1 paths to 'lb:50051'...5823worker2 # [ 157.574447] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:404965824client # [ 158.457043] nix[1345]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'...5825worker2 # [ 157.590046] nix-daemon[936]: accepted connection from pid 1580, user nix-grpc-daemon (trusted)5826worker2 # [ 157.796332] nix-daemon[936]: accepted connection from pid 1714, user nix-grpc-daemon (trusted)5827worker2 # [ 157.817152] nix-daemon[936]: reaped child process 1717, status = succeeded5828worker2 # [ 157.996352] nix-daemon[936]: accepted connection from pid 1719, user nix-grpc-daemon (trusted)5829worker2 # [ 158.017771] nix-daemon[936]: reaped child process 1722, status = succeeded5830lb # [ 158.705166] niks3-server[1132]: 2026/09/20 12:23:31 INFO Received uploads request method=POST path=/api/pending_closures5831worker2 # [ 158.041618] nix-grpc-daemon[1659]: time=2026-09-20T12:23:31.553Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5832worker2 # [ 158.043189] nix-grpc-daemon[1659]: time=2026-09-20T12:23:31.553Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)"5833lb # [ 158.734234] niks3-server[1132]: 2026/09/20 12:23:31 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst5834lb # [ 158.750981] niks3-server[1132]: 2026/09/20 12:23:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign5835worker2 # [ 158.082247] nix-grpc-daemon[1659]: time=2026-09-20T12:23:31.594Z level=INFO msg="Uploading 1 narinfos"5836lb # [ 158.764947] niks3-server[1132]: 2026/09/20 12:23:31 INFO Signed narinfos id=38 count=15837lb # [ 158.776598] niks3-server[1132]: 2026/09/20 12:23:31 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls5838worker2 # [ 158.107130] nix-grpc-daemon[1659]: time=2026-09-20T12:23:31.620Z level=INFO msg="Upload complete. (500ms)"5839lb # [ 158.783957] niks3-server[1132]: 2026/09/20 12:23:31 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete5840lb # [ 158.789386] niks3-server[1132]: 2026/09/20 12:23:31 INFO Completed upload id=385841worker2 # [ 158.114033] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40496 duration_s=0 paths=1 nar_bytes_in=4165842lb # [ 158.792829] niks3-server[1132]: 2026/09/20 12:23:31 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo5843worker2 # [ 158.125080] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:404965844worker2 # [ 158.136415] nix-daemon[936]: accepted connection from pid 1580, user nix-grpc-daemon (trusted)5845worker2 # [ 158.141720] nix-daemon[936]: reaped child process 1712, status = succeeded5846client # [ 159.032791] nix[1345]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051...5847worker2 # [ 158.155599] nix-daemon[936]: accepted connection from pid 1580, user nix-grpc-daemon (trusted)5848worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&25849worker2 # Checking switch inhibitors... done5850worker2 # [ 160.802177] nixos[1757]: switching to system configuration /nix/store/x3shrq1ffilhixgmbncqj99ry14cv69l-nixos-system-worker2-test5851worker2 # [ 160.805389] systemd[1]: Stopped target Remote File Systems.5852worker2 # activating the configuration...5853worker2 # [ 160.809499] systemd[1]: Stopped target Local File Systems.5854worker2 # setting up /etc...5855worker2 # [ 161.593079] systemd[1]: Reload requested from client PID 1757 ('.switch-to-conf') (unit backdoor.service)...5856worker2 # [ 161.596076] systemd[1]: Reloading...5857worker2 # [ 162.488393] systemd[1]: Reloading finished in 889 ms.5858worker2 # restarting sysinit-reactivation.target5859worker2 # [ 162.569501] systemd[1]: Starting Re-setup tmpfiles on a system that is already running....5860worker2 # [ 162.722846] systemd[1]: Finished Re-setup tmpfiles on a system that is already running..5861worker2 # [ 162.728413] systemd[1]: Reached target Reactivate sysinit units.5862worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service5863worker2 # [ 162.735313] systemd[1]: Reloading D-Bus System Message Bus...5864worker2 # [ 162.739373] dbus-broker-launch[574]: Caught SIGHUP, trigger reload.5865worker2 # [ 162.747396] systemd[1]: Reloading Nix worker-protocol over gRPC...5866worker2 # [ 162.768526] dbus-broker-launch[574]: Looking up NSS user entry for 'systemd-timesync'...5867worker2 # [ 162.771623] dbus-broker-launch[574]: NSS returned no entry for 'systemd-timesync'5868worker2 # [ 162.778019] dbus-broker-launch[574]: Invalid user-name in /nix/store/c8a4kacp5lnfwrcxrjmwczbl8ba32m4a-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"5869worker2 # [ 162.801406] systemd[1]: Reloaded Nix worker-protocol over gRPC.5870worker2 # [ 162.825874] systemd[1]: Reloaded D-Bus System Message Bus.5871worker2 # [ 162.828582] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os5872worker2 # [ 162.836731] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki5873worker2 # [ 162.838050] systemd[1]: Reached target Local File Systems.5874worker2 # [ 162.838825] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore5875worker2 # [ 162.852136] systemd[1]: Starting Create SUID/SGID Wrappers...5876worker2 # [ 162.853036] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met.5877worker2 # [ 162.854211] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc675878worker2 # [ 162.865400] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore5879worker2 # [ 162.872229] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os5880worker2 # [ 162.875549] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os5881worker2 # [ 162.880305] systemd[1]: Reached target Remote File Systems.5882worker2 # [ 163.356130] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully.5883worker2 # [ 163.362836] systemd[1]: Finished Create SUID/SGID Wrappers.5884worker2 # [ 163.365746] systemd[1]: Starting linger-users.service...5885worker2 # [ 163.421836] systemd[1]: linger-users.service: Deactivated successfully.5886worker2 # [ 163.422861] systemd[1]: Finished linger-users.service.5887worker2 # [ 163.518134] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:37Z level=info event=draining5888worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service5889worker2 # [ 163.684500] nixos[1757]: finished switching to system configuration /nix/store/x3shrq1ffilhixgmbncqj99ry14cv69l-nixos-system-worker2-test5890worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 3.71 seconds)5891worker2: must succeed: pgrep -f 'read -t [9]0 x'5892worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.06 seconds)5893client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain >&25894client # don't know how to build these paths:5895client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5896worker1 # [ 164.569290] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:37Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:417845897worker1 # [ 164.578102] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417845898worker2 # [ 164.032721] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:37Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=05899worker2 # [ 164.034902] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:37Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.3:50051 clients=15900worker1 # [ 164.648623] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417845901worker1 # [ 164.665077] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417845902client # copying 1 paths to 'lb:50051'...5903worker1 # [ 164.682531] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417845904client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'...5905worker1 # [ 164.707580] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)5906worker1 # [ 164.918900] nix-daemon[919]: accepted connection from pid 1569, user nix-grpc-daemon (trusted)5907worker1 # [ 164.939675] nix-daemon[919]: reaped child process 1572, status = succeeded5908worker1 # [ 165.117333] nix-daemon[919]: accepted connection from pid 1574, user nix-grpc-daemon (trusted)5909worker1 # [ 165.135769] nix-daemon[919]: reaped child process 1577, status = succeeded5910lb # [ 165.258201] niks3-server[1132]: 2026/09/20 12:23:38 INFO Received uploads request method=POST path=/api/pending_closures5911worker1 # [ 165.159577] nix-grpc-daemon[1490]: time=2026-09-20T12:23:38.189Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5912worker1 # [ 165.161237] nix-grpc-daemon[1490]: time=2026-09-20T12:23:38.189Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)"5913lb # [ 165.286195] niks3-server[1132]: 2026/09/20 12:23:38 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst5914worker1 # [ 165.193001] nix-grpc-daemon[1490]: time=2026-09-20T12:23:38.223Z level=INFO msg="Uploading 1 narinfos"5915lb # [ 165.303927] niks3-server[1132]: 2026/09/20 12:23:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign5916lb # [ 165.313701] niks3-server[1132]: 2026/09/20 12:23:38 INFO Signed narinfos id=39 count=15917worker1 # [ 165.216384] nix-grpc-daemon[1490]: time=2026-09-20T12:23:38.246Z level=INFO msg="Upload complete. (490ms)"5918lb # [ 165.324502] niks3-server[1132]: 2026/09/20 12:23:38 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls5919lb # [ 165.329936] niks3-server[1132]: 2026/09/20 12:23:38 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete5920lb # [ 165.335071] niks3-server[1132]: 2026/09/20 12:23:38 INFO Completed upload id=395921lb # [ 165.337622] niks3-server[1132]: 2026/09/20 12:23:38 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo5922worker1 # [ 165.224125] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=0 paths=1 nar_bytes_in=5285923worker1 # [ 165.237247] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417845924worker1 # [ 165.243389] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)5925worker1 # [ 165.248444] nix-daemon[919]: reaped child process 1567, status = succeeded5926client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051...5927worker1 # [ 165.260591] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)5928client # farm-a> node-a: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5929lb # [ 165.542268] envoy[882]: conn peer=[2001:db8:1::1]:40376 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2931 tx=755 ms=35725930client # farm-a> LOG-farm-a-during-drain5931worker1 # [ 166.109573] nix-daemon[919]: reaped child process 1581, status = succeeded5932worker1 # [ 166.285044] nix-daemon[919]: accepted connection from pid 1596, user nix-grpc-daemon (trusted)5933worker1 # [ 166.302741] nix-daemon[919]: reaped child process 1599, status = succeeded5934lb # [ 166.427535] niks3-server[1132]: 2026/09/20 12:23:39 INFO Received uploads request method=POST path=/api/pending_closures5935worker1 # [ 166.332542] nix-grpc-daemon[1490]: time=2026-09-20T12:23:39.359Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5936worker1 # [ 166.337106] nix-grpc-daemon[1490]: time=2026-09-20T12:23:39.359Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)"5937lb # [ 166.466176] niks3-server[1132]: 2026/09/20 12:23:39 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5938lb # [ 166.485027] niks3-server[1132]: 2026/09/20 12:23:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign5939worker1 # [ 166.389805] nix-grpc-daemon[1490]: time=2026-09-20T12:23:39.418Z level=INFO msg="Uploading 1 narinfos"5940lb # [ 166.499590] niks3-server[1132]: 2026/09/20 12:23:39 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst5941lb # [ 166.513627] niks3-server[1132]: 2026/09/20 12:23:39 INFO Signed narinfos id=40 count=15942worker1 # [ 166.414435] nix-grpc-daemon[1490]: time=2026-09-20T12:23:39.444Z level=INFO msg="Upload complete. (341ms)"5943lb # [ 166.522182] niks3-server[1132]: 2026/09/20 12:23:39 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls5944lb # [ 166.529537] niks3-server[1132]: 2026/09/20 12:23:39 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete5945lb # [ 166.534320] niks3-server[1132]: 2026/09/20 12:23:39 INFO Completed upload id=405946lb # [ 166.536649] niks3-server[1132]: 2026/09/20 12:23:39 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo5947worker1 # [ 166.425224] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=15948worker1 # [ 166.444209] nix-daemon[919]: reaped child process 1579, status = succeeded5949worker2 # [ 165.900738] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:39Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=05950worker2 # [ 165.904179] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:39Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.3:50051 clients=15951worker1 # [ 166.518729] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417845952worker1 # [ 166.534685] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417845953worker1 # [ 166.561976] nix-grpc-daemon[1490]: time=2026-09-20T12:23:39.592Z level=INFO msg="All 1 paths already cached"5954client # copying 1 paths to 'lb:50051'...5955worker1 # [ 166.571912] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417845956client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'...5957worker1 # [ 166.597104] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)5958worker1 # [ 166.806430] nix-daemon[919]: accepted connection from pid 1603, user nix-grpc-daemon (trusted)5959worker1 # [ 166.827351] nix-daemon[919]: reaped child process 1606, status = succeeded5960worker1 # [ 167.004260] nix-daemon[919]: accepted connection from pid 1608, user nix-grpc-daemon (trusted)5961worker1 # [ 167.022654] nix-daemon[919]: reaped child process 1611, status = succeeded5962lb # [ 167.142741] niks3-server[1132]: 2026/09/20 12:23:40 INFO Received uploads request method=POST path=/api/pending_closures5963worker1 # [ 167.044925] nix-grpc-daemon[1490]: time=2026-09-20T12:23:40.074Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5964worker1 # [ 167.046459] nix-grpc-daemon[1490]: time=2026-09-20T12:23:40.074Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)"5965lb # [ 167.171411] niks3-server[1132]: 2026/09/20 12:23:40 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst5966worker1 # [ 167.077714] nix-grpc-daemon[1490]: time=2026-09-20T12:23:40.107Z level=INFO msg="Uploading 1 narinfos"5967lb # [ 167.188352] niks3-server[1132]: 2026/09/20 12:23:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign5968lb # [ 167.199259] niks3-server[1132]: 2026/09/20 12:23:40 INFO Signed narinfos id=41 count=15969lb # [ 167.203526] niks3-server[1132]: 2026/09/20 12:23:40 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls5970lb # [ 167.212384] niks3-server[1132]: 2026/09/20 12:23:40 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete5971lb # [ 167.217630] niks3-server[1132]: 2026/09/20 12:23:40 INFO Completed upload id=415972worker1 # [ 167.109762] nix-grpc-daemon[1490]: time=2026-09-20T12:23:40.140Z level=INFO msg="Upload complete. (494ms)"5973lb # [ 167.220455] niks3-server[1132]: 2026/09/20 12:23:40 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo5974worker1 # [ 167.116271] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=0 paths=1 nar_bytes_in=6405975worker1 # [ 167.130059] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417845976worker1 # [ 167.138110] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)5977worker1 # [ 167.143095] nix-daemon[919]: reaped child process 1601, status = succeeded5978client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051...5979client # farm-b> node-a: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv5980worker1 # [ 167.156634] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)5981client # farm-b> LOG-farm-b-during-drain5982worker1 # [ 167.989754] nix-daemon[919]: reaped child process 1615, status = succeeded5983worker1 # [ 168.164768] nix-daemon[919]: accepted connection from pid 1630, user nix-grpc-daemon (trusted)5984worker1 # [ 168.186720] nix-daemon[919]: reaped child process 1633, status = succeeded5985lb # [ 168.306552] niks3-server[1132]: 2026/09/20 12:23:41 INFO Received uploads request method=POST path=/api/pending_closures5986worker1 # [ 168.210871] nix-grpc-daemon[1490]: time=2026-09-20T12:23:41.240Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5987worker1 # [ 168.212553] nix-grpc-daemon[1490]: time=2026-09-20T12:23:41.240Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)"5988lb # [ 168.344277] niks3-server[1132]: 2026/09/20 12:23:41 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv5989lb # [ 168.356980] niks3-server[1132]: 2026/09/20 12:23:41 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst5990lb # [ 168.368369] niks3-server[1132]: 2026/09/20 12:23:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign5991worker1 # [ 168.266366] nix-grpc-daemon[1490]: time=2026-09-20T12:23:41.296Z level=INFO msg="Uploading 1 narinfos"5992lb # [ 168.377131] niks3-server[1132]: 2026/09/20 12:23:41 INFO Signed narinfos id=42 count=15993lb # [ 168.379739] niks3-server[1132]: 2026/09/20 12:23:41 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls5994lb # [ 168.390967] niks3-server[1132]: 2026/09/20 12:23:41 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete5995lb # [ 168.398446] niks3-server[1132]: 2026/09/20 12:23:41 INFO Completed upload id=425996worker1 # [ 168.291073] nix-grpc-daemon[1490]: time=2026-09-20T12:23:41.321Z level=INFO msg="Upload complete. (324ms)"5997lb # [ 168.401840] niks3-server[1132]: 2026/09/20 12:23:41 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo5998worker2 # [ 167.738437] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:41Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=05999worker2 # [ 167.741946] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:41Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.3:50051 clients=16000worker1 # [ 168.301137] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=16001worker1 # [ 168.320908] nix-daemon[919]: reaped child process 1613, status = succeeded6002worker1 # [ 168.353192] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846003worker1 # [ 168.370095] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417846004worker1 # [ 168.386575] nix-grpc-daemon[1490]: time=2026-09-20T12:23:41.416Z level=INFO msg="All 2 paths already cached"6005client # copying 1 paths to 'lb:50051'...6006worker1 # [ 168.397899] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417846007client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'...6008worker1 # [ 168.421434] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6009worker1 # [ 168.632627] nix-daemon[919]: accepted connection from pid 1637, user nix-grpc-daemon (trusted)6010worker1 # [ 168.653805] nix-daemon[919]: reaped child process 1640, status = succeeded6011worker1 # [ 168.830034] nix-daemon[919]: accepted connection from pid 1642, user nix-grpc-daemon (trusted)6012worker1 # [ 168.849093] nix-daemon[919]: reaped child process 1645, status = succeeded6013lb # [ 168.967457] niks3-server[1132]: 2026/09/20 12:23:41 INFO Received uploads request method=POST path=/api/pending_closures6014worker1 # [ 168.869532] nix-grpc-daemon[1490]: time=2026-09-20T12:23:41.899Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6015worker1 # [ 168.871053] nix-grpc-daemon[1490]: time=2026-09-20T12:23:41.899Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)"6016lb # [ 168.997108] niks3-server[1132]: 2026/09/20 12:23:41 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst6017lb # [ 169.009496] niks3-server[1132]: 2026/09/20 12:23:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign6018worker1 # [ 168.909638] nix-grpc-daemon[1490]: time=2026-09-20T12:23:41.939Z level=INFO msg="Uploading 1 narinfos"6019lb # [ 169.017340] niks3-server[1132]: 2026/09/20 12:23:41 INFO Signed narinfos id=43 count=16020lb # [ 169.025544] niks3-server[1132]: 2026/09/20 12:23:41 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls6021lb # [ 169.035385] niks3-server[1132]: 2026/09/20 12:23:42 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete6022lb # [ 169.042628] niks3-server[1132]: 2026/09/20 12:23:42 INFO Completed upload id=436023worker1 # [ 168.935126] nix-grpc-daemon[1490]: time=2026-09-20T12:23:41.965Z level=INFO msg="Upload complete. (495ms)"6024lb # [ 169.045801] niks3-server[1132]: 2026/09/20 12:23:42 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo6025worker1 # [ 168.943201] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=0 paths=1 nar_bytes_in=7686026worker1 # [ 168.957378] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846027worker1 # [ 168.963941] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6028worker1 # [ 168.969636] nix-daemon[919]: reaped child process 1635, status = succeeded6029client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051...6030worker1 # [ 168.981337] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6031client # farm-top> node-a: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6032worker1 # [ 170.207873] nix-daemon[919]: reaped child process 1649, status = succeeded6033client # farm-top> LOG-farm-top-during-drain6034worker1 # [ 170.369940] nix-daemon[919]: accepted connection from pid 1664, user nix-grpc-daemon (trusted)6035worker1 # [ 170.388686] nix-daemon[919]: reaped child process 1667, status = succeeded6036lb # [ 170.510335] niks3-server[1132]: 2026/09/20 12:23:43 INFO Received uploads request method=POST path=/api/pending_closures6037worker1 # [ 170.412421] nix-grpc-daemon[1490]: time=2026-09-20T12:23:43.441Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6038worker1 # [ 170.413924] nix-grpc-daemon[1490]: time=2026-09-20T12:23:43.441Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)"6039lb # [ 170.546128] niks3-server[1132]: 2026/09/20 12:23:43 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6040lb # [ 170.560087] niks3-server[1132]: 2026/09/20 12:23:43 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst6041lb # [ 170.571263] niks3-server[1132]: 2026/09/20 12:23:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign6042worker1 # [ 170.469372] nix-grpc-daemon[1490]: time=2026-09-20T12:23:43.499Z level=INFO msg="Uploading 1 narinfos"6043lb # [ 170.579832] niks3-server[1132]: 2026/09/20 12:23:43 INFO Signed narinfos id=44 count=16044lb # [ 170.585688] niks3-server[1132]: 2026/09/20 12:23:43 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls6045lb # [ 170.594734] niks3-server[1132]: 2026/09/20 12:23:43 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete6046lb # [ 170.601410] niks3-server[1132]: 2026/09/20 12:23:43 INFO Completed upload id=446047worker1 # [ 170.493977] nix-grpc-daemon[1490]: time=2026-09-20T12:23:43.524Z level=INFO msg="Upload complete. (300ms)"6048lb # [ 170.604204] niks3-server[1132]: 2026/09/20 12:23:43 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo6049worker1 # [ 170.505148] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=16050worker1 # [ 170.522661] nix-daemon[919]: reaped child process 1647, status = succeeded6051client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain >&2, in 6.23 seconds)6052client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain outPath6053client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain outPath, in 0.24 seconds)6054worker2: must succeed: pkill -f 'read -t [9]0 x'6055worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.06 seconds)6056client: waiting for success: ! systemctl is-active sw6057worker2 # [ 170.408430] nix-daemon[936]: reaped child process 1726, status = succeeded6058worker2 # [ 170.583340] nix-daemon[936]: accepted connection from pid 1903, user nix-grpc-daemon (trusted)6059worker2 # [ 170.604140] nix-daemon[936]: reaped child process 1906, status = succeeded6060lb # [ 171.291120] niks3-server[1132]: 2026/09/20 12:23:44 INFO Received uploads request method=POST path=/api/pending_closures6061worker2 # [ 170.626163] nix-grpc-daemon[1659]: time=2026-09-20T12:23:44.138Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6062worker2 # [ 170.628558] nix-grpc-daemon[1659]: time=2026-09-20T12:23:44.140Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)"6063lb # [ 171.336504] niks3-server[1132]: 2026/09/20 12:23:44 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6064worker2 # [ 170.677399] nix-grpc-daemon[1659]: time=2026-09-20T12:23:44.188Z level=INFO msg="Uploading 1 narinfos"6065lb # [ 171.351579] niks3-server[1132]: 2026/09/20 12:23:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign6066lb # [ 171.362121] niks3-server[1132]: 2026/09/20 12:23:44 INFO Signed narinfos id=45 count=16067lb # [ 171.372477] niks3-server[1132]: 2026/09/20 12:23:44 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete6068worker2 # [ 170.704452] nix-grpc-daemon[1659]: time=2026-09-20T12:23:44.217Z level=INFO msg="Upload complete. (321ms)"6069lb # [ 171.387544] niks3-server[1132]: 2026/09/20 12:23:44 INFO Completed upload id=456070worker2 # [ 170.716788] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40496 duration_s=12 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=16071lb # [ 171.393744] niks3-server[1132]: 2026/09/20 12:23:44 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls6072lb # [ 171.404253] niks3-server[1132]: 2026/09/20 12:23:44 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv6073worker2 # [ 170.731080] nix-daemon[936]: reaped child process 1724, status = succeeded6074worker2 # [ 170.734763] nix-grpc-daemon[1659]: time=2026-09-20T12:23:44.247Z level=WARN msg="Failed to register uploaded object" key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo error="server returned 404: object is not pending upload\n"6075client # [ 171.641883] systemd[1]: sw.service: Deactivated successfully.6076client # [ 171.647218] systemd[1]: sw.service: Consumed 160ms CPU time over 14.584s wall clock time, 37M memory peak, 8.8K incoming IP traffic, 9K outgoing IP traffic.6077client: (finished: waiting for success: ! systemctl is-active sw, in 1.13 seconds)6078client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6079client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.06 seconds)6080worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6081worker2 # [ 171.520970] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:45Z level=info event=signal_exit6082worker2 # [ 171.562992] systemd[1]: Starting scheduler-token.service...6083worker2 # [ 171.650310] systemd[1]: scheduler-token.service: Deactivated successfully.6084worker2 # [ 171.653702] systemd[1]: Finished scheduler-token.service.6085worker2 # [ 171.654999] systemd[1]: scheduler-token.service: Consumed 32ms CPU time over 90ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6086worker2 # [ 171.745057] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:45Z level=info event=scheduler_yield6087lb # [ 172.429146] niks3-server[1132]: 2026/09/20 12:23:45 INFO lead: released remote=192.168.1.4:330306088worker1 # [ 172.323566] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:45Z level=info event=scheduler_restarting addr=lb:500516089worker2 # [ 171.755682] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:45Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:404846090worker2 # [ 171.762743] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:45Z level=info event=worker_gone addr=192.168.1.3:500516091worker2 # [ 171.767011] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:45Z level=info event=scheduler_restarting addr=lb:500516092worker2 # [ 171.771099] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:45Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:404846093worker2 # [ 171.775657] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:45Z level=info event=worker_gone addr=192.168.1.4:500516094worker1 # [ 172.529099] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:45Z level=info event=scheduler_connected addr=lb:500516095worker2 # [ 171.963740] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:45Z level=info event=scheduler_connected addr=lb:500516096worker1 # [ 172.534625] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6097worker2 # [ 171.969214] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6098lb # [ 172.834592] niks3-server[1132]: 2026/09/20 12:23:45 INFO lead: acquired remote=192.168.1.3:455906099worker1 # [ 172.731721] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:45Z level=info event=scheduler_take_over6100worker2 # [ 172.170083] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:45Z level=info event=scheduler_connected addr=lb:500516101worker1 # [ 172.736713] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:45Z level=info event=scheduler_connected addr=lb:500516102worker1 # [ 172.742575] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6103worker2 # [ 172.175326] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6104worker1 # [ 172.943320] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:45Z level=info event=scheduler_connected addr=lb:500516105worker2 # [ 172.376951] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:45Z level=info event=scheduler_connected addr=lb:500516106worker1 # [ 172.948859] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6107worker2 # [ 172.381934] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:45Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6108worker1 # [ 173.149757] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:46Z level=info event=scheduler_connected addr=lb:500516109worker2 # [ 172.582874] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:46Z level=info event=scheduler_connected addr=lb:500516110worker1 # [ 173.156088] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6111worker2 # [ 172.588369] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6112worker2 # [ 172.756396] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:46Z level=info event=scheduler_restarting6113worker2 # [ 172.782735] nix-daemon[936]: reaped child process 1631, status = succeeded6114worker2 # [ 172.786873] nix-grpc-daemon[1580]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6115worker1 # [ 173.358797] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:46Z level=info event=scheduler_connected addr=lb:500516116worker1 # [ 173.363293] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:46Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6117worker2 # [ 172.801560] nix-grpc-daemon[1580]: E0000 00:00:1789907026.299437 1623 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 19, started true, shutdown true6118worker2 # [ 172.817159] nix-grpc-daemon[1580]: E0000 00:00:1789907026.306736 1623 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true6119worker2 # [ 172.822967] nix-grpc-daemon[1580]: ts=2026-09-20T12:23:46Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6120worker1 # [ 173.559503] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:46Z level=info event=scheduler_connected addr=lb:500516121worker1 # [ 173.578104] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:46Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:443606122worker1 # [ 173.583658] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:46Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=06123worker2 # [ 173.554538] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6124worker2 # [ 173.559868] systemd[1]: nix-grpc-daemon.service: Consumed 865ms CPU time over 28.428s wall clock time, 32M memory peak, 26.3K incoming IP traffic, 36.5K outgoing IP traffic.6125worker2 # [ 173.583327] systemd[1]: Starting Nix worker-protocol over gRPC...6126worker2 # [ 173.782987] nix-grpc-daemon[1923]: ts=2026-09-20T12:23:47Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6127worker2 # [ 173.785607] nix-grpc-daemon[1923]: ts=2026-09-20T12:23:47Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6128worker2 # [ 173.789830] nix-grpc-daemon[1923]: ts=2026-09-20T12:23:47Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16129worker2 # [ 173.797999] nix-grpc-daemon[1923]: ts=2026-09-20T12:23:47Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6130worker2 # [ 173.799800] systemd[1]: Started Nix worker-protocol over gRPC.6131worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.34 seconds)6132lb: waiting until exactly no endpoints of aarch64-linux are unhealthy6133lb: must succeed: curl -sf localhost:9901/clusters6134worker2 # [ 173.822638] nix-grpc-daemon[1923]: ts=2026-09-20T12:23:47Z level=info event=scheduler_connected addr=lb:500516135worker1 # [ 174.404782] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:47Z level=info event=oidc_jwks provider=mock keys=16136worker1 # [ 174.405198] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:47Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:443606137worker1 # [ 174.405382] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:47Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=06138lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6139lb # [ 175.542549] envoy[882]: conn peer=[2001:db8:1::1]:32800 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9612 tx=3860 ms=59536140lb # [ 175.547564] envoy[882]: conn peer=[2001:db8:1::1]:40386 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2939 tx=1377 ms=143176141lb # [ 175.552504] envoy[882]: conn peer=[2001:db8:1::4]:54732 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=4556 tx=516 ms=272686142lb: must succeed: curl -sf localhost:9901/clusters6143lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6144lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.16 seconds)6145lb: must succeed: curl -sf localhost:9901/clusters6146lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6147worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6148worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6149(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 19.02 seconds)6150subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6151client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16152worker2 # [ 175.395748] nix-grpc-daemon[1923]: ts=2026-09-20T12:23:48Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:482066153worker2 # [ 175.411010] nix-daemon[936]: accepted connection from pid 1923, user nix-grpc-daemon (trusted)6154worker1 # [ 176.001076] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417846155worker1 # [ 176.046765] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:49Z level=debug event=want client=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=06156worker1 # [ 176.054089] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:49Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=16157worker1 # [ 176.060459] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:49Z level=debug event=want client=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=06158worker1 # [ 176.066760] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:49Z level=debug event=want client=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=06159worker1 # [ 176.092670] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846160worker1 # [ 176.108352] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417846161worker1 # [ 176.122886] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417846162worker1 # [ 176.138699] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6163worker1 # [ 176.343886] nix-daemon[919]: accepted connection from pid 1676, user nix-grpc-daemon (trusted)6164worker1 # [ 176.365979] nix-daemon[919]: reaped child process 1679, status = succeeded6165worker1 # [ 176.544964] nix-daemon[919]: accepted connection from pid 1681, user nix-grpc-daemon (trusted)6166worker1 # [ 176.563424] nix-daemon[919]: reaped child process 1684, status = succeeded6167lb # [ 176.683012] niks3-server[1132]: 2026/09/20 12:23:49 INFO Received uploads request method=POST path=/api/pending_closures6168worker1 # [ 176.587088] nix-grpc-daemon[1490]: time=2026-09-20T12:23:49.616Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6169worker1 # [ 176.588675] nix-grpc-daemon[1490]: time=2026-09-20T12:23:49.616Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)"6170lb # [ 176.719062] niks3-server[1132]: 2026/09/20 12:23:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6171worker1 # [ 176.622292] nix-grpc-daemon[1490]: time=2026-09-20T12:23:49.652Z level=INFO msg="Uploading 1 narinfos"6172lb # [ 176.736225] niks3-server[1132]: 2026/09/20 12:23:49 INFO Signed narinfos id=46 count=16173lb # [ 176.738976] niks3-server[1132]: 2026/09/20 12:23:49 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst6174worker1 # [ 176.647323] nix-grpc-daemon[1490]: time=2026-09-20T12:23:49.677Z level=INFO msg="Upload complete. (492ms)"6175lb # [ 176.755195] niks3-server[1132]: 2026/09/20 12:23:49 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls6176lb # [ 176.759581] niks3-server[1132]: 2026/09/20 12:23:49 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6177lb # [ 176.763806] niks3-server[1132]: 2026/09/20 12:23:49 INFO Completed upload id=466178worker1 # [ 176.657086] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=0 paths=1 nar_bytes_in=5126179lb # [ 176.765878] niks3-server[1132]: 2026/09/20 12:23:49 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo6180worker1 # [ 176.667541] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846181worker1 # [ 176.675596] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6182worker1 # [ 176.680829] nix-daemon[919]: reaped child process 1674, status = succeeded6183worker1 # [ 176.694391] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6184worker1 # [ 177.931351] nix-daemon[919]: reaped child process 1688, status = succeeded6185worker1 # [ 178.101746] nix-daemon[919]: accepted connection from pid 1703, user nix-grpc-daemon (trusted)6186worker1 # [ 178.121879] nix-daemon[919]: reaped child process 1706, status = succeeded6187lb # [ 178.241365] niks3-server[1132]: 2026/09/20 12:23:51 INFO Received uploads request method=POST path=/api/pending_closures6188worker1 # [ 178.144541] nix-grpc-daemon[1490]: time=2026-09-20T12:23:51.174Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6189worker1 # [ 178.146382] nix-grpc-daemon[1490]: time=2026-09-20T12:23:51.174Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)"6190lb # [ 178.283749] niks3-server[1132]: 2026/09/20 12:23:51 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv6191lb # [ 178.295980] niks3-server[1132]: 2026/09/20 12:23:51 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6192worker1 # [ 178.200692] nix-grpc-daemon[1490]: time=2026-09-20T12:23:51.230Z level=INFO msg="Uploading 1 narinfos"6193lb # [ 178.307784] niks3-server[1132]: 2026/09/20 12:23:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6194lb # [ 178.313962] niks3-server[1132]: 2026/09/20 12:23:51 INFO Signed narinfos id=47 count=16195lb # [ 178.318660] niks3-server[1132]: 2026/09/20 12:23:51 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls6196lb # [ 178.326990] niks3-server[1132]: 2026/09/20 12:23:51 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6197lb # [ 178.333592] niks3-server[1132]: 2026/09/20 12:23:51 INFO Completed upload id=476198worker1 # [ 178.226258] nix-grpc-daemon[1490]: time=2026-09-20T12:23:51.256Z level=INFO msg="Upload complete. (321ms)"6199lb # [ 178.336619] niks3-server[1132]: 2026/09/20 12:23:51 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo6200worker1 # [ 178.251225] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=1 outputs=16201worker1 # [ 178.266680] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:51Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=16202worker1 # [ 178.270810] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:51Z level=debug event=want client=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=06203worker1 # [ 178.274198] nix-daemon[919]: reaped child process 1686, status = succeeded6204worker1 # [ 178.297359] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846205worker1 # [ 178.313818] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417846206worker1 # [ 178.329611] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417846207worker1 # [ 178.344131] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6208worker1 # [ 178.546863] nix-daemon[919]: accepted connection from pid 1712, user nix-grpc-daemon (trusted)6209worker1 # [ 178.567705] nix-daemon[919]: reaped child process 1715, status = succeeded6210worker1 # [ 178.744339] nix-daemon[919]: accepted connection from pid 1717, user nix-grpc-daemon (trusted)6211worker1 # [ 178.763444] nix-daemon[919]: reaped child process 1720, status = succeeded6212lb # [ 178.881681] niks3-server[1132]: 2026/09/20 12:23:51 INFO Received uploads request method=POST path=/api/pending_closures6213worker1 # [ 178.780116] nix-grpc-daemon[1490]: time=2026-09-20T12:23:51.810Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6214worker1 # [ 178.781647] nix-grpc-daemon[1490]: time=2026-09-20T12:23:51.810Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)"6215lb # [ 178.911152] niks3-server[1132]: 2026/09/20 12:23:51 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst6216worker1 # [ 178.816580] nix-grpc-daemon[1490]: time=2026-09-20T12:23:51.846Z level=INFO msg="Uploading 1 narinfos"6217lb # [ 178.928983] niks3-server[1132]: 2026/09/20 12:23:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6218lb # [ 178.938896] niks3-server[1132]: 2026/09/20 12:23:51 INFO Signed narinfos id=48 count=16219lb # [ 178.944120] niks3-server[1132]: 2026/09/20 12:23:51 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls6220lb # [ 178.953718] niks3-server[1132]: 2026/09/20 12:23:51 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6221worker1 # [ 178.845796] nix-grpc-daemon[1490]: time=2026-09-20T12:23:51.876Z level=INFO msg="Upload complete. (485ms)"6222lb # [ 178.958590] niks3-server[1132]: 2026/09/20 12:23:51 INFO Completed upload id=486223worker1 # [ 178.852401] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=0 paths=1 nar_bytes_in=5126224lb # [ 178.961868] niks3-server[1132]: 2026/09/20 12:23:51 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo6225worker1 # [ 178.863548] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846226worker1 # [ 178.874834] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6227worker1 # [ 178.883371] nix-daemon[919]: reaped child process 1710, status = succeeded6228worker1 # [ 178.889416] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6229worker1 # [ 179.911421] nix-daemon[919]: reaped child process 1724, status = succeeded6230worker1 # [ 180.073772] nix-daemon[919]: accepted connection from pid 1739, user nix-grpc-daemon (trusted)6231worker1 # [ 180.094543] nix-daemon[919]: reaped child process 1742, status = succeeded6232lb # [ 180.215648] niks3-server[1132]: 2026/09/20 12:23:53 INFO Received uploads request method=POST path=/api/pending_closures6233worker1 # [ 180.405026] nix-grpc-daemon[1490]: time=2026-09-20T12:23:53.432Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6234worker1 # [ 180.418980] nix-grpc-daemon[1490]: time=2026-09-20T12:23:53.432Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)"6235lb # [ 180.562961] niks3-server[1132]: 2026/09/20 12:23:53 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6236lb # [ 180.581812] niks3-server[1132]: 2026/09/20 12:23:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6237lb # [ 180.595334] niks3-server[1132]: 2026/09/20 12:23:53 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv6238lb # [ 180.602813] niks3-server[1132]: 2026/09/20 12:23:53 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls6239worker1 # [ 180.495650] nix-grpc-daemon[1490]: time=2026-09-20T12:23:53.525Z level=INFO msg="Uploading 1 narinfos"6240lb # [ 180.608279] niks3-server[1132]: 2026/09/20 12:23:53 INFO Signed narinfos id=49 count=16241lb # [ 180.619122] niks3-server[1132]: 2026/09/20 12:23:53 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6242lb # [ 180.632658] niks3-server[1132]: 2026/09/20 12:23:53 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo6243lb # [ 180.646596] niks3-server[1132]: 2026/09/20 12:23:53 INFO Completed upload id=496244worker1 # [ 180.539320] nix-grpc-daemon[1490]: time=2026-09-20T12:23:53.569Z level=INFO msg="Upload complete. (641ms)"6245worker1 # [ 180.557328] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=2 outputs=16246worker1 # [ 180.579147] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:53Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=16247worker1 # [ 180.592352] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846248worker1 # [ 180.599367] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:53Z level=debug event=want client=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=06249worker1 # [ 180.606039] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417846250worker1 # [ 180.617357] nix-daemon[919]: reaped child process 1722, status = succeeded6251worker1 # [ 180.624263] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6252worker1 # [ 180.626943] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417846253worker1 # [ 180.819828] nix-daemon[919]: accepted connection from pid 1748, user nix-grpc-daemon (trusted)6254worker1 # [ 180.840386] nix-daemon[919]: reaped child process 1751, status = succeeded6255worker1 # [ 181.018588] nix-daemon[919]: accepted connection from pid 1753, user nix-grpc-daemon (trusted)6256worker1 # [ 181.038533] nix-daemon[919]: reaped child process 1756, status = succeeded6257lb # [ 181.157151] niks3-server[1132]: 2026/09/20 12:23:54 INFO Received uploads request method=POST path=/api/pending_closures6258worker1 # [ 181.062072] nix-grpc-daemon[1490]: time=2026-09-20T12:23:54.092Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6259worker1 # [ 181.063670] nix-grpc-daemon[1490]: time=2026-09-20T12:23:54.092Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)"6260lb # [ 181.194443] niks3-server[1132]: 2026/09/20 12:23:54 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst6261lb # [ 181.207360] niks3-server[1132]: 2026/09/20 12:23:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6262worker1 # [ 181.106959] nix-grpc-daemon[1490]: time=2026-09-20T12:23:54.137Z level=INFO msg="Uploading 1 narinfos"6263lb # [ 181.214454] niks3-server[1132]: 2026/09/20 12:23:54 INFO Signed narinfos id=50 count=16264lb # [ 181.221185] niks3-server[1132]: 2026/09/20 12:23:54 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls6265lb # [ 181.230732] niks3-server[1132]: 2026/09/20 12:23:54 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6266lb # [ 181.241164] niks3-server[1132]: 2026/09/20 12:23:54 INFO Completed upload id=506267lb # [ 181.244616] niks3-server[1132]: 2026/09/20 12:23:54 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo6268worker1 # [ 181.133830] nix-grpc-daemon[1490]: time=2026-09-20T12:23:54.164Z level=INFO msg="Upload complete. (498ms)"6269worker1 # [ 181.144668] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=0 paths=1 nar_bytes_in=5126270worker1 # [ 181.160705] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846271worker1 # [ 181.168940] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6272worker1 # [ 181.174814] nix-daemon[919]: reaped child process 1746, status = succeeded6273worker1 # [ 181.190660] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6274worker1 # [ 182.037760] nix-daemon[919]: reaped child process 1760, status = succeeded6275worker1 # [ 182.226167] nix-daemon[919]: accepted connection from pid 1775, user nix-grpc-daemon (trusted)6276worker1 # [ 182.246000] nix-daemon[919]: reaped child process 1778, status = succeeded6277lb # [ 182.365637] niks3-server[1132]: 2026/09/20 12:23:55 INFO Received uploads request method=POST path=/api/pending_closures6278worker1 # [ 182.274640] nix-grpc-daemon[1490]: time=2026-09-20T12:23:55.302Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6279worker1 # [ 182.277053] nix-grpc-daemon[1490]: time=2026-09-20T12:23:55.303Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)"6280lb # [ 182.405494] niks3-server[1132]: 2026/09/20 12:23:55 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv6281lb # [ 182.432170] niks3-server[1132]: 2026/09/20 12:23:55 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6282lb # [ 182.444594] niks3-server[1132]: 2026/09/20 12:23:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6283worker1 # [ 182.335998] nix-grpc-daemon[1490]: time=2026-09-20T12:23:55.365Z level=INFO msg="Uploading 1 narinfos"6284lb # [ 182.447555] niks3-server[1132]: 2026/09/20 12:23:55 INFO Signed narinfos id=51 count=16285lb # [ 182.456663] niks3-server[1132]: 2026/09/20 12:23:55 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls6286lb # [ 182.462388] niks3-server[1132]: 2026/09/20 12:23:55 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6287lb # [ 182.467501] niks3-server[1132]: 2026/09/20 12:23:55 INFO Completed upload id=516288worker1 # [ 182.359713] nix-grpc-daemon[1490]: time=2026-09-20T12:23:55.390Z level=INFO msg="Upload complete. (355ms)"6289lb # [ 182.469588] niks3-server[1132]: 2026/09/20 12:23:55 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo6290worker1 # [ 182.379866] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=3 outputs=16291worker1 # [ 182.397786] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:55Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=16292worker1 # [ 182.408215] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846293worker1 # [ 182.411752] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:55Z level=debug event=want client=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=06294worker1 # [ 182.417729] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417846295worker1 # [ 182.422467] nix-daemon[919]: reaped child process 1758, status = succeeded6296worker1 # [ 182.424654] nix-grpc-daemon[1490]: time=2026-09-20T12:23:55.452Z level=INFO msg="All 1 paths already cached"6297worker1 # [ 182.429345] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417846298worker1 # [ 182.440336] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6299worker1 # [ 182.636145] nix-daemon[919]: accepted connection from pid 1784, user nix-grpc-daemon (trusted)6300worker1 # [ 182.657109] nix-daemon[919]: reaped child process 1787, status = succeeded6301worker1 # [ 182.837401] nix-daemon[919]: accepted connection from pid 1789, user nix-grpc-daemon (trusted)6302worker1 # [ 182.858330] nix-daemon[919]: reaped child process 1792, status = succeeded6303lb # [ 182.979387] niks3-server[1132]: 2026/09/20 12:23:55 INFO Received uploads request method=POST path=/api/pending_closures6304worker1 # [ 182.885024] nix-grpc-daemon[1490]: time=2026-09-20T12:23:55.915Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6305worker1 # [ 182.886574] nix-grpc-daemon[1490]: time=2026-09-20T12:23:55.915Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)"6306lb # [ 183.016423] niks3-server[1132]: 2026/09/20 12:23:55 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst6307lb # [ 183.031663] niks3-server[1132]: 2026/09/20 12:23:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6308worker1 # [ 182.932630] nix-grpc-daemon[1490]: time=2026-09-20T12:23:55.962Z level=INFO msg="Uploading 1 narinfos"6309lb # [ 183.043713] niks3-server[1132]: 2026/09/20 12:23:56 INFO Signed narinfos id=52 count=16310lb # [ 183.050940] niks3-server[1132]: 2026/09/20 12:23:56 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls6311lb # [ 183.059732] niks3-server[1132]: 2026/09/20 12:23:56 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6312lb # [ 183.069715] niks3-server[1132]: 2026/09/20 12:23:56 INFO Completed upload id=526313worker1 # [ 182.962113] nix-grpc-daemon[1490]: time=2026-09-20T12:23:55.992Z level=INFO msg="Upload complete. (508ms)"6314lb # [ 183.072420] niks3-server[1132]: 2026/09/20 12:23:56 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo6315worker1 # [ 182.972862] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=0 paths=1 nar_bytes_in=6246316worker1 # [ 182.985129] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846317worker1 # [ 182.998364] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6318worker1 # [ 183.003763] nix-daemon[919]: reaped child process 1782, status = succeeded6319worker1 # [ 183.013978] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6320worker1 # [ 184.467471] nix-daemon[919]: reaped child process 1796, status = succeeded6321worker1 # [ 184.649049] nix-daemon[919]: accepted connection from pid 1811, user nix-grpc-daemon (trusted)6322worker1 # [ 184.670380] nix-daemon[919]: reaped child process 1814, status = succeeded6323lb # [ 184.793639] niks3-server[1132]: 2026/09/20 12:23:57 INFO Received uploads request method=POST path=/api/pending_closures6324worker1 # [ 184.695302] nix-grpc-daemon[1490]: time=2026-09-20T12:23:57.725Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6325worker1 # [ 184.700515] nix-grpc-daemon[1490]: time=2026-09-20T12:23:57.727Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)"6326lb # [ 184.839523] niks3-server[1132]: 2026/09/20 12:23:57 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv6327worker1 # [ 184.744578] nix-grpc-daemon[1490]: time=2026-09-20T12:23:57.774Z level=INFO msg="Uploading 1 narinfos"6328lb # [ 184.853535] niks3-server[1132]: 2026/09/20 12:23:57 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst6329lb # [ 184.863112] niks3-server[1132]: 2026/09/20 12:23:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6330lb # [ 184.873521] niks3-server[1132]: 2026/09/20 12:23:57 INFO Signed narinfos id=53 count=16331lb # [ 184.875812] niks3-server[1132]: 2026/09/20 12:23:57 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6332lb # [ 184.879994] niks3-server[1132]: 2026/09/20 12:23:57 INFO Completed upload id=536333worker1 # [ 184.771744] nix-grpc-daemon[1490]: time=2026-09-20T12:23:57.802Z level=INFO msg="Upload complete. (343ms)"6334lb # [ 184.882571] niks3-server[1132]: 2026/09/20 12:23:57 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo6335lb # [ 184.886775] niks3-server[1132]: 2026/09/20 12:23:57 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls6336worker1 # [ 184.786997] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=4 outputs=16337worker1 # [ 184.808483] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:57Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=16338worker1 # [ 184.817288] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846339worker1 # [ 184.821179] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:57Z level=debug event=want client=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=06340worker1 # [ 184.825076] nix-daemon[919]: reaped child process 1794, status = succeeded6341worker1 # [ 184.833095] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417846342worker1 # [ 184.846784] nix-grpc-daemon[1490]: time=2026-09-20T12:23:57.877Z level=INFO msg="All 1 paths already cached"6343worker1 # [ 184.855429] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417846344worker1 # [ 184.867897] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6345worker1 # [ 185.063273] nix-daemon[919]: accepted connection from pid 1820, user nix-grpc-daemon (trusted)6346worker1 # [ 185.080989] nix-daemon[919]: reaped child process 1823, status = succeeded6347worker1 # [ 185.262171] nix-daemon[919]: accepted connection from pid 1825, user nix-grpc-daemon (trusted)6348worker1 # [ 185.280427] nix-daemon[919]: reaped child process 1828, status = succeeded6349lb # [ 185.398916] niks3-server[1132]: 2026/09/20 12:23:58 INFO Received uploads request method=POST path=/api/pending_closures6350worker1 # [ 185.300816] nix-grpc-daemon[1490]: time=2026-09-20T12:23:58.330Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6351worker1 # [ 185.302302] nix-grpc-daemon[1490]: time=2026-09-20T12:23:58.330Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)"6352lb # [ 185.427709] niks3-server[1132]: 2026/09/20 12:23:58 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst6353worker1 # [ 185.335033] nix-grpc-daemon[1490]: time=2026-09-20T12:23:58.365Z level=INFO msg="Uploading 1 narinfos"6354lb # [ 185.446694] niks3-server[1132]: 2026/09/20 12:23:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6355lb # [ 185.456924] niks3-server[1132]: 2026/09/20 12:23:58 INFO Signed narinfos id=54 count=16356lb # [ 185.462168] niks3-server[1132]: 2026/09/20 12:23:58 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls6357lb # [ 185.471192] niks3-server[1132]: 2026/09/20 12:23:58 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6358lb # [ 185.475938] niks3-server[1132]: 2026/09/20 12:23:58 INFO Completed upload id=546359worker1 # [ 185.368135] nix-grpc-daemon[1490]: time=2026-09-20T12:23:58.398Z level=INFO msg="Upload complete. (486ms)"6360lb # [ 185.478616] niks3-server[1132]: 2026/09/20 12:23:58 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo6361worker1 # [ 185.376141] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=0 paths=1 nar_bytes_in=6246362worker1 # [ 185.387512] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846363worker1 # [ 185.393429] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6364worker1 # [ 185.398526] nix-daemon[919]: reaped child process 1818, status = succeeded6365worker1 # [ 185.411774] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6366worker1 # [ 186.445464] nix-daemon[919]: reaped child process 1832, status = succeeded6367worker1 # [ 186.626020] nix-daemon[919]: accepted connection from pid 1847, user nix-grpc-daemon (trusted)6368worker1 # [ 186.645100] nix-daemon[919]: reaped child process 1850, status = succeeded6369lb # [ 186.766848] niks3-server[1132]: 2026/09/20 12:23:59 INFO Received uploads request method=POST path=/api/pending_closures6370worker1 # [ 186.669646] nix-grpc-daemon[1490]: time=2026-09-20T12:23:59.698Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6371worker1 # [ 186.671272] nix-grpc-daemon[1490]: time=2026-09-20T12:23:59.698Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)"6372lb # [ 186.810733] niks3-server[1132]: 2026/09/20 12:23:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6373worker1 # [ 186.709689] nix-grpc-daemon[1490]: time=2026-09-20T12:23:59.739Z level=INFO msg="Uploading 1 narinfos"6374lb # [ 186.828882] niks3-server[1132]: 2026/09/20 12:23:59 INFO Signed narinfos id=55 count=16375lb # [ 186.836139] niks3-server[1132]: 2026/09/20 12:23:59 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst6376worker1 # [ 186.745344] nix-grpc-daemon[1490]: time=2026-09-20T12:23:59.775Z level=INFO msg="Upload complete. (323ms)"6377lb # [ 186.846367] niks3-server[1132]: 2026/09/20 12:23:59 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv6378lb # [ 186.859471] niks3-server[1132]: 2026/09/20 12:23:59 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls6379lb # [ 186.863522] niks3-server[1132]: 2026/09/20 12:23:59 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6380lb # [ 186.867848] niks3-server[1132]: 2026/09/20 12:23:59 INFO Completed upload id=556381worker1 # [ 186.759257] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=5 outputs=16382lb # [ 186.870998] niks3-server[1132]: 2026/09/20 12:23:59 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo6383worker1 # [ 186.780428] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:59Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=16384worker1 # [ 186.784682] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846385worker1 # [ 186.787816] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:59Z level=debug event=want client=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=06386worker1 # [ 186.792879] nix-daemon[919]: reaped child process 1830, status = succeeded6387worker1 # [ 186.796234] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417846388worker1 # [ 186.807443] nix-grpc-daemon[1490]: time=2026-09-20T12:23:59.837Z level=INFO msg="All 1 paths already cached"6389worker1 # [ 186.815070] nix-grpc-daemon[1352]: ts=2026-09-20T12:23:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417846390worker1 # [ 186.825438] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6391worker1 # [ 187.020182] nix-daemon[919]: accepted connection from pid 1856, user nix-grpc-daemon (trusted)6392worker1 # [ 187.038830] nix-daemon[919]: reaped child process 1859, status = succeeded6393worker1 # [ 187.218330] nix-daemon[919]: accepted connection from pid 1861, user nix-grpc-daemon (trusted)6394worker1 # [ 187.238412] nix-daemon[919]: reaped child process 1864, status = succeeded6395lb # [ 187.357960] niks3-server[1132]: 2026/09/20 12:24:00 INFO Received uploads request method=POST path=/api/pending_closures6396worker1 # [ 187.259660] nix-grpc-daemon[1490]: time=2026-09-20T12:24:00.289Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6397worker1 # [ 187.261385] nix-grpc-daemon[1490]: time=2026-09-20T12:24:00.289Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)"6398lb # [ 187.384576] niks3-server[1132]: 2026/09/20 12:24:00 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst6399lb # [ 187.398982] niks3-server[1132]: 2026/09/20 12:24:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6400worker1 # [ 187.298854] nix-grpc-daemon[1490]: time=2026-09-20T12:24:00.329Z level=INFO msg="Uploading 1 narinfos"6401lb # [ 187.412400] niks3-server[1132]: 2026/09/20 12:24:00 INFO Signed narinfos id=56 count=16402lb # [ 187.415574] niks3-server[1132]: 2026/09/20 12:24:00 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls6403lb # [ 187.426565] niks3-server[1132]: 2026/09/20 12:24:00 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6404lb # [ 187.433456] niks3-server[1132]: 2026/09/20 12:24:00 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo6405lb # [ 187.439440] niks3-server[1132]: 2026/09/20 12:24:00 INFO Completed upload id=566406worker1 # [ 187.332123] nix-grpc-daemon[1490]: time=2026-09-20T12:24:00.362Z level=INFO msg="Upload complete. (494ms)"6407worker1 # [ 187.341014] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=0 paths=1 nar_bytes_in=6246408worker1 # [ 187.354402] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846409worker1 # [ 187.361686] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6410worker1 # [ 187.366702] nix-daemon[919]: reaped child process 1854, status = succeeded6411worker1 # [ 187.378183] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6412worker1 # [ 188.804520] nix-daemon[919]: reaped child process 1868, status = succeeded6413worker1 # [ 188.971214] nix-daemon[919]: accepted connection from pid 1883, user nix-grpc-daemon (trusted)6414worker1 # [ 188.989009] nix-daemon[919]: reaped child process 1886, status = succeeded6415lb # [ 189.111580] niks3-server[1132]: 2026/09/20 12:24:02 INFO Received uploads request method=POST path=/api/pending_closures6416worker1 # [ 189.016384] nix-grpc-daemon[1490]: time=2026-09-20T12:24:02.044Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6417worker1 # [ 189.017901] nix-grpc-daemon[1490]: time=2026-09-20T12:24:02.044Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)"6418lb # [ 189.146661] niks3-server[1132]: 2026/09/20 12:24:02 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv6419lb # [ 189.161330] niks3-server[1132]: 2026/09/20 12:24:02 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst6420worker1 # [ 189.067474] nix-grpc-daemon[1490]: time=2026-09-20T12:24:02.097Z level=INFO msg="Uploading 1 narinfos"6421lb # [ 189.176852] niks3-server[1132]: 2026/09/20 12:24:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6422lb # [ 189.186668] niks3-server[1132]: 2026/09/20 12:24:02 INFO Signed narinfos id=57 count=16423lb # [ 189.194016] niks3-server[1132]: 2026/09/20 12:24:02 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls6424worker1 # [ 189.090662] nix-grpc-daemon[1490]: time=2026-09-20T12:24:02.121Z level=INFO msg="Upload complete. (299ms)"6425lb # [ 189.198831] niks3-server[1132]: 2026/09/20 12:24:02 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6426lb # [ 189.204799] niks3-server[1132]: 2026/09/20 12:24:02 INFO Completed upload id=576427lb # [ 189.208398] niks3-server[1132]: 2026/09/20 12:24:02 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo6428worker1 # [ 189.103695] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=6 outputs=16429worker1 # [ 189.123088] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:02Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=16430worker1 # [ 189.129110] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846431worker1 # [ 189.134298] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:02Z level=debug event=want client=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=06432worker1 # [ 189.139725] nix-daemon[919]: reaped child process 1866, status = succeeded6433worker1 # [ 189.141951] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417846434worker1 # [ 189.150072] nix-grpc-daemon[1490]: time=2026-09-20T12:24:02.180Z level=INFO msg="All 2 paths already cached"6435worker1 # [ 189.157608] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417846436worker1 # [ 189.171264] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6437worker1 # [ 189.356196] nix-daemon[919]: accepted connection from pid 1892, user nix-grpc-daemon (trusted)6438worker1 # [ 189.376590] nix-daemon[919]: reaped child process 1895, status = succeeded6439worker1 # [ 189.551961] nix-daemon[919]: accepted connection from pid 1897, user nix-grpc-daemon (trusted)6440worker1 # [ 189.572905] nix-daemon[919]: reaped child process 1900, status = succeeded6441lb # [ 189.691582] niks3-server[1132]: 2026/09/20 12:24:02 INFO Received uploads request method=POST path=/api/pending_closures6442worker1 # [ 189.593777] nix-grpc-daemon[1490]: time=2026-09-20T12:24:02.623Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6443worker1 # [ 189.595337] nix-grpc-daemon[1490]: time=2026-09-20T12:24:02.623Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)"6444lb # [ 189.722232] niks3-server[1132]: 2026/09/20 12:24:02 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst6445worker1 # [ 189.627035] nix-grpc-daemon[1490]: time=2026-09-20T12:24:02.657Z level=INFO msg="Uploading 1 narinfos"6446lb # [ 189.739405] niks3-server[1132]: 2026/09/20 12:24:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6447lb # [ 189.750664] niks3-server[1132]: 2026/09/20 12:24:02 INFO Signed narinfos id=58 count=16448lb # [ 189.754775] niks3-server[1132]: 2026/09/20 12:24:02 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls6449worker1 # [ 189.656715] nix-grpc-daemon[1490]: time=2026-09-20T12:24:02.687Z level=INFO msg="Upload complete. (473ms)"6450lb # [ 189.764393] niks3-server[1132]: 2026/09/20 12:24:02 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6451lb # [ 189.769623] niks3-server[1132]: 2026/09/20 12:24:02 INFO Completed upload id=586452worker1 # [ 189.664160] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=0 paths=1 nar_bytes_in=7526453lb # [ 189.772840] niks3-server[1132]: 2026/09/20 12:24:02 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo6454worker1 # [ 189.672861] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846455worker1 # [ 189.683304] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6456worker1 # [ 189.688481] nix-daemon[919]: reaped child process 1890, status = succeeded6457worker1 # [ 189.702149] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6458worker1 # [ 190.749641] nix-daemon[919]: reaped child process 1904, status = succeeded6459worker1 # [ 190.928835] nix-daemon[919]: accepted connection from pid 1919, user nix-grpc-daemon (trusted)6460worker1 # [ 190.949597] nix-daemon[919]: reaped child process 1922, status = succeeded6461lb # [ 191.070927] niks3-server[1132]: 2026/09/20 12:24:04 INFO Received uploads request method=POST path=/api/pending_closures6462worker1 # [ 190.973311] nix-grpc-daemon[1490]: time=2026-09-20T12:24:04.003Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6463worker1 # [ 190.976679] nix-grpc-daemon[1490]: time=2026-09-20T12:24:04.005Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)"6464lb # [ 191.107499] niks3-server[1132]: 2026/09/20 12:24:04 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv6465lb # [ 191.123372] niks3-server[1132]: 2026/09/20 12:24:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6466worker1 # [ 191.021970] nix-grpc-daemon[1490]: time=2026-09-20T12:24:04.052Z level=INFO msg="Uploading 1 narinfos"6467lb # [ 191.134987] niks3-server[1132]: 2026/09/20 12:24:04 INFO Signed narinfos id=59 count=16468lb # [ 191.142486] niks3-server[1132]: 2026/09/20 12:24:04 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls6469lb # [ 191.153474] niks3-server[1132]: 2026/09/20 12:24:04 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst6470lb # [ 191.160062] niks3-server[1132]: 2026/09/20 12:24:04 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6471lb # [ 191.164783] niks3-server[1132]: 2026/09/20 12:24:04 INFO Completed upload id=596472worker1 # [ 191.057078] nix-grpc-daemon[1490]: time=2026-09-20T12:24:04.087Z level=INFO msg="Upload complete. (343ms)"6473lb # [ 191.167838] niks3-server[1132]: 2026/09/20 12:24:04 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo6474worker1 # [ 191.073171] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=7 outputs=16475worker1 # [ 191.092944] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:04Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=16476worker1 # [ 191.097424] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846477worker1 # [ 191.102491] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417846478worker1 # [ 191.107350] nix-daemon[919]: reaped child process 1902, status = succeeded6479worker1 # [ 191.110897] nix-grpc-daemon[1490]: time=2026-09-20T12:24:04.141Z level=INFO msg="All 2 paths already cached"6480worker1 # [ 191.118581] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417846481worker1 # [ 191.131152] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6482worker1 # [ 191.327372] nix-daemon[919]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)6483worker1 # [ 191.344811] nix-daemon[919]: reaped child process 1929, status = succeeded6484worker1 # [ 191.527238] nix-daemon[919]: accepted connection from pid 1931, user nix-grpc-daemon (trusted)6485worker1 # [ 191.545068] nix-daemon[919]: reaped child process 1934, status = succeeded6486lb # [ 191.664220] niks3-server[1132]: 2026/09/20 12:24:04 INFO Received uploads request method=POST path=/api/pending_closures6487worker1 # [ 191.565147] nix-grpc-daemon[1490]: time=2026-09-20T12:24:04.595Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6488worker1 # [ 191.566637] nix-grpc-daemon[1490]: time=2026-09-20T12:24:04.595Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)"6489lb # [ 191.693973] niks3-server[1132]: 2026/09/20 12:24:04 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst6490worker1 # [ 191.595400] nix-grpc-daemon[1490]: time=2026-09-20T12:24:04.625Z level=INFO msg="Uploading 1 narinfos"6491lb # [ 191.706884] niks3-server[1132]: 2026/09/20 12:24:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6492lb # [ 191.718100] niks3-server[1132]: 2026/09/20 12:24:04 INFO Signed narinfos id=60 count=16493lb # [ 191.727313] niks3-server[1132]: 2026/09/20 12:24:04 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls6494worker1 # [ 191.624889] nix-grpc-daemon[1490]: time=2026-09-20T12:24:04.655Z level=INFO msg="Upload complete. (480ms)"6495lb # [ 191.733002] niks3-server[1132]: 2026/09/20 12:24:04 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6496lb # [ 191.739043] niks3-server[1132]: 2026/09/20 12:24:04 INFO Completed upload id=606497worker1 # [ 191.631208] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=0 paths=1 nar_bytes_in=7526498lb # [ 191.742847] niks3-server[1132]: 2026/09/20 12:24:04 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo6499worker1 # [ 191.640510] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846500worker1 # [ 191.653629] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6501worker1 # [ 191.656459] nix-daemon[919]: reaped child process 1924, status = succeeded6502worker1 # [ 191.669252] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6503worker1 # [ 193.318592] nix-daemon[919]: reaped child process 1938, status = succeeded6504worker1 # [ 193.498200] nix-daemon[919]: accepted connection from pid 1953, user nix-grpc-daemon (trusted)6505worker1 # [ 193.517923] nix-daemon[919]: reaped child process 1956, status = succeeded6506lb # [ 193.637808] niks3-server[1132]: 2026/09/20 12:24:06 INFO Received uploads request method=POST path=/api/pending_closures6507worker1 # [ 193.540406] nix-grpc-daemon[1490]: time=2026-09-20T12:24:06.570Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6508worker1 # [ 193.546425] nix-grpc-daemon[1490]: time=2026-09-20T12:24:06.570Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)"6509lb # [ 193.679419] niks3-server[1132]: 2026/09/20 12:24:06 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst6510lb # [ 193.692586] niks3-server[1132]: 2026/09/20 12:24:06 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv6511lb # [ 193.703006] niks3-server[1132]: 2026/09/20 12:24:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6512worker1 # [ 193.600853] nix-grpc-daemon[1490]: time=2026-09-20T12:24:06.630Z level=INFO msg="Uploading 1 narinfos"6513lb # [ 193.711254] niks3-server[1132]: 2026/09/20 12:24:06 INFO Signed narinfos id=61 count=16514lb # [ 193.716726] niks3-server[1132]: 2026/09/20 12:24:06 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls6515lb # [ 193.725318] niks3-server[1132]: 2026/09/20 12:24:06 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6516lb # [ 193.732238] niks3-server[1132]: 2026/09/20 12:24:06 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo6517lb # [ 193.737004] niks3-server[1132]: 2026/09/20 12:24:06 INFO Completed upload id=616518worker1 # [ 193.629648] nix-grpc-daemon[1490]: time=2026-09-20T12:24:06.660Z level=INFO msg="Upload complete. (348ms)"6519worker1 # [ 193.645191] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=8 outputs=16520worker1 # [ 193.659982] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:06Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=16521worker1 # [ 193.664853] nix-daemon[919]: reaped child process 1936, status = succeeded6522worker1 # [ 193.685264] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846523worker1 # [ 193.700716] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417846524worker1 # [ 193.714705] nix-grpc-daemon[1490]: time=2026-09-20T12:24:06.745Z level=INFO msg="All 2 paths already cached"6525worker1 # [ 193.724085] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417846526worker1 # [ 193.738046] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6527worker1 # [ 193.935470] nix-daemon[919]: accepted connection from pid 1960, user nix-grpc-daemon (trusted)6528worker1 # [ 193.953471] nix-daemon[919]: reaped child process 1963, status = succeeded6529worker1 # [ 194.132375] nix-daemon[919]: accepted connection from pid 1965, user nix-grpc-daemon (trusted)6530worker1 # [ 194.150087] nix-daemon[919]: reaped child process 1968, status = succeeded6531lb # [ 194.270844] niks3-server[1132]: 2026/09/20 12:24:07 INFO Received uploads request method=POST path=/api/pending_closures6532worker1 # [ 194.175435] nix-grpc-daemon[1490]: time=2026-09-20T12:24:07.205Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6533worker1 # [ 194.177108] nix-grpc-daemon[1490]: time=2026-09-20T12:24:07.205Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)"6534lb # [ 194.301016] niks3-server[1132]: 2026/09/20 12:24:07 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst6535worker1 # [ 194.214490] nix-grpc-daemon[1490]: time=2026-09-20T12:24:07.244Z level=INFO msg="Uploading 1 narinfos"6536lb # [ 194.316529] niks3-server[1132]: 2026/09/20 12:24:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6537lb # [ 194.328093] niks3-server[1132]: 2026/09/20 12:24:07 INFO Signed narinfos id=62 count=16538lb # [ 194.335019] niks3-server[1132]: 2026/09/20 12:24:07 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls6539lb # [ 194.341818] niks3-server[1132]: 2026/09/20 12:24:07 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6540lb # [ 194.349988] niks3-server[1132]: 2026/09/20 12:24:07 INFO Completed upload id=626541worker1 # [ 194.239337] nix-grpc-daemon[1490]: time=2026-09-20T12:24:07.269Z level=INFO msg="Upload complete. (485ms)"6542lb # [ 194.352954] niks3-server[1132]: 2026/09/20 12:24:07 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo6543worker1 # [ 194.248410] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:07Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=0 paths=1 nar_bytes_in=7526544worker1 # [ 194.262533] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417846545worker1 # [ 194.268956] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6546worker1 # [ 194.273882] nix-daemon[919]: reaped child process 1958, status = succeeded6547worker1 # [ 194.288174] nix-daemon[919]: accepted connection from pid 1352, user nix-grpc-daemon (trusted)6548worker1 # [ 195.132676] nix-daemon[919]: reaped child process 1972, status = succeeded6549worker1 # [ 195.322403] nix-daemon[919]: accepted connection from pid 1987, user nix-grpc-daemon (trusted)6550worker1 # [ 195.341572] nix-daemon[919]: reaped child process 1990, status = succeeded6551lb # [ 195.461574] niks3-server[1132]: 2026/09/20 12:24:08 INFO Received uploads request method=POST path=/api/pending_closures6552worker1 # [ 195.366997] nix-grpc-daemon[1490]: time=2026-09-20T12:24:08.394Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6553worker1 # [ 195.368628] nix-grpc-daemon[1490]: time=2026-09-20T12:24:08.394Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)"6554lb # [ 195.504207] niks3-server[1132]: 2026/09/20 12:24:08 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst6555lb # [ 195.514082] niks3-server[1132]: 2026/09/20 12:24:08 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv6556worker1 # [ 195.415228] nix-grpc-daemon[1490]: time=2026-09-20T12:24:08.445Z level=INFO msg="Uploading 1 narinfos"6557lb # [ 195.523109] niks3-server[1132]: 2026/09/20 12:24:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6558lb # [ 195.529169] niks3-server[1132]: 2026/09/20 12:24:08 INFO Signed narinfos id=63 count=16559lb # [ 195.530927] niks3-server[1132]: 2026/09/20 12:24:08 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls6560lb # [ 195.539730] niks3-server[1132]: 2026/09/20 12:24:08 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6561worker1 # [ 195.437207] nix-grpc-daemon[1490]: time=2026-09-20T12:24:08.467Z level=INFO msg="Upload complete. (335ms)"6562lb # [ 195.547566] niks3-server[1132]: 2026/09/20 12:24:08 INFO Completed upload id=636563lb # [ 195.550310] niks3-server[1132]: 2026/09/20 12:24:08 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo6564worker1 # [ 195.449023] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41784 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=9 outputs=16565worker1 # [ 195.465379] nix-daemon[919]: reaped child process 1970, status = succeeded6566client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 19.83 seconds)6567worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6568worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6569client: must fail: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features '["gpu"]' 2>&16570worker2 # [ 195.281823] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:08Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:482066571worker2 # [ 195.288771] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482066572worker2 # [ 195.305119] nix-grpc-daemon[1923]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 34 ms (attempt 1/5)6573worker2 # [ 195.339368] nix-grpc-daemon[1923]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 48 ms (attempt 2/5)6574worker2 # [ 195.395092] nix-grpc-daemon[1923]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 360 ms (attempt 3/5)6575worker2 # [ 195.755610] nix-grpc-daemon[1923]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 317 ms (attempt 4/5)6576worker2 # [ 196.081197] nix-grpc-daemon[1923]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6577worker1 # [ 196.684850] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:09Z level=debug event=want client=2 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=06578lb # [ 205.547291] envoy[882]: conn peer=[2001:db8:1::1]:50474 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27170 tx=10809 ms=195066579client: (finished: must fail: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features '["gpu"]' 2>&1, in 31.13 seconds)6580(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 51.04 seconds)6581subtest: low disk drains a worker and builds go to the other6582worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6583worker1 # [ 227.773544] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:40Z level=info event=unhealthy reason=min_free6584worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.12 seconds)6585worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6586worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 0.04 seconds)6587client: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain >&26588worker1 # [ 228.147519] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:417846589client # don't know how to build these paths:6590client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6591worker2 # [ 227.594246] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482066592worker1 # [ 228.194558] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:41Z level=debug event=want client=3 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=06593worker1 # [ 228.201032] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:41Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=16594worker2 # [ 227.674546] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482066595worker2 # [ 227.691173] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482066596client # copying 1 paths to 'lb:50051'...6597client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'...6598worker2 # [ 227.707698] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:482066599worker2 # [ 227.725795] nix-daemon[936]: accepted connection from pid 1923, user nix-grpc-daemon (trusted)6600worker2 # [ 227.755535] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:41Z level=info event=niks3_push_started pid=19796601worker2 # [ 227.976492] nix-grpc-daemon[1979]: time=2026-09-20T12:24:41.489Z level=INFO msg="Configuring client TLS" cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/worker2.pem key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/worker2.key ca=""6602worker2 # [ 228.156292] nix-daemon[936]: accepted connection from pid 1992, user nix-grpc-daemon (trusted)6603worker2 # [ 228.174729] nix-daemon[936]: reaped child process 1995, status = succeeded6604worker2 # [ 228.348533] nix-daemon[936]: accepted connection from pid 1997, user nix-grpc-daemon (trusted)6605worker2 # [ 228.365546] nix-daemon[936]: reaped child process 2000, status = succeeded6606lb # [ 229.055931] niks3-server[1132]: 2026/09/20 12:24:42 INFO Received uploads request method=POST path=/api/pending_closures6607worker2 # [ 228.392085] nix-grpc-daemon[1979]: time=2026-09-20T12:24:41.904Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6608worker2 # [ 228.393625] nix-grpc-daemon[1979]: time=2026-09-20T12:24:41.905Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)"6609lb # [ 229.100394] niks3-server[1132]: 2026/09/20 12:24:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6610worker2 # [ 228.436317] nix-grpc-daemon[1979]: time=2026-09-20T12:24:41.948Z level=INFO msg="Uploading 1 narinfos"6611lb # [ 229.115743] niks3-server[1132]: 2026/09/20 12:24:42 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst6612lb # [ 229.126007] niks3-server[1132]: 2026/09/20 12:24:42 INFO Signed narinfos id=64 count=16613worker2 # [ 228.458106] nix-grpc-daemon[1979]: time=2026-09-20T12:24:41.971Z level=INFO msg="Upload complete. (477ms)"6614lb # [ 229.134700] niks3-server[1132]: 2026/09/20 12:24:42 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls6615worker2 # [ 228.464331] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48206 duration_s=0 paths=1 nar_bytes_in=5126616lb # [ 229.140323] niks3-server[1132]: 2026/09/20 12:24:42 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6617lb # [ 229.145729] niks3-server[1132]: 2026/09/20 12:24:42 INFO Completed upload id=646618lb # [ 229.148289] niks3-server[1132]: 2026/09/20 12:24:42 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo6619worker2 # [ 228.479010] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482066620worker2 # [ 228.484663] nix-daemon[936]: accepted connection from pid 1923, user nix-grpc-daemon (trusted)6621worker2 # [ 228.489249] nix-daemon[936]: reaped child process 1977, status = succeeded6622client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051...6623worker2 # [ 228.499599] nix-daemon[936]: accepted connection from pid 1923, user nix-grpc-daemon (trusted)6624client # farm-a> worker2-next: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6625client # farm-a> LOG-farm-a-drain6626worker2 # [ 229.730032] nix-daemon[936]: reaped child process 2004, status = succeeded6627worker2 # [ 229.892534] nix-daemon[936]: accepted connection from pid 2019, user nix-grpc-daemon (trusted)6628worker2 # [ 229.910846] nix-daemon[936]: reaped child process 2022, status = succeeded6629lb # [ 230.600826] niks3-server[1132]: 2026/09/20 12:24:43 INFO Received uploads request method=POST path=/api/pending_closures6630worker2 # [ 229.936446] nix-grpc-daemon[1979]: time=2026-09-20T12:24:43.447Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6631worker2 # [ 229.937974] nix-grpc-daemon[1979]: time=2026-09-20T12:24:43.447Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)"6632lb # [ 230.645634] niks3-server[1132]: 2026/09/20 12:24:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6633worker2 # [ 229.980168] nix-grpc-daemon[1979]: time=2026-09-20T12:24:43.492Z level=INFO msg="Uploading 1 narinfos"6634lb # [ 230.659969] niks3-server[1132]: 2026/09/20 12:24:43 INFO Signed narinfos id=65 count=16635lb # [ 230.669106] niks3-server[1132]: 2026/09/20 12:24:43 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6636lb # [ 230.682894] niks3-server[1132]: 2026/09/20 12:24:43 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6637lb # [ 230.687662] niks3-server[1132]: 2026/09/20 12:24:43 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6638lb # [ 230.692550] niks3-server[1132]: 2026/09/20 12:24:43 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls6639worker2 # [ 230.011771] nix-grpc-daemon[1979]: time=2026-09-20T12:24:43.524Z level=INFO msg="Upload complete. (299ms)"6640lb # [ 230.695934] niks3-server[1132]: 2026/09/20 12:24:43 INFO Completed upload id=656641lb # [ 230.698619] niks3-server[1132]: 2026/09/20 12:24:43 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo6642worker2 # [ 230.021104] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48206 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=10 outputs=16643worker2 # [ 230.037369] nix-daemon[936]: reaped child process 2002, status = succeeded6644worker1 # [ 230.603719] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:43Z level=debug event=want client=3 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=06645worker1 # [ 230.609937] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:43Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=16646worker2 # [ 230.081526] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482066647worker2 # [ 230.097383] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482066648client # copying 1 paths to 'lb:50051'...6649worker2 # [ 230.111995] nix-grpc-daemon[1979]: time=2026-09-20T12:24:43.625Z level=INFO msg="All 1 paths already cached"6650client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'...6651worker2 # [ 230.118599] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:482066652worker2 # [ 230.132905] nix-daemon[936]: accepted connection from pid 1923, user nix-grpc-daemon (trusted)6653worker2 # [ 230.343345] nix-daemon[936]: accepted connection from pid 2026, user nix-grpc-daemon (trusted)6654worker2 # [ 230.363224] nix-daemon[936]: reaped child process 2029, status = succeeded6655worker2 # [ 230.535820] nix-daemon[936]: accepted connection from pid 2031, user nix-grpc-daemon (trusted)6656worker2 # [ 230.552440] nix-daemon[936]: reaped child process 2034, status = succeeded6657lb # [ 231.241805] niks3-server[1132]: 2026/09/20 12:24:44 INFO Received uploads request method=POST path=/api/pending_closures6658worker2 # [ 230.575858] nix-grpc-daemon[1979]: time=2026-09-20T12:24:44.088Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6659worker2 # [ 230.577513] nix-grpc-daemon[1979]: time=2026-09-20T12:24:44.088Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)"6660lb # [ 231.269464] niks3-server[1132]: 2026/09/20 12:24:44 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst6661lb # [ 231.282653] niks3-server[1132]: 2026/09/20 12:24:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign6662worker2 # [ 230.615036] nix-grpc-daemon[1979]: time=2026-09-20T12:24:44.127Z level=INFO msg="Uploading 1 narinfos"6663lb # [ 231.291800] niks3-server[1132]: 2026/09/20 12:24:44 INFO Signed narinfos id=66 count=16664lb # [ 231.297910] niks3-server[1132]: 2026/09/20 12:24:44 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls6665lb # [ 231.305521] niks3-server[1132]: 2026/09/20 12:24:44 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete6666worker2 # [ 230.635225] nix-grpc-daemon[1979]: time=2026-09-20T12:24:44.148Z level=INFO msg="Upload complete. (485ms)"6667lb # [ 231.312923] niks3-server[1132]: 2026/09/20 12:24:44 INFO Completed upload id=666668worker2 # [ 230.641188] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48206 duration_s=0 paths=1 nar_bytes_in=6326669lb # [ 231.316769] niks3-server[1132]: 2026/09/20 12:24:44 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo6670worker2 # [ 230.655447] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482066671worker2 # [ 230.661164] nix-daemon[936]: accepted connection from pid 1923, user nix-grpc-daemon (trusted)6672worker2 # [ 230.665822] nix-daemon[936]: reaped child process 2024, status = succeeded6673client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051...6674worker2 # [ 230.675503] nix-daemon[936]: accepted connection from pid 1923, user nix-grpc-daemon (trusted)6675client # farm-b> worker2-next: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6676client # farm-b> LOG-farm-b-drain6677worker2 # [ 231.718559] nix-daemon[936]: reaped child process 2038, status = succeeded6678worker2 # [ 231.886231] nix-daemon[936]: accepted connection from pid 2053, user nix-grpc-daemon (trusted)6679worker2 # [ 231.905499] nix-daemon[936]: reaped child process 2056, status = succeeded6680lb # [ 232.593011] niks3-server[1132]: 2026/09/20 12:24:45 INFO Received uploads request method=POST path=/api/pending_closures6681worker2 # [ 231.927733] nix-grpc-daemon[1979]: time=2026-09-20T12:24:45.439Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6682worker2 # [ 231.929365] nix-grpc-daemon[1979]: time=2026-09-20T12:24:45.439Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)"6683lb # [ 232.633943] niks3-server[1132]: 2026/09/20 12:24:45 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst6684worker2 # [ 231.969582] nix-grpc-daemon[1979]: time=2026-09-20T12:24:45.482Z level=INFO msg="Uploading 1 narinfos"6685lb # [ 232.644618] niks3-server[1132]: 2026/09/20 12:24:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign6686lb # [ 232.655687] niks3-server[1132]: 2026/09/20 12:24:45 INFO Signed narinfos id=67 count=16687lb # [ 232.661586] niks3-server[1132]: 2026/09/20 12:24:45 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6688lb # [ 232.666830] niks3-server[1132]: 2026/09/20 12:24:45 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls6689lb # [ 232.676391] niks3-server[1132]: 2026/09/20 12:24:45 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete6690worker2 # [ 232.005989] nix-grpc-daemon[1979]: time=2026-09-20T12:24:45.519Z level=INFO msg="Upload complete. (317ms)"6691lb # [ 232.682428] niks3-server[1132]: 2026/09/20 12:24:45 INFO Completed upload id=676692lb # [ 232.686548] niks3-server[1132]: 2026/09/20 12:24:45 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo6693worker2 # [ 232.016982] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48206 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=11 outputs=16694worker1 # [ 232.593255] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:45Z level=debug event=want client=3 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=06695worker2 # [ 232.032896] nix-daemon[936]: reaped child process 2036, status = succeeded6696worker1 # [ 232.593672] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:45Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=16697worker2 # [ 232.069048] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482066698worker2 # [ 232.085852] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482066699client # copying 1 paths to 'lb:50051'...6700worker2 # [ 232.101560] nix-grpc-daemon[1979]: time=2026-09-20T12:24:45.613Z level=INFO msg="All 2 paths already cached"6701client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'...6702worker2 # [ 232.107462] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:482066703worker2 # [ 232.121476] nix-daemon[936]: accepted connection from pid 1923, user nix-grpc-daemon (trusted)6704worker2 # [ 232.327422] nix-daemon[936]: accepted connection from pid 2060, user nix-grpc-daemon (trusted)6705worker2 # [ 232.346347] nix-daemon[936]: reaped child process 2063, status = succeeded6706worker2 # [ 232.524109] nix-daemon[936]: accepted connection from pid 2065, user nix-grpc-daemon (trusted)6707worker2 # [ 232.541591] nix-daemon[936]: reaped child process 2068, status = succeeded6708lb # [ 233.230158] niks3-server[1132]: 2026/09/20 12:24:46 INFO Received uploads request method=POST path=/api/pending_closures6709worker2 # [ 232.564102] nix-grpc-daemon[1979]: time=2026-09-20T12:24:46.076Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6710worker2 # [ 232.565617] nix-grpc-daemon[1979]: time=2026-09-20T12:24:46.077Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)"6711lb # [ 233.259237] niks3-server[1132]: 2026/09/20 12:24:46 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst6712worker2 # [ 232.599375] nix-grpc-daemon[1979]: time=2026-09-20T12:24:46.112Z level=INFO msg="Uploading 1 narinfos"6713lb # [ 233.274551] niks3-server[1132]: 2026/09/20 12:24:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign6714lb # [ 233.284846] niks3-server[1132]: 2026/09/20 12:24:46 INFO Signed narinfos id=68 count=16715worker2 # [ 232.621212] nix-grpc-daemon[1979]: time=2026-09-20T12:24:46.134Z level=INFO msg="Upload complete. (483ms)"6716lb # [ 233.297312] niks3-server[1132]: 2026/09/20 12:24:46 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls6717worker2 # [ 232.628119] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48206 duration_s=0 paths=1 nar_bytes_in=7606718lb # [ 233.304756] niks3-server[1132]: 2026/09/20 12:24:46 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete6719lb # [ 233.310866] niks3-server[1132]: 2026/09/20 12:24:46 INFO Completed upload id=686720worker2 # [ 232.636872] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482066721lb # [ 233.313582] niks3-server[1132]: 2026/09/20 12:24:46 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo6722worker2 # [ 232.647344] nix-daemon[936]: accepted connection from pid 1923, user nix-grpc-daemon (trusted)6723worker2 # [ 232.649332] nix-daemon[936]: reaped child process 2058, status = succeeded6724client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051...6725worker2 # [ 232.662355] nix-daemon[936]: accepted connection from pid 1923, user nix-grpc-daemon (trusted)6726client # farm-top> worker2-next: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6727client # farm-top> LOG-farm-top-drain6728worker2 # [ 233.684564] nix-daemon[936]: reaped child process 2072, status = succeeded6729worker2 # [ 233.850523] nix-daemon[936]: accepted connection from pid 2087, user nix-grpc-daemon (trusted)6730worker2 # [ 233.868853] nix-daemon[936]: reaped child process 2090, status = succeeded6731lb # [ 234.556814] niks3-server[1132]: 2026/09/20 12:24:47 INFO Received uploads request method=POST path=/api/pending_closures6732worker2 # [ 233.892385] nix-grpc-daemon[1979]: time=2026-09-20T12:24:47.403Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6733worker2 # [ 233.893880] nix-grpc-daemon[1979]: time=2026-09-20T12:24:47.403Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)"6734lb # [ 234.589223] niks3-server[1132]: 2026/09/20 12:24:47 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6735lb # [ 234.601678] niks3-server[1132]: 2026/09/20 12:24:47 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst6736worker2 # [ 233.940598] nix-grpc-daemon[1979]: time=2026-09-20T12:24:47.453Z level=INFO msg="Uploading 1 narinfos"6737lb # [ 234.615563] niks3-server[1132]: 2026/09/20 12:24:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign6738lb # [ 234.626032] niks3-server[1132]: 2026/09/20 12:24:47 INFO Signed narinfos id=69 count=16739lb # [ 234.632249] niks3-server[1132]: 2026/09/20 12:24:47 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls6740lb # [ 234.640979] niks3-server[1132]: 2026/09/20 12:24:47 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete6741lb # [ 234.646366] niks3-server[1132]: 2026/09/20 12:24:47 INFO Completed upload id=696742worker2 # [ 233.970416] nix-grpc-daemon[1979]: time=2026-09-20T12:24:47.483Z level=INFO msg="Upload complete. (299ms)"6743lb # [ 234.649418] niks3-server[1132]: 2026/09/20 12:24:47 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo6744worker2 # [ 233.981030] nix-grpc-daemon[1923]: ts=2026-09-20T12:24:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48206 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=12 outputs=16745worker2 # [ 233.997236] nix-daemon[936]: reaped child process 2070, status = succeeded6746client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?ca-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ca.pem&client-cert=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.pem&client-key=/nix/store/1k2a3bdhnmd3p212243kr1xfai6n0fph-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain >&2, in 6.71 seconds)6747client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain outPath6748client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain outPath, in 0.23 seconds)6749worker1: must succeed: rm /nix/.rw-store/fill6750worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.15 seconds)6751worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'6752lb # [ 235.547518] envoy[882]: conn peer=[2001:db8:1::1]:44846 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=914 tx=540 ms=308596753worker1 # [ 235.778543] nix-grpc-daemon[1352]: ts=2026-09-20T12:24:48Z level=info event=healthy reason=min_free6754lb # [ 235.554359] envoy[882]: conn peer=[2001:db8:1::1]:57038 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9510 tx=3859 ms=64186755worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.12 seconds)6756(finished: subtest: low disk drains a worker and builds go to the other, in 9.48 seconds)6757(finished: run the VM test script, in 240.05 seconds)6758test script finished in 240.35s6759cleanup6760kill QemuMachine (pid 47)6761client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6762kill QemuMachine (pid 48)6763lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6764kill QemuMachine (pid 51)6765worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6766kill QemuMachine (pid 53)6767worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14)6768(finished: cleanup, in 1.25 seconds)6769additionally exposed symbols:6770 client, lb, worker1, worker2,6771 vlan1,6772 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh