vm-test-run-nix-grpc-farm
checks.aarch64-linux.farm
· build #550
· raw
1tribuchet: building on eliza2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker2 # Disk image does not exist, creating the virtualisation disk image...13client # Disk image does not exist, creating the virtualisation disk image...14worker1 # Disk image does not exist, creating the virtualisation disk image...15client: QEMU running (pid 49)16worker2: QEMU running (pid 52)17lb # Disk image does not exist, creating the virtualisation disk image...18worker2 # Formatting '/build/vm-state-worker2/tmp.k8jyAQ2Elt', fmt=raw size=107374182419worker1: QEMU running (pid 51)20client # Formatting '/build/vm-state-client/tmp.oZB7z0BJub', fmt=raw size=107374182421worker1 # Formatting '/build/vm-state-worker1/tmp.2Rph7fzrws', fmt=raw size=107374182422lb: QEMU running (pid 50)23client # mke2fs 1.47.4 (6-Mar-2025)24worker1 # mke2fs 1.47.4 (6-Mar-2025)25worker2 # mke2fs 1.47.4 (6-Mar-2025)26lb # Formatting '/build/vm-state-lb/tmp.o1Lpa73xCK', fmt=raw size=107374182427client # Discarding device blocks: 0/262144 done28worker2 # Discarding device blocks: 0/262144 done29lb # mke2fs 1.47.4 (6-Mar-2025)30client # Creating filesystem with 262144 4k blocks and 65536 inodes31worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes32lb # Discarding device blocks: 0/262144 done33client # Filesystem UUID: b2ad9f67-78cc-48c1-ae94-b1f72b21168934worker2 # Filesystem UUID: 000d82cb-86e1-4e34-961e-58a6d4ea511a35lb # Creating filesystem with 262144 4k blocks and 65536 inodes36client # Superblock backups stored on blocks:37worker2 # Superblock backups stored on blocks:38lb # Filesystem UUID: 9056aea3-a3a4-4d88-b405-10eb7271847b39client # 32768, 98304, 163840, 22937640worker2 # 32768, 98304, 163840, 22937641lb # Superblock backups stored on blocks:42client # 43worker2 # 44lb # 32768, 98304, 163840, 22937645client # Allocating group tables: 0/8 done46worker2 # Allocating group tables: 0/8 done47lb # 48client # Writing inode tables: 0/8 done49worker2 # Writing inode tables: 0/8 done50lb # Allocating group tables: 0/8 done51client # Creating journal (8192 blocks): done52worker2 # Creating journal (8192 blocks): done53lb # Writing inode tables: 0/8 done54client # Writing superblocks and filesystem accounting information: 0/8 done55worker2 # Writing superblocks and filesystem accounting information: 0/8 done56lb # Creating journal (8192 blocks): done57client # 58worker2 # 59lb # Writing superblocks and filesystem accounting information: 0/8 done60client # Virtualisation disk image created.61worker2 # Virtualisation disk image created.62lb # 63client # Starting virtiofs daemons...64worker2 # Starting virtiofs daemons...65lb # Virtualisation disk image created.66client # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)67worker2 # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)68lb # Starting virtiofs daemons...69client # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether70worker2 # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether71lb # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)72client # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)73worker2 # [2026-09-24T11:47:27Z INFO virtiofsd] Waiting for vhost-user socket connection...74lb # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether75client # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether76worker2 # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)77lb # [2026-09-24T11:47:27Z INFO virtiofsd] Waiting for vhost-user socket connection...78client # [2026-09-24T11:47:27Z INFO virtiofsd] Waiting for vhost-user socket connection...79worker2 # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)80lb # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)81client # [2026-09-24T11:47:27Z INFO virtiofsd] Waiting for vhost-user socket connection...82worker2 # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether83lb # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether84client # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)85worker2 # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether86lb # [2026-09-24T11:47:27Z INFO virtiofsd] Waiting for vhost-user socket connection...87client # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether88worker2 # [2026-09-24T11:47:27Z INFO virtiofsd] Waiting for vhost-user socket connection...89lb # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)90client # [2026-09-24T11:47:27Z INFO virtiofsd] Waiting for vhost-user socket connection...91worker2 # [2026-09-24T11:47:27Z INFO virtiofsd] Waiting for vhost-user socket connection...92lb # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether93client # [2026-09-24T11:47:28Z INFO virtiofsd] Client connected, servicing requests94worker2 # [2026-09-24T11:47:28Z INFO virtiofsd] Client connected, servicing requests95lb # [2026-09-24T11:47:27Z INFO virtiofsd] Waiting for vhost-user socket connection...96client # [2026-09-24T11:47:28Z INFO virtiofsd] Client connected, servicing requests97worker2 # [2026-09-24T11:47:28Z INFO virtiofsd] Client connected, servicing requests98lb # [2026-09-24T11:47:28Z INFO virtiofsd] Client connected, servicing requests99client # [2026-09-24T11:47:28Z INFO virtiofsd] Client connected, servicing requests100worker2 # [2026-09-24T11:47:28Z INFO virtiofsd] Client connected, servicing requests101lb # [2026-09-24T11:47:28Z INFO virtiofsd] Client connected, servicing requests102worker1 # Discarding device blocks: 0/262144 done103lb # [2026-09-24T11:47:28Z INFO virtiofsd] Client connected, servicing requests104worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes105(finished: start all VMs, in 0.53 seconds)106worker1 # Filesystem UUID: 71b369b9-0d5d-48d6-ab99-5eb237670f2d107client: waiting for unit network-addresses-eth1.service108worker1 # Superblock backups stored on blocks:109client: waiting for the VM to finish booting110worker1 # 32768, 98304, 163840, 229376111worker1 # 112worker1 # Allocating group tables: 0/8 done113worker1 # Writing inode tables: 0/8 done114worker1 # Creating journal (8192 blocks): done115worker1 # Writing superblocks and filesystem accounting information: 0/8 done116worker1 # 117worker1 # Virtualisation disk image created.118worker1 # Starting virtiofs daemons...119worker1 # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)120worker1 # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether121worker1 # [2026-09-24T11:47:27Z INFO virtiofsd] Waiting for vhost-user socket connection...122worker1 # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)123worker1 # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether124worker1 # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)125worker1 # [2026-09-24T11:47:27Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether126worker1 # [2026-09-24T11:47:27Z INFO virtiofsd] Waiting for vhost-user socket connection...127worker1 # [2026-09-24T11:47:27Z INFO virtiofsd] Waiting for vhost-user socket connection...128worker1 # [2026-09-24T11:47:28Z INFO virtiofsd] Client connected, servicing requests129worker1 # [2026-09-24T11:47:28Z INFO virtiofsd] Client connected, servicing requests130worker1 # [2026-09-24T11:47:28Z INFO virtiofsd] Client connected, servicing requests131worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]132worker2 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026133worker2 # [ 0.000000] KASLR enabled134worker2 # [ 0.000000] random: crng init done135worker2 # [ 0.000000] Machine model: linux,dummy-virt136worker2 # [ 0.000000] efi: UEFI not found.137worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT138worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]139worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]140worker2 # [ 0.000000] Zone ranges:141worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]142worker2 # [ 0.000000] DMA32 empty143worker2 # [ 0.000000] Normal empty144worker2 # [ 0.000000] Device empty145worker2 # [ 0.000000] Movable zone start for each node146worker2 # [ 0.000000] Early memory node ranges147worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]148worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]149worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000150worker2 # [ 0.000000] psci: probing for conduit method from DT.151worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware.152worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs153worker2 # [ 0.000000] psci: Trusted OS migration not required154worker2 # [ 0.000000] psci: SMC Calling Convention v1.1155worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)156worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296157worker2 # [ 0.000000] Detected PIPT I-cache on CPU0158worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)159worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface160worker2 # [ 0.000000] CPU features: detected: Spectre-v4161worker2 # [ 0.000000] CPU features: detected: Spectre-BHB162worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38163worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23164worker2 # [ 0.000000] alternatives: applying boot alternatives165worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/52cfs3js2grn34r92x00f7g27cmri1c1-nixos-system-worker2-test/init regInfo=/nix/store/c8pq5g7iyqyy30bsg7hpn4bc0827rjp4-closure-info/registration console=ttyAMA0,115200n8 console=tty0166worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/c8pq5g7iyqyy30bsg7hpn4bc0827rjp4-closure-info/registration", will be passed to user space.167worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes168worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)169worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)170worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB171worker2 # [ 0.000000] software IO TLB: area num 1.172worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB173worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)174worker2 # [ 0.000000] Fallback order for Node 0: 0175worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216176worker2 # [ 0.000000] Policy zone: DMA177worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off178worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1179worker2 # [ 0.000000] allocated 3145728 bytes of page_ext180worker2 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages181worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups182worker2 # [ 0.000000] rcu: Hierarchical RCU implementation.183worker2 # [ 0.000000] rcu: RCU event tracing is enabled.184worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.185worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled.186worker2 # [ 0.000000] Rude variant of Tasks RCU enabled.187worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled.188worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.189worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1190worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.191worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.192worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.193worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0194worker2 # [ 0.000000] GICv3: 256 SPIs implemented195worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented196worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq197worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI198worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0199worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000200worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]201worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)202worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)203worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000204worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000205worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.206worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns207worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).208worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns209worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns210worker2 # [ 0.000034] arm-pv: using stolen time PV211worker2 # [ 0.000676] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)212worker2 # [ 0.000847] Console: colour dummy device 80x25213worker2 # [ 0.000854] printk: legacy console [tty0] enabled214worker2 # [ 0.001044] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)215worker2 # [ 0.001051] pid_max: default: 32768 minimum: 301216worker2 # [ 0.001132] LSM: initializing lsm=capability,landlock,yama,bpf,ima217worker2 # [ 0.001342] landlock: Up and running.218worker2 # [ 0.001345] Yama: becoming mindful.219worker2 # [ 0.001965] LSM support for eBPF active220worker2 # [ 0.002115] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)221worker2 # [ 0.002148] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)222worker2 # [ 0.004044] rcu: Hierarchical SRCU implementation.223worker2 # [ 0.004049] rcu: Max phase no-delay instances is 1000.224worker2 # [ 0.005276] fsl-mc MSI: its@8080000 domain created225worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]226worker2 # [ 0.005368] EFI services will not be available.227worker2 # [ 0.005436] smp: Bringing up secondary CPUs ...228worker2 # [ 0.005444] smp: Brought up 1 node, 1 CPU229worker1 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026230worker1 # [ 0.000000] KASLR enabled231worker2 # [ 0.005447] SMP: Total of 1 processors activated.232worker1 # [ 0.000000] random: crng init done233worker2 # [ 0.005453] CPU: All CPU(s) started at EL1234worker1 # [ 0.000000] Machine model: linux,dummy-virt235worker2 # [ 0.005463] CPU features: detected: Branch Target Identification236worker1 # [ 0.000000] efi: UEFI not found.237worker2 # [ 0.005467] CPU features: detected: ARMv8.4 Translation Table Level238worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT239worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]240worker2 # [ 0.005470] CPU features: detected: Instruction cache invalidation not required for I/D coherence241worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]242worker1 # [ 0.000000] Zone ranges:243worker2 # [ 0.005474] CPU features: detected: Data cache clean to the PoU not required for I/D coherence244worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]245worker2 # [ 0.005478] CPU features: detected: Common not Private translations246worker1 # [ 0.000000] DMA32 empty247worker1 # [ 0.000000] Normal empty248worker2 # [ 0.005481] CPU features: detected: CRC32 instructions249worker1 # [ 0.000000] Device empty250worker1 # [ 0.000000] Movable zone start for each node251worker2 # [ 0.005484] CPU features: detected: Data cache clean to Point of Deep Persistence252worker1 # [ 0.000000] Early memory node ranges253worker2 # [ 0.005487] CPU features: detected: Data cache clean to Point of Persistence254worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]255worker2 # [ 0.005490] CPU features: detected: Data independent timing control (DIT)256worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]257worker2 # [ 0.005493] CPU features: detected: E0PD258worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000259worker2 # [ 0.005496] CPU features: detected: Enhanced Counter Virtualization260worker1 # [ 0.000000] psci: probing for conduit method from DT.261worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware.262worker2 # [ 0.005499] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)263worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs264worker2 # [ 0.005503] CPU features: detected: Enhanced Virtualization Traps265worker1 # [ 0.000000] psci: Trusted OS migration not required266worker2 # [ 0.005506] CPU features: detected: Fine Grained Traps267worker1 # [ 0.000000] psci: SMC Calling Convention v1.1268worker2 # [ 0.005510] CPU features: detected: Generic authentication (architected QARMA5 algorithm)269worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)270worker2 # [ 0.005515] CPU features: detected: RCpc load-acquire (LDAPR)271worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296272worker2 # [ 0.005518] CPU features: detected: LSE atomic instructions273worker1 # [ 0.000000] Detected PIPT I-cache on CPU0274worker2 # [ 0.005521] CPU features: detected: Privileged Access Never275worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)276worker2 # [ 0.005523] CPU features: detected: PMUv3277worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface278worker2 # [ 0.005526] CPU features: detected: RAS Extension Support279worker1 # [ 0.000000] CPU features: detected: Spectre-v4280worker2 # [ 0.005529] CPU features: detected: RASv1p1 Extension Support281worker1 # [ 0.000000] CPU features: detected: Spectre-BHB282worker2 # [ 0.005532] CPU features: detected: Random Number Generator283client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]284worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38285lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]286worker2 # [ 0.005534] CPU features: detected: Speculation barrier (SB)287worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23288worker2 # [ 0.005537] CPU features: detected: Stage-2 Force Write-Back289client # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026290worker1 # [ 0.000000] alternatives: applying boot alternatives291client # [ 0.000000] KASLR enabled292worker2 # [ 0.005540] CPU features: detected: TLB range maintenance instructions293lb # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026294client # [ 0.000000] random: crng init done295lb # [ 0.000000] KASLR enabled296client # [ 0.000000] Machine model: linux,dummy-virt297lb # [ 0.000000] random: crng init done298worker2 # [ 0.005545] CPU features: detected: Speculative Store Bypassing Safe (SSBS)299client # [ 0.000000] efi: UEFI not found.300lb # [ 0.000000] Machine model: linux,dummy-virt301worker2 # [ 0.005587] alternatives: applying system-wide alternatives302lb # [ 0.000000] efi: UEFI not found.303client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT304worker2 # [ 0.008575] CPU features: detected: BBM Level 2 without TLB conflict abort305lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT306client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]307worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/bbgklh2m8ldfjpxz72brjqxd9czyy42j-nixos-system-worker1-test/init regInfo=/nix/store/1ani0qysqmhjvlrkf03zfl4p3nr0rimv-closure-info/registration console=ttyAMA0,115200n8 console=tty0308lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]309client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]310client # [ 0.000000] Zone ranges:311worker2 # [ 0.008762] Memory: 1405648K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124632K reserved, 32768K cma-reserved)312lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]313worker2 # [ 0.009142] devtmpfs: initialized314lb # [ 0.000000] Zone ranges:315client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]316client # [ 0.000000] DMA32 empty317lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]318client # [ 0.000000] Normal empty319worker2 # [ 0.011017] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)320lb # [ 0.000000] DMA32 empty321client # [ 0.000000] Device empty322worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/1ani0qysqmhjvlrkf03zfl4p3nr0rimv-closure-info/registration", will be passed to user space.323lb # [ 0.000000] Normal empty324client # [ 0.000000] Movable zone start for each node325worker2 # [ 0.011038] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).326lb # [ 0.000000] Device empty327client # [ 0.000000] Early memory node ranges328lb # [ 0.000000] Movable zone start for each node329worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes330worker2 # [ 0.011213] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL331lb # [ 0.000000] Early memory node ranges332client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]333worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)334worker2 # [ 0.011217] 0 pages in range for non-PLT usage335lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]336worker2 # [ 0.011218] 508272 pages in range for PLT usage337client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]338worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)339worker2 # [ 0.011338] pinctrl core: initialized pinctrl subsystem340lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]341client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000342worker2 # [ 0.012169] DMI not present or invalid.343client # [ 0.000000] psci: probing for conduit method from DT.344lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000345worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB346worker1 # [ 0.000000] software IO TLB: area num 1.347worker2 # [ 0.015832] NET: Registered PF_NETLINK/PF_ROUTE protocol family348client # [ 0.000000] psci: PSCIv1.3 detected in firmware.349lb # [ 0.000000] psci: probing for conduit method from DT.350worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB351client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs352lb # [ 0.000000] psci: PSCIv1.3 detected in firmware.353worker2 # [ 0.018053] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations354client # [ 0.000000] psci: Trusted OS migration not required355lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs356worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)357worker2 # [ 0.018248] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations358client # [ 0.000000] psci: SMC Calling Convention v1.1359worker1 # [ 0.000000] Fallback order for Node 0: 0360lb # [ 0.000000] psci: Trusted OS migration not required361lb # [ 0.000000] psci: SMC Calling Convention v1.1362worker2 # [ 0.018450] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations363worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216364client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)365worker1 # [ 0.000000] Policy zone: DMA366worker2 # [ 0.018471] audit: initializing netlink subsys (disabled)367lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)368client # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296369worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off370worker2 # [ 0.019077] thermal_sys: Registered thermal governor 'fair_share'371client # [ 0.000000] Detected PIPT I-cache on CPU0372lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296373worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1374worker2 # [ 0.019079] thermal_sys: Registered thermal governor 'bang_bang'375lb # [ 0.000000] Detected PIPT I-cache on CPU0376worker1 # [ 0.000000] allocated 3145728 bytes of page_ext377client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)378worker2 # [ 0.019083] thermal_sys: Registered thermal governor 'step_wise'379worker1 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages380client # [ 0.000000] CPU features: detected: GICv3 CPU interface381lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)382worker2 # [ 0.019085] thermal_sys: Registered thermal governor 'user_space'383worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups384client # [ 0.000000] CPU features: detected: Spectre-v4385lb # [ 0.000000] CPU features: detected: GICv3 CPU interface386client # [ 0.000000] CPU features: detected: Spectre-BHB387worker2 # [ 0.019090] thermal_sys: Registered thermal governor 'power_allocator'388lb # [ 0.000000] CPU features: detected: Spectre-v4389worker1 # [ 0.000000] rcu: Hierarchical RCU implementation.390worker1 # [ 0.000000] rcu: RCU event tracing is enabled.391lb # [ 0.000000] CPU features: detected: Spectre-BHB392client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38393worker2 # [ 0.019117] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1394lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38395worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.396worker2 # [ 0.019125] cpuidle: using governor ladder397client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23398worker2 # [ 0.019131] cpuidle: using governor menu399worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled.400client # [ 0.000000] alternatives: applying boot alternatives401lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23402worker1 # [ 0.000000] Rude variant of Tasks RCU enabled.403lb # [ 0.000000] alternatives: applying boot alternatives404worker2 # [ 0.019320] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.405worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled.406worker2 # [ 0.019336] ASID allocator initialised with 65536 entries407worker2 # [ 0.020590] Serial: AMBA PL011 UART driver408worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.409worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1410worker2 # [ 0.026037] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1411worker2 # [ 0.026163] printk: console [ttyAMA0] enabled412worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.413client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/il2md5jw6xiq27dyq30m5mshj1x05gp3-nixos-system-client-test/init regInfo=/nix/store/hc76hxvrgbcg9dc4yqibrrl8l50khmqc-closure-info/registration console=ttyAMA0,115200n8 console=tty0414lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/5hmvppg1fs8xkiccvl0m50n4dxjz92fr-nixos-system-lb-test/init regInfo=/nix/store/4s18yqhkjka9yjskv51fw7r9nw16dgfd-closure-info/registration console=ttyAMA0,115200n8 console=tty0415worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.416client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/hc76hxvrgbcg9dc4yqibrrl8l50khmqc-closure-info/registration", will be passed to user space.417worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.418client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes419worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0420lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/4s18yqhkjka9yjskv51fw7r9nw16dgfd-closure-info/registration", will be passed to user space.421worker1 # [ 0.000000] GICv3: 256 SPIs implemented422client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)423worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented424lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes425worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq426client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)427lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)428worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI429worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0430lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)431client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB432worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000433client # [ 0.000000] software IO TLB: area num 1.434lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB435worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]436lb # [ 0.000000] software IO TLB: area num 1.437client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB438lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB439worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)440client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)441client # [ 0.000000] Fallback order for Node 0: 0442lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)443worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)444worker2 # [ 0.150300] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages445lb # [ 0.000000] Fallback order for Node 0: 0446client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216447worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000448client # [ 0.000000] Policy zone: DMA449lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216450worker2 # [ 0.150318] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page451lb # [ 0.000000] Policy zone: DMA452worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000453client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off454worker2 # [ 0.150324] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages455lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off456client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1457worker2 # [ 0.150328] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page458worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.459lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1460client # [ 0.000000] allocated 3145728 bytes of page_ext461worker2 # [ 0.150333] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages462lb # [ 0.000000] allocated 3145728 bytes of page_ext463client # [ 0.000000] ftrace: allocating 74950 entries in 294 pages464worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns465lb # [ 0.000000] ftrace: allocating 74950 entries in 294 pages466worker2 # [ 0.150337] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page467client # [ 0.000000] ftrace: allocated 294 pages with 4 groups468worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).469client # [ 0.000000] rcu: Hierarchical RCU implementation.470lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups471worker2 # [ 0.150342] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages472lb # [ 0.000000] rcu: Hierarchical RCU implementation.473client # [ 0.000000] rcu: RCU event tracing is enabled.474worker2 # [ 0.150346] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page475worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns476lb # [ 0.000000] rcu: RCU event tracing is enabled.477client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.478worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns479lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.480client # [ 0.000000] Trampoline variant of Tasks RCU enabled.481worker1 # [ 0.000031] arm-pv: using stolen time PV482client # [ 0.000000] Rude variant of Tasks RCU enabled.483lb # [ 0.000000] Trampoline variant of Tasks RCU enabled.484lb # [ 0.000000] Rude variant of Tasks RCU enabled.485client # [ 0.000000] Tracing variant of Tasks RCU enabled.486worker2 # [ 0.157655] fbcon: Taking over console487worker1 # [ 0.000533] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)488lb # [ 0.000000] Tracing variant of Tasks RCU enabled.489worker2 # [ 0.157670] ACPI: Interpreter disabled.490worker1 # [ 0.000711] Console: colour dummy device 80x25491client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.492worker2 # [ 0.159515] iommu: Default domain type: Translated493lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.494worker1 # [ 0.000719] printk: legacy console [tty0] enabled495client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1496worker2 # [ 0.159525] iommu: DMA domain TLB invalidation policy: strict mode497lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1498worker1 # [ 0.001119] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)499client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.500worker2 # [ 0.161177] SCSI subsystem initialized501worker1 # [ 0.001125] pid_max: default: 32768 minimum: 301502lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.503client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.504worker1 # [ 0.001201] LSM: initializing lsm=capability,landlock,yama,bpf,ima505lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.506worker1 # [ 0.001358] landlock: Up and running.507worker1 # [ 0.001361] Yama: becoming mindful.508client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.509worker1 # [ 0.001889] LSM support for eBPF active510lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.511client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0512client # [ 0.000000] GICv3: 256 SPIs implemented513worker1 # [ 0.002025] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)514lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0515client # [ 0.000000] GICv3: 0 Extended SPIs implemented516lb # [ 0.000000] GICv3: 256 SPIs implemented517client # [ 0.000000] Root IRQ handler: gic_handle_irq518worker1 # [ 0.002056] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)519lb # [ 0.000000] GICv3: 0 Extended SPIs implemented520client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI521lb # [ 0.000000] Root IRQ handler: gic_handle_irq522worker1 # [ 0.003954] rcu: Hierarchical SRCU implementation.523client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0524worker1 # [ 0.003960] rcu: Max phase no-delay instances is 1000.525lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI526lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0527worker1 # [ 0.005220] fsl-mc MSI: its@8080000 domain created528client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000529worker2 # [ 0.166570] usbcore: registered new interface driver usbfs530worker1 # [ 0.005309] EFI services will not be available.531client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]532lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000533worker1 # [ 0.005375] smp: Bringing up secondary CPUs ...534worker2 # [ 0.166601] usbcore: registered new interface driver hub535lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]536worker2 # [ 0.166617] usbcore: registered new device driver usb537worker1 # [ 0.005382] smp: Brought up 1 node, 1 CPU538client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)539worker1 # [ 0.005386] SMP: Total of 1 processors activated.540worker2 # [ 0.166881] pps_core: LinuxPPS API ver. 1 registered541lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)542worker1 # [ 0.005392] CPU: All CPU(s) started at EL1543client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)544worker2 # [ 0.166888] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>545worker1 # [ 0.005403] CPU features: detected: Branch Target Identification546client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000547worker2 # [ 0.166898] PTP clock support registered548lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)549worker1 # [ 0.005408] CPU features: detected: ARMv8.4 Translation Table Level550worker2 # [ 0.166949] EDAC MC: Ver: 3.0.0551lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000552client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000553worker2 # [ 0.171433] scmi_core: SCMI protocol bus registered554worker1 # [ 0.005410] CPU features: detected: Instruction cache invalidation not required for I/D coherence555lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000556client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.557worker2 # [ 0.172388] FPGA manager framework558lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.559worker1 # [ 0.005415] CPU features: detected: Data cache clean to the PoU not required for I/D coherence560worker2 # [ 0.173309] vgaarb: loaded561client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns562worker1 # [ 0.005418] CPU features: detected: Common not Private translations563client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).564lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns565worker2 # [ 0.173929] clocksource: Switched to clocksource arch_sys_counter566worker1 # [ 0.005422] CPU features: detected: CRC32 instructions567lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).568worker1 # [ 0.005424] CPU features: detected: Data cache clean to Point of Deep Persistence569client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns570worker1 # [ 0.005428] CPU features: detected: Data cache clean to Point of Persistence571lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns572client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns573client # [ 0.000030] arm-pv: using stolen time PV574worker1 # [ 0.005431] CPU features: detected: Data independent timing control (DIT)575lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns576worker1 # [ 0.005433] CPU features: detected: E0PD577lb # [ 0.000031] arm-pv: using stolen time PV578client # [ 0.000454] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)579worker1 # [ 0.005436] CPU features: detected: Enhanced Counter Virtualization580client # [ 0.000616] Console: colour dummy device 80x25581lb # [ 0.000458] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)582worker2 # [ 0.178764] VFS: Disk quotas dquot_6.6.0583worker1 # [ 0.005439] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)584client # [ 0.000624] printk: legacy console [tty0] enabled585lb # [ 0.000622] Console: colour dummy device 80x25586worker1 # [ 0.005443] CPU features: detected: Enhanced Virtualization Traps587worker2 # [ 0.178800] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)588lb # [ 0.000630] printk: legacy console [tty0] enabled589client # [ 0.000813] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)590worker1 # [ 0.005446] CPU features: detected: Fine Grained Traps591client # [ 0.000820] pid_max: default: 32768 minimum: 301592lb # [ 0.000824] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)593worker1 # [ 0.005449] CPU features: detected: Generic authentication (architected QARMA5 algorithm)594client # [ 0.000892] LSM: initializing lsm=capability,landlock,yama,bpf,ima595lb # [ 0.000831] pid_max: default: 32768 minimum: 301596worker1 # [ 0.005454] CPU features: detected: RCpc load-acquire (LDAPR)597client # [ 0.001019] landlock: Up and running.598client # [ 0.001021] Yama: becoming mindful.599lb # [ 0.000902] LSM: initializing lsm=capability,landlock,yama,bpf,ima600worker1 # [ 0.005457] CPU features: detected: LSE atomic instructions601worker2 # [ 0.182380] netfs: FS-Cache loaded602lb # [ 0.001029] landlock: Up and running.603client # [ 0.001506] LSM support for eBPF active604worker2 # [ 0.182479] pnp: PnP ACPI: disabled605lb # [ 0.001032] Yama: becoming mindful.606worker1 # [ 0.005460] CPU features: detected: Privileged Access Never607worker1 # [ 0.005463] CPU features: detected: PMUv3608client # [ 0.001637] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)609lb # [ 0.001513] LSM support for eBPF active610worker1 # [ 0.005466] CPU features: detected: RAS Extension Support611lb # [ 0.001658] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)612client # [ 0.001669] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)613worker1 # [ 0.005469] CPU features: detected: RASv1p1 Extension Support614client # [ 0.003534] rcu: Hierarchical SRCU implementation.615lb # [ 0.001692] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)616worker1 # [ 0.005471] CPU features: detected: Random Number Generator617client # [ 0.003538] rcu: Max phase no-delay instances is 1000.618lb # [ 0.003611] rcu: Hierarchical SRCU implementation.619client # [ 0.004800] fsl-mc MSI: its@8080000 domain created620worker1 # [ 0.005474] CPU features: detected: Speculation barrier (SB)621lb # [ 0.003617] rcu: Max phase no-delay instances is 1000.622client # [ 0.004889] EFI services will not be available.623worker1 # [ 0.005476] CPU features: detected: Stage-2 Force Write-Back624lb # [ 0.004888] fsl-mc MSI: its@8080000 domain created625client # [ 0.004954] smp: Bringing up secondary CPUs ...626lb # [ 0.004987] EFI services will not be available.627worker1 # [ 0.005479] CPU features: detected: TLB range maintenance instructions628client # [ 0.004962] smp: Brought up 1 node, 1 CPU629lb # [ 0.005071] smp: Bringing up secondary CPUs ...630client # [ 0.004965] SMP: Total of 1 processors activated.631worker1 # [ 0.005484] CPU features: detected: Speculative Store Bypassing Safe (SSBS)632lb # [ 0.005079] smp: Brought up 1 node, 1 CPU633client # [ 0.004972] CPU: All CPU(s) started at EL1634worker1 # [ 0.005522] alternatives: applying system-wide alternatives635lb # [ 0.005083] SMP: Total of 1 processors activated.636client # [ 0.004982] CPU features: detected: Branch Target Identification637lb # [ 0.005088] CPU: All CPU(s) started at EL1638worker2 # [ 0.186324] NET: Registered PF_INET protocol family639worker1 # [ 0.008481] CPU features: detected: BBM Level 2 without TLB conflict abort640client # [ 0.004987] CPU features: detected: ARMv8.4 Translation Table Level641lb # [ 0.005098] CPU features: detected: Branch Target Identification642worker2 # [ 0.186580] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)643lb # [ 0.005103] CPU features: detected: ARMv8.4 Translation Table Level644client # [ 0.004990] CPU features: detected: Instruction cache invalidation not required for I/D coherence645worker1 # [ 0.008651] Memory: 1405632K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124664K reserved, 32768K cma-reserved)646worker1 # [ 0.009051] devtmpfs: initialized647lb # [ 0.005106] CPU features: detected: Instruction cache invalidation not required for I/D coherence648client # [ 0.004994] CPU features: detected: Data cache clean to the PoU not required for I/D coherence649worker1 # [ 0.010826] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)650client # [ 0.004998] CPU features: detected: Common not Private translations651lb # [ 0.005110] CPU features: detected: Data cache clean to the PoU not required for I/D coherence652client # [ 0.005001] CPU features: detected: CRC32 instructions653worker1 # [ 0.010869] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).654lb # [ 0.005113] CPU features: detected: Common not Private translations655client # [ 0.005004] CPU features: detected: Data cache clean to Point of Deep Persistence656worker1 # [ 0.011065] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL657lb # [ 0.005117] CPU features: detected: CRC32 instructions658client # [ 0.005007] CPU features: detected: Data cache clean to Point of Persistence659lb # [ 0.005120] CPU features: detected: Data cache clean to Point of Deep Persistence660client # [ 0.005010] CPU features: detected: Data independent timing control (DIT)661lb # [ 0.005123] CPU features: detected: Data cache clean to Point of Persistence662client # [ 0.005014] CPU features: detected: E0PD663lb # [ 0.005127] CPU features: detected: Data independent timing control (DIT)664client # [ 0.005016] CPU features: detected: Enhanced Counter Virtualization665lb # [ 0.005129] CPU features: detected: E0PD666client # [ 0.005019] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)667lb # [ 0.005132] CPU features: detected: Enhanced Counter Virtualization668client # [ 0.005023] CPU features: detected: Enhanced Virtualization Traps669lb # [ 0.005135] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)670client # [ 0.005026] CPU features: detected: Fine Grained Traps671lb # [ 0.005138] CPU features: detected: Enhanced Virtualization Traps672lb # [ 0.005142] CPU features: detected: Fine Grained Traps673client # [ 0.005029] CPU features: detected: Generic authentication (architected QARMA5 algorithm)674client # [ 0.005034] CPU features: detected: RCpc load-acquire (LDAPR)675lb # [ 0.005146] CPU features: detected: Generic authentication (architected QARMA5 algorithm)676client # [ 0.005037] CPU features: detected: LSE atomic instructions677lb # [ 0.005150] CPU features: detected: RCpc load-acquire (LDAPR)678client # [ 0.005040] CPU features: detected: Privileged Access Never679worker1 # [ 0.011069] 0 pages in range for non-PLT usage680lb # [ 0.005153] CPU features: detected: LSE atomic instructions681client # [ 0.005043] CPU features: detected: PMUv3682worker1 # [ 0.011070] 508272 pages in range for PLT usage683lb # [ 0.005156] CPU features: detected: Privileged Access Never684client # [ 0.005045] CPU features: detected: RAS Extension Support685worker1 # [ 0.011179] pinctrl core: initialized pinctrl subsystem686lb # [ 0.005158] CPU features: detected: PMUv3687client # [ 0.005048] CPU features: detected: RASv1p1 Extension Support688worker1 # [ 0.011921] DMI not present or invalid.689lb # [ 0.005162] CPU features: detected: RAS Extension Support690client # [ 0.005051] CPU features: detected: Random Number Generator691worker1 # [ 0.014972] NET: Registered PF_NETLINK/PF_ROUTE protocol family692lb # [ 0.005164] CPU features: detected: RASv1p1 Extension Support693client # [ 0.005053] CPU features: detected: Speculation barrier (SB)694worker1 # [ 0.017127] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations695lb # [ 0.005167] CPU features: detected: Random Number Generator696client # [ 0.005056] CPU features: detected: Stage-2 Force Write-Back697lb # [ 0.005169] CPU features: detected: Speculation barrier (SB)698worker1 # [ 0.017317] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations699client # [ 0.005059] CPU features: detected: TLB range maintenance instructions700lb # [ 0.005172] CPU features: detected: Stage-2 Force Write-Back701worker1 # [ 0.017514] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations702client # [ 0.005063] CPU features: detected: Speculative Store Bypassing Safe (SSBS)703lb # [ 0.005175] CPU features: detected: TLB range maintenance instructions704worker1 # [ 0.017534] audit: initializing netlink subsys (disabled)705lb # [ 0.005180] CPU features: detected: Speculative Store Bypassing Safe (SSBS)706worker1 # [ 0.018033] thermal_sys: Registered thermal governor 'fair_share'707lb # [ 0.005218] alternatives: applying system-wide alternatives708worker1 # [ 0.018035] thermal_sys: Registered thermal governor 'bang_bang'709worker1 # [ 0.018038] thermal_sys: Registered thermal governor 'step_wise'710lb # [ 0.008266] CPU features: detected: BBM Level 2 without TLB conflict abort711worker1 # [ 0.018041] thermal_sys: Registered thermal governor 'user_space'712worker1 # [ 0.018046] thermal_sys: Registered thermal governor 'power_allocator'713lb # [ 0.008425] Memory: 1405676K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124636K reserved, 32768K cma-reserved)714lb # [ 0.008805] devtmpfs: initialized715worker1 # [ 0.018070] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1716worker1 # [ 0.018078] cpuidle: using governor ladder717lb # [ 0.010585] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)718worker1 # [ 0.018084] cpuidle: using governor menu719worker1 # [ 0.018266] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.720worker1 # [ 0.018280] ASID allocator initialised with 65536 entries721worker1 # [ 0.019466] Serial: AMBA PL011 UART driver722client # [ 0.005101] alternatives: applying system-wide alternatives723worker1 # [ 0.024635] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1724worker1 # [ 0.024753] printk: console [ttyAMA0] enabled725client # [ 0.008060] CPU features: detected: BBM Level 2 without TLB conflict abort726worker1 # [ 0.145293] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages727client # [ 0.008236] Memory: 1405644K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124660K reserved, 32768K cma-reserved)728client # [ 0.008602] devtmpfs: initialized729worker1 # [ 0.145311] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page730worker1 # [ 0.145317] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages731client # [ 0.010301] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)732worker1 # [ 0.145321] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page733client # [ 0.010322] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).734worker1 # [ 0.145326] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages735worker2 # [ 0.215042] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)736client # [ 0.010517] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL737worker1 # [ 0.145330] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page738client # [ 0.010521] 0 pages in range for non-PLT usage739lb # [ 0.010608] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).740worker2 # [ 0.215080] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)741client # [ 0.010522] 508272 pages in range for PLT usage742worker1 # [ 0.145335] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages743lb # [ 0.010794] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL744worker2 # [ 0.215105] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)745client # [ 0.010612] pinctrl core: initialized pinctrl subsystem746lb # [ 0.010799] 0 pages in range for non-PLT usage747worker1 # [ 0.145339] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page748client # [ 0.011345] DMI not present or invalid.749lb # [ 0.010800] 508272 pages in range for PLT usage750worker2 # [ 0.215176] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)751client # [ 0.014624] NET: Registered PF_NETLINK/PF_ROUTE protocol family752lb # [ 0.010890] pinctrl core: initialized pinctrl subsystem753worker2 # [ 0.215322] TCP: Hash tables configured (established 16384 bind 16384)754lb # [ 0.011659] DMI not present or invalid.755client # [ 0.016790] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations756lb # [ 0.014928] NET: Registered PF_NETLINK/PF_ROUTE protocol family757worker2 # [ 0.215398] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)758worker1 # [ 0.152662] fbcon: Taking over console759client # [ 0.016976] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations760worker1 # [ 0.152675] ACPI: Interpreter disabled.761worker2 # [ 0.215432] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)762lb # [ 0.017200] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations763worker1 # [ 0.154471] iommu: Default domain type: Translated764client # [ 0.017171] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations765worker2 # [ 0.215468] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)766lb # [ 0.017377] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations767client # [ 0.017194] audit: initializing netlink subsys (disabled)768worker1 # [ 0.154481] iommu: DMA domain TLB invalidation policy: strict mode769worker2 # [ 0.215554] NET: Registered PF_UNIX/PF_LOCAL protocol family770client # [ 0.017746] thermal_sys: Registered thermal governor 'fair_share'771lb # [ 0.017578] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations772worker1 # [ 0.156123] SCSI subsystem initialized773worker2 # [ 0.215607] NET: Registered PF_XDP protocol family774lb # [ 0.017599] audit: initializing netlink subsys (disabled)775worker2 # [ 0.215623] PCI: CLS 0 bytes, default 64776client # [ 0.017748] thermal_sys: Registered thermal governor 'bang_bang'777worker2 # [ 0.215857] Trying to unpack rootfs image as initramfs...778lb # [ 0.018150] thermal_sys: Registered thermal governor 'fair_share'779client # [ 0.017752] thermal_sys: Registered thermal governor 'step_wise'780lb # [ 0.018153] thermal_sys: Registered thermal governor 'bang_bang'781client # [ 0.017754] thermal_sys: Registered thermal governor 'user_space'782lb # [ 0.018156] thermal_sys: Registered thermal governor 'step_wise'783client # [ 0.017760] thermal_sys: Registered thermal governor 'power_allocator'784lb # [ 0.018159] thermal_sys: Registered thermal governor 'user_space'785client # [ 0.017784] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1786client # [ 0.017792] cpuidle: using governor ladder787lb # [ 0.018164] thermal_sys: Registered thermal governor 'power_allocator'788client # [ 0.017798] cpuidle: using governor menu789lb # [ 0.018188] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1790client # [ 0.017983] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.791lb # [ 0.018196] cpuidle: using governor ladder792client # [ 0.017997] ASID allocator initialised with 65536 entries793lb # [ 0.018202] cpuidle: using governor menu794client # [ 0.019181] Serial: AMBA PL011 UART driver795lb # [ 0.018380] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.796lb # [ 0.018396] ASID allocator initialised with 65536 entries797client # [ 0.024308] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1798client # [ 0.024431] printk: console [ttyAMA0] enabled799worker1 # [ 0.163753] usbcore: registered new interface driver usbfs800lb # [ 0.019613] Serial: AMBA PL011 UART driver801worker2 # [ 0.231376] kvm [1]: HYP mode not available802worker1 # [ 0.163792] usbcore: registered new interface driver hub803lb # [ 0.024837] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1804worker1 # [ 0.163808] usbcore: registered new device driver usb805client # [ 0.145777] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages806lb # [ 0.024955] printk: console [ttyAMA0] enabled807worker1 # [ 0.164085] pps_core: LinuxPPS API ver. 1 registered808client # [ 0.145797] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page809worker1 # [ 0.164092] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>810lb # [ 0.147194] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages811client # [ 0.145802] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages812worker1 # [ 0.164101] PTP clock support registered813lb # [ 0.147216] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page814client # [ 0.145814] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page815worker1 # [ 0.164139] EDAC MC: Ver: 3.0.0816lb # [ 0.147221] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages817client # [ 0.145818] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages818worker1 # [ 0.168677] scmi_core: SCMI protocol bus registered819lb # [ 0.147225] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page820client # [ 0.145823] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page821worker1 # [ 0.169616] FPGA manager framework822lb # [ 0.147229] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages823client # [ 0.145827] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages824worker1 # [ 0.170563] vgaarb: loaded825lb # [ 0.147233] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page826client # [ 0.145832] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page827lb # [ 0.147238] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages828worker1 # [ 0.171185] clocksource: Switched to clocksource arch_sys_counter829lb # [ 0.147242] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page830worker1 # [ 0.171925] VFS: Disk quotas dquot_6.6.0831client # [ 0.153197] fbcon: Taking over console832worker1 # [ 0.171951] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)833client # [ 0.153212] ACPI: Interpreter disabled.834worker1 # [ 0.174313] netfs: FS-Cache loaded835client # [ 0.155059] iommu: Default domain type: Translated836worker1 # [ 0.174430] pnp: PnP ACPI: disabled837lb # [ 0.154795] fbcon: Taking over console838client # [ 0.155069] iommu: DMA domain TLB invalidation policy: strict mode839lb # [ 0.154810] ACPI: Interpreter disabled.840client # [ 0.156827] SCSI subsystem initialized841lb # [ 0.156741] iommu: Default domain type: Translated842lb # [ 0.156751] iommu: DMA domain TLB invalidation policy: strict mode843lb # [ 0.158432] SCSI subsystem initialized844client # [ 0.162045] usbcore: registered new interface driver usbfs845client # [ 0.162077] usbcore: registered new interface driver hub846worker1 # [ 0.180271] NET: Registered PF_INET protocol family847client # [ 0.162093] usbcore: registered new device driver usb848client # [ 0.162377] pps_core: LinuxPPS API ver. 1 registered849worker1 # [ 0.180524] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)850lb # [ 0.164431] usbcore: registered new interface driver usbfs851client # [ 0.162383] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>852lb # [ 0.164464] usbcore: registered new interface driver hub853client # [ 0.162393] PTP clock support registered854lb # [ 0.164479] usbcore: registered new device driver usb855client # [ 0.162440] EDAC MC: Ver: 3.0.0856lb # [ 0.164797] pps_core: LinuxPPS API ver. 1 registered857client # [ 0.167172] scmi_core: SCMI protocol bus registered858lb # [ 0.164803] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>859client # [ 0.168151] FPGA manager framework860lb # [ 0.164821] PTP clock support registered861client # [ 0.169153] vgaarb: loaded862lb # [ 0.164877] EDAC MC: Ver: 3.0.0863lb # [ 0.169675] scmi_core: SCMI protocol bus registered864client # [ 0.169780] clocksource: Switched to clocksource arch_sys_counter865lb # [ 0.170669] FPGA manager framework866lb # [ 0.171670] vgaarb: loaded867lb # [ 0.172327] clocksource: Switched to clocksource arch_sys_counter868lb # [ 0.175628] VFS: Disk quotas dquot_6.6.0869client # [ 0.173734] VFS: Disk quotas dquot_6.6.0870lb # [ 0.175658] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)871client # [ 0.173769] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)872client # [ 0.177499] netfs: FS-Cache loaded873client # [ 0.177619] pnp: PnP ACPI: disabled874lb # [ 0.179451] netfs: FS-Cache loaded875lb # [ 0.179557] pnp: PnP ACPI: disabled876client # [ 0.181499] NET: Registered PF_INET protocol family877client # [ 0.181749] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)878lb # [ 0.183528] NET: Registered PF_INET protocol family879lb # [ 0.183782] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)880worker1 # [ 0.209579] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)881worker1 # [ 0.209621] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)882worker1 # [ 0.209648] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)883worker1 # [ 0.209722] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)884worker1 # [ 0.209865] TCP: Hash tables configured (established 16384 bind 16384)885worker1 # [ 0.209943] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)886worker1 # [ 0.209980] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)887worker1 # [ 0.210015] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)888worker1 # [ 0.210128] NET: Registered PF_UNIX/PF_LOCAL protocol family889worker1 # [ 0.210154] NET: Registered PF_XDP protocol family890worker1 # [ 0.210172] PCI: CLS 0 bytes, default 64891worker1 # [ 0.210404] Trying to unpack rootfs image as initramfs...892worker1 # [ 0.224825] kvm [1]: HYP mode not available893client # [ 0.211234] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)894client # [ 0.211285] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)895client # [ 0.211328] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)896lb # [ 0.213744] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)897client # [ 0.211398] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)898lb # [ 0.213789] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)899client # [ 0.211541] TCP: Hash tables configured (established 16384 bind 16384)900client # [ 0.211633] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)901lb # [ 0.213816] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)902client # [ 0.211671] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)903lb # [ 0.213890] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)904client # [ 0.211708] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)905lb # [ 0.214037] TCP: Hash tables configured (established 16384 bind 16384)906client # [ 0.211803] NET: Registered PF_UNIX/PF_LOCAL protocol family907lb # [ 0.214124] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)908client # [ 0.211823] NET: Registered PF_XDP protocol family909client # [ 0.211841] PCI: CLS 0 bytes, default 64910lb # [ 0.214160] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)911client # [ 0.212102] Trying to unpack rootfs image as initramfs...912lb # [ 0.214196] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)913lb # [ 0.214295] NET: Registered PF_UNIX/PF_LOCAL protocol family914lb # [ 0.214323] NET: Registered PF_XDP protocol family915lb # [ 0.214340] PCI: CLS 0 bytes, default 64916lb # [ 0.214591] Trying to unpack rootfs image as initramfs...917worker2 # [ 0.322460] Initialise system trusted keyrings918worker2 # [ 0.323239] workingset: timestamp_bits=42 max_order=19 bucket_order=0919worker2 # [ 0.324584] squashfs: version 4.0 (2009/01/31) Phillip Lougher920worker2 # [ 0.325361] 9p: Installing v9fs 9p2000 file system support921client # [ 0.227529] kvm [1]: HYP mode not available922lb # [ 0.230279] kvm [1]: HYP mode not available923worker2 # [ 0.354078] Key type asymmetric registered924worker2 # [ 0.354109] Asymmetric key parser 'x509' registered925worker2 # [ 0.354190] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)926worker2 # [ 0.356329] io scheduler mq-deadline registered927worker2 # [ 0.356340] io scheduler kyber registered928worker2 # [ 0.366073] pl061_gpio 9030000.pl061: PL061 GPIO chip registered929worker2 # [ 0.367523] ledtrig-cpu: registered to indicate activity on CPUs930worker2 # [ 0.367906] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:931worker2 # [ 0.367933] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000932worker2 # [ 0.367946] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000933worker2 # [ 0.367955] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000934worker2 # [ 0.367979] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits935worker2 # [ 0.368005] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]936worker2 # [ 0.368080] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00937worker2 # [ 0.368090] pci_bus 0000:00: root bus resource [bus 00-ff]938worker2 # [ 0.368096] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]939worker2 # [ 0.368101] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]940worker2 # [ 0.368106] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]941worker2 # [ 0.368194] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint942worker2 # [ 0.368641] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint943worker2 # [ 0.368824] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]944worker2 # [ 0.368841] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]945worker2 # [ 0.368870] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]946worker2 # [ 0.368887] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]947worker2 # [ 0.369335] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint948worker2 # [ 0.369515] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]949worker2 # [ 0.369530] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]950worker2 # [ 0.369559] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]951worker1 # [ 0.315749] Initialise system trusted keyrings952worker1 # [ 0.316501] workingset: timestamp_bits=42 max_order=19 bucket_order=0953worker2 # [ 0.389315] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint954worker2 # [ 0.389501] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]955worker1 # [ 0.317745] squashfs: version 4.0 (2009/01/31) Phillip Lougher956worker2 # [ 0.389518] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]957worker2 # [ 0.389548] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]958worker1 # [ 0.318521] 9p: Installing v9fs 9p2000 file system support959worker2 # [ 0.398073] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint960worker2 # [ 0.398276] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]961worker2 # [ 0.398293] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]962worker2 # [ 0.398322] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]963worker2 # [ 0.398341] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]964worker2 # [ 0.398826] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint965worker2 # [ 0.399011] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]966worker2 # [ 0.399041] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]967worker2 # [ 0.399482] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint968worker2 # [ 0.399665] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]969worker2 # [ 0.399695] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]970worker2 # [ 0.400072] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint971worker2 # [ 0.400250] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]972worker1 # [ 0.339158] Key type asymmetric registered973worker2 # [ 0.400554] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint974worker1 # [ 0.339180] Asymmetric key parser 'x509' registered975worker2 # [ 0.400738] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]976worker2 # [ 0.400768] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]977client # [ 0.322662] Initialise system trusted keyrings978worker2 # [ 0.401207] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint979worker2 # [ 0.401390] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]980client # [ 0.323494] workingset: timestamp_bits=42 max_order=19 bucket_order=0981worker2 # [ 0.401419] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]982client # [ 0.324826] squashfs: version 4.0 (2009/01/31) Phillip Lougher983worker2 # [ 0.401858] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint984client # [ 0.325698] 9p: Installing v9fs 9p2000 file system support985lb # [ 0.326408] Initialise system trusted keyrings986lb # [ 0.327168] workingset: timestamp_bits=42 max_order=19 bucket_order=0987worker1 # [ 0.347280] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)988lb # [ 0.328582] squashfs: version 4.0 (2009/01/31) Phillip Lougher989worker1 # [ 0.348297] io scheduler mq-deadline registered990worker2 # [ 0.419320] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]991worker1 # [ 0.348308] io scheduler kyber registered992worker2 # [ 0.419354] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]993worker2 # [ 0.419832] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint994worker2 # [ 0.420118] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]995worker2 # [ 0.420135] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]996lb # [ 0.329361] 9p: Installing v9fs 9p2000 file system support997worker2 # [ 0.420165] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]998worker2 # [ 0.420634] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint999worker2 # [ 0.420817] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1000worker2 # [ 0.420832] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1001worker2 # [ 0.420862] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1002worker2 # [ 0.421432] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1003worker2 # [ 0.421442] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1004worker2 # [ 0.421448] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1005worker2 # [ 0.421492] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1006worker1 # [ 0.359316] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1007worker2 # [ 0.421539] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1008worker2 # [ 0.421586] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1009worker2 # [ 0.421632] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1010worker1 # [ 0.360687] ledtrig-cpu: registered to indicate activity on CPUs1011worker2 # [ 0.421678] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1012worker1 # [ 0.361047] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1013worker2 # [ 0.421724] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1014worker1 # [ 0.361065] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001015worker2 # [ 0.421770] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1016worker1 # [ 0.361078] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001017worker2 # [ 0.421815] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1018worker1 # [ 0.361086] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001019worker2 # [ 0.421861] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1020worker1 # [ 0.361106] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1021worker1 # [ 0.361143] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1022worker1 # [ 0.361222] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001023worker1 # [ 0.361232] pci_bus 0000:00: root bus resource [bus 00-ff]1024worker1 # [ 0.361238] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1025worker1 # [ 0.361243] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1026worker1 # [ 0.361248] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1027client # [ 0.354527] Key type asymmetric registered1028worker1 # [ 0.361300] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1029client # [ 0.354554] Asymmetric key parser 'x509' registered1030worker1 # [ 0.361735] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1031client # [ 0.354663] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)1032worker1 # [ 0.361920] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1033worker2 # [ 0.445958] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1034client # [ 0.356828] io scheduler mq-deadline registered1035client # [ 0.356839] io scheduler kyber registered1036worker1 # [ 0.361937] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1037worker2 # [ 0.446038] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1038lb # [ 0.357388] Key type asymmetric registered1039lb # [ 0.357415] Asymmetric key parser 'x509' registered1040worker1 # [ 0.361966] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1041worker2 # [ 0.446062] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1042worker1 # [ 0.361982] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1043lb # [ 0.357499] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)1044worker2 # [ 0.446085] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1045lb # [ 0.359586] io scheduler mq-deadline registered1046worker1 # [ 0.362434] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1047worker2 # [ 0.446107] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1048lb # [ 0.359598] io scheduler kyber registered1049worker1 # [ 0.362615] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1050worker2 # [ 0.446129] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1051worker1 # [ 0.362631] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1052worker2 # [ 0.446152] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1053worker1 # [ 0.362660] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1054worker2 # [ 0.446174] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1055worker1 # [ 0.363116] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1056worker2 # [ 0.446200] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1057worker2 # [ 0.446223] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1058worker2 # [ 0.446245] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1059worker2 # [ 0.446269] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1060worker2 # [ 0.446292] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1061worker2 # [ 0.446313] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1062worker1 # [ 0.383352] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1063worker2 # [ 0.446333] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1064worker1 # [ 0.383371] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1065worker2 # [ 0.446354] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1066worker1 # [ 0.383401] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1067worker2 # [ 0.446375] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1068worker1 # [ 0.383865] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1069worker2 # [ 0.446396] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1070worker1 # [ 0.384050] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1071worker2 # [ 0.446417] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1072worker1 # [ 0.384066] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1073worker2 # [ 0.446447] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1074worker1 # [ 0.384095] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1075worker2 # [ 0.446456] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1076worker1 # [ 0.384113] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1077worker2 # [ 0.446461] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1078worker1 # [ 0.384562] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1079worker2 # [ 0.447281] pci 0000:00:07.0: enabling device (0000 -> 0002)1080worker1 # [ 0.384747] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1081worker1 # [ 0.384777] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1082worker1 # [ 0.385230] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1083worker1 # [ 0.385415] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1084client # [ 0.369965] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1085worker1 # [ 0.385444] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1086client # [ 0.370665] ledtrig-cpu: registered to indicate activity on CPUs1087worker1 # [ 0.385835] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1088client # [ 0.371024] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1089worker1 # [ 0.386016] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1090lb # [ 0.373036] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1091client # [ 0.371041] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001092lb # [ 0.373643] ledtrig-cpu: registered to indicate activity on CPUs1093worker1 # [ 0.386274] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1094client # [ 0.371053] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001095worker1 # [ 0.386462] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1096lb # [ 0.374007] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1097worker1 # [ 0.386492] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1098client # [ 0.371062] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001099lb # [ 0.374023] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001100worker1 # [ 0.386951] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1101client # [ 0.371084] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1102lb # [ 0.374043] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001103worker1 # [ 0.387135] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1104client # [ 0.371112] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1105worker1 # [ 0.387165] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1106lb # [ 0.374052] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001107client # [ 0.371199] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001108lb # [ 0.374071] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1109client # [ 0.371209] pci_bus 0000:00: root bus resource [bus 00-ff]1110client # [ 0.371215] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1111lb # [ 0.374092] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1112client # [ 0.371221] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1113lb # [ 0.374165] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001114client # [ 0.371226] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1115lb # [ 0.374175] pci_bus 0000:00: root bus resource [bus 00-ff]1116lb # [ 0.374180] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1117client # [ 0.371301] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1118lb # [ 0.374185] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1119client # [ 0.371770] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1120lb # [ 0.374190] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1121client # [ 0.371958] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1122client # [ 0.371976] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1123lb # [ 0.374450] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1124client # [ 0.372007] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1125lb # [ 0.374897] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1126client # [ 0.372024] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1127lb # [ 0.375086] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1128client # [ 0.372525] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1129lb # [ 0.375103] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1130client # [ 0.372709] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1131lb # [ 0.375133] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1132worker1 # [ 0.411696] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1133client # [ 0.372725] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1134lb # [ 0.375150] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1135worker1 # [ 0.411901] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1136client # [ 0.372755] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1137lb # [ 0.375630] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1138worker1 # [ 0.411932] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1139client # [ 0.373208] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1140lb # [ 0.375815] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1141worker1 # [ 0.412409] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1142client # [ 0.373391] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1143lb # [ 0.375832] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1144worker1 # [ 0.412694] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1145client # [ 0.373408] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1146lb # [ 0.375861] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1147worker1 # [ 0.412712] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1148client # [ 0.373438] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1149lb # [ 0.376315] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1150worker1 # [ 0.412741] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1151client # [ 0.373916] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1152lb # [ 0.376520] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1153worker2 # [ 0.492179] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1154worker1 # [ 0.413198] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1155client # [ 0.374101] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1156lb # [ 0.376537] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1157worker1 # [ 0.413382] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1158client # [ 0.374117] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1159lb # [ 0.376567] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1160worker1 # [ 0.413398] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1161client # [ 0.374147] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1162worker1 # [ 0.413428] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1163lb # [ 0.377026] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1164client # [ 0.374167] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1165lb # [ 0.377211] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1166worker1 # [ 0.414018] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1167client # [ 0.374664] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1168lb # [ 0.377226] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1169worker1 # [ 0.414029] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1170client # [ 0.374861] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1171lb # [ 0.377256] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1172worker1 # [ 0.414035] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1173client # [ 0.374892] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1174lb # [ 0.377273] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1175worker1 # [ 0.414081] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1176client # [ 0.375371] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1177lb # [ 0.377753] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1178client # [ 0.375563] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1179worker1 # [ 0.414128] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1180lb # [ 0.377949] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1181client # [ 0.375593] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1182worker1 # [ 0.414176] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1183lb # [ 0.377981] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1184client # [ 0.375994] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1185worker1 # [ 0.414224] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1186lb # [ 0.378435] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1187worker2 # [ 0.495179] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1188client # [ 0.376176] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1189lb # [ 0.378624] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1190worker1 # [ 0.414272] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1191worker2 # [ 0.497347] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1192client # [ 0.376437] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1193lb # [ 0.378654] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1194worker2 # [ 0.499378] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1195worker1 # [ 0.414319] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1196client # [ 0.376625] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1197lb # [ 0.379044] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1198worker2 # [ 0.501395] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1199worker1 # [ 0.414367] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1200client # [ 0.376655] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1201lb # [ 0.379225] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1202worker1 # [ 0.414413] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1203client # [ 0.377104] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1204lb # [ 0.379521] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1205client # [ 0.377290] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1206worker1 # [ 0.414460] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1207lb # [ 0.379711] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1208client # [ 0.377320] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1209lb # [ 0.379741] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1210worker1 # [ 0.414532] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1211worker1 # [ 0.414579] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1212lb # [ 0.380193] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1213worker1 # [ 0.414601] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1214worker1 # [ 0.414622] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1215worker1 # [ 0.414644] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1216worker1 # [ 0.414666] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1217worker1 # [ 0.414687] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1218worker1 # [ 0.414709] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1219worker1 # [ 0.414735] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1220worker1 # [ 0.414757] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1221worker1 # [ 0.414779] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1222worker1 # [ 0.414802] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1223worker1 # [ 0.414824] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1224client # [ 0.421884] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1225worker1 # [ 0.414862] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1226client # [ 0.422106] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1227worker1 # [ 0.414884] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1228client # [ 0.422140] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1229worker1 # [ 0.414906] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1230worker1 # [ 0.414927] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1231client # [ 0.422663] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1232worker1 # [ 0.414948] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1233client # [ 0.422962] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1234worker2 # [ 0.519598] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1235worker1 # [ 0.414970] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1236client # [ 0.422984] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1237worker2 # [ 0.521377] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1238worker1 # [ 0.414997] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1239client # [ 0.423015] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1240worker1 # [ 0.415006] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1241client # [ 0.423515] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1242worker1 # [ 0.415011] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1243client # [ 0.423701] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1244worker2 # [ 0.525050] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1245client # [ 0.423718] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1246client # [ 0.423749] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1247client # [ 0.424334] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1248lb # [ 0.428483] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1249client # [ 0.424347] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1250lb # [ 0.428531] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1251client # [ 0.424353] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1252lb # [ 0.429075] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1253client # [ 0.424399] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1254lb # [ 0.429274] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1255client # [ 0.424447] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1256lb # [ 0.429305] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1257client # [ 0.424496] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1258lb # [ 0.429773] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1259lb # [ 0.430069] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1260client # [ 0.424545] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1261worker2 # [ 0.532161] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1262lb # [ 0.430087] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1263client # [ 0.424594] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1264lb # [ 0.430116] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1265worker1 # [ 0.459936] pci 0000:00:07.0: enabling device (0000 -> 0002)1266client # [ 0.424643] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1267lb # [ 0.430582] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1268client # [ 0.424691] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1269lb # [ 0.430768] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1270lb # [ 0.430784] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1271client # [ 0.424739] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1272lb # [ 0.430814] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1273client # [ 0.424787] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1274lb # [ 0.431416] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1275client # [ 0.424862] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1276lb # [ 0.431428] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1277client # [ 0.424909] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1278lb # [ 0.431433] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1279client # [ 0.424932] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1280lb # [ 0.431479] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1281client # [ 0.424955] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1282client # [ 0.424978] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1283lb # [ 0.431527] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1284worker2 # [ 0.538061] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1285client # [ 0.425001] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1286worker2 # [ 0.540358] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1287lb # [ 0.431575] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1288client # [ 0.425024] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1289lb # [ 0.431624] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1290client # [ 0.425047] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1291lb # [ 0.431672] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1292client # [ 0.425074] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1293client # [ 0.425097] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1294lb # [ 0.431720] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1295client # [ 0.425121] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1296lb # [ 0.431768] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1297client # [ 0.425144] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1298lb # [ 0.431814] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1299client # [ 0.425168] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1300lb # [ 0.431860] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1301client # [ 0.425191] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1302client # [ 0.425214] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1303lb # [ 0.431936] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1304client # [ 0.425236] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1305lb # [ 0.431983] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1306client # [ 0.425259] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1307lb # [ 0.432004] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1308client # [ 0.425282] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1309lb # [ 0.432026] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1310client # [ 0.425304] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1311lb # [ 0.432047] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1312client # [ 0.425333] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1313lb # [ 0.432069] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1314client # [ 0.425343] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1315lb # [ 0.432090] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1316client # [ 0.425347] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1317lb # [ 0.432112] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1318worker1 # [ 0.480921] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1319lb # [ 0.432137] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1320worker1 # [ 0.482991] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1321client # [ 0.464501] pci 0000:00:07.0: enabling device (0000 -> 0002)1322lb # [ 0.432159] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1323worker2 # [ 0.558374] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1324lb # [ 0.432181] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1325lb # [ 0.432203] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1326lb # [ 0.432225] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1327worker2 # [ 0.561015] msm_serial: driver initialized1328lb # [ 0.432246] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1329worker2 # [ 0.561163] SuperH (H)SCI(F) driver initialized1330worker2 # [ 0.561216] STM32 USART driver initialized1331lb # [ 0.432268] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1332lb # [ 0.432288] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1333lb # [ 0.432310] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1334lb # [ 0.432348] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1335lb # [ 0.432368] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1336lb # [ 0.432395] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1337lb # [ 0.432405] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1338lb # [ 0.432410] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1339lb # [ 0.433235] pci 0000:00:07.0: enabling device (0000 -> 0002)1340worker1 # [ 0.493201] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1341worker1 # [ 0.495142] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1342worker1 # [ 0.504915] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1343worker1 # [ 0.507101] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1344client # [ 0.489028] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1345lb # [ 0.495417] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1346client # [ 0.495222] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1347client # [ 0.497301] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1348worker1 # [ 0.517122] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1349worker1 # [ 0.519031] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1350worker2 # [ 0.592880] loop: module loaded1351worker2 # [ 0.593050] virtio_blk virtio2: 1/0/0 default/read/poll queues1352worker2 # [ 0.593804] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1353worker1 # [ 0.522432] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1354lb # [ 0.505700] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1355worker2 # [ 0.598492] megasas: 07.734.00.00-rc11356worker2 # [ 0.599244] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1357client # [ 0.507190] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1358client # [ 0.509379] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1359worker1 # [ 0.529391] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1360lb # [ 0.510117] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1361lb # [ 0.512039] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1362client # [ 0.520910] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1363worker2 # [ 0.609968] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001364worker2 # [ 0.610010] Intel/Sharp Extended Query Table at 0x00311365worker1 # [ 0.540560] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1366worker2 # [ 0.611739] Using buffer write method1367worker2 # [ 0.611801] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1368worker2 # [ 0.614743] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001369worker2 # [ 0.614764] Intel/Sharp Extended Query Table at 0x00311370worker2 # [ 0.616257] Using buffer write method1371lb # [ 0.525894] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1372worker2 # [ 0.616282] Concatenating MTD devices:1373worker2 # [ 0.616286] (0): "0.flash"1374lb # [ 0.528009] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1375worker2 # [ 0.616290] (1): "0.flash"1376client # [ 0.526965] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1377worker2 # [ 0.616312] into device "0.flash"1378client # [ 0.529006] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1379lb # [ 0.531287] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1380worker1 # [ 0.554328] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1381lb # [ 0.534214] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1382lb # [ 0.536103] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1383worker1 # [ 0.556784] msm_serial: driver initialized1384worker1 # [ 0.556913] SuperH (H)SCI(F) driver initialized1385worker1 # [ 0.556966] STM32 USART driver initialized1386client # [ 0.538988] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1387client # [ 0.541042] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1388client # [ 0.544823] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1389lb # [ 0.545751] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1390lb # [ 0.547990] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1391client # [ 0.563000] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1392client # [ 0.565606] msm_serial: driver initialized1393client # [ 0.565756] SuperH (H)SCI(F) driver initialized1394lb # [ 0.565602] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1395worker1 # [ 0.591436] loop: module loaded1396worker1 # [ 0.591629] virtio_blk virtio2: 1/0/0 default/read/poll queues1397worker1 # [ 0.592385] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1398lb # [ 0.573657] msm_serial: driver initialized1399lb # [ 0.573819] SuperH (H)SCI(F) driver initialized1400lb # [ 0.573876] STM32 USART driver initialized1401client # [ 0.573896] STM32 USART driver initialized1402worker1 # [ 0.599833] megasas: 07.734.00.00-rc11403worker1 # [ 0.600519] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1404worker1 # [ 0.602488] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001405worker1 # [ 0.602512] Intel/Sharp Extended Query Table at 0x00311406worker1 # [ 0.607499] Using buffer write method1407worker1 # [ 0.607563] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1408worker1 # [ 0.609476] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001409worker1 # [ 0.609499] Intel/Sharp Extended Query Table at 0x00311410worker1 # [ 0.611183] Using buffer write method1411worker1 # [ 0.623258] Concatenating MTD devices:1412worker1 # [ 0.623263] (0): "0.flash"1413client # [ 0.603067] loop: module loaded1414lb # [ 0.603143] loop: module loaded1415worker1 # [ 0.623267] (1): "0.flash"1416worker1 # [ 0.623273] into device "0.flash"1417client # [ 0.603259] virtio_blk virtio2: 1/0/0 default/read/poll queues1418lb # [ 0.603357] virtio_blk virtio2: 1/0/0 default/read/poll queues1419client # [ 0.604140] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1420lb # [ 0.604146] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1421client # [ 0.610390] megasas: 07.734.00.00-rc11422client # [ 0.611108] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1423client # [ 0.613108] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001424client # [ 0.613134] Intel/Sharp Extended Query Table at 0x00311425lb # [ 0.616971] megasas: 07.734.00.00-rc11426lb # [ 0.617725] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1427client # [ 0.623002] Using buffer write method1428lb # [ 0.619612] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001429client # [ 0.623072] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1430lb # [ 0.619639] Intel/Sharp Extended Query Table at 0x00311431lb # [ 0.621260] Using buffer write method1432client # [ 0.624721] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001433client # [ 0.624747] Intel/Sharp Extended Query Table at 0x00311434lb # [ 0.621321] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1435client # [ 0.629829] Using buffer write method1436client # [ 0.629851] Concatenating MTD devices:1437client # [ 0.629855] (0): "0.flash"1438lb # [ 0.623117] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001439client # [ 0.629859] (1): "0.flash"1440client # [ 0.629868] into device "0.flash"1441lb # [ 0.623140] Intel/Sharp Extended Query Table at 0x00311442lb # [ 0.641218] Using buffer write method1443lb # [ 0.641255] Concatenating MTD devices:1444lb # [ 0.641259] (0): "0.flash"1445lb # [ 0.641263] (1): "0.flash"1446lb # [ 0.641273] into device "0.flash"1447worker2 # [ 0.868862] Freeing initrd memory: 26944K1448worker2 # [ 0.874793] tun: Universal TUN/TAP device driver, 1.61449worker2 # [ 0.878594] thunder_xcv, ver 1.01450worker2 # [ 0.878637] thunder_bgx, ver 1.01451worker2 # [ 0.878659] nicpf, ver 1.01452worker2 # [ 0.879228] e1000: Intel(R) PRO/1000 Network Driver1453worker2 # [ 0.879236] e1000: Copyright (c) 1999-2006 Intel Corporation.1454worker2 # [ 0.879274] e1000e: Intel(R) PRO/1000 Network Driver1455worker2 # [ 0.879281] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1456worker2 # [ 0.879311] igb: Intel(R) Gigabit Ethernet Network Driver1457worker2 # [ 0.879317] igb: Copyright (c) 2007-2014 Intel Corporation.1458worker2 # [ 0.879340] igbvf: Intel(R) Gigabit Virtual Function Network Driver1459worker2 # [ 0.879345] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1460worker2 # [ 0.879479] sky2: driver version 1.301461worker2 # [ 0.881089] usbcore: registered new interface driver usb-storage1462worker2 # [ 0.881177] usbcore: registered new interface driver usbserial_generic1463worker2 # [ 0.881192] usbserial: USB Serial support registered for generic1464worker2 # [ 0.881793] hv_vmbus: registering driver hyperv_keyboard1465worker2 # [ 0.882705] ehci-pci 0000:00:07.0: EHCI Host Controller1466worker2 # [ 0.882730] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11467worker2 # [ 0.883051] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001468worker2 # [ 0.895507] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001469worker2 # [ 0.895818] hub 1-0:1.0: USB hub found1470worker2 # [ 0.895837] hub 1-0:1.0: 6 ports detected1471worker2 # [ 0.898644] rtc-pl031 9010000.pl031: registered as rtc01472worker2 # [ 0.898676] rtc-pl031 9010000.pl031: setting system clock to 2026-09-24T11:47:29 UTC (1790250449)1473worker2 # [ 0.898976] i2c_dev: i2c /dev entries driver1474worker2 # [ 0.904381] sdhci: Secure Digital Host Controller Interface driver1475worker2 # [ 0.904393] sdhci: Copyright(c) Pierre Ossman1476worker2 # [ 0.904659] Synopsys Designware Multimedia Card Interface Driver1477worker2 # [ 0.905036] sdhci-pltfm: SDHCI platform and OF driver helper1478worker2 # [ 0.909295] hid: raw HID events driver (C) Jiri Kosina1479worker2 # [ 0.909537] usbcore: registered new interface driver usbhid1480worker2 # [ 0.909545] usbhid: USB HID core driver1481worker2 # [ 0.912433] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1482worker2 # [ 0.915080] drop_monitor: Initializing network drop monitor service1483worker2 # [ 0.915220] NET: Registered PF_INET6 protocol family1484worker2 # [ 0.917140] Segment Routing with IPv61485worker2 # [ 0.917158] In-situ OAM (IOAM) with IPv61486worker2 # [ 0.917185] NET: Registered PF_PACKET protocol family1487worker2 # [ 0.919018] 9pnet: Installing 9P2000 support1488worker2 # [ 0.919063] Key type dns_resolver registered1489worker2 # [ 0.925617] registered taskstats version 11490worker2 # [ 0.925761] Loading compiled-in X.509 certificates1491worker2 # [ 0.934413] Demotion targets for Node 0: null1492worker2 # [ 0.934513] Key type .fscrypt registered1493worker2 # [ 0.934520] Key type fscrypt-provisioning registered1494worker2 # [ 0.934612] ima: No TPM chip found, activating TPM-bypass!1495worker2 # [ 0.934631] ima: Allocated hash algorithm: sha11496worker2 # [ 0.934654] ima: No architecture policies found1497worker2 # [ 0.938858] input: gpio-keys as /devices/platform/gpio-keys/input/input01498worker1 # [ 0.870065] Freeing initrd memory: 26976K1499worker1 # [ 0.876148] tun: Universal TUN/TAP device driver, 1.61500worker1 # [ 0.879843] thunder_xcv, ver 1.01501worker2 # [ 0.957060] clk: Disabling unused clocks1502worker1 # [ 0.879885] thunder_bgx, ver 1.01503worker1 # [ 0.879906] nicpf, ver 1.01504worker2 # [ 0.957083] PM: genpd: Disabling unused power domains1505worker1 # [ 0.880433] e1000: Intel(R) PRO/1000 Network Driver1506worker1 # [ 0.880440] e1000: Copyright (c) 1999-2006 Intel Corporation.1507worker1 # [ 0.880463] e1000e: Intel(R) PRO/1000 Network Driver1508worker1 # [ 0.880471] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1509worker1 # [ 0.880499] igb: Intel(R) Gigabit Ethernet Network Driver1510worker2 # [ 0.961264] Freeing unused kernel memory: 4736K1511worker1 # [ 0.880505] igb: Copyright (c) 2007-2014 Intel Corporation.1512worker2 # [ 0.961537] Run /init as init process1513worker1 # [ 0.880543] igbvf: Intel(R) Gigabit Virtual Function Network Driver1514worker1 # [ 0.880549] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1515worker1 # [ 0.880683] sky2: driver version 1.301516worker1 # [ 0.882205] usbcore: registered new interface driver usb-storage1517worker1 # [ 0.882292] usbcore: registered new interface driver usbserial_generic1518worker1 # [ 0.882305] usbserial: USB Serial support registered for generic1519worker1 # [ 0.882958] hv_vmbus: registering driver hyperv_keyboard1520worker1 # [ 0.883936] ehci-pci 0000:00:07.0: EHCI Host Controller1521worker1 # [ 0.883961] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11522worker1 # [ 0.884210] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001523worker1 # [ 0.896273] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001524worker1 # [ 0.896607] hub 1-0:1.0: USB hub found1525worker1 # [ 0.896637] hub 1-0:1.0: 6 ports detected1526client # [ 0.880834] Freeing initrd memory: 26972K1527worker1 # [ 0.899449] rtc-pl031 9010000.pl031: registered as rtc01528worker1 # [ 0.899477] rtc-pl031 9010000.pl031: setting system clock to 2026-09-24T11:47:29 UTC (1790250449)1529worker1 # [ 0.899773] i2c_dev: i2c /dev entries driver1530worker2 # [ 0.976229] systemd[1]: Successfully made /usr/ read-only.1531lb # [ 0.885018] Freeing initrd memory: 26948K1532client # [ 0.886806] tun: Universal TUN/TAP device driver, 1.61533worker1 # [ 0.904935] sdhci: Secure Digital Host Controller Interface driver1534worker1 # [ 0.904946] sdhci: Copyright(c) Pierre Ossman1535worker1 # [ 0.905206] Synopsys Designware Multimedia Card Interface Driver1536worker1 # [ 0.905564] sdhci-pltfm: SDHCI platform and OF driver helper1537worker1 # [ 0.907061] hid: raw HID events driver (C) Jiri Kosina1538lb # [ 0.891198] tun: Universal TUN/TAP device driver, 1.61539worker1 # [ 0.910652] usbcore: registered new interface driver usbhid1540worker1 # [ 0.910663] usbhid: USB HID core driver1541client # [ 0.890559] thunder_xcv, ver 1.01542client # [ 0.890601] thunder_bgx, ver 1.01543client # [ 0.890624] nicpf, ver 1.01544worker1 # [ 0.912934] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1545client # [ 0.891194] e1000: Intel(R) PRO/1000 Network Driver1546lb # [ 0.894980] thunder_xcv, ver 1.01547lb # [ 0.895027] thunder_bgx, ver 1.01548client # [ 0.891202] e1000: Copyright (c) 1999-2006 Intel Corporation.1549worker1 # [ 0.914447] drop_monitor: Initializing network drop monitor service1550lb # [ 0.895055] nicpf, ver 1.01551client # [ 0.891226] e1000e: Intel(R) PRO/1000 Network Driver1552worker1 # [ 0.914589] NET: Registered PF_INET6 protocol family1553lb # [ 0.895650] e1000: Intel(R) PRO/1000 Network Driver1554client # [ 0.891233] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1555worker1 # [ 0.917777] Segment Routing with IPv61556lb # [ 0.895658] e1000: Copyright (c) 1999-2006 Intel Corporation.1557client # [ 0.891263] igb: Intel(R) Gigabit Ethernet Network Driver1558worker1 # [ 0.917794] In-situ OAM (IOAM) with IPv61559lb # [ 0.895682] e1000e: Intel(R) PRO/1000 Network Driver1560client # [ 0.891268] igb: Copyright (c) 2007-2014 Intel Corporation.1561worker1 # [ 0.917821] NET: Registered PF_PACKET protocol family1562lb # [ 0.895690] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1563client # [ 0.891292] igbvf: Intel(R) Gigabit Virtual Function Network Driver1564lb # [ 0.895722] igb: Intel(R) Gigabit Ethernet Network Driver1565client # [ 0.891310] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1566lb # [ 0.895727] igb: Copyright (c) 2007-2014 Intel Corporation.1567client # [ 0.891462] sky2: driver version 1.301568lb # [ 0.895750] igbvf: Intel(R) Gigabit Virtual Function Network Driver1569client # [ 0.893016] usbcore: registered new interface driver usb-storage1570lb # [ 0.895756] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1571client # [ 0.893166] usbcore: registered new interface driver usbserial_generic1572lb # [ 0.895892] sky2: driver version 1.301573client # [ 0.893180] usbserial: USB Serial support registered for generic1574worker1 # [ 0.919594] 9pnet: Installing 9P2000 support1575worker1 # [ 0.919639] Key type dns_resolver registered1576client # [ 0.893765] hv_vmbus: registering driver hyperv_keyboard1577client # [ 0.894445] ehci-pci 0000:00:07.0: EHCI Host Controller1578client # [ 0.894470] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11579lb # [ 0.904802] usbcore: registered new interface driver usb-storage1580lb # [ 0.904864] usbcore: registered new interface driver usbserial_generic1581client # [ 0.894633] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001582lb # [ 0.904879] usbserial: USB Serial support registered for generic1583client # [ 0.906416] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001584worker1 # [ 0.926347] registered taskstats version 11585worker1 # [ 0.926489] Loading compiled-in X.509 certificates1586client # [ 0.906716] hub 1-0:1.0: USB hub found1587lb # [ 0.905504] hv_vmbus: registering driver hyperv_keyboard1588client # [ 0.906732] hub 1-0:1.0: 6 ports detected1589lb # [ 0.909742] ehci-pci 0000:00:07.0: EHCI Host Controller1590lb # [ 0.909770] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11591client # [ 0.910366] rtc-pl031 9010000.pl031: registered as rtc01592lb # [ 0.910041] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001593client # [ 0.910400] rtc-pl031 9010000.pl031: setting system clock to 2026-09-24T11:47:29 UTC (1790250449)1594client # [ 0.910702] i2c_dev: i2c /dev entries driver1595lb # [ 0.912736] rtc-pl031 9010000.pl031: registered as rtc01596lb # [ 0.912767] rtc-pl031 9010000.pl031: setting system clock to 2026-09-24T11:47:29 UTC (1790250449)1597lb # [ 0.913109] i2c_dev: i2c /dev entries driver1598worker1 # [ 0.935137] Demotion targets for Node 0: null1599worker1 # [ 0.935942] Key type .fscrypt registered1600worker1 # [ 0.935951] Key type fscrypt-provisioning registered1601worker1 # [ 0.936047] ima: No TPM chip found, activating TPM-bypass!1602client # [ 0.915969] sdhci: Secure Digital Host Controller Interface driver1603worker1 # [ 0.936067] ima: Allocated hash algorithm: sha11604client # [ 0.915981] sdhci: Copyright(c) Pierre Ossman1605worker1 # [ 0.936089] ima: No architecture policies found1606client # [ 0.916260] Synopsys Designware Multimedia Card Interface Driver1607client # [ 0.916636] sdhci-pltfm: SDHCI platform and OF driver helper1608worker1 # [ 0.939564] input: gpio-keys as /devices/platform/gpio-keys/input/input01609lb # [ 0.918572] sdhci: Secure Digital Host Controller Interface driver1610client # [ 0.920873] hid: raw HID events driver (C) Jiri Kosina1611lb # [ 0.918585] sdhci: Copyright(c) Pierre Ossman1612client # [ 0.921119] usbcore: registered new interface driver usbhid1613lb # [ 0.918857] Synopsys Designware Multimedia Card Interface Driver1614client # [ 0.921126] usbhid: USB HID core driver1615lb # [ 0.919236] sdhci-pltfm: SDHCI platform and OF driver helper1616lb # [ 0.920368] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001617lb # [ 0.920689] hub 1-0:1.0: USB hub found1618lb # [ 0.920751] hub 1-0:1.0: 6 ports detected1619client # [ 0.924063] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1620client # [ 0.925556] drop_monitor: Initializing network drop monitor service1621lb # [ 0.926158] hid: raw HID events driver (C) Jiri Kosina1622client # [ 0.925736] NET: Registered PF_INET6 protocol family1623lb # [ 0.926414] usbcore: registered new interface driver usbhid1624lb # [ 0.926422] usbhid: USB HID core driver1625client # [ 0.928824] Segment Routing with IPv61626client # [ 0.928844] In-situ OAM (IOAM) with IPv61627client # [ 0.928872] NET: Registered PF_PACKET protocol family1628lb # [ 0.929422] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1629lb # [ 0.930950] drop_monitor: Initializing network drop monitor service1630lb # [ 0.931082] NET: Registered PF_INET6 protocol family1631client # [ 0.930716] 9pnet: Installing 9P2000 support1632lb # [ 0.934143] Segment Routing with IPv61633client # [ 0.930763] Key type dns_resolver registered1634lb # [ 0.934164] In-situ OAM (IOAM) with IPv61635lb # [ 0.934196] NET: Registered PF_PACKET protocol family1636lb # [ 0.935904] 9pnet: Installing 9P2000 support1637lb # [ 0.935951] Key type dns_resolver registered1638client # [ 0.937331] registered taskstats version 11639client # [ 0.937474] Loading compiled-in X.509 certificates1640worker1 # [ 0.957295] clk: Disabling unused clocks1641worker1 # [ 0.957324] PM: genpd: Disabling unused power domains1642worker1 # [ 0.961436] Freeing unused kernel memory: 4736K1643worker1 # [ 0.961627] Run /init as init process1644lb # [ 0.942655] registered taskstats version 11645lb # [ 0.942796] Loading compiled-in X.509 certificates1646client # [ 0.946120] Demotion targets for Node 0: null1647client # [ 0.946228] Key type .fscrypt registered1648client # [ 0.946235] Key type fscrypt-provisioning registered1649client # [ 0.946327] ima: No TPM chip found, activating TPM-bypass!1650client # [ 0.946346] ima: Allocated hash algorithm: sha11651client # [ 0.946369] ima: No architecture policies found1652client # [ 0.950568] input: gpio-keys as /devices/platform/gpio-keys/input/input01653lb # [ 0.951597] Demotion targets for Node 0: null1654lb # [ 0.951711] Key type .fscrypt registered1655lb # [ 0.951717] Key type fscrypt-provisioning registered1656lb # [ 0.951816] ima: No TPM chip found, activating TPM-bypass!1657lb # [ 0.951835] ima: Allocated hash algorithm: sha11658lb # [ 0.951857] ima: No architecture policies found1659lb # [ 0.955898] input: gpio-keys as /devices/platform/gpio-keys/input/input01660worker1 # [ 0.977349] systemd[1]: Successfully made /usr/ read-only.1661client # [ 0.972087] clk: Disabling unused clocks1662client # [ 0.972108] PM: genpd: Disabling unused power domains1663lb # [ 0.975026] clk: Disabling unused clocks1664client # [ 0.976195] Freeing unused kernel memory: 4736K1665lb # [ 0.975050] PM: genpd: Disabling unused power domains1666client # [ 0.976403] Run /init as init process1667lb # [ 0.979195] Freeing unused kernel memory: 4736K1668lb # [ 0.979404] Run /init as init process1669client # [ 0.992019] systemd[1]: Successfully made /usr/ read-only.1670lb # [ 0.994569] systemd[1]: Successfully made /usr/ read-only.1671worker2 # [ 1.142019] usb 1-1: new high-speed USB device number 2 using ehci-pci1672worker1 # [ 1.143310] usb 1-1: new high-speed USB device number 2 using ehci-pci1673client # [ 1.153861] usb 1-1: new high-speed USB device number 2 using ehci-pci1674lb # [ 1.168357] usb 1-1: new high-speed USB device number 2 using ehci-pci1675worker2 # [ 1.294309] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11676worker2 # [ 1.311143] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1677worker2 # [ 1.323296] systemd[1]: Detected virtualization qemu.1678worker2 # [ 1.325463] systemd[1]: Detected architecture arm64.1679worker2 # [ 1.327442] systemd[1]: Running in initrd.1680worker2 # [ 1.330157] systemd[1]: Initializing machine ID from random generator.1681worker2 # [ 1.333093] systemd[1]: Hostname set to <worker2>.1682worker1 # [ 1.295637] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11683worker2 # [ 1.378234] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01684worker1 # [ 1.311755] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1685worker1 # [ 1.321213] systemd[1]: Detected virtualization qemu.1686worker1 # [ 1.323431] systemd[1]: Detected architecture arm64.1687worker1 # [ 1.325406] systemd[1]: Running in initrd.1688worker1 # [ 1.328117] systemd[1]: Initializing machine ID from random generator.1689client # [ 1.306229] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11690worker1 # [ 1.331119] systemd[1]: Hostname set to <worker1>.1691lb # [ 1.320941] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11692client # [ 1.326713] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1693client # [ 1.339050] systemd[1]: Detected virtualization qemu.1694lb # [ 1.329434] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1695client # [ 1.341159] systemd[1]: Detected architecture arm64.1696client # [ 1.343139] systemd[1]: Running in initrd.1697lb # [ 1.341942] systemd[1]: Detected virtualization qemu.1698lb # [ 1.342046] systemd[1]: Detected architecture arm64.1699client # [ 1.345844] systemd[1]: Initializing machine ID from random generator.1700lb # [ 1.342070] systemd[1]: Running in initrd.1701client # [ 1.348882] systemd[1]: Hostname set to <client>.1702lb # [ 1.343062] systemd[1]: Initializing machine ID from random generator.1703lb # [ 1.343369] systemd[1]: Hostname set to <lb>.1704worker1 # [ 1.379492] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01705client # [ 1.394082] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01706lb # [ 1.400627] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01707worker2 # [ 1.502017] usb 1-2: new high-speed USB device number 3 using ehci-pci1708worker1 # [ 1.499234] usb 1-2: new high-speed USB device number 3 using ehci-pci1709client # [ 1.517822] usb 1-2: new high-speed USB device number 3 using ehci-pci1710lb # [ 1.524401] usb 1-2: new high-speed USB device number 3 using ehci-pci1711worker2 # [ 1.639382] systemd[1]: bpf-restrict-fs: LSM BPF program attached1712worker2 # [ 1.664966] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21713worker2 # [ 1.670656] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01714worker1 # [ 1.649136] systemd[1]: bpf-restrict-fs: LSM BPF program attached1715worker1 # [ 1.660969] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21716worker1 # [ 1.666261] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01717worker2 # [ 1.755015] systemd[1]: Queued start job for default target Initrd Default Target.1718client # [ 1.663787] systemd[1]: bpf-restrict-fs: LSM BPF program attached1719worker2 # [ 1.764397] systemd[1]: Created slice Slice /system/modprobe.1720lb # [ 1.670472] systemd[1]: bpf-restrict-fs: LSM BPF program attached1721worker2 # [ 1.765602] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1722worker2 # [ 1.767052] systemd[1]: Expecting device /dev/disk/by-label/nixos...1723worker2 # [ 1.767089] systemd[1]: Reached target Path Units.1724worker2 # [ 1.767110] systemd[1]: Reached target Slice Units.1725worker2 # [ 1.767129] systemd[1]: Reached target Swaps.1726worker2 # [ 1.767149] systemd[1]: Reached target Timer Units.1727worker2 # [ 1.767361] systemd[1]: Listening on D-Bus System Message Bus Socket.1728worker2 # [ 1.767522] systemd[1]: Listening on Journal Socket (/dev/log).1729worker2 # [ 1.767663] systemd[1]: Listening on Journal Sockets.1730worker2 # [ 1.767795] systemd[1]: Listening on udev Control Socket.1731worker2 # [ 1.767908] systemd[1]: Listening on udev Kernel Socket.1732worker2 # [ 1.767931] systemd[1]: Reached target Socket Units.1733client # [ 1.680849] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21734worker2 # [ 1.779070] systemd[1]: Starting Create List of Static Device Nodes...1735worker2 # [ 1.780206] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1736lb # [ 1.687672] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21737client # [ 1.690189] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01738worker2 # [ 1.790129] systemd[1]: Mounting Kernel Configuration File System...1739lb # [ 1.694556] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01740worker2 # [ 1.798217] systemd[1]: Starting Journal Service...1741worker2 # [ 1.822390] systemd[1]: Starting Load Kernel Modules...1742worker2 # [ 1.823359] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1743worker1 # [ 1.757821] systemd[1]: Queued start job for default target Initrd Default Target.1744worker1 # [ 1.769605] systemd[1]: Created slice Slice /system/modprobe.1745worker1 # [ 1.770760] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1746worker1 # [ 1.772113] systemd[1]: Expecting device /dev/disk/by-label/nixos...1747worker2 # [ 1.845619] systemd-journald[72]: Collecting audit messages is disabled.1748worker1 # [ 1.773164] systemd[1]: Reached target Path Units.1749worker1 # [ 1.773948] systemd[1]: Reached target Slice Units.1750worker1 # [ 1.774744] systemd[1]: Reached target Swaps.1751worker1 # [ 1.775512] systemd[1]: Reached target Timer Units.1752worker1 # [ 1.776493] systemd[1]: Listening on D-Bus System Message Bus Socket.1753worker1 # [ 1.777673] systemd[1]: Listening on Journal Socket (/dev/log).1754worker1 # [ 1.778759] systemd[1]: Listening on Journal Sockets.1755worker1 # [ 1.779802] systemd[1]: Listening on udev Control Socket.1756worker1 # [ 1.779923] systemd[1]: Listening on udev Kernel Socket.1757worker1 # [ 1.779945] systemd[1]: Reached target Socket Units.1758worker1 # [ 1.783982] systemd[1]: Starting Create List of Static Device Nodes...1759worker2 # [ 1.854292] systemd[1]: Starting Coldplug All udev Devices...1760worker1 # [ 1.785103] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1761worker1 # [ 1.792556] systemd[1]: Mounting Kernel Configuration File System...1762client # [ 1.775497] systemd[1]: Queued start job for default target Initrd Default Target.1763lb # [ 1.776851] systemd[1]: Queued start job for default target Initrd Default Target.1764worker2 # [ 1.870088] systemd[1]: Finished Create List of Static Device Nodes.1765worker2 # [ 1.871052] systemd[1]: Mounted Kernel Configuration File System.1766worker2 # [ 1.873892] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1767worker1 # [ 1.803603] systemd[1]: Starting Journal Service...1768client # [ 1.784364] systemd[1]: Created slice Slice /system/modprobe.1769client # [ 1.785613] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1770client # [ 1.787046] systemd[1]: Expecting device /dev/disk/by-label/nixos...1771client # [ 1.788097] systemd[1]: Reached target Path Units.1772lb # [ 1.788130] systemd[1]: Created slice Slice /system/modprobe.1773client # [ 1.788902] systemd[1]: Reached target Slice Units.1774client # [ 1.789715] systemd[1]: Reached target Swaps.1775lb # [ 1.789413] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1776client # [ 1.790530] systemd[1]: Reached target Timer Units.1777lb # [ 1.790698] systemd[1]: Expecting device /dev/disk/by-label/nixos...1778lb # [ 1.791748] systemd[1]: Reached target Path Units.1779client # [ 1.791579] systemd[1]: Listening on D-Bus System Message Bus Socket.1780lb # [ 1.792600] systemd[1]: Reached target Slice Units.1781client # [ 1.792823] systemd[1]: Listening on Journal Socket (/dev/log).1782lb # [ 1.793429] systemd[1]: Reached target Swaps.1783client # [ 1.793996] systemd[1]: Listening on Journal Sockets.1784lb # [ 1.794267] systemd[1]: Reached target Timer Units.1785client # [ 1.794991] systemd[1]: Listening on udev Control Socket.1786lb # [ 1.795267] systemd[1]: Listening on D-Bus System Message Bus Socket.1787client # [ 1.796036] systemd[1]: Listening on udev Kernel Socket.1788lb # [ 1.796550] systemd[1]: Listening on Journal Socket (/dev/log).1789client # [ 1.796929] systemd[1]: Reached target Socket Units.1790lb # [ 1.797646] systemd[1]: Listening on Journal Sockets.1791lb # [ 1.798616] systemd[1]: Listening on udev Control Socket.1792lb # [ 1.799626] systemd[1]: Listening on udev Kernel Socket.1793client # [ 1.799631] systemd[1]: Starting Create List of Static Device Nodes...1794client # [ 1.800817] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1795lb # [ 1.800570] systemd[1]: Reached target Socket Units.1796lb # [ 1.803013] systemd[1]: Starting Create List of Static Device Nodes...1797lb # [ 1.804140] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1798worker1 # [ 1.827466] systemd[1]: Starting Load Kernel Modules...1799worker1 # [ 1.828421] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1800client # [ 1.810018] systemd[1]: Mounting Kernel Configuration File System...1801lb # [ 1.812510] systemd[1]: Mounting Kernel Configuration File System...1802worker2 # [ 1.905766] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1803client # [ 1.818049] systemd[1]: Starting Journal Service...1804lb # [ 1.820636] systemd[1]: Starting Journal Service...1805worker1 # [ 1.847457] systemd[1]: Starting Coldplug All udev Devices...1806worker1 # [ 1.850456] systemd-journald[72]: Collecting audit messages is disabled.1807worker2 # [ 1.926060] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1808worker2 # [ 1.932164] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01809client # [ 1.842025] systemd[1]: Starting Load Kernel Modules...1810worker2 # [ 1.932424] [drm] features: -virgl +edid -resource_blob -host_visible1811worker2 # [ 1.932434] [drm] features: -context_init1812worker2 # [ 1.933142] [drm] number of scanouts: 11813client # [ 1.842937] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1814worker2 # [ 1.933160] [drm] number of cap sets: 01815worker1 # [ 1.871382] systemd[1]: Finished Create List of Static Device Nodes.1816worker2 # [ 1.946675] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1817worker1 # [ 1.875550] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1818worker2 # [ 1.949328] systemd[1]: Starting Create Static Device Nodes in /dev...1819worker1 # [ 1.876754] systemd[1]: Mounted Kernel Configuration File System.1820lb # [ 1.856522] systemd[1]: Starting Load Kernel Modules...1821lb # [ 1.856657] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1822client # [ 1.858102] systemd[1]: Starting Coldplug All udev Devices...1823worker2 # [ 1.954340] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1824worker2 # [ 1.954355] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01825lb # [ 1.872511] systemd[1]: Starting Coldplug All udev Devices...1826client # [ 1.874488] systemd-journald[72]: Collecting audit messages is disabled.1827client # [ 1.878174] systemd[1]: Finished Create List of Static Device Nodes.1828client # [ 1.878845] systemd[1]: Mounted Kernel Configuration File System.1829lb # [ 1.882527] systemd-journald[72]: Collecting audit messages is disabled.1830lb # [ 1.888535] systemd[1]: Finished Create List of Static Device Nodes.1831lb # [ 1.889255] systemd[1]: Mounted Kernel Configuration File System.1832worker2 # [ 1.978171] Console: switching to colour frame buffer device 160x501833worker1 # [ 1.911010] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1834client # [ 1.899059] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1835lb # [ 1.899219] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1836worker1 # [ 1.923331] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1837worker1 # [ 1.932514] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01838worker1 # [ 1.932756] [drm] features: -virgl +edid -resource_blob -host_visible1839worker2 # [ 2.006519] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1840worker1 # [ 1.932766] [drm] features: -context_init1841worker1 # [ 1.933479] [drm] number of scanouts: 11842worker1 # [ 1.933497] [drm] number of cap sets: 01843client # [ 1.916814] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1844worker2 # [ 2.009864] systemd[1]: Finished Create Static Device Nodes in /dev.1845worker2 # [ 2.010145] systemd[1]: Reached target Preparation for Local File Systems.1846worker2 # [ 2.010180] systemd[1]: Reached target Local File Systems.1847lb # [ 1.920596] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1848worker2 # [ 2.015558] systemd[1]: Starting Rule-based Manager for Device Events and Files...1849lb # [ 1.924437] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1850worker1 # [ 1.951938] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1851lb # [ 1.930771] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01852lb # [ 1.931000] [drm] features: -virgl +edid -resource_blob -host_visible1853lb # [ 1.931010] [drm] features: -context_init1854lb # [ 1.931741] [drm] number of scanouts: 11855lb # [ 1.931760] [drm] number of cap sets: 01856worker2 # [ 2.011681] systemd-modules-load[74]: Inserted module 'dm_mod'1857client # [ 1.930101] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1858client # [ 1.936234] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01859client # [ 1.936466] [drm] features: -virgl +edid -resource_blob -host_visible1860client # [ 1.936476] [drm] features: -context_init1861worker1 # [ 1.955545] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1862client # [ 1.937170] [drm] number of scanouts: 11863worker1 # [ 1.955563] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01864client # [ 1.937189] [drm] number of cap sets: 01865worker1 # [ 1.961330] systemd[1]: Starting Create Static Device Nodes in /dev...1866worker2 # [ 2.034093] systemd[1]: Finished Load Kernel Modules.1867worker2 # [ 2.034365] systemd[1]: Started Journal Service.1868worker2 # [ 2.021417] systemd-modules-load[74]: Module 'virtio_balloon' is built in1869lb # [ 1.949002] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1870lb # [ 1.951577] systemd[1]: Starting Create Static Device Nodes in /dev...1871worker2 # [ 2.028399] systemd-modules-load[74]: Module 'virtio_console' is built in1872client # [ 1.954439] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1873worker2 # [ 2.032371] systemd-modules-load[74]: Inserted module 'virtio_gpu'1874client # [ 1.958154] systemd[1]: Starting Create Static Device Nodes in /dev...1875worker2 # [ 2.033493] systemd-modules-load[74]: Module 'virtio_rng' is built in1876worker2 # [ 2.034557] systemd[1]: Starting Apply Kernel Variables...1877worker2 # [ 2.040261] systemd[1]: Starting Create System Files and Directories...1878lb # [ 1.964770] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1879lb # [ 1.964789] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01880worker1 # [ 1.987514] Console: switching to colour frame buffer device 160x501881client # [ 1.970161] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1882worker1 # [ 1.994145] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1883client # [ 1.970177] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01884lb # [ 1.988649] Console: switching to colour frame buffer device 160x501885client # [ 1.990081] Console: switching to colour frame buffer device 160x501886worker1 # [ 2.020662] systemd[1]: Finished Load Kernel Modules.1887worker1 # [ 2.024096] systemd[1]: Starting Apply Kernel Variables...1888lb # [ 2.009063] systemd[1]: Finished Create Static Device Nodes in /dev.1889lb # [ 2.010300] systemd[1]: Reached target Preparation for Local File Systems.1890lb # [ 2.011210] systemd[1]: Reached target Local File Systems.1891worker1 # [ 2.021846] systemd-modules-load[74]: Inserted module 'dm_mod'1892worker2 # [ 2.093045] systemd[1]: Finished Apply Kernel Variables.1893worker1 # [ 2.038254] systemd[1]: Finished Create Static Device Nodes in /dev.1894lb # [ 2.017860] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1895client # [ 2.018474] systemd[1]: Finished Create Static Device Nodes in /dev.1896client # [ 2.018851] systemd[1]: Reached target Preparation for Local File Systems.1897client # [ 2.018875] systemd[1]: Reached target Local File Systems.1898lb # [ 2.021506] systemd[1]: Starting Rule-based Manager for Device Events and Files...1899client # [ 2.023138] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1900worker1 # [ 2.043637] systemd[1]: Started Journal Service.1901worker1 # [ 2.031769] systemd-modules-load[74]: Module 'virtio_balloon' is built in1902client # [ 2.027160] systemd[1]: Starting Rule-based Manager for Device Events and Files...1903worker2 # [ 2.106294] systemd[1]: Finished Create System Files and Directories.1904worker1 # [ 2.037940] systemd-modules-load[74]: Module 'virtio_console' is built in1905worker1 # [ 2.039053] systemd-modules-load[74]: Inserted module 'virtio_gpu'1906worker2 # [ 2.113877] systemd-udevd[78]: Using default interface naming scheme 'v261'.1907lb # [ 2.040896] systemd[1]: Finished Load Kernel Modules.1908client # [ 2.029857] systemd-modules-load[73]: Inserted module 'dm_mod'1909worker1 # [ 2.052260] systemd-modules-load[74]: Module 'virtio_rng' is built in1910worker1 # [ 2.053387] systemd[1]: Reached target Preparation for Local File Systems.1911client # [ 2.048728] systemd[1]: Finished Load Kernel Modules.1912worker1 # [ 2.055638] systemd[1]: Reached target Local File Systems.1913client # [ 2.049025] systemd[1]: Started Journal Service.1914client # [ 2.036281] systemd-modules-load[73]: Module 'virtio_balloon' is built in1915worker1 # [ 2.060419] systemd[1]: Starting Create System Files and Directories...1916lb # [ 2.048897] systemd[1]: Starting Apply Kernel Variables...1917client # [ 2.040428] systemd-modules-load[73]: Module 'virtio_console' is built in1918worker1 # [ 2.064440] systemd[1]: Starting Rule-based Manager for Device Events and Files...1919client # [ 2.044296] systemd-modules-load[73]: Inserted module 'virtio_gpu'1920client # [ 2.046728] systemd-modules-load[73]: Module 'virtio_rng' is built in1921worker1 # [ 2.069005] systemd[1]: Finished Apply Kernel Variables.1922worker2 # [ 2.141854] systemd[1]: Started Rule-based Manager for Device Events and Files.1923client # [ 2.052393] systemd[1]: Starting Apply Kernel Variables...1924client # [ 2.056581] systemd[1]: Starting Create System Files and Directories...1925lb # [ 2.072526] systemd[1]: Started Journal Service.1926lb # [ 2.059593] systemd-modules-load[73]: Inserted module 'dm_mod'1927lb # [ 2.060824] systemd-modules-load[73]: Module 'virtio_balloon' is built in1928lb # [ 2.061939] systemd-modules-load[73]: Module 'virtio_console' is built in1929lb # [ 2.063015] systemd-modules-load[73]: Inserted module 'virtio_gpu'1930worker1 # [ 2.102657] systemd[1]: Finished Create System Files and Directories.1931lb # [ 2.088142] systemd-modules-load[73]: Module 'virtio_rng' is built in1932lb # [ 2.089465] systemd[1]: Starting Create System Files and Directories...1933lb # [ 2.096691] systemd[1]: Finished Apply Kernel Variables.1934worker2 # [ 2.193542] systemd[1]: Starting Virtual Console Setup...1935worker1 # [ 2.129286] systemd-udevd[81]: Using default interface naming scheme 'v261'.1936client # [ 2.109155] systemd[1]: Finished Apply Kernel Variables.1937lb # [ 2.114176] systemd[1]: Finished Create System Files and Directories.1938client # [ 2.113595] systemd[1]: Finished Create System Files and Directories.1939lb # [ 2.121610] systemd-udevd[79]: Using default interface naming scheme 'v261'.1940client # [ 2.125347] systemd-udevd[78]: Using default interface naming scheme 'v261'.1941worker1 # [ 2.157578] systemd[1]: Started Rule-based Manager for Device Events and Files.1942client # [ 2.149943] systemd[1]: Started Rule-based Manager for Device Events and Files.1943lb # [ 2.152737] systemd[1]: Started Rule-based Manager for Device Events and Files.1944worker2 # [ 2.248506] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1945worker2 # [ 2.252037] systemd[1]: Finished Virtual Console Setup.1946worker1 # [ 2.213586] systemd[1]: Starting Virtual Console Setup...1947lb # [ 2.212114] systemd[1]: Starting Virtual Console Setup...1948client # [ 2.212109] systemd[1]: Starting Virtual Console Setup...1949worker1 # [ 2.265337] systemd-vconsole-setup[102]: Configuration of first virtual console was skipped, ignoring remaining ones.1950worker1 # [ 2.268050] systemd[1]: Finished Virtual Console Setup.1951lb # [ 2.260530] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1952client # [ 2.260475] systemd-vconsole-setup[100]: Configuration of first virtual console was skipped, ignoring remaining ones.1953lb # [ 2.263743] systemd[1]: Finished Virtual Console Setup.1954client # [ 2.263976] systemd[1]: Finished Virtual Console Setup.1955worker2 # [ 2.877595] systemd[1]: Finished Coldplug All udev Devices.1956worker2 # [ 2.878546] systemd[1]: Reached target System Initialization.1957worker2 # [ 2.879386] systemd[1]: Reached target Basic System.1958worker1 # [ 2.863386] systemd[1]: Finished Coldplug All udev Devices.1959worker1 # [ 2.864404] systemd[1]: Reached target System Initialization.1960worker1 # [ 2.865215] systemd[1]: Reached target Basic System.1961client # [ 2.855241] systemd[1]: Finished Coldplug All udev Devices.1962client # [ 2.856281] systemd[1]: Reached target System Initialization.1963client # [ 2.857098] systemd[1]: Reached target Basic System.1964lb # [ 2.896740] systemd[1]: Finished Coldplug All udev Devices.1965lb # [ 2.900714] systemd[1]: Reached target System Initialization.1966lb # [ 2.901619] systemd[1]: Reached target Basic System.1967worker2 # [ 3.037909] (udev-worker)[95]: Network interface NamePolicy= disabled on kernel command line.1968worker2 # [ 3.057429] (udev-worker)[90]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1969worker2 # [ 3.064201] (udev-worker)[90]: Network interface NamePolicy= disabled on kernel command line.1970client # [ 2.991933] (udev-worker)[103]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1971worker1 # [ 3.016768] (udev-worker)[99]: Network interface NamePolicy= disabled on kernel command line.1972worker1 # [ 3.042653] (udev-worker)[101]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1973worker1 # [ 3.048293] (udev-worker)[101]: Network interface NamePolicy= disabled on kernel command line.1974client # [ 3.032946] (udev-worker)[103]: Network interface NamePolicy= disabled on kernel command line.1975worker2 # [ 3.142759] systemd[1]: Found device /dev/disk/by-label/nixos.1976client # [ 3.050853] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line.1977worker2 # [ 3.147065] systemd[1]: Reached target Initrd Root Device.1978worker2 # [ 3.151904] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1979lb # [ 3.060581] (udev-worker)[96]: Network interface NamePolicy= disabled on kernel command line.1980lb # [ 3.068113] (udev-worker)[93]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1981lb # [ 3.108536] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line.1982worker2 # [ 3.200545] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks1983worker2 # [ 3.205895] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1984client # [ 3.117200] systemd[1]: Found device /dev/disk/by-label/nixos.1985worker2 # [ 3.210986] systemd[1]: Mounting /sysroot...1986worker1 # [ 3.140146] systemd[1]: Found device /dev/disk/by-label/nixos.1987worker1 # [ 3.142044] systemd[1]: Reached target Initrd Root Device.1988worker1 # [ 3.145315] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1989client # [ 3.125364] systemd[1]: Reached target Initrd Root Device.1990client # [ 3.130813] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1991worker2 # [ 3.270152] EXT4-fs (vda): mounted filesystem 000d82cb-86e1-4e34-961e-58a6d4ea511a r/w with ordered data mode. Quota mode: none.1992worker2 # [ 3.259533] systemd[1]: Mounted /sysroot.1993worker2 # [ 3.261366] systemd[1]: Reached target Initrd Root File System.1994worker2 # [ 3.265381] systemd[1]: Starting Mountpoints Configured in the Real Root...1995worker1 # [ 3.197216] systemd-fsck[111]: nixos: clean, 12/65536 files, 13019/262144 blocks1996lb # [ 3.180632] systemd[1]: Found device /dev/disk/by-label/nixos.1997client # [ 3.179805] systemd-fsck[109]: nixos: clean, 12/65536 files, 13019/262144 blocks1998lb # [ 3.184155] systemd[1]: Reached target Initrd Root Device.1999worker1 # [ 3.205578] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.2000client # [ 3.187161] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.
loading 1,996 hidden lines…
3997lb: must succeed: curl -sf localhost:9901/clusters3998lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)3999worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4000worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4001client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&24002worker2 # [ 57.427304] nix-grpc-daemon[784]: ts=2026-09-24T11:48:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:602844003client # don't know how to build these paths:4004client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4005worker2 # [ 57.438064] nix-grpc-daemon[784]: ts=2026-09-24T11:48:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602844006worker2 # [ 57.446017] nix-grpc-daemon[854]: time=2026-09-24T11:48:26.064Z level=INFO msg="All 2 paths already cached"4007worker1 # [ 57.392210] nix-grpc-daemon[488]: ts=2026-09-24T11:48:26Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=04008worker1 # [ 57.392645] nix-grpc-daemon[488]: ts=2026-09-24T11:48:26Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.4:50051 clients=14009worker2 # [ 57.479779] nix-grpc-daemon[784]: ts=2026-09-24T11:48:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602844010worker2 # [ 57.495957] nix-grpc-daemon[784]: ts=2026-09-24T11:48:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602844011worker2 # [ 57.511377] nix-grpc-daemon[854]: time=2026-09-24T11:48:26.130Z level=INFO msg="All 2 paths already cached"4012client # copying 1 paths to 'lb:50051'...4013client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'...4014worker2 # [ 57.524148] nix-grpc-daemon[784]: ts=2026-09-24T11:48:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:602844015worker2 # [ 57.541026] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4016worker2 # [ 57.704124] nix-daemon[841]: accepted connection from pid 970, user nix-grpc-daemon (trusted)4017worker2 # [ 57.722221] nix-daemon[841]: reaped child process 973, status = succeeded4018lb # [ 57.639655] niks3-server[1007]: 2026/09/24 11:48:26 INFO Received push request method=POST path=/api/pushes4019worker2 # [ 57.742184] nix-grpc-daemon[854]: time=2026-09-24T11:48:26.360Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4020worker2 # [ 57.743891] nix-grpc-daemon[854]: time=2026-09-24T11:48:26.360Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)"4021lb # [ 57.677076] niks3-server[1007]: 2026/09/24 11:48:26 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst4022worker2 # [ 57.784110] nix-grpc-daemon[854]: time=2026-09-24T11:48:26.403Z level=INFO msg="Uploading 1 narinfos"4023lb # [ 57.692734] niks3-server[1007]: 2026/09/24 11:48:26 INFO Received sign narinfos request method=POST path=/api/pushes/7/sign4024lb # [ 57.700520] niks3-server[1007]: 2026/09/24 11:48:26 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls4025lb # [ 57.708595] niks3-server[1007]: 2026/09/24 11:48:26 INFO Signed narinfos id=7 count=14026lb # [ 57.714926] niks3-server[1007]: 2026/09/24 11:48:26 INFO Received complete push request method=POST path=/api/pushes/7/complete4027worker2 # [ 57.841140] nix-grpc-daemon[854]: time=2026-09-24T11:48:26.460Z level=INFO msg="Upload complete. (282ms)"4028lb # [ 57.750769] niks3-server[1007]: 2026/09/24 11:48:26 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo4029worker2 # [ 57.854627] nix-grpc-daemon[784]: ts=2026-09-24T11:48:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=0 paths=1 nar_bytes_in=7524030worker2 # [ 57.872877] nix-grpc-daemon[784]: ts=2026-09-24T11:48:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602844031worker2 # [ 57.884285] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4032worker2 # [ 57.891428] nix-daemon[841]: reaped child process 968, status = succeeded4033client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051...4034worker2 # [ 57.900583] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4035client # farm-top> worker2: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4036client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4037lb # [ 59.025407] envoy[907]: conn peer=[2001:db8:1::1]:34734 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=94038lb # [ 59.030639] envoy[907]: conn peer=[2001:db8:1::1]:34744 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9549 tx=4431 ms=59964039lb # [ 59.035681] envoy[907]: conn peer=[2001:db8:1::1]:44576 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=480 ms=774040client # farm-top> LOG-farm-top-t1b4041worker2 # [ 59.406294] nix-daemon[841]: reaped child process 977, status = succeeded4042worker2 # [ 59.475115] nix-daemon[841]: accepted connection from pid 993, user nix-grpc-daemon (trusted)4043worker2 # [ 59.500904] nix-daemon[841]: reaped child process 996, status = succeeded4044lb # [ 59.407155] niks3-server[1007]: 2026/09/24 11:48:28 INFO Received push request method=POST path=/api/pushes4045worker2 # [ 59.509488] nix-grpc-daemon[854]: time=2026-09-24T11:48:28.128Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4046worker2 # [ 59.514026] nix-grpc-daemon[854]: time=2026-09-24T11:48:28.130Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)"4047lb # [ 59.454550] niks3-server[1007]: 2026/09/24 11:48:28 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls4048worker2 # [ 59.559512] nix-grpc-daemon[854]: time=2026-09-24T11:48:28.178Z level=INFO msg="Uploading 1 narinfos"4049lb # [ 59.470672] niks3-server[1007]: 2026/09/24 11:48:28 INFO Received sign narinfos request method=POST path=/api/pushes/8/sign4050lb # [ 59.477685] niks3-server[1007]: 2026/09/24 11:48:28 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst4051lb # [ 59.490273] niks3-server[1007]: 2026/09/24 11:48:28 INFO Signed narinfos id=8 count=14052lb # [ 59.496691] niks3-server[1007]: 2026/09/24 11:48:28 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4053lb # [ 59.499671] niks3-server[1007]: 2026/09/24 11:48:28 INFO Received complete push request method=POST path=/api/pushes/8/complete4054worker2 # [ 59.595813] nix-grpc-daemon[854]: time=2026-09-24T11:48:28.214Z level=INFO msg="Upload complete. (281ms)"4055lb # [ 59.505873] niks3-server[1007]: 2026/09/24 11:48:28 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo4056worker2 # [ 59.609099] nix-grpc-daemon[784]: ts=2026-09-24T11:48:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=14057worker2 # [ 59.625332] nix-daemon[841]: reaped child process 975, status = succeeded4058client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.44 seconds)4059client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath4060client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.18 seconds)4061lb: must succeed: curl -sf localhost:9901/clusters4062lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4063lb: must succeed: curl -sf localhost:9901/clusters4064lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4065worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4066worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4067(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.95 seconds)4068subtest: outputs read back through any worker carry the cache signature4069client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&24070client # don't know how to build these paths:4071client # /nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv4072worker2 # [ 60.224862] nix-grpc-daemon[784]: ts=2026-09-24T11:48:28Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:602844073worker2 # [ 60.233406] nix-grpc-daemon[784]: ts=2026-09-24T11:48:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602844074worker1 # [ 60.184674] nix-grpc-daemon[488]: ts=2026-09-24T11:48:28Z level=debug event=want client=4 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv cached=04075worker1 # [ 60.192798] nix-grpc-daemon[488]: ts=2026-09-24T11:48:28Z level=debug event=assigned drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker=192.168.1.3:50051 clients=14076worker1 # [ 60.199805] nix-grpc-daemon[488]: ts=2026-09-24T11:48:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764077worker1 # [ 60.215993] nix-grpc-daemon[488]: ts=2026-09-24T11:48:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357764078client # copying 1 paths to 'lb:50051'...4079client # copying path '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' to 'lb:50051'...4080worker1 # [ 60.233290] nix-grpc-daemon[488]: ts=2026-09-24T11:48:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357764081worker1 # [ 60.249066] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4082worker1 # [ 60.277668] nix-grpc-daemon[488]: ts=2026-09-24T11:48:28Z level=info event=niks3_push_started pid=9044083worker1 # [ 60.447821] nix-grpc-daemon[904]: time=2026-09-24T11:48:29.062Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""4084worker1 # [ 60.561121] nix-daemon[854]: accepted connection from pid 917, user nix-grpc-daemon (trusted)4085worker1 # [ 60.581554] nix-daemon[854]: reaped child process 920, status = succeeded4086lb # [ 60.567420] niks3-server[1007]: 2026/09/24 11:48:29 INFO Received push request method=POST path=/api/pushes4087worker1 # [ 60.597919] nix-grpc-daemon[904]: time=2026-09-24T11:48:29.213Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4088worker1 # [ 60.599454] nix-grpc-daemon[904]: time=2026-09-24T11:48:29.213Z level=INFO msg="Uploading 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv (512B)"4089lb # [ 60.615163] niks3-server[1007]: 2026/09/24 11:48:29 INFO Received sign narinfos request method=POST path=/api/pushes/9/sign4090worker1 # [ 60.647948] nix-grpc-daemon[904]: time=2026-09-24T11:48:29.262Z level=INFO msg="Uploading 1 narinfos"4091lb # [ 60.628449] niks3-server[1007]: 2026/09/24 11:48:29 INFO Signed narinfos id=9 count=14092lb # [ 60.631444] niks3-server[1007]: 2026/09/24 11:48:29 INFO Registered completed upload object_key=nar/13mhmmy2qay13imbbdi287vhryz1hwnyg0w6agd9s0ph37wnsqll.nar.zst4093lb # [ 60.647961] niks3-server[1007]: 2026/09/24 11:48:29 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.ls4094worker1 # [ 60.682044] nix-grpc-daemon[904]: time=2026-09-24T11:48:29.297Z level=INFO msg="Upload complete. (230ms)"4095lb # [ 60.661858] niks3-server[1007]: 2026/09/24 11:48:29 INFO Received complete push request method=POST path=/api/pushes/9/complete4096lb # [ 60.667592] niks3-server[1007]: 2026/09/24 11:48:29 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.narinfo4097worker1 # [ 60.693477] nix-grpc-daemon[488]: ts=2026-09-24T11:48:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=0 paths=1 nar_bytes_in=5124098worker1 # [ 60.707398] nix-grpc-daemon[488]: ts=2026-09-24T11:48:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764099worker1 # [ 60.713978] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4100worker1 # [ 60.718939] nix-daemon[854]: reaped child process 902, status = succeeded4101worker1 # [ 60.730432] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4102client # building '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' on lb:50051...4103client # farm-a> node-a: building 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv4104client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4105client # farm-a> LOG-farm-a-sig4106worker1 # [ 61.798098] nix-daemon[854]: reaped child process 924, status = succeeded4107worker1 # [ 61.909554] nix-daemon[854]: accepted connection from pid 940, user nix-grpc-daemon (trusted)4108worker1 # [ 61.931324] nix-daemon[854]: reaped child process 943, status = succeeded4109lb # [ 61.919562] niks3-server[1007]: 2026/09/24 11:48:30 INFO Received push request method=POST path=/api/pushes4110worker1 # [ 61.952441] nix-grpc-daemon[904]: time=2026-09-24T11:48:30.567Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4111worker1 # [ 61.954037] nix-grpc-daemon[904]: time=2026-09-24T11:48:30.567Z level=INFO msg="Uploading 2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a (128B)"4112lb # [ 61.968861] niks3-server[1007]: 2026/09/24 11:48:30 INFO Registered completed upload object_key=log/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv4113lb # [ 61.981780] niks3-server[1007]: 2026/09/24 11:48:30 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst4114worker1 # [ 62.011067] nix-grpc-daemon[904]: time=2026-09-24T11:48:30.626Z level=INFO msg="Uploading 1 narinfos"4115lb # [ 61.993584] niks3-server[1007]: 2026/09/24 11:48:30 INFO Received sign narinfos request method=POST path=/api/pushes/10/sign4116lb # [ 62.001315] niks3-server[1007]: 2026/09/24 11:48:30 INFO Signed narinfos id=10 count=14117lb # [ 62.006351] niks3-server[1007]: 2026/09/24 11:48:30 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.ls4118lb # [ 62.015404] niks3-server[1007]: 2026/09/24 11:48:30 INFO Received complete push request method=POST path=/api/pushes/10/complete4119lb # [ 62.023250] niks3-server[1007]: 2026/09/24 11:48:30 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.narinfo4120worker1 # [ 62.043537] nix-grpc-daemon[904]: time=2026-09-24T11:48:30.659Z level=INFO msg="Upload complete. (298ms)"4121worker1 # [ 62.063782] nix-grpc-daemon[488]: ts=2026-09-24T11:48:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=1 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv assign_id=5 outputs=14122worker1 # [ 62.088410] nix-grpc-daemon[488]: ts=2026-09-24T11:48:30Z level=debug event=want client=4 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv cached=04123worker1 # [ 62.092439] nix-grpc-daemon[488]: ts=2026-09-24T11:48:30Z level=debug event=assigned drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker=192.168.1.3:50051 clients=14124worker1 # [ 62.100503] nix-grpc-daemon[488]: ts=2026-09-24T11:48:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764125worker1 # [ 62.106026] nix-daemon[854]: reaped child process 922, status = succeeded4126worker1 # [ 62.107926] nix-grpc-daemon[488]: ts=2026-09-24T11:48:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357764127worker1 # [ 62.116939] nix-grpc-daemon[904]: time=2026-09-24T11:48:30.732Z level=INFO msg="All 1 paths already cached"4128client # copying 1 paths to 'lb:50051'...4129client # copying path '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' to 'lb:50051'...4130worker1 # [ 62.127154] nix-grpc-daemon[488]: ts=2026-09-24T11:48:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357764131worker1 # [ 62.139407] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4132worker1 # [ 62.284158] nix-daemon[854]: accepted connection from pid 949, user nix-grpc-daemon (trusted)4133worker1 # [ 62.306344] nix-daemon[854]: reaped child process 952, status = succeeded4134lb # [ 62.291968] niks3-server[1007]: 2026/09/24 11:48:30 INFO Received push request method=POST path=/api/pushes4135worker1 # [ 62.323930] nix-grpc-daemon[904]: time=2026-09-24T11:48:30.939Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4136worker1 # [ 62.325652] nix-grpc-daemon[904]: time=2026-09-24T11:48:30.939Z level=INFO msg="Uploading z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv (624B)"4137lb # [ 62.331628] niks3-server[1007]: 2026/09/24 11:48:30 INFO Registered completed upload object_key=nar/0cnhkgsbmxv5hj7zw3fa59qfxikw6251mlhz2gz1v6pgn8c8shwp.nar.zst4138worker1 # [ 62.367031] nix-grpc-daemon[904]: time=2026-09-24T11:48:30.982Z level=INFO msg="Uploading 1 narinfos"4139lb # [ 62.351638] niks3-server[1007]: 2026/09/24 11:48:30 INFO Received sign narinfos request method=POST path=/api/pushes/11/sign4140lb # [ 62.359217] niks3-server[1007]: 2026/09/24 11:48:30 INFO Signed narinfos id=11 count=14141lb # [ 62.364925] niks3-server[1007]: 2026/09/24 11:48:30 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.ls4142lb # [ 62.377001] niks3-server[1007]: 2026/09/24 11:48:30 INFO Received complete push request method=POST path=/api/pushes/11/complete4143lb # [ 62.386531] niks3-server[1007]: 2026/09/24 11:48:30 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.narinfo4144worker1 # [ 62.413003] nix-grpc-daemon[904]: time=2026-09-24T11:48:31.028Z level=INFO msg="Upload complete. (259ms)"4145worker1 # [ 62.425641] nix-grpc-daemon[488]: ts=2026-09-24T11:48:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=0 paths=1 nar_bytes_in=6244146worker1 # [ 62.444821] nix-grpc-daemon[488]: ts=2026-09-24T11:48:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764147worker1 # [ 62.453239] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4148worker1 # [ 62.459590] nix-daemon[854]: reaped child process 947, status = succeeded4149client # building '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' on lb:50051...4150client # farm-b> node-a: building z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv4151worker1 # [ 62.475009] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4152client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4153client # farm-b> LOG-farm-b-sig4154worker1 # [ 63.390601] nix-daemon[854]: reaped child process 956, status = succeeded4155worker1 # [ 63.465457] nix-daemon[854]: accepted connection from pid 972, user nix-grpc-daemon (trusted)4156worker1 # [ 63.489069] nix-daemon[854]: reaped child process 975, status = succeeded4157lb # [ 63.473864] niks3-server[1007]: 2026/09/24 11:48:32 INFO Received push request method=POST path=/api/pushes4158worker1 # [ 63.504441] nix-grpc-daemon[904]: time=2026-09-24T11:48:32.119Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4159worker1 # [ 63.506137] nix-grpc-daemon[904]: time=2026-09-24T11:48:32.119Z level=INFO msg="Uploading h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b (176B)"4160worker1 # [ 63.545456] nix-grpc-daemon[904]: time=2026-09-24T11:48:32.160Z level=INFO msg="Uploading 1 narinfos"4161lb # [ 63.512722] niks3-server[1007]: 2026/09/24 11:48:32 INFO Received sign narinfos request method=POST path=/api/pushes/12/sign4162lb # [ 63.530898] niks3-server[1007]: 2026/09/24 11:48:32 INFO Signed narinfos id=12 count=14163lb # [ 63.540089] niks3-server[1007]: 2026/09/24 11:48:32 INFO Registered completed upload object_key=log/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv4164lb # [ 63.550769] niks3-server[1007]: 2026/09/24 11:48:32 INFO Registered completed upload object_key=nar/0f75gpfkjj0g9xi2113yvvpdw0l16spc2f1xik1ifnzfbrvx4pfk.nar.zst4165worker1 # [ 63.587036] nix-grpc-daemon[904]: time=2026-09-24T11:48:32.202Z level=INFO msg="Upload complete. (294ms)"4166lb # [ 63.567214] niks3-server[1007]: 2026/09/24 11:48:32 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.ls4167lb # [ 63.571403] niks3-server[1007]: 2026/09/24 11:48:32 INFO Received complete push request method=POST path=/api/pushes/12/complete4168lb # [ 63.576790] niks3-server[1007]: 2026/09/24 11:48:32 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.narinfo4169worker1 # [ 63.601354] nix-grpc-daemon[488]: ts=2026-09-24T11:48:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=1 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv assign_id=6 outputs=14170worker1 # [ 63.625490] nix-grpc-daemon[488]: ts=2026-09-24T11:48:32Z level=debug event=want client=4 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv cached=04171worker1 # [ 63.633376] nix-grpc-daemon[488]: ts=2026-09-24T11:48:32Z level=debug event=assigned drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker=192.168.1.3:50051 clients=14172worker1 # [ 63.639578] nix-grpc-daemon[488]: ts=2026-09-24T11:48:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764173worker1 # [ 63.645043] nix-daemon[854]: reaped child process 954, status = succeeded4174worker1 # [ 63.649677] nix-grpc-daemon[488]: ts=2026-09-24T11:48:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357764175worker1 # [ 63.686057] nix-grpc-daemon[904]: time=2026-09-24T11:48:32.301Z level=INFO msg="All 2 paths already cached"4176client # copying 1 paths to 'lb:50051'...4177client # copying path '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' to 'lb:50051'...4178worker1 # [ 63.696942] nix-grpc-daemon[488]: ts=2026-09-24T11:48:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357764179worker1 # [ 63.714074] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4180worker1 # [ 63.867041] nix-daemon[854]: accepted connection from pid 981, user nix-grpc-daemon (trusted)4181worker1 # [ 63.888107] nix-daemon[854]: reaped child process 984, status = succeeded4182lb # [ 63.873505] niks3-server[1007]: 2026/09/24 11:48:32 INFO Received push request method=POST path=/api/pushes4183worker1 # [ 63.905317] nix-grpc-daemon[904]: time=2026-09-24T11:48:32.520Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4184worker1 # [ 63.906999] nix-grpc-daemon[904]: time=2026-09-24T11:48:32.520Z level=INFO msg="Uploading knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv (752B)"4185lb # [ 63.908629] niks3-server[1007]: 2026/09/24 11:48:32 INFO Registered completed upload object_key=nar/0lpagyrngav67x80d1hq0njjlkmqdkvqawraz8dsp36ag50v6iwf.nar.zst4186worker1 # [ 63.937406] nix-grpc-daemon[904]: time=2026-09-24T11:48:32.552Z level=INFO msg="Uploading 1 narinfos"4187lb # [ 63.923533] niks3-server[1007]: 2026/09/24 11:48:32 INFO Received sign narinfos request method=POST path=/api/pushes/13/sign4188lb # [ 63.940203] niks3-server[1007]: 2026/09/24 11:48:32 INFO Signed narinfos id=13 count=14189lb # [ 63.943208] niks3-server[1007]: 2026/09/24 11:48:32 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.ls4190worker1 # [ 63.973627] nix-grpc-daemon[904]: time=2026-09-24T11:48:32.589Z level=INFO msg="Upload complete. (242ms)"4191lb # [ 63.953598] niks3-server[1007]: 2026/09/24 11:48:32 INFO Received complete push request method=POST path=/api/pushes/13/complete4192lb # [ 63.958165] niks3-server[1007]: 2026/09/24 11:48:32 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.narinfo4193worker1 # [ 63.984359] nix-grpc-daemon[488]: ts=2026-09-24T11:48:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=0 paths=1 nar_bytes_in=7524194worker1 # [ 64.004220] nix-grpc-daemon[488]: ts=2026-09-24T11:48:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764195worker1 # [ 64.012257] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4196worker1 # [ 64.018170] nix-daemon[854]: reaped child process 979, status = succeeded4197client # building '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' on lb:50051...4198worker1 # [ 64.029585] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4199client # farm-top> node-a: building knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv4200client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4201client # farm-top> LOG-farm-top-sig4202worker1 # [ 65.341942] nix-daemon[854]: reaped child process 988, status = succeeded4203worker1 # [ 65.418285] nix-daemon[854]: accepted connection from pid 1004, user nix-grpc-daemon (trusted)4204worker1 # [ 65.443526] nix-daemon[854]: reaped child process 1007, status = succeeded4205lb # [ 65.427288] niks3-server[1007]: 2026/09/24 11:48:34 INFO Received push request method=POST path=/api/pushes4206worker1 # [ 65.459347] nix-grpc-daemon[904]: time=2026-09-24T11:48:34.073Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4207worker1 # [ 65.462098] nix-grpc-daemon[904]: time=2026-09-24T11:48:34.073Z level=INFO msg="Uploading 4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top (232B)"4208lb # [ 65.466529] niks3-server[1007]: 2026/09/24 11:48:34 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.ls4209worker1 # [ 65.501211] nix-grpc-daemon[904]: time=2026-09-24T11:48:34.116Z level=INFO msg="Uploading 1 narinfos"4210lb # [ 65.480678] niks3-server[1007]: 2026/09/24 11:48:34 INFO Received sign narinfos request method=POST path=/api/pushes/14/sign4211lb # [ 65.496393] niks3-server[1007]: 2026/09/24 11:48:34 INFO Signed narinfos id=14 count=14212lb # [ 65.499740] niks3-server[1007]: 2026/09/24 11:48:34 INFO Registered completed upload object_key=nar/07ar364rwqqniwi2nkdpqlgv7ws7702n5lqz703ackfmssrh40v8.nar.zst4213lb # [ 65.514207] niks3-server[1007]: 2026/09/24 11:48:34 INFO Registered completed upload object_key=log/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv4214lb # [ 65.527111] niks3-server[1007]: 2026/09/24 11:48:34 INFO Received complete push request method=POST path=/api/pushes/14/complete4215lb # [ 65.531590] niks3-server[1007]: 2026/09/24 11:48:34 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.narinfo4216worker1 # [ 65.547578] nix-grpc-daemon[904]: time=2026-09-24T11:48:34.163Z level=INFO msg="Upload complete. (305ms)"4217worker1 # [ 65.564727] nix-grpc-daemon[488]: ts=2026-09-24T11:48:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=1 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv assign_id=7 outputs=14218worker1 # [ 65.582230] nix-daemon[854]: reaped child process 986, status = succeeded4219client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2, in 5.67 seconds)4220client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath4221client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath, in 0.14 seconds)4222client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top4223worker1 # [ 65.879989] nix-grpc-daemon[488]: ts=2026-09-24T11:48:34Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:423404224worker1 # [ 65.887927] nix-grpc-daemon[488]: ts=2026-09-24T11:48:34Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:423404225client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.16 seconds)4226client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top4227worker2 # [ 66.110534] nix-grpc-daemon[784]: ts=2026-09-24T11:48:34Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:461524228client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked):4229client # /nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a4230client # /nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b4231client # /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top4232worker2 # [ 66.132816] nix-grpc-daemon[784]: ts=2026-09-24T11:48:34Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:461524233worker2 # [ 66.142207] nix-grpc-daemon[784]: copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'http://lb:5751'...4234worker2 # [ 66.163621] nix-grpc-daemon[784]: copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'http://lb:5751'...4235worker2 # [ 66.180192] nix-grpc-daemon[784]: copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'http://lb:5751'...4236client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.25 seconds)4237client: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top4238client # finding garbage collector roots...4239client # removing stale temporary roots file "/nix/var/nix/temproots/831"4240client # removing stale temporary roots file "/nix/var/nix/temproots/851"4241client # removing stale temporary roots file "/nix/var/nix/temproots/887"4242client # removing stale temporary roots file "/nix/var/nix/temproots/810"4243client # removing stale temporary roots file "/nix/var/nix/temproots/872"4244client # removing stale temporary roots file "/nix/var/nix/temproots/864"4245client # removing stale temporary roots file "/nix/var/nix/temproots/843"4246client # removing stale temporary roots file "/nix/var/nix/temproots/796"4247client # deleting specified paths...4248client # deleting '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top'4249client # deleting unused links...4250client # note: hard linking is currently saving 0.0 KiB4251worker2 # [ 66.556675] nix-grpc-daemon[784]: ts=2026-09-24T11:48:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:602844252worker1 # [ 66.497319] nix-grpc-daemon[488]: ts=2026-09-24T11:48:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357764253worker1 # [ 66.509779] nix-grpc-daemon[904]: time=2026-09-24T11:48:35.124Z level=INFO msg="All 1 paths already cached"4254worker2 # [ 66.589742] nix-grpc-daemon[784]: ts=2026-09-24T11:48:35Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:602844255worker2 # [ 66.599518] nix-grpc-daemon[784]: ts=2026-09-24T11:48:35Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:602844256client # copying 3 paths...4257worker1 # [ 66.544221] nix-grpc-daemon[488]: ts=2026-09-24T11:48:35Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:357764258client # copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'grpc://lb:50051'...4259worker2 # [ 66.646330] nix-grpc-daemon[784]: ts=2026-09-24T11:48:35Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:602844260worker2 # [ 66.654719] nix-grpc-daemon[784]: ts=2026-09-24T11:48:35Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=0 paths=1 nar_bytes_out=2324261worker2 # [ 66.664053] nix-grpc-daemon[784]: ts=2026-09-24T11:48:35Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:602844262worker2 # [ 66.671293] nix-grpc-daemon[784]: ts=2026-09-24T11:48:35Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=0 paths=1 nar_bytes_out=1764263worker1 # [ 66.610369] nix-grpc-daemon[488]: ts=2026-09-24T11:48:35Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:357764264worker1 # [ 66.618975] nix-grpc-daemon[488]: ts=2026-09-24T11:48:35Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=0 paths=1 nar_bytes_out=1284265client # copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'grpc://lb:50051'...4266client # copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'grpc://lb:50051'...4267client: (finished: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.51 seconds)4268(finished: subtest: outputs read back through any worker carry the cache signature, in 6.73 seconds)4269subtest: two clients wanting the same drv build it once4270client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup4271client # [ 66.678839] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup.4272client # Running as unit: dedup1.service; invocation ID: b6469cffc4d240f799ca4f125b27449f4273client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 0.05 seconds)4274client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup4275worker1 # [ 66.879276] nix-grpc-daemon[488]: ts=2026-09-24T11:48:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:357764276client # [ 66.864867] nix[940]: don't know how to build these paths:4277client # [ 66.866698] nix[940]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4278worker1 # [ 66.895839] nix-grpc-daemon[488]: ts=2026-09-24T11:48:35Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04279worker1 # [ 66.899190] nix-grpc-daemon[488]: ts=2026-09-24T11:48:35Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.4:50051 clients=14280worker2 # [ 66.970373] nix-grpc-daemon[784]: ts=2026-09-24T11:48:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602844281worker2 # [ 66.985173] nix-grpc-daemon[784]: ts=2026-09-24T11:48:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602844282client # [ 66.905986] nix[940]: copying 1 paths to 'lb:50051'...4283client # [ 66.907227] nix[940]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'...4284worker2 # [ 67.000124] nix-grpc-daemon[784]: ts=2026-09-24T11:48:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:602844285worker2 # [ 67.015277] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4286worker2 # [ 67.191437] nix-daemon[841]: accepted connection from pid 1003, user nix-grpc-daemon (trusted)4287worker2 # [ 67.213043] nix-daemon[841]: reaped child process 1006, status = succeeded4288lb # [ 67.126765] niks3-server[1007]: 2026/09/24 11:48:35 INFO Received push request method=POST path=/api/pushes4289worker2 # [ 67.234056] nix-grpc-daemon[854]: time=2026-09-24T11:48:35.851Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4290worker2 # [ 67.235760] nix-grpc-daemon[854]: time=2026-09-24T11:48:35.851Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)"4291lb # [ 67.172991] niks3-server[1007]: 2026/09/24 11:48:35 INFO Received sign narinfos request method=POST path=/api/pushes/15/sign4292worker2 # [ 67.276319] nix-grpc-daemon[854]: time=2026-09-24T11:48:35.894Z level=INFO msg="Uploading 1 narinfos"4293lb # [ 67.189477] niks3-server[1007]: 2026/09/24 11:48:35 INFO Signed narinfos id=15 count=14294lb # [ 67.199107] niks3-server[1007]: 2026/09/24 11:48:35 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls4295lb # [ 67.209326] niks3-server[1007]: 2026/09/24 11:48:35 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst4296lb # [ 67.224932] niks3-server[1007]: 2026/09/24 11:48:35 INFO Received complete push request method=POST path=/api/pushes/15/complete4297worker2 # [ 67.322690] nix-grpc-daemon[854]: time=2026-09-24T11:48:35.941Z level=INFO msg="Upload complete. (288ms)"4298lb # [ 67.231953] niks3-server[1007]: 2026/09/24 11:48:35 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo4299worker2 # [ 67.337003] nix-grpc-daemon[784]: ts=2026-09-24T11:48:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=0 paths=1 nar_bytes_in=4324300worker2 # [ 67.352403] nix-grpc-daemon[784]: ts=2026-09-24T11:48:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602844301worker2 # [ 67.364166] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4302worker2 # [ 67.372463] nix-daemon[841]: reaped child process 1001, status = succeeded4303client # [ 67.287452] nix[940]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4304worker2 # [ 67.384310] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4305client # [ 67.763860] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup.4306client # Running as unit: dedup2.service; invocation ID: ee050d7c83ca468ebf3046701b871b874307client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 1.10 seconds)4308worker1 # [ 68.021734] nix-grpc-daemon[488]: ts=2026-09-24T11:48:36Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:357764309client # [ 68.007952] nix[956]: don't know how to build these paths:4310client # [ 68.009272] nix[956]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4311worker1 # [ 68.037658] nix-grpc-daemon[488]: ts=2026-09-24T11:48:36Z level=debug event=want client=6 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04312worker2 # [ 68.114692] nix-grpc-daemon[784]: ts=2026-09-24T11:48:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602844313client # [ 68.029492] nix[956]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4314lb # [ 69.025066] envoy[907]: conn peer=[2001:db8:1::1]:44584 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4198 tx=1921 ms=21924315client: must succeed: sleep 24316lb # [ 69.030215] envoy[907]: conn peer=[2001:db8:1::1]:44592 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9502 tx=4399 ms=54234317lb # [ 69.030343] envoy[907]: conn peer=[2001:db8:1::1]:59124 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=384 tx=409 ms=804318lb # [ 69.030464] envoy[907]: conn peer=[2001:db8:1::1]:59118 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1051 tx=2887 ms=1684319lb # [ 69.030583] envoy[907]: conn peer=[2001:db8:1::1]:59128 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=375 ms=644320lb # [ 69.030698] envoy[907]: conn peer=[2001:db8:1::1]:59130 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=330 ms=474321client: (finished: must succeed: sleep 2, in 2.05 seconds)4322client: waiting for success: ! systemctl is-active dedup1 dedup24323worker2 # [ 71.417117] nix-daemon[841]: reaped child process 1010, status = succeeded4324worker2 # [ 71.509732] nix-daemon[841]: accepted connection from pid 1040, user nix-grpc-daemon (trusted)4325worker2 # [ 71.535765] nix-daemon[841]: reaped child process 1043, status = succeeded4326lb # [ 71.443333] niks3-server[1007]: 2026/09/24 11:48:40 INFO Received push request method=POST path=/api/pushes4327worker2 # [ 71.548488] nix-grpc-daemon[854]: time=2026-09-24T11:48:40.166Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4328worker2 # [ 71.550060] nix-grpc-daemon[854]: time=2026-09-24T11:48:40.166Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)"4329lb # [ 71.497851] niks3-server[1007]: 2026/09/24 11:48:40 INFO Received sign narinfos request method=POST path=/api/pushes/16/sign4330worker2 # [ 71.599301] nix-grpc-daemon[854]: time=2026-09-24T11:48:40.218Z level=INFO msg="Uploading 1 narinfos"4331lb # [ 71.513623] niks3-server[1007]: 2026/09/24 11:48:40 INFO Signed narinfos id=16 count=14332lb # [ 71.525854] niks3-server[1007]: 2026/09/24 11:48:40 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4333lb # [ 71.536442] niks3-server[1007]: 2026/09/24 11:48:40 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4334lb # [ 71.553117] niks3-server[1007]: 2026/09/24 11:48:40 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls4335lb # [ 71.562285] niks3-server[1007]: 2026/09/24 11:48:40 INFO Received complete push request method=POST path=/api/pushes/16/complete4336lb # [ 71.565944] niks3-server[1007]: 2026/09/24 11:48:40 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo4337worker2 # [ 71.658577] nix-grpc-daemon[854]: time=2026-09-24T11:48:40.277Z level=INFO msg="Upload complete. (259ms)"4338worker2 # [ 71.673531] nix-grpc-daemon[784]: ts=2026-09-24T11:48:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=14339worker2 # [ 71.692593] nix-grpc-daemon[784]: ts=2026-09-24T11:48:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=14340worker2 # [ 71.700620] nix-daemon[841]: reaped child process 1008, status = succeeded4341client # [ 71.638999] systemd[1]: dedup1.service: Deactivated successfully.4342client # [ 71.642775] systemd[1]: dedup1.service: Consumed 169ms CPU time over 4.963s wall clock time, 39.5M memory peak, 7.3K incoming IP traffic, 8.4K outgoing IP traffic.4343client # [ 71.652591] systemd[1]: dedup2.service: Deactivated successfully.4344client # [ 71.657461] systemd[1]: dedup2.service: Consumed 212ms CPU time over 3.888s wall clock time, 33.2M memory peak, 5.9K incoming IP traffic, 5.6K outgoing IP traffic.4345client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.13 seconds)4346client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4347client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.06 seconds)4348worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4349worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4350worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4351worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4352worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4353worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4354worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4355worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4356(finished: subtest: two clients wanting the same drv build it once, in 6.11 seconds)4357subtest: --timeout and --max-silent-time reach the builder4358worker1 # [ 72.999870] nix-grpc-daemon[488]: ts=2026-09-24T11:48:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:357764359worker1 # [ 73.017085] nix-grpc-daemon[488]: ts=2026-09-24T11:48:41Z level=debug event=want client=7 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv cached=04360worker1 # [ 73.024560] nix-grpc-daemon[488]: ts=2026-09-24T11:48:41Z level=debug event=assigned drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv worker=192.168.1.3:50051 clients=14361worker1 # [ 73.034002] nix-grpc-daemon[488]: ts=2026-09-24T11:48:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764362worker1 # [ 73.045302] nix-grpc-daemon[488]: ts=2026-09-24T11:48:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357764363worker1 # [ 73.060665] nix-grpc-daemon[488]: ts=2026-09-24T11:48:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357764364worker1 # [ 73.076623] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4365worker1 # [ 73.236623] nix-daemon[854]: accepted connection from pid 1038, user nix-grpc-daemon (trusted)4366worker1 # [ 73.255370] nix-daemon[854]: reaped child process 1041, status = succeeded4367lb # [ 73.243737] niks3-server[1007]: 2026/09/24 11:48:41 INFO Received push request method=POST path=/api/pushes4368worker1 # [ 73.275037] nix-grpc-daemon[904]: time=2026-09-24T11:48:41.889Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4369worker1 # [ 73.276678] nix-grpc-daemon[904]: time=2026-09-24T11:48:41.890Z level=INFO msg="Uploading sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv (424B)"4370lb # [ 73.265889] niks3-server[1007]: 2026/09/24 11:48:41 INFO Registered completed upload object_key=nar/1m12xmj6yzmb9y2j8ly0yvh5fyzy1nqk7ajhqwr4wn3b5h0bsch5.nar.zst4371worker1 # [ 73.292270] nix-grpc-daemon[904]: time=2026-09-24T11:48:41.907Z level=INFO msg="Uploading 1 narinfos"4372lb # [ 73.272260] niks3-server[1007]: 2026/09/24 11:48:41 INFO Received sign narinfos request method=POST path=/api/pushes/17/sign4373lb # [ 73.274332] niks3-server[1007]: 2026/09/24 11:48:41 INFO Signed narinfos id=17 count=14374lb # [ 73.281256] niks3-server[1007]: 2026/09/24 11:48:41 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.ls4375lb # [ 73.289046] niks3-server[1007]: 2026/09/24 11:48:41 INFO Received complete push request method=POST path=/api/pushes/17/complete4376worker1 # [ 73.317543] nix-grpc-daemon[904]: time=2026-09-24T11:48:41.933Z level=INFO msg="Upload complete. (225ms)"4377lb # [ 73.297056] niks3-server[1007]: 2026/09/24 11:48:41 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.narinfo4378worker1 # [ 73.324243] nix-grpc-daemon[488]: ts=2026-09-24T11:48:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=0 paths=1 nar_bytes_in=4244379worker1 # [ 73.332671] nix-grpc-daemon[488]: ts=2026-09-24T11:48:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764380worker1 # [ 73.342479] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4381worker1 # [ 73.347154] nix-daemon[854]: reaped child process 1036, status = succeeded4382worker1 # [ 73.360613] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4383lb # [ 79.025587] envoy[907]: conn peer=[2001:db8:1::1]:59144 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1430 tx=1197 ms=35994384lb # [ 79.030607] envoy[907]: conn peer=[2001:db8:1::1]:59136 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3018 tx=1541 ms=47374385worker1 # [ 79.374735] nix-grpc-daemon[488]: ts=2026-09-24T11:48:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=6 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv assign_id=9 outputs=04386worker1 # [ 79.401132] nix-daemon[854]: reaped child process 1043, status = succeeded4387worker1 # [ 79.402668] nix-daemon[854]: reaped child process 1045, status = succeeded4388worker1 # [ 79.613086] nix-grpc-daemon[488]: ts=2026-09-24T11:48:48Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:357764389worker1 # [ 79.625929] nix-grpc-daemon[488]: ts=2026-09-24T11:48:48Z level=debug event=want client=8 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv cached=04390worker2 # [ 79.701353] nix-grpc-daemon[784]: ts=2026-09-24T11:48:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602844391worker1 # [ 79.632289] nix-grpc-daemon[488]: ts=2026-09-24T11:48:48Z level=debug event=assigned drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv worker=192.168.1.4:50051 clients=14392worker2 # [ 79.718094] nix-grpc-daemon[784]: ts=2026-09-24T11:48:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602844393worker2 # [ 79.728946] nix-grpc-daemon[784]: ts=2026-09-24T11:48:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:602844394worker2 # [ 79.744006] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4395worker2 # [ 79.910462] nix-daemon[841]: accepted connection from pid 1056, user nix-grpc-daemon (trusted)4396worker2 # [ 79.927552] nix-daemon[841]: reaped child process 1059, status = succeeded4397lb # [ 79.844841] niks3-server[1007]: 2026/09/24 11:48:48 INFO Received push request method=POST path=/api/pushes4398worker2 # [ 79.948409] nix-grpc-daemon[854]: time=2026-09-24T11:48:48.566Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4399worker2 # [ 79.950029] nix-grpc-daemon[854]: time=2026-09-24T11:48:48.567Z level=INFO msg="Uploading pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv (424B)"4400lb # [ 79.880733] niks3-server[1007]: 2026/09/24 11:48:48 INFO Received sign narinfos request method=POST path=/api/pushes/18/sign4401worker2 # [ 79.981182] nix-grpc-daemon[854]: time=2026-09-24T11:48:48.600Z level=INFO msg="Uploading 1 narinfos"4402lb # [ 79.892578] niks3-server[1007]: 2026/09/24 11:48:48 INFO Signed narinfos id=18 count=14403lb # [ 79.901330] niks3-server[1007]: 2026/09/24 11:48:48 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.ls4404lb # [ 79.911629] niks3-server[1007]: 2026/09/24 11:48:48 INFO Registered completed upload object_key=nar/14f5yrnllfav3svmw4v4svw15r4zlf5yv1p972jy14mmq6gdw943.nar.zst4405lb # [ 79.925671] niks3-server[1007]: 2026/09/24 11:48:48 INFO Received complete push request method=POST path=/api/pushes/18/complete4406worker2 # [ 80.022503] nix-grpc-daemon[854]: time=2026-09-24T11:48:48.641Z level=INFO msg="Upload complete. (260ms)"4407lb # [ 79.932719] niks3-server[1007]: 2026/09/24 11:48:48 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.narinfo4408worker2 # [ 80.033348] nix-grpc-daemon[784]: ts=2026-09-24T11:48:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=0 paths=1 nar_bytes_in=4244409worker2 # [ 80.048531] nix-grpc-daemon[784]: ts=2026-09-24T11:48:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602844410worker2 # [ 80.058640] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4411worker2 # [ 80.065104] nix-daemon[841]: reaped child process 1054, status = succeeded4412worker2 # [ 80.081333] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4413worker2 # [ 85.699204] nix-grpc-daemon[784]: ts=2026-09-24T11:48:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=5 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv assign_id=10 outputs=04414worker2 # [ 85.739060] nix-daemon[841]: reaped child process 1063, status = succeeded4415worker2 # [ 85.741828] nix-daemon[841]: reaped child process 1061, status = succeeded4416worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4417worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4418worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4419worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4420(finished: subtest: --timeout and --max-silent-time reach the builder, in 13.17 seconds)4421subtest: a failing build is counted with its own reason4422worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4423worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4424worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4425worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4426worker2 # [ 86.361505] nix-grpc-daemon[784]: ts=2026-09-24T11:48:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:602844427worker1 # [ 86.310793] nix-grpc-daemon[488]: ts=2026-09-24T11:48:54Z level=debug event=want client=9 drv=j4rjl229w5xzbr9614xcipiv7652fdka-fail-boom.drv cached=04428worker1 # [ 86.311219] nix-grpc-daemon[488]: ts=2026-09-24T11:48:54Z level=debug event=assigned drv=j4rjl229w5xzbr9614xcipiv7652fdka-fail-boom.drv worker=192.168.1.3:50051 clients=14429worker1 # [ 86.323581] nix-grpc-daemon[488]: ts=2026-09-24T11:48:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764430worker1 # [ 86.341576] nix-grpc-daemon[488]: ts=2026-09-24T11:48:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357764431worker1 # [ 86.355551] nix-grpc-daemon[488]: ts=2026-09-24T11:48:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357764432worker1 # [ 86.371125] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4433worker1 # [ 86.534000] nix-daemon[854]: accepted connection from pid 1078, user nix-grpc-daemon (trusted)4434worker1 # [ 86.558294] nix-daemon[854]: reaped child process 1081, status = succeeded4435lb # [ 86.538077] niks3-server[1007]: 2026/09/24 11:48:55 INFO Received push request method=POST path=/api/pushes4436worker1 # [ 86.568593] nix-grpc-daemon[904]: time=2026-09-24T11:48:55.183Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4437worker1 # [ 86.570397] nix-grpc-daemon[904]: time=2026-09-24T11:48:55.183Z level=INFO msg="Uploading j4rjl229w5xzbr9614xcipiv7652fdka-fail-boom.drv (400B)"4438lb # [ 86.571370] niks3-server[1007]: 2026/09/24 11:48:55 INFO Received sign narinfos request method=POST path=/api/pushes/19/sign4439worker1 # [ 86.601867] nix-grpc-daemon[904]: time=2026-09-24T11:48:55.217Z level=INFO msg="Uploading 1 narinfos"4440lb # [ 86.582640] niks3-server[1007]: 2026/09/24 11:48:55 INFO Signed narinfos id=19 count=14441lb # [ 86.593640] niks3-server[1007]: 2026/09/24 11:48:55 INFO Registered completed upload object_key=nar/0d79vqzwc4s1agk431w3csq2p4brahnzmanwj14fjf383ld797i6.nar.zst4442lb # [ 86.604309] niks3-server[1007]: 2026/09/24 11:48:55 INFO Registered completed upload object_key=j4rjl229w5xzbr9614xcipiv7652fdka.ls4443lb # [ 86.620507] niks3-server[1007]: 2026/09/24 11:48:55 INFO Received complete push request method=POST path=/api/pushes/19/complete4444worker1 # [ 86.650166] nix-grpc-daemon[904]: time=2026-09-24T11:48:55.265Z level=INFO msg="Upload complete. (262ms)"4445lb # [ 86.631186] niks3-server[1007]: 2026/09/24 11:48:55 INFO Registered completed upload object_key=j4rjl229w5xzbr9614xcipiv7652fdka.narinfo4446worker1 # [ 86.661448] nix-grpc-daemon[488]: ts=2026-09-24T11:48:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=0 paths=1 nar_bytes_in=4004447worker1 # [ 86.676961] nix-grpc-daemon[488]: ts=2026-09-24T11:48:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764448worker1 # [ 86.689993] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4449worker1 # [ 86.698650] nix-daemon[854]: reaped child process 1076, status = succeeded4450worker1 # [ 86.713853] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4451worker1 # [ 87.727755] nix-grpc-daemon[488]: ts=2026-09-24T11:48:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=1 drv=j4rjl229w5xzbr9614xcipiv7652fdka-fail-boom.drv assign_id=11 outputs=04452worker1 # [ 87.765161] nix-daemon[854]: reaped child process 1083, status = succeeded4453worker1 # [ 87.767272] nix-daemon[854]: reaped child process 1085, status = succeeded4454worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4455worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4456worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4457worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4458worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4459worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4460worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4461worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4462worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4463worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4464worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4465worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4466worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4467worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4468worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4469worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4470(finished: subtest: a failing build is counted with its own reason, in 2.34 seconds)4471subtest: the first client leaving does not fail the second4472client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share4473client # [ 88.315071] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share.4474client # Running as unit: share1.service; invocation ID: 753364c502a44d5baf8f7c4e9177356a4475client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 0.07 seconds)4476client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share4477worker1 # [ 88.538742] nix-grpc-daemon[488]: ts=2026-09-24T11:48:57Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:357764478client # [ 88.525627] nix[1045]: don't know how to build these paths:4479client # [ 88.526652] nix[1045]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4480worker1 # [ 88.556324] nix-grpc-daemon[488]: ts=2026-09-24T11:48:57Z level=debug event=want client=10 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=04481worker1 # [ 88.562822] nix-grpc-daemon[488]: ts=2026-09-24T11:48:57Z level=debug event=assigned drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker=192.168.1.4:50051 clients=14482worker2 # [ 88.640670] nix-grpc-daemon[784]: ts=2026-09-24T11:48:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602844483worker2 # [ 88.653139] nix-grpc-daemon[784]: ts=2026-09-24T11:48:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602844484client # [ 88.586735] nix[1045]: copying 1 paths to 'lb:50051'...4485client # [ 88.596322] nix[1045]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' to 'lb:50051'...4486worker2 # [ 88.700730] nix-grpc-daemon[784]: ts=2026-09-24T11:48:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:602844487worker2 # [ 88.709719] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4488worker2 # [ 88.829447] nix-daemon[841]: accepted connection from pid 1104, user nix-grpc-daemon (trusted)4489lb # [ 88.760922] niks3-server[1007]: 2026/09/24 11:48:57 INFO Received push request method=POST path=/api/pushes4490worker2 # [ 88.854903] nix-daemon[841]: reaped child process 1107, status = succeeded4491worker2 # [ 88.859531] nix-grpc-daemon[854]: time=2026-09-24T11:48:57.477Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4492worker2 # [ 88.861372] nix-grpc-daemon[854]: time=2026-09-24T11:48:57.477Z level=INFO msg="Uploading rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv (432B)"4493worker2 # [ 88.892566] nix-grpc-daemon[854]: time=2026-09-24T11:48:57.510Z level=INFO msg="Uploading 1 narinfos"4494lb # [ 88.792899] niks3-server[1007]: 2026/09/24 11:48:57 INFO Received sign narinfos request method=POST path=/api/pushes/20/sign4495lb # [ 88.805462] niks3-server[1007]: 2026/09/24 11:48:57 INFO Signed narinfos id=20 count=14496lb # [ 88.813460] niks3-server[1007]: 2026/09/24 11:48:57 INFO Registered completed upload object_key=nar/0chdjpc9an2ckdblh08lnyrmzvw3h8qs76cn5ydwjwvgwkrm731l.nar.zst4497lb # [ 88.827230] niks3-server[1007]: 2026/09/24 11:48:57 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.ls4498worker2 # [ 88.930058] nix-grpc-daemon[854]: time=2026-09-24T11:48:57.549Z level=INFO msg="Upload complete. (212ms)"4499lb # [ 88.838499] niks3-server[1007]: 2026/09/24 11:48:57 INFO Received complete push request method=POST path=/api/pushes/20/complete4500worker2 # [ 88.936631] nix-grpc-daemon[784]: ts=2026-09-24T11:48:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=0 paths=1 nar_bytes_in=4324501lb # [ 88.847430] niks3-server[1007]: 2026/09/24 11:48:57 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.narinfo4502worker2 # [ 88.944865] nix-grpc-daemon[784]: ts=2026-09-24T11:48:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602844503worker2 # [ 88.956289] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4504worker2 # [ 88.959830] nix-daemon[841]: reaped child process 1102, status = succeeded4505client # [ 88.877830] nix[1045]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051...4506worker2 # [ 88.974232] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4507lb # [ 89.028091] envoy[907]: conn peer=[2001:db8:1::1]:51124 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2971 tx=931 ms=64014508lb # [ 89.033186] envoy[907]: conn peer=[2001:db8:1::1]:51132 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2972 tx=959 ms=60324509lb # [ 89.038017] envoy[907]: conn peer=[2001:db8:1::1]:38494 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2924 tx=1264 ms=14614510client # [ 89.415187] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share.4511client # Running as unit: share2.service; invocation ID: 714b3fd757c6456fa54763b59a933dad4512client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 1.11 seconds)4513worker1 # [ 89.667518] nix-grpc-daemon[488]: ts=2026-09-24T11:48:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:357764514client # [ 89.653982] nix[1061]: don't know how to build these paths:4515client # [ 89.655205] nix[1061]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4516worker1 # [ 89.684627] nix-grpc-daemon[488]: ts=2026-09-24T11:48:58Z level=debug event=want client=11 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=04517worker2 # [ 89.760655] nix-grpc-daemon[784]: ts=2026-09-24T11:48:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602844518client # [ 89.674943] nix[1061]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051...4519client: must succeed: sleep 2; systemctl kill -s INT share14520client # [ 92.797437] systemd[1]: share1.service: Sent signal SIGINT to main process 1045 (nix) on client request.4521client # [ 92.807943] nix[1045]: shutting downrxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)4522client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.08 seconds)4523client: waiting for success: ! systemctl is-active share14524client # [ 92.817614] nix[1045]: [mutex.cc : 2506] RAW: thread should hold at least a read lock on Mutex 0xeec3073fc6784525client # [ 92.844798] systemd-coredump[1076]: Process 1045 (nix) of user 0 terminated abnormally with signal 6/ABRT, processing...4526client # [ 92.871361] systemd[1]: Created slice Slice /system/systemd-coredump.4527client # [ 92.878764] systemd[1]: Started Process Core Dump (PID 1076/UID 0).4528client # [ 95.120346] systemd-coredump[1080]: Cannot store coredump of 1045 (nix): No space left on device4529client # [ 95.239574] systemd-coredump[1080]: Process 1045 (nix) of user 0 terminated abnormally without generating a coredump.4530client # [ 95.254164] systemd[1]: systemd-coredump@0-1-1076_1077-0.service: Deactivated successfully.4531client # [ 95.263196] systemd[1]: systemd-coredump@0-1-1076_1077-0.service: Consumed 1.940s CPU time over 2.377s wall clock time, 969M memory peak, 926.3M written to disk.4532client # [ 95.267809] systemd[1]: share1.service: Main process exited, code=dumped, status=6/ABRT4533client # [ 95.271524] systemd[1]: share1.service: Failed with result 'core-dump'.4534client # [ 95.272905] systemd[1]: share1.service: Consumed 379ms CPU time over 6.956s wall clock time, 41.5M memory peak, 4K written to disk, 6.3K incoming IP traffic, 8.1K outgoing IP traffic.4535client: (finished: waiting for success: ! systemctl is-active share1, in 3.24 seconds)4536client: waiting for success: ! systemctl is-active share24537worker2 # [ 96.621977] nix-daemon[841]: reaped child process 1111, status = succeeded4538worker2 # [ 96.772558] nix-daemon[841]: accepted connection from pid 1145, user nix-grpc-daemon (trusted)4539worker2 # [ 96.793012] nix-daemon[841]: reaped child process 1148, status = succeeded4540lb # [ 96.706778] niks3-server[1007]: 2026/09/24 11:49:05 INFO Received push request method=POST path=/api/pushes4541worker2 # [ 96.812465] nix-grpc-daemon[854]: time=2026-09-24T11:49:05.430Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4542worker2 # [ 96.814378] nix-grpc-daemon[854]: time=2026-09-24T11:49:05.430Z level=INFO msg="Uploading fjww0fd4rwk1s35jav3cyal62l24n356-slow-share (120B)"4543worker2 # [ 96.863140] nix-grpc-daemon[854]: time=2026-09-24T11:49:05.481Z level=INFO msg="Uploading 1 narinfos"4544lb # [ 96.763985] niks3-server[1007]: 2026/09/24 11:49:05 INFO Received sign narinfos request method=POST path=/api/pushes/21/sign4545lb # [ 96.779132] niks3-server[1007]: 2026/09/24 11:49:05 INFO Signed narinfos id=21 count=14546lb # [ 96.786615] niks3-server[1007]: 2026/09/24 11:49:05 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst4547lb # [ 96.801455] niks3-server[1007]: 2026/09/24 11:49:05 INFO Registered completed upload object_key=log/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4548lb # [ 96.810335] niks3-server[1007]: 2026/09/24 11:49:05 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.ls4549lb # [ 96.818653] niks3-server[1007]: 2026/09/24 11:49:05 INFO Received complete push request method=POST path=/api/pushes/21/complete4550worker2 # [ 96.917118] nix-grpc-daemon[854]: time=2026-09-24T11:49:05.536Z level=INFO msg="Upload complete. (324ms)"4551lb # [ 96.827933] niks3-server[1007]: 2026/09/24 11:49:05 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.narinfo4552worker2 # [ 96.929233] nix-grpc-daemon[784]: ts=2026-09-24T11:49:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=7 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=12 outputs=14553worker2 # [ 96.948527] nix-grpc-daemon[784]: ts=2026-09-24T11:49:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=7 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=12 outputs=14554worker2 # [ 96.957488] nix-daemon[841]: reaped child process 1109, status = succeeded4555client # [ 96.871621] systemd[1]: share2.service: Deactivated successfully.4556client # [ 96.876078] systemd[1]: share2.service: Consumed 203ms CPU time over 7.460s wall clock time, 33.3M memory peak, 4K written to disk, 5.9K incoming IP traffic, 5.7K outgoing IP traffic.4557client: (finished: waiting for success: ! systemctl is-active share2, in 1.12 seconds)4558client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }4559client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.05 seconds)4560(finished: subtest: the first client leaving does not fail the second, in 9.22 seconds)4561subtest: build hook: nix-daemon with builders = grpc://lb4562client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log4563client # [ 97.616242] systemd[1]: Started Nix Daemon.4564client # [ 97.731653] nix-daemon[1127]: accepted connection from pid 1124, user root (trusted)4565worker2 # [ 97.967584] nix-grpc-daemon[784]: ts=2026-09-24T11:49:06Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:602844566worker1 # [ 97.909407] nix-grpc-daemon[488]: ts=2026-09-24T11:49:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357764567worker1 # [ 97.924137] nix-grpc-daemon[488]: ts=2026-09-24T11:49:06Z level=debug event=want client=12 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=04568worker1 # [ 97.931283] nix-grpc-daemon[488]: ts=2026-09-24T11:49:06Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.3:50051 clients=14569worker1 # [ 97.940684] nix-grpc-daemon[488]: ts=2026-09-24T11:49:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764570worker1 # [ 97.952514] nix-grpc-daemon[488]: ts=2026-09-24T11:49:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357764571worker1 # [ 97.967150] nix-grpc-daemon[488]: ts=2026-09-24T11:49:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357764572worker1 # [ 97.982969] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4573worker1 # [ 98.139122] nix-daemon[854]: accepted connection from pid 1137, user nix-grpc-daemon (trusted)4574worker1 # [ 98.159009] nix-daemon[854]: reaped child process 1140, status = succeeded4575lb # [ 98.143662] niks3-server[1007]: 2026/09/24 11:49:06 INFO Received push request method=POST path=/api/pushes4576worker1 # [ 98.175286] nix-grpc-daemon[904]: time=2026-09-24T11:49:06.790Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4577worker1 # [ 98.176885] nix-grpc-daemon[904]: time=2026-09-24T11:49:06.790Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)"4578lb # [ 98.175981] niks3-server[1007]: 2026/09/24 11:49:06 INFO Received sign narinfos request method=POST path=/api/pushes/22/sign4579worker1 # [ 98.206166] nix-grpc-daemon[904]: time=2026-09-24T11:49:06.821Z level=INFO msg="Uploading 1 narinfos"4580lb # [ 98.188564] niks3-server[1007]: 2026/09/24 11:49:06 INFO Signed narinfos id=22 count=14581lb # [ 98.196797] niks3-server[1007]: 2026/09/24 11:49:06 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls4582lb # [ 98.210346] niks3-server[1007]: 2026/09/24 11:49:06 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst4583lb # [ 98.219126] niks3-server[1007]: 2026/09/24 11:49:06 INFO Received complete push request method=POST path=/api/pushes/22/complete4584worker1 # [ 98.249170] nix-grpc-daemon[904]: time=2026-09-24T11:49:06.863Z level=INFO msg="Upload complete. (248ms)"4585lb # [ 98.228094] niks3-server[1007]: 2026/09/24 11:49:06 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo4586worker1 # [ 98.256383] nix-grpc-daemon[488]: ts=2026-09-24T11:49:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=0 paths=1 nar_bytes_in=5124587worker1 # [ 98.276568] nix-grpc-daemon[488]: ts=2026-09-24T11:49:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764588worker1 # [ 98.289967] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4589worker1 # [ 98.293821] nix-daemon[854]: reaped child process 1135, status = succeeded4590worker1 # [ 98.309633] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4591lb # [ 99.026605] envoy[907]: conn peer=[2001:db8:1::1]:38510 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3016 tx=813 ms=67434592lb # [ 99.026870] envoy[907]: conn peer=[2001:db8:1::1]:38516 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1443 tx=1194 ms=72064593worker1 # [ 99.343890] nix-daemon[854]: reaped child process 1144, status = succeeded4594worker1 # [ 99.456759] nix-daemon[854]: accepted connection from pid 1160, user nix-grpc-daemon (trusted)4595worker1 # [ 99.477960] nix-daemon[854]: reaped child process 1163, status = succeeded4596lb # [ 99.457102] niks3-server[1007]: 2026/09/24 11:49:08 INFO Received push request method=POST path=/api/pushes4597worker1 # [ 99.491122] nix-grpc-daemon[904]: time=2026-09-24T11:49:08.105Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4598worker1 # [ 99.493820] nix-grpc-daemon[904]: time=2026-09-24T11:49:08.105Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)"4599lb # [ 99.487880] niks3-server[1007]: 2026/09/24 11:49:08 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls4600worker1 # [ 99.520338] nix-grpc-daemon[904]: time=2026-09-24T11:49:08.135Z level=INFO msg="Uploading 1 narinfos"4601lb # [ 99.499143] niks3-server[1007]: 2026/09/24 11:49:08 INFO Received sign narinfos request method=POST path=/api/pushes/23/sign4602lb # [ 99.505560] niks3-server[1007]: 2026/09/24 11:49:08 INFO Signed narinfos id=23 count=14603lb # [ 99.510223] niks3-server[1007]: 2026/09/24 11:49:08 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv4604lb # [ 99.517269] niks3-server[1007]: 2026/09/24 11:49:08 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4605lb # [ 99.519798] niks3-server[1007]: 2026/09/24 11:49:08 INFO Received complete push request method=POST path=/api/pushes/23/complete4606lb # [ 99.525975] niks3-server[1007]: 2026/09/24 11:49:08 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo4607worker1 # [ 99.545615] nix-grpc-daemon[904]: time=2026-09-24T11:49:08.161Z level=INFO msg="Upload complete. (224ms)"4608worker1 # [ 99.556915] nix-grpc-daemon[488]: ts=2026-09-24T11:49:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=13 outputs=14609worker1 # [ 99.568445] nix-grpc-daemon[488]: ts=2026-09-24T11:49:08Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:357764610worker1 # [ 99.572477] nix-daemon[854]: reaped child process 1142, status = succeeded4611worker2 # [ 99.666272] nix-grpc-daemon[784]: ts=2026-09-24T11:49:08Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:602844612worker2 # [ 99.678813] nix-grpc-daemon[784]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'...4613worker2 # [ 99.697425] nix-grpc-daemon[784]: ts=2026-09-24T11:49:08Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=0 paths=1 nar_bytes_out=1284614worker1 # [ 99.870323] nix-grpc-daemon[488]: ts=2026-09-24T11:49:08Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:357764615worker2 # [ 99.945689] nix-grpc-daemon[784]: ts=2026-09-24T11:49:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602844616worker2 # [ 99.957252] nix-grpc-daemon[854]: time=2026-09-24T11:49:08.575Z level=INFO msg="All 1 paths already cached"4617worker1 # [ 99.903956] nix-grpc-daemon[488]: ts=2026-09-24T11:49:08Z level=debug event=want client=13 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=04618worker1 # [ 99.908759] nix-grpc-daemon[488]: ts=2026-09-24T11:49:08Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.3:50051 clients=14619worker1 # [ 99.912960] nix-grpc-daemon[488]: ts=2026-09-24T11:49:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764620worker1 # [ 99.923461] nix-grpc-daemon[488]: ts=2026-09-24T11:49:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357764621worker1 # [ 99.941179] nix-grpc-daemon[904]: time=2026-09-24T11:49:08.556Z level=INFO msg="All 1 paths already cached"4622worker1 # [ 99.954239] nix-grpc-daemon[488]: ts=2026-09-24T11:49:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357764623worker1 # [ 99.968924] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4624worker1 # [ 100.119753] nix-daemon[854]: accepted connection from pid 1169, user nix-grpc-daemon (trusted)4625worker1 # [ 100.139120] nix-daemon[854]: reaped child process 1172, status = succeeded4626lb # [ 100.122470] niks3-server[1007]: 2026/09/24 11:49:08 INFO Received push request method=POST path=/api/pushes4627worker1 # [ 100.152125] nix-grpc-daemon[904]: time=2026-09-24T11:49:08.767Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4628worker1 # [ 100.154031] nix-grpc-daemon[904]: time=2026-09-24T11:49:08.767Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)"4629lb # [ 100.161358] niks3-server[1007]: 2026/09/24 11:49:08 INFO Received sign narinfos request method=POST path=/api/pushes/24/sign4630worker1 # [ 100.190525] nix-grpc-daemon[904]: time=2026-09-24T11:49:08.805Z level=INFO msg="Uploading 1 narinfos"4631lb # [ 100.176990] niks3-server[1007]: 2026/09/24 11:49:08 INFO Signed narinfos id=24 count=14632lb # [ 100.183858] niks3-server[1007]: 2026/09/24 11:49:08 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls4633worker1 # [ 100.216353] nix-grpc-daemon[904]: time=2026-09-24T11:49:08.831Z level=INFO msg="Upload complete. (231ms)"4634lb # [ 100.195780] niks3-server[1007]: 2026/09/24 11:49:08 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst4635lb # [ 100.208256] niks3-server[1007]: 2026/09/24 11:49:08 INFO Received complete push request method=POST path=/api/pushes/24/complete4636lb # [ 100.212167] niks3-server[1007]: 2026/09/24 11:49:08 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo4637worker1 # [ 100.224191] nix-grpc-daemon[488]: ts=2026-09-24T11:49:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=0 paths=1 nar_bytes_in=6244638worker1 # [ 100.242645] nix-grpc-daemon[488]: ts=2026-09-24T11:49:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764639worker1 # [ 100.250122] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4640worker1 # [ 100.255303] nix-daemon[854]: reaped child process 1167, status = succeeded4641worker1 # [ 100.268193] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4642worker1 # [ 100.945025] nix-daemon[854]: reaped child process 1176, status = succeeded4643worker1 # [ 101.051264] nix-daemon[854]: accepted connection from pid 1192, user nix-grpc-daemon (trusted)4644worker1 # [ 101.068942] nix-daemon[854]: reaped child process 1195, status = succeeded4645lb # [ 101.055477] niks3-server[1007]: 2026/09/24 11:49:09 INFO Received push request method=POST path=/api/pushes4646worker1 # [ 101.087396] nix-grpc-daemon[904]: time=2026-09-24T11:49:09.700Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4647worker1 # [ 101.089211] nix-grpc-daemon[904]: time=2026-09-24T11:49:09.700Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)"4648lb # [ 101.095810] niks3-server[1007]: 2026/09/24 11:49:09 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst4649lb # [ 101.111738] niks3-server[1007]: 2026/09/24 11:49:09 INFO Received sign narinfos request method=POST path=/api/pushes/25/sign4650worker1 # [ 101.137068] nix-grpc-daemon[904]: time=2026-09-24T11:49:09.752Z level=INFO msg="Uploading 1 narinfos"4651lb # [ 101.121664] niks3-server[1007]: 2026/09/24 11:49:09 INFO Signed narinfos id=25 count=14652lb # [ 101.127613] niks3-server[1007]: 2026/09/24 11:49:09 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls4653lb # [ 101.138679] niks3-server[1007]: 2026/09/24 11:49:09 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv4654lb # [ 101.143040] niks3-server[1007]: 2026/09/24 11:49:09 INFO Received complete push request method=POST path=/api/pushes/25/complete4655worker1 # [ 101.163692] nix-grpc-daemon[904]: time=2026-09-24T11:49:09.779Z level=INFO msg="Upload complete. (283ms)"4656lb # [ 101.146920] niks3-server[1007]: 2026/09/24 11:49:09 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo4657worker1 # [ 101.177172] nix-grpc-daemon[488]: ts=2026-09-24T11:49:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=0 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=14 outputs=14658worker2 # [ 101.251912] nix-grpc-daemon[784]: ts=2026-09-24T11:49:09Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:602844659worker2 # [ 101.263874] nix-grpc-daemon[784]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'...4660worker1 # [ 101.197756] nix-daemon[854]: reaped child process 1174, status = succeeded4661worker1 # [ 101.232432] nix-grpc-daemon[488]: ts=2026-09-24T11:49:09Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:357764662worker1 # [ 101.239679] nix-grpc-daemon[488]: ts=2026-09-24T11:49:09Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=0 paths=1 nar_bytes_out=1764663worker1 # [ 101.481103] nix-grpc-daemon[488]: ts=2026-09-24T11:49:10Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:357764664worker2 # [ 101.556100] nix-grpc-daemon[784]: ts=2026-09-24T11:49:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602844665worker2 # [ 101.566874] nix-grpc-daemon[854]: time=2026-09-24T11:49:10.185Z level=INFO msg="All 2 paths already cached"4666worker1 # [ 101.514634] nix-grpc-daemon[488]: ts=2026-09-24T11:49:10Z level=debug event=want client=14 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=04667worker1 # [ 101.523813] nix-grpc-daemon[488]: ts=2026-09-24T11:49:10Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.3:50051 clients=14668worker1 # [ 101.532265] nix-grpc-daemon[488]: ts=2026-09-24T11:49:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764669worker1 # [ 101.539820] nix-grpc-daemon[488]: ts=2026-09-24T11:49:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357764670worker1 # [ 101.550468] nix-grpc-daemon[904]: time=2026-09-24T11:49:10.166Z level=INFO msg="All 2 paths already cached"4671worker1 # [ 101.561413] nix-grpc-daemon[488]: ts=2026-09-24T11:49:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357764672worker1 # [ 101.575622] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4673worker1 # [ 101.725017] nix-daemon[854]: accepted connection from pid 1201, user nix-grpc-daemon (trusted)4674worker1 # [ 101.742607] nix-daemon[854]: reaped child process 1204, status = succeeded4675lb # [ 101.729821] niks3-server[1007]: 2026/09/24 11:49:10 INFO Received push request method=POST path=/api/pushes4676worker1 # [ 101.762249] nix-grpc-daemon[904]: time=2026-09-24T11:49:10.377Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4677worker1 # [ 101.763825] nix-grpc-daemon[904]: time=2026-09-24T11:49:10.377Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)"4678lb # [ 101.762158] niks3-server[1007]: 2026/09/24 11:49:10 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls4679worker1 # [ 101.793249] nix-grpc-daemon[904]: time=2026-09-24T11:49:10.408Z level=INFO msg="Uploading 1 narinfos"4680lb # [ 101.776401] niks3-server[1007]: 2026/09/24 11:49:10 INFO Received sign narinfos request method=POST path=/api/pushes/26/sign4681lb # [ 101.785696] niks3-server[1007]: 2026/09/24 11:49:10 INFO Signed narinfos id=26 count=14682lb # [ 101.795029] niks3-server[1007]: 2026/09/24 11:49:10 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst4683worker1 # [ 101.821979] nix-grpc-daemon[904]: time=2026-09-24T11:49:10.437Z level=INFO msg="Upload complete. (230ms)"4684lb # [ 101.801735] niks3-server[1007]: 2026/09/24 11:49:10 INFO Received complete push request method=POST path=/api/pushes/26/complete4685lb # [ 101.805665] niks3-server[1007]: 2026/09/24 11:49:10 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo4686worker1 # [ 101.832268] nix-grpc-daemon[488]: ts=2026-09-24T11:49:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=0 paths=1 nar_bytes_in=7524687worker1 # [ 101.847934] nix-grpc-daemon[488]: ts=2026-09-24T11:49:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764688worker1 # [ 101.854621] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4689worker1 # [ 101.860304] nix-daemon[854]: reaped child process 1199, status = succeeded4690worker1 # [ 101.873914] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4691worker1 # [ 102.752299] nix-daemon[854]: reaped child process 1208, status = succeeded4692worker1 # [ 102.852603] nix-daemon[854]: accepted connection from pid 1224, user nix-grpc-daemon (trusted)4693worker1 # [ 102.873908] nix-daemon[854]: reaped child process 1227, status = succeeded4694lb # [ 102.853563] niks3-server[1007]: 2026/09/24 11:49:11 INFO Received push request method=POST path=/api/pushes4695worker1 # [ 102.884510] nix-grpc-daemon[904]: time=2026-09-24T11:49:11.499Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4696worker1 # [ 102.886327] nix-grpc-daemon[904]: time=2026-09-24T11:49:11.499Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)"4697lb # [ 102.905714] niks3-server[1007]: 2026/09/24 11:49:11 INFO Received sign narinfos request method=POST path=/api/pushes/27/sign4698worker1 # [ 102.933841] nix-grpc-daemon[904]: time=2026-09-24T11:49:11.549Z level=INFO msg="Uploading 1 narinfos"4699lb # [ 102.918345] niks3-server[1007]: 2026/09/24 11:49:11 INFO Signed narinfos id=27 count=14700lb # [ 102.923717] niks3-server[1007]: 2026/09/24 11:49:11 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv4701lb # [ 102.931118] niks3-server[1007]: 2026/09/24 11:49:11 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst4702lb # [ 102.941369] niks3-server[1007]: 2026/09/24 11:49:11 INFO Received complete push request method=POST path=/api/pushes/27/complete4703lb # [ 102.946619] niks3-server[1007]: 2026/09/24 11:49:11 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo4704worker1 # [ 102.966601] nix-grpc-daemon[904]: time=2026-09-24T11:49:11.582Z level=INFO msg="Upload complete. (282ms)"4705lb # [ 102.950879] niks3-server[1007]: 2026/09/24 11:49:11 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls4706worker1 # [ 102.981195] nix-grpc-daemon[488]: ts=2026-09-24T11:49:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=15 outputs=14707worker2 # [ 103.057633] nix-grpc-daemon[784]: ts=2026-09-24T11:49:11Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:602844708worker2 # [ 103.066177] nix-grpc-daemon[784]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'...4709worker1 # [ 103.002253] nix-daemon[854]: reaped child process 1206, status = succeeded4710worker2 # [ 103.106598] nix-grpc-daemon[784]: ts=2026-09-24T11:49:11Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:602844711worker2 # [ 103.114070] nix-grpc-daemon[784]: ts=2026-09-24T11:49:11Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=0 paths=1 nar_bytes_out=2324712client # [ 103.114750] nix-daemon[1127]: reaped child process 1129, status = succeeded4713client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log, in 5.65 seconds)4714client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top4715client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.04 seconds)4716client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4717client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds)4718client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4719client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)4720(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 5.77 seconds)4721subtest: an input only one worker has still reaches the builder4722client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only4723client # this derivation will be built:4724client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv4725client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'...4726client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only, in 0.31 seconds)4727client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure4728client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.12 seconds)4729worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4730worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.20 seconds)4731worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only4732worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.03 seconds)4733client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&24734client # [ 104.101003] nix-daemon[1127]: accepted connection from pid 1210, user root (trusted)4735client # this derivation will be built:4736client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4737client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4738worker1 # [ 104.282239] nix-grpc-daemon[488]: ts=2026-09-24T11:49:12Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:357764739worker1 # [ 104.290407] nix-grpc-daemon[488]: ts=2026-09-24T11:49:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357764740worker1 # [ 104.470822] nix-daemon[854]: accepted connection from pid 1234, user nix-grpc-daemon (trusted)4741worker1 # [ 104.493614] nix-daemon[854]: reaped child process 1237, status = succeeded4742lb # [ 104.473567] niks3-server[1007]: 2026/09/24 11:49:13 INFO Received push request method=POST path=/api/pushes4743worker1 # [ 104.512879] nix-grpc-daemon[904]: time=2026-09-24T11:49:13.127Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4744worker1 # [ 104.515488] nix-grpc-daemon[904]: time=2026-09-24T11:49:13.127Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)"4745lb # [ 104.513061] niks3-server[1007]: 2026/09/24 11:49:13 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4746worker1 # [ 104.547828] nix-grpc-daemon[904]: time=2026-09-24T11:49:13.162Z level=INFO msg="Uploading 1 narinfos"4747lb # [ 104.528876] niks3-server[1007]: 2026/09/24 11:49:13 INFO Received sign narinfos request method=POST path=/api/pushes/28/sign4748lb # [ 104.541283] niks3-server[1007]: 2026/09/24 11:49:13 INFO Signed narinfos id=28 count=14749lb # [ 104.548917] niks3-server[1007]: 2026/09/24 11:49:13 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls4750lb # [ 104.561893] niks3-server[1007]: 2026/09/24 11:49:13 INFO Received complete push request method=POST path=/api/pushes/28/complete4751worker1 # [ 104.591193] nix-grpc-daemon[904]: time=2026-09-24T11:49:13.206Z level=INFO msg="Upload complete. (302ms)"4752lb # [ 104.571928] niks3-server[1007]: 2026/09/24 11:49:13 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo4753worker1 # [ 104.616911] nix-grpc-daemon[488]: ts=2026-09-24T11:49:13Z level=debug event=want client=15 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=04754worker1 # [ 104.623802] nix-grpc-daemon[488]: ts=2026-09-24T11:49:13Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=14755worker2 # [ 104.701639] nix-grpc-daemon[784]: ts=2026-09-24T11:49:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602844756worker2 # [ 104.714781] nix-grpc-daemon[784]: ts=2026-09-24T11:49:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602844757client # copying 2 paths to 'lb:50051'...4758client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'...4759worker2 # [ 104.734652] nix-grpc-daemon[784]: ts=2026-09-24T11:49:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:602844760client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'...4761worker2 # [ 104.750156] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4762worker2 # [ 104.915466] nix-daemon[841]: accepted connection from pid 1158, user nix-grpc-daemon (trusted)4763worker2 # [ 104.942985] nix-daemon[841]: reaped child process 1161, status = succeeded4764lb # [ 104.849374] niks3-server[1007]: 2026/09/24 11:49:13 INFO Received push request method=POST path=/api/pushes4765worker2 # [ 104.958982] nix-grpc-daemon[854]: time=2026-09-24T11:49:13.577Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4766worker2 # [ 104.963358] nix-grpc-daemon[854]: time=2026-09-24T11:49:13.579Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)"4767worker2 # [ 104.966177] nix-grpc-daemon[854]: time=2026-09-24T11:49:13.580Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)"4768lb # [ 104.903847] niks3-server[1007]: 2026/09/24 11:49:13 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst4769lb # [ 104.918974] niks3-server[1007]: 2026/09/24 11:49:13 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls4770lb # [ 104.928523] niks3-server[1007]: 2026/09/24 11:49:13 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst4771worker2 # [ 105.028849] nix-grpc-daemon[854]: time=2026-09-24T11:49:13.647Z level=INFO msg="Uploading 2 narinfos"4772lb # [ 104.938548] niks3-server[1007]: 2026/09/24 11:49:13 INFO Received sign narinfos request method=POST path=/api/pushes/29/sign4773lb # [ 104.944656] niks3-server[1007]: 2026/09/24 11:49:13 INFO Signed narinfos id=29 count=24774lb # [ 104.952668] niks3-server[1007]: 2026/09/24 11:49:13 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls4775lb # [ 104.965195] niks3-server[1007]: 2026/09/24 11:49:13 INFO Received complete push request method=POST path=/api/pushes/29/complete4776worker2 # [ 105.064397] nix-grpc-daemon[854]: time=2026-09-24T11:49:13.683Z level=INFO msg="Upload complete. (295ms)"4777lb # [ 104.974854] niks3-server[1007]: 2026/09/24 11:49:13 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo4778lb # [ 104.979355] niks3-server[1007]: 2026/09/24 11:49:13 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4779worker2 # [ 105.076446] nix-grpc-daemon[784]: ts=2026-09-24T11:49:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=0 paths=2 nar_bytes_in=9844780worker2 # [ 105.096497] nix-grpc-daemon[784]: ts=2026-09-24T11:49:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602844781worker2 # [ 105.105868] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4782worker2 # [ 105.111645] nix-daemon[841]: reaped child process 1156, status = succeeded4783worker2 # [ 105.130005] nix-grpc-daemon[784]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'...4784client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051...4785client # uses-input-w1onlya> worker2: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4786worker2 # [ 105.151256] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4787client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds4788worker2 # [ 106.236888] nix-daemon[841]: reaped child process 1166, status = succeeded4789worker2 # [ 106.350372] nix-daemon[841]: accepted connection from pid 1182, user nix-grpc-daemon (trusted)4790worker2 # [ 106.370128] nix-daemon[841]: reaped child process 1185, status = succeeded4791lb # [ 106.282784] niks3-server[1007]: 2026/09/24 11:49:14 INFO Received push request method=POST path=/api/pushes4792worker2 # [ 106.408573] nix-grpc-daemon[854]: time=2026-09-24T11:49:15.023Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4793lb # [ 106.335653] niks3-server[1007]: 2026/09/24 11:49:14 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls4794worker2 # [ 106.445018] nix-grpc-daemon[854]: time=2026-09-24T11:49:15.063Z level=INFO msg="Uploading 1 narinfos"4795lb # [ 106.351973] niks3-server[1007]: 2026/09/24 11:49:14 INFO Received sign narinfos request method=POST path=/api/pushes/30/sign4796lb # [ 106.364653] niks3-server[1007]: 2026/09/24 11:49:14 INFO Signed narinfos id=30 count=14797lb # [ 106.372303] niks3-server[1007]: 2026/09/24 11:49:14 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4798lb # [ 106.381876] niks3-server[1007]: 2026/09/24 11:49:14 INFO Received complete push request method=POST path=/api/pushes/30/complete4799worker2 # [ 106.487139] nix-grpc-daemon[854]: time=2026-09-24T11:49:15.106Z level=INFO msg="Upload complete. (316ms)"4800lb # [ 106.396808] niks3-server[1007]: 2026/09/24 11:49:15 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo4801client # copying 1 paths...4802worker2 # [ 106.501230] nix-grpc-daemon[784]: ts=2026-09-24T11:49:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=16 outputs=14803worker2 # [ 106.520529] nix-grpc-daemon[784]: ts=2026-09-24T11:49:15Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:602844804worker2 # [ 106.526720] nix-daemon[841]: reaped child process 1163, status = succeeded4805client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'...4806worker2 # [ 106.553257] nix-grpc-daemon[784]: ts=2026-09-24T11:49:15Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:602844807worker2 # [ 106.560352] nix-grpc-daemon[784]: ts=2026-09-24T11:49:15Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=0 paths=1 nar_bytes_out=1204808client # [ 106.554322] nix-daemon[1127]: reaped child process 1213, status = succeeded4809client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.65 seconds)4810client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&24811client # [ 106.732103] nix-daemon[1127]: accepted connection from pid 1232, user root (trusted)4812client # this derivation will be built:4813client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4814worker2 # [ 106.981411] nix-grpc-daemon[784]: ts=2026-09-24T11:49:15Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:602844815client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4816worker1 # [ 106.916914] nix-grpc-daemon[488]: ts=2026-09-24T11:49:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357764817worker1 # [ 106.930040] nix-grpc-daemon[904]: time=2026-09-24T11:49:15.544Z level=INFO msg="All 1 paths already cached"4818worker1 # [ 106.944157] nix-grpc-daemon[488]: ts=2026-09-24T11:49:15Z level=debug event=want client=16 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=04819worker1 # [ 106.951870] nix-grpc-daemon[488]: ts=2026-09-24T11:49:15Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=14820worker1 # [ 106.962211] nix-grpc-daemon[488]: ts=2026-09-24T11:49:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764821worker1 # [ 106.974186] nix-grpc-daemon[488]: ts=2026-09-24T11:49:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:357764822client # copying 1 paths to 'lb:50051'...4823client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'...4824worker1 # [ 106.996835] nix-grpc-daemon[488]: ts=2026-09-24T11:49:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:357764825worker1 # [ 107.011847] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4826worker1 # [ 107.033422] nix-grpc-daemon[488]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 93 ms (attempt 1/5)4827worker1 # [ 107.134276] nix-grpc-daemon[488]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 173 ms (attempt 2/5)4828worker1 # [ 107.309824] nix-grpc-daemon[488]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 59 ms (attempt 3/5)4829worker1 # [ 107.370715] nix-grpc-daemon[488]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 52 ms (attempt 4/5)4830worker1 # [ 107.425178] nix-grpc-daemon[488]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4831worker1 # [ 107.433251] nix-grpc-daemon[488]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4832worker1 # [ 107.440842] nix-grpc-daemon[488]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'...4833worker1 # [ 107.612836] nix-daemon[854]: accepted connection from pid 1250, user nix-grpc-daemon (trusted)4834worker1 # [ 107.633623] nix-daemon[854]: reaped child process 1253, status = succeeded4835lb # [ 107.619027] niks3-server[1007]: 2026/09/24 11:49:16 INFO Received push request method=POST path=/api/pushes4836worker1 # [ 107.663875] nix-grpc-daemon[904]: time=2026-09-24T11:49:16.278Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4837worker1 # [ 107.667066] nix-grpc-daemon[904]: time=2026-09-24T11:49:16.279Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)"4838lb # [ 107.662229] niks3-server[1007]: 2026/09/24 11:49:16 INFO Received sign narinfos request method=POST path=/api/pushes/31/sign4839worker1 # [ 107.690514] nix-grpc-daemon[904]: time=2026-09-24T11:49:16.305Z level=INFO msg="Uploading 1 narinfos"4840lb # [ 107.675685] niks3-server[1007]: 2026/09/24 11:49:16 INFO Signed narinfos id=31 count=14841lb # [ 107.686394] niks3-server[1007]: 2026/09/24 11:49:16 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst4842lb # [ 107.700919] niks3-server[1007]: 2026/09/24 11:49:16 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls4843lb # [ 107.712405] niks3-server[1007]: 2026/09/24 11:49:16 INFO Received complete push request method=POST path=/api/pushes/31/complete4844worker1 # [ 107.732869] nix-grpc-daemon[904]: time=2026-09-24T11:49:16.348Z level=INFO msg="Upload complete. (272ms)"4845lb # [ 107.718446] niks3-server[1007]: 2026/09/24 11:49:16 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo4846worker1 # [ 107.745571] nix-grpc-daemon[488]: ts=2026-09-24T11:49:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=0 paths=1 nar_bytes_in=5684847worker1 # [ 107.757147] nix-grpc-daemon[488]: ts=2026-09-24T11:49:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:357764848worker1 # [ 107.770759] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4849worker1 # [ 107.777679] nix-daemon[854]: reaped child process 1243, status = succeeded4850client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051...4851worker1 # [ 107.793835] nix-daemon[854]: accepted connection from pid 488, user nix-grpc-daemon (trusted)4852client # uses-input-w1onlyb> node-a: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4853client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds4854worker1 # [ 108.827497] nix-daemon[854]: reaped child process 1257, status = succeeded4855worker1 # [ 108.932680] nix-daemon[854]: accepted connection from pid 1273, user nix-grpc-daemon (trusted)4856worker1 # [ 108.954400] nix-daemon[854]: reaped child process 1276, status = succeeded4857lb # [ 108.938459] niks3-server[1007]: 2026/09/24 11:49:17 INFO Received push request method=POST path=/api/pushes4858worker1 # [ 108.995738] nix-grpc-daemon[904]: time=2026-09-24T11:49:17.607Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4859lb # [ 108.995248] niks3-server[1007]: 2026/09/24 11:49:17 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls4860worker1 # [ 109.028164] nix-grpc-daemon[904]: time=2026-09-24T11:49:17.643Z level=INFO msg="Uploading 1 narinfos"4861lb # [ 109.007394] niks3-server[1007]: 2026/09/24 11:49:17 INFO Received sign narinfos request method=POST path=/api/pushes/32/sign4862lb # [ 109.018812] niks3-server[1007]: 2026/09/24 11:49:17 INFO Signed narinfos id=32 count=14863lb # [ 109.024913] niks3-server[1007]: 2026/09/24 11:49:17 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4864lb # [ 109.039802] niks3-server[1007]: 2026/09/24 11:49:17 INFO Received complete push request method=POST path=/api/pushes/32/complete4865worker1 # [ 109.067771] nix-grpc-daemon[904]: time=2026-09-24T11:49:17.683Z level=INFO msg="Upload complete. (263ms)"4866lb # [ 109.047786] envoy[907]: conn peer=[2001:db8:1::1]:46350 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3440 tx=1942 ms=17474867lb # [ 109.052251] envoy[907]: conn peer=[2001:db8:1::1]:46364 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=334 ms=524868lb # [ 109.057385] envoy[907]: conn peer=[2001:db8:1::1]:46366 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3874 tx=2156 ms=13874869client # copying 1 paths...4870lb # [ 109.062312] envoy[907]: conn peer=[2001:db8:1::1]:54170 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=375 ms=294871worker1 # [ 109.081321] nix-grpc-daemon[488]: ts=2026-09-24T11:49:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=17 outputs=14872lb # [ 109.065758] envoy[907]: conn peer=[2001:db8:1::1]:54184 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4339 tx=2414 ms=15804873lb # [ 109.071552] envoy[907]: conn peer=[2001:db8:1::1]:54186 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=412 tx=406 ms=304874lb # [ 109.075157] envoy[907]: conn peer=[2001:db8:1::1]:54202 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3872 tx=2070 ms=22254875lb # [ 109.078271] envoy[907]: conn peer=[2001:db8:1::1]:54204 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=328 ms=284876lb # [ 109.081809] niks3-server[1007]: 2026/09/24 11:49:17 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo4877worker1 # [ 109.103813] nix-grpc-daemon[488]: ts=2026-09-24T11:49:17Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:357764878worker1 # [ 109.109587] nix-daemon[854]: reaped child process 1255, status = succeeded4879client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'...4880worker2 # [ 109.211071] nix-grpc-daemon[784]: ts=2026-09-24T11:49:17Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:602844881worker2 # [ 109.224728] nix-grpc-daemon[784]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'...4882worker2 # [ 109.242249] nix-grpc-daemon[784]: ts=2026-09-24T11:49:17Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=0 paths=1 nar_bytes_out=1204883client # [ 109.243198] nix-daemon[1127]: reaped child process 1235, status = succeeded4884client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2, in 2.69 seconds)4885client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&24886client # [ 109.442491] nix-daemon[1127]: accepted connection from pid 1254, user root (trusted)4887client # this derivation will be built:4888client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4889worker2 # [ 109.794954] nix-grpc-daemon[784]: ts=2026-09-24T11:49:18Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:602844890client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4891worker2 # [ 109.804941] nix-grpc-daemon[784]: ts=2026-09-24T11:49:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602844892worker2 # [ 109.813684] nix-grpc-daemon[854]: time=2026-09-24T11:49:18.429Z level=INFO msg="All 1 paths already cached"4893worker1 # [ 109.759860] nix-grpc-daemon[488]: ts=2026-09-24T11:49:18Z level=debug event=want client=17 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=04894worker1 # [ 109.760546] nix-grpc-daemon[488]: ts=2026-09-24T11:49:18Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=14895worker2 # [ 109.849216] nix-grpc-daemon[784]: ts=2026-09-24T11:49:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602844896worker2 # [ 109.869766] nix-grpc-daemon[784]: ts=2026-09-24T11:49:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:602844897worker2 # [ 109.888076] nix-grpc-daemon[854]: time=2026-09-24T11:49:18.507Z level=INFO msg="All 1 paths already cached"4898client # copying 1 paths to 'lb:50051'...4899worker2 # [ 109.903451] nix-grpc-daemon[784]: ts=2026-09-24T11:49:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:602844900client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'...4901worker2 # [ 109.927780] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4902worker2 # [ 110.155789] nix-daemon[841]: accepted connection from pid 1190, user nix-grpc-daemon (trusted)4903worker2 # [ 110.177682] nix-daemon[841]: reaped child process 1193, status = succeeded4904lb # [ 110.099979] niks3-server[1007]: 2026/09/24 11:49:18 INFO Received push request method=POST path=/api/pushes4905worker2 # [ 110.204095] nix-grpc-daemon[854]: time=2026-09-24T11:49:18.823Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4906worker2 # [ 110.209012] nix-grpc-daemon[854]: time=2026-09-24T11:49:18.823Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)"4907lb # [ 110.148658] niks3-server[1007]: 2026/09/24 11:49:18 INFO Received sign narinfos request method=POST path=/api/pushes/33/sign4908worker2 # [ 110.247171] nix-grpc-daemon[854]: time=2026-09-24T11:49:18.865Z level=INFO msg="Uploading 1 narinfos"4909lb # [ 110.162349] niks3-server[1007]: 2026/09/24 11:49:18 INFO Signed narinfos id=33 count=14910lb # [ 110.166974] niks3-server[1007]: 2026/09/24 11:49:18 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst4911lb # [ 110.175915] niks3-server[1007]: 2026/09/24 11:49:18 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls4912lb # [ 110.185593] niks3-server[1007]: 2026/09/24 11:49:18 INFO Received complete push request method=POST path=/api/pushes/33/complete4913lb # [ 110.189065] niks3-server[1007]: 2026/09/24 11:49:18 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo4914worker2 # [ 110.277179] nix-grpc-daemon[854]: time=2026-09-24T11:49:18.896Z level=INFO msg="Upload complete. (324ms)"4915worker2 # [ 110.290047] nix-grpc-daemon[784]: ts=2026-09-24T11:49:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=0 paths=1 nar_bytes_in=5684916worker2 # [ 110.304769] nix-grpc-daemon[784]: ts=2026-09-24T11:49:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:602844917worker2 # [ 110.317175] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4918worker2 # [ 110.321525] nix-daemon[841]: reaped child process 1188, status = succeeded4919client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051...4920worker2 # [ 110.334229] nix-daemon[841]: accepted connection from pid 784, user nix-grpc-daemon (trusted)4921client # uses-input-w1onlyc> worker2: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4922client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds4923worker2 # [ 111.445470] nix-daemon[841]: reaped child process 1197, status = succeeded4924worker2 # [ 111.517014] nix-daemon[841]: accepted connection from pid 1213, user nix-grpc-daemon (trusted)4925worker2 # [ 111.538663] nix-daemon[841]: reaped child process 1216, status = succeeded4926lb # [ 111.446204] niks3-server[1007]: 2026/09/24 11:49:20 INFO Received push request method=POST path=/api/pushes4927worker2 # [ 111.548510] nix-grpc-daemon[854]: time=2026-09-24T11:49:20.166Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4928lb # [ 111.475592] niks3-server[1007]: 2026/09/24 11:49:20 INFO Received sign narinfos request method=POST path=/api/pushes/34/sign4929worker2 # [ 111.573773] nix-grpc-daemon[854]: time=2026-09-24T11:49:20.192Z level=INFO msg="Uploading 1 narinfos"4930lb # [ 111.490345] niks3-server[1007]: 2026/09/24 11:49:20 INFO Signed narinfos id=34 count=14931lb # [ 111.498521] niks3-server[1007]: 2026/09/24 11:49:20 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls4932lb # [ 111.508357] niks3-server[1007]: 2026/09/24 11:49:20 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4933lb # [ 111.522461] niks3-server[1007]: 2026/09/24 11:49:20 INFO Received complete push request method=POST path=/api/pushes/34/complete4934worker2 # [ 111.621661] nix-grpc-daemon[854]: time=2026-09-24T11:49:20.240Z level=INFO msg="Upload complete. (272ms)"4935lb # [ 111.532076] niks3-server[1007]: 2026/09/24 11:49:20 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo4936worker2 # [ 111.628574] nix-grpc-daemon[784]: ts=2026-09-24T11:49:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60284 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=18 outputs=14937client # copying 1 paths...4938worker2 # [ 111.637366] nix-daemon[841]: reaped child process 1195, status = succeeded4939worker2 # [ 111.639212] nix-grpc-daemon[784]: ts=2026-09-24T11:49:20Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:602844940client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'...4941worker1 # [ 111.601595] nix-grpc-daemon[488]: ts=2026-09-24T11:49:20Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:357764942worker1 # [ 111.616752] nix-grpc-daemon[488]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'...4943worker1 # [ 111.636183] nix-grpc-daemon[488]: ts=2026-09-24T11:49:20Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:35776 duration_s=0 paths=1 nar_bytes_out=1204944client # [ 111.705180] nix-daemon[1127]: reaped child process 1257, status = succeeded4945client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.46 seconds)4946(finished: subtest: an input only one worker has still reaches the builder, in 8.47 seconds)4947subtest: upload whose reference the worker lacks is completed from the cache4948client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache4949client # this derivation will be built:4950client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv4951client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'...4952client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache, in 0.33 seconds)4953client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache4954client # this derivation will be built:4955client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv4956client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'...4957client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache, in 0.29 seconds)4958client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure4959client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.12 seconds)4960worker1: must succeed: nix-store --import < /tmp/shared/ref.closure4961worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.20 seconds)4962worker2 # [ 112.817855] systemd[1]: Stopping Nix worker-protocol over gRPC...4963worker2 # [ 113.270070] nix-grpc-daemon[784]: ts=2026-09-24T11:49:21Z level=info event=signal_exit4964worker2 # [ 115.243894] nix-grpc-daemon[784]: ts=2026-09-24T11:49:23Z level=info event=scheduler_restarting4965worker1 # [ 115.185077] nix-grpc-daemon[488]: ts=2026-09-24T11:49:23Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:451384966worker1 # [ 115.193481] nix-grpc-daemon[488]: ts=2026-09-24T11:49:23Z level=info event=worker_gone addr=192.168.1.4:500514967worker2 # [ 115.281465] nix-grpc-daemon[784]: ts=2026-09-24T11:49:23Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4968worker2 # [ 115.311266] nix-daemon[841]: reaped child process 843, status = succeeded4969worker2 # [ 116.419795] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4970worker2 # [ 116.425469] systemd[1]: Stopped Nix worker-protocol over gRPC.4971worker2 # [ 116.433459] systemd[1]: nix-grpc-daemon.service: Consumed 3.301s CPU time over 1min 21.376s wall clock time, 67.5M memory peak, 183.6K incoming IP traffic, 316.4K outgoing IP traffic.4972lb: waiting until exactly ['worker1'] are healthy members of aarch64-linux4973lb: must succeed: curl -sf localhost:9901/clusters4974lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4975lb: (finished: waiting until exactly ['worker1'] are healthy members of aarch64-linux, in 0.07 seconds)4976client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&24977worker1 # [ 116.674205] nix-grpc-daemon[488]: ts=2026-09-24T11:49:25Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:357764978worker1 # [ 116.684957] nix-grpc-daemon[488]: ts=2026-09-24T11:49:25Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:357764979client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4980client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.26 seconds)4981worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4982worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.03 seconds)4983worker2 # [ 116.875749] systemd[1]: Starting scheduler-token.service...4984worker2 # [ 116.940974] systemd[1]: scheduler-token.service: Deactivated successfully.4985worker2 # [ 116.948473] systemd[1]: Finished scheduler-token.service.4986worker2 # [ 116.950024] systemd[1]: scheduler-token.service: Consumed 36ms CPU time over 68ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4987worker2 # [ 116.955515] systemd[1]: Starting Nix worker-protocol over gRPC...4988worker2 # [ 117.076494] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:25Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4989worker2 # [ 117.079041] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:25Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4990worker2 # [ 117.083075] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:25Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=14991worker2 # [ 117.088059] nix-grpc-daemon[1240]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR4992worker2 # [ 117.089525] nix-grpc-daemon[1240]: E0000 00:00:1790250565.706849 1240 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4993worker2 # [ 117.094173] nix-grpc-daemon[1240]: E0000 00:00:1790250565.713205 1240 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4994worker2 # [ 117.136165] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:25Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket4995worker2 # [ 117.137923] nix-grpc-daemon[1240]: E0000 00:00:1790250565.722429 1283 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4996worker2 # [ 117.142947] nix-grpc-daemon[1240]: E0000 00:00:1790250565.740899 1283 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4997worker2 # [ 117.145892] nix-grpc-daemon[1240]: E0000 00:00:1790250565.741062 1283 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4998worker2 # [ 117.148474] systemd[1]: Started Nix worker-protocol over gRPC.4999worker2 # [ 117.149276] nix-grpc-daemon[1240]: E0000 00:00:1790250565.761034 1283 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5000worker2 # [ 117.157269] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:25Z level=info event=scheduler_connected addr=lb:500515001worker1 # [ 117.089882] nix-grpc-daemon[488]: ts=2026-09-24T11:49:25Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:451385002worker1 # [ 117.105285] nix-grpc-daemon[488]: ts=2026-09-24T11:49:25Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05003worker1 # [ 117.136637] systemd[1]: Stopping Nix worker-protocol over gRPC...5004worker1 # [ 117.439967] nix-grpc-daemon[488]: ts=2026-09-24T11:49:26Z level=info event=signal_exit5005lb # [ 117.837343] niks3-server[1007]: 2026/09/24 11:49:26 INFO lead: released remote=192.168.1.3:460245006worker2 # [ 118.930259] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:27Z level=info event=scheduler_restarting addr=lb:500515007worker1 # [ 118.861697] nix-grpc-daemon[488]: ts=2026-09-24T11:49:27Z level=info event=scheduler_restarting5008lb # [ 118.838211] envoy[907]: [2026-09-24 11:49:27.439][907][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream}5009worker1 # [ 118.862067] nix-grpc-daemon[488]: ts=2026-09-24T11:49:27Z level=info event=scheduler_restarting addr=lb:500515010worker1 # [ 118.872627] nix-grpc-daemon[488]: ts=2026-09-24T11:49:27Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:451385011worker1 # [ 118.882134] nix-grpc-daemon[488]: ts=2026-09-24T11:49:27Z level=info event=worker_gone addr=192.168.1.4:500515012worker1 # [ 118.886702] nix-grpc-daemon[488]: ts=2026-09-24T11:49:27Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:451385013worker1 # [ 118.892666] nix-grpc-daemon[488]: ts=2026-09-24T11:49:27Z level=info event=worker_gone addr=192.168.1.3:500515014worker1 # [ 118.921416] nix-daemon[854]: reaped child process 856, status = succeeded5015worker2 # [ 119.130477] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:27Z level=info event=scheduler_connected addr=lb:500515016lb # [ 119.036299] envoy[907]: conn peer=[2001:db8:1::1]:54216 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3740 tx=2095 ms=22785017worker2 # [ 119.136210] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5018lb # [ 119.044230] envoy[907]: conn peer=[2001:db8:1::1]:54228 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=329 ms=565019lb # [ 119.051072] envoy[907]: conn peer=[2001:db8:1::1]:54242 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3740 tx=2125 ms=19315020lb # [ 119.057181] envoy[907]: conn peer=[2001:db8:1::1]:33346 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=329 ms=595021lb # [ 119.062258] envoy[907]: conn peer=[2001:db8:1::4]:38950 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=14368 tx=1587 ms=799665022worker1 # [ 119.087489] nix-grpc-daemon[488]: ts=2026-09-24T11:49:27Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5023lb # [ 119.070151] envoy[907]: conn peer=[2001:db8:1::1]:33352 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=523 ms=345024worker2 # [ 119.337072] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:27Z level=info event=scheduler_connected addr=lb:500515025worker2 # [ 119.342363] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5026worker2 # [ 119.543292] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:28Z level=info event=scheduler_connected addr=lb:500515027worker2 # [ 119.548723] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5028worker2 # [ 119.749844] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:28Z level=info event=scheduler_connected addr=lb:500515029worker2 # [ 119.755080] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5030worker2 # [ 119.958236] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:28Z level=info event=scheduler_connected addr=lb:500515031worker2 # [ 119.962422] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5032worker2 # [ 120.159385] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:28Z level=info event=scheduler_connected addr=lb:500515033worker2 # [ 120.164743] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5034worker1 # [ 120.290114] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5035worker2 # [ 120.365689] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:28Z level=info event=scheduler_connected addr=lb:500515036worker1 # [ 120.297216] systemd[1]: Stopped Nix worker-protocol over gRPC.5037worker2 # [ 120.371032] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5038worker1 # [ 120.299908] systemd[1]: nix-grpc-daemon.service: Consumed 3.545s CPU time over 1min 50.233s wall clock time, 64.9M memory peak, 447.1K incoming IP traffic, 371.2K outgoing IP traffic.5039lb: waiting until exactly ['worker2'] are healthy members of aarch64-linux5040lb: must succeed: curl -sf localhost:9901/clusters5041lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5042lb: (finished: waiting until exactly ['worker2'] are healthy members of aarch64-linux, in 0.07 seconds)5043client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&25044worker2 # [ 120.572437] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:29Z level=info event=scheduler_connected addr=lb:500515045worker2 # [ 120.577759] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5046worker2 # [ 120.704165] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:496865047worker2 # [ 120.719404] nix-daemon[841]: accepted connection from pid 1240, user nix-grpc-daemon (trusted)5048worker2 # [ 120.746039] nix-grpc-daemon[1240]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 3 ms (attempt 1/5)5049worker2 # [ 120.754621] nix-grpc-daemon[1240]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 144 ms (attempt 2/5)5050worker2 # [ 120.778840] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:29Z level=info event=scheduler_connected addr=lb:500515051worker2 # [ 120.784181] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5052worker2 # [ 120.900954] nix-grpc-daemon[1240]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 323 ms (attempt 3/5)5053worker2 # [ 120.987063] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:29Z level=info event=scheduler_connected addr=lb:500515054worker2 # [ 120.991263] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5055worker2 # [ 121.187796] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:29Z level=info event=scheduler_connected addr=lb:500515056worker2 # [ 121.193022] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5057worker2 # [ 121.225928] nix-grpc-daemon[1240]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 438 ms (attempt 4/5)5058worker2 # [ 121.393794] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:30Z level=info event=scheduler_connected addr=lb:500515059worker2 # [ 121.399094] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5060worker2 # [ 121.600498] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:30Z level=info event=scheduler_connected addr=lb:500515061worker2 # [ 121.606449] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5062worker2 # [ 121.666622] nix-grpc-daemon[1240]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5063worker2 # [ 121.673672] nix-grpc-daemon[1240]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5064client # copying 2 paths...5065client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'...5066worker2 # [ 121.698196] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:496865067client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'...5068worker2 # [ 121.721070] nix-daemon[841]: accepted connection from pid 1240, user nix-grpc-daemon (trusted)5069worker2 # [ 121.752526] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:30Z level=info event=niks3_push_started pid=13005070worker2 # [ 121.808873] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:30Z level=info event=scheduler_connected addr=lb:500515071worker2 # [ 121.822237] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5072worker2 # [ 121.917972] nix-grpc-daemon[1300]: time=2026-09-24T11:49:30.536Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""5073worker2 # [ 122.017180] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:30Z level=info event=scheduler_connected addr=lb:500515074worker2 # [ 122.040292] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5075worker2 # [ 122.046997] nix-daemon[841]: accepted connection from pid 1313, user nix-grpc-daemon (trusted)5076worker2 # [ 122.069640] nix-daemon[841]: reaped child process 1316, status = succeeded5077lb # [ 121.985128] niks3-server[1007]: 2026/09/24 11:49:30 INFO Received push request method=POST path=/api/pushes5078worker2 # [ 122.088137] nix-grpc-daemon[1300]: time=2026-09-24T11:49:30.705Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5079worker2 # [ 122.089851] nix-grpc-daemon[1300]: time=2026-09-24T11:49:30.705Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)"5080worker2 # [ 122.091684] nix-grpc-daemon[1300]: time=2026-09-24T11:49:30.705Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)"5081lb # [ 122.013963] niks3-server[1007]: 2026/09/24 11:49:30 INFO lead: acquired remote=192.168.1.4:574865082worker2 # [ 122.123568] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:30Z level=info event=scheduler_take_over5083lb # [ 122.103258] niks3-server[1007]: 2026/09/24 11:49:30 INFO Received sign narinfos request method=POST path=/api/pushes/35/sign5084worker2 # [ 122.203795] nix-grpc-daemon[1300]: time=2026-09-24T11:49:30.821Z level=INFO msg="Uploading 2 narinfos"5085lb # [ 122.113266] niks3-server[1007]: 2026/09/24 11:49:30 INFO Signed narinfos id=35 count=25086lb # [ 122.122888] niks3-server[1007]: 2026/09/24 11:49:30 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls5087lb # [ 122.128241] niks3-server[1007]: 2026/09/24 11:49:30 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst5088worker2 # [ 122.220557] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:30Z level=info event=scheduler_connected addr=lb:500515089worker2 # [ 122.224436] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5090lb # [ 122.142389] niks3-server[1007]: 2026/09/24 11:49:30 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls5091lb # [ 122.152484] niks3-server[1007]: 2026/09/24 11:49:30 INFO Received complete push request method=POST path=/api/pushes/35/complete5092lb # [ 122.164431] niks3-server[1007]: 2026/09/24 11:49:30 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5093lb # [ 122.166998] niks3-server[1007]: 2026/09/24 11:49:30 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo5094lb # [ 122.171943] niks3-server[1007]: 2026/09/24 11:49:30 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5095worker2 # [ 122.281079] nix-grpc-daemon[1300]: time=2026-09-24T11:49:30.900Z level=INFO msg="Upload complete. (359ms)"5096worker2 # [ 122.292872] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49686 duration_s=0 paths=2 nar_bytes_in=3125097worker2 # [ 122.311285] nix-daemon[841]: reaped child process 1298, status = succeeded5098client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.86 seconds)5099worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache5100worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.03 seconds)5101worker2 # [ 122.425650] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:31Z level=info event=scheduler_connected addr=lb:500515102worker1 # [ 122.364970] systemd[1]: Starting Nix worker-protocol over gRPC...5103worker2 # [ 122.444201] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:31Z level=info event=oidc_jwks provider=mock keys=15104worker2 # [ 122.449664] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:31Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:499145105worker2 # [ 122.455023] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:31Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05106worker1 # [ 122.505426] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:31Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5107worker1 # [ 122.508338] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:31Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5108worker1 # [ 122.512446] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:31Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15109worker1 # [ 122.519351] nix-grpc-daemon[1302]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5110worker1 # [ 122.544452] nix-grpc-daemon[1302]: E0000 00:00:1790250571.132739 1302 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5111worker1 # [ 122.546837] nix-grpc-daemon[1302]: E0000 00:00:1790250571.134835 1302 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5112lb: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux5113lb: must succeed: curl -sf localhost:9901/clusters5114worker1 # [ 122.569989] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:31Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket5115worker1 # [ 122.572283] nix-grpc-daemon[1302]: E0000 00:00:1790250571.145339 1345 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5116worker1 # [ 122.574804] nix-grpc-daemon[1302]: E0000 00:00:1790250571.168067 1345 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5117worker1 # [ 122.577148] nix-grpc-daemon[1302]: E0000 00:00:1790250571.168229 1345 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5118worker1 # [ 122.579466] nix-grpc-daemon[1302]: E0000 00:00:1790250571.184042 1345 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5119worker1 # [ 122.585028] systemd[1]: Started Nix worker-protocol over gRPC.5120worker1 # [ 122.587824] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:31Z level=info event=scheduler_connected addr=lb:500515121worker2 # [ 122.659729] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:31Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:499145122worker2 # [ 122.666292] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:31Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05123lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5124lb: must succeed: curl -sf localhost:9901/clusters5125lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5126lb: (finished: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux, in 1.15 seconds)5127(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.98 seconds)5128subtest: a niks3 restart does not cost the leader its role5129lb: must succeed: curl -sf localhost:9901/clusters5130lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5131lb: must succeed: curl -sf localhost:9901/clusters5132lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5133lb # [ 123.813805] systemd[1]: Stopping niks3 server...5134lb # [ 123.814484] niks3-server[1007]: 2026/09/24 11:49:32 INFO Shutdown signal received, draining in-flight requests timeout=10s5135lb # [ 123.814568] niks3-server[1007]: 2026/09/24 11:49:32 INFO lead: released remote=192.168.1.4:574865136worker1 # [ 123.835651] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:32Z level=info event=scheduler_lock_lost status=2005137worker2 # [ 123.906528] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:32Z level=info event=scheduler_lock_lost status=2005138lb # [ 123.841003] systemd[1]: niks3.service: Deactivated successfully.5139lb # [ 123.842691] systemd[1]: Stopped niks3 server.5140lb # [ 123.844371] systemd[1]: niks3.service: Consumed 687ms CPU time over 1min 36.292s wall clock time, 27.8M memory peak, 99.1K incoming IP traffic, 143.2K outgoing IP traffic.5141lb # [ 123.853387] systemd[1]: Starting niks3 server...5142lb # [ 123.907286] niks3-server[1164]: 2026/09/24 11:49:32 goose: no migrations to run. current version: 202609231200005143lb # [ 123.913597] niks3-server[1164]: 2026/09/24 11:49:32 OK 1_commit_pending_closure.sql (4.42ms)5144lb # [ 123.916836] niks3-server[1164]: 2026/09/24 11:49:32 OK 2_object_stats_trigger.sql (3.19ms)5145lb # [ 123.919848] niks3-server[1164]: 2026/09/24 11:49:32 OK 3_commit_push.sql (2.7ms)5146lb # [ 123.921826] niks3-server[1164]: 2026/09/24 11:49:32 goose: up to current file version: 35147lb # [ 123.923537] niks3-server[1164]: 2026/09/24 11:49:32 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key5148lb # [ 123.933852] niks3-server[1164]: 2026/09/24 11:49:32 INFO Read proxy enabled — serving cache objects from S35149lb # [ 123.935219] niks3-server[1164]: 2026/09/24 11:49:32 INFO Using socket-activated listener address=0.0.0.0:57515150lb # [ 123.940472] niks3-server[1164]: 2026/09/24 11:49:32 INFO systemd watchdog enabled interval=15s5151lb # [ 123.942227] niks3-server[1164]: 2026/09/24 11:49:32 INFO Starting HTTP server address=0.0.0.0:57515152lb # [ 123.943568] systemd[1]: Started niks3 server.5153lb: waiting for unit niks3.service5154lb: (finished: waiting for unit niks3.service, in 0.03 seconds)5155lb: must succeed: curl -sf localhost:9901/clusters5156lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5157worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5158worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5159lb: must succeed: curl -sf localhost:9901/clusters5160lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5161lb: must succeed: curl -sf localhost:9901/clusters5162lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5163worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5164worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.06 seconds)5165client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&25166worker1 # [ 124.452155] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:33Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:402945167worker1 # [ 124.467721] nix-daemon[854]: accepted connection from pid 1302, user nix-grpc-daemon (trusted)5168client # don't know how to build these paths:5169client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5170worker1 # [ 124.491531] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:402945171worker1 # [ 124.507306] nix-grpc-daemon[1302]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 6 ms (attempt 1/5)5172worker1 # [ 124.522488] nix-grpc-daemon[1302]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 121 ms (attempt 2/5)5173worker1 # [ 124.652798] nix-grpc-daemon[1302]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 295 ms (attempt 3/5)5174lb # [ 124.820395] niks3-server[1164]: 2026/09/24 11:49:33 INFO lead: acquired remote=192.168.1.4:463425175worker1 # [ 124.950879] nix-grpc-daemon[1302]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 744 ms (attempt 4/5)5176worker2 # [ 125.475486] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:34Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:499305177worker1 # [ 125.696347] nix-grpc-daemon[1302]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5178worker1 # [ 125.698618] nix-grpc-daemon[1302]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5179worker2 # [ 125.814099] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:34Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=05180worker2 # [ 125.821942] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:34Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.4:50051 clients=15181worker2 # [ 125.833714] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:496865182worker2 # [ 125.845350] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:496865183client # copying 1 paths to 'lb:50051'...5184worker2 # [ 125.861300] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:496865185client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'...5186worker2 # [ 125.878566] nix-daemon[841]: accepted connection from pid 1240, user nix-grpc-daemon (trusted)5187worker2 # [ 126.039997] nix-daemon[841]: accepted connection from pid 1335, user nix-grpc-daemon (trusted)5188worker2 # [ 126.060880] nix-daemon[841]: reaped child process 1338, status = succeeded5189lb # [ 125.974889] niks3-server[1164]: 2026/09/24 11:49:34 INFO Received push request method=POST path=/api/pushes5190worker2 # [ 126.082123] nix-grpc-daemon[1300]: time=2026-09-24T11:49:34.700Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5191worker2 # [ 126.083794] nix-grpc-daemon[1300]: time=2026-09-24T11:49:34.700Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)"5192lb # [ 126.018699] niks3-server[1164]: 2026/09/24 11:49:34 INFO Received sign narinfos request method=POST path=/api/pushes/36/sign5193lb # [ 126.046375] niks3-server[1164]: 2026/09/24 11:49:34 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls5194lb # [ 126.082126] niks3-server[1164]: 2026/09/24 11:49:34 INFO Signed narinfos id=36 count=15195worker2 # [ 126.175610] nix-grpc-daemon[1300]: time=2026-09-24T11:49:34.793Z level=INFO msg="Uploading 1 narinfos"5196lb # [ 126.085463] niks3-server[1164]: 2026/09/24 11:49:34 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst5197lb # [ 126.096119] niks3-server[1164]: 2026/09/24 11:49:34 INFO Received complete push request method=POST path=/api/pushes/36/complete5198worker2 # [ 126.201703] nix-grpc-daemon[1300]: time=2026-09-24T11:49:34.820Z level=INFO msg="Upload complete. (306ms)"5199lb # [ 126.113776] niks3-server[1164]: 2026/09/24 11:49:34 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.narinfo5200worker2 # [ 126.216660] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49686 duration_s=0 paths=1 nar_bytes_in=5445201worker2 # [ 126.231997] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:496865202worker2 # [ 126.241807] nix-daemon[841]: accepted connection from pid 1240, user nix-grpc-daemon (trusted)5203worker2 # [ 126.256227] nix-daemon[841]: reaped child process 1333, status = succeeded5204client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051...5205client # farm-a> worker2: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5206worker2 # [ 126.266120] nix-daemon[841]: accepted connection from pid 1240, user nix-grpc-daemon (trusted)5207client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5208client # farm-a> LOG-farm-a-after-niks3-restart5209worker2 # [ 127.737607] nix-daemon[841]: reaped child process 1342, status = succeeded5210worker2 # [ 127.852432] nix-daemon[841]: accepted connection from pid 1358, user nix-grpc-daemon (trusted)5211worker2 # [ 127.872738] nix-daemon[841]: reaped child process 1361, status = succeeded5212lb # [ 127.787209] niks3-server[1164]: 2026/09/24 11:49:36 INFO Received push request method=POST path=/api/pushes5213worker2 # [ 127.894344] nix-grpc-daemon[1300]: time=2026-09-24T11:49:36.511Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5214worker2 # [ 127.896192] nix-grpc-daemon[1300]: time=2026-09-24T11:49:36.511Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)"5215lb # [ 127.839655] niks3-server[1164]: 2026/09/24 11:49:36 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5216worker2 # [ 127.945821] nix-grpc-daemon[1300]: time=2026-09-24T11:49:36.564Z level=INFO msg="Uploading 1 narinfos"5217lb # [ 127.856678] niks3-server[1164]: 2026/09/24 11:49:36 INFO Received sign narinfos request method=POST path=/api/pushes/37/sign5218lb # [ 127.865571] niks3-server[1164]: 2026/09/24 11:49:36 INFO Signed narinfos id=37 count=15219lb # [ 127.871688] niks3-server[1164]: 2026/09/24 11:49:36 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls5220lb # [ 127.879135] niks3-server[1164]: 2026/09/24 11:49:36 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5221lb # [ 127.891818] niks3-server[1164]: 2026/09/24 11:49:36 INFO Received complete push request method=POST path=/api/pushes/37/complete5222lb # [ 127.901633] niks3-server[1164]: 2026/09/24 11:49:36 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.narinfo5223worker2 # [ 127.998567] nix-grpc-daemon[1300]: time=2026-09-24T11:49:36.617Z level=INFO msg="Upload complete. (318ms)"5224worker2 # [ 128.019788] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49686 duration_s=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=15225worker2 # [ 128.039691] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:36Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=05226worker2 # [ 128.052955] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:36Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.4:50051 clients=15227worker2 # [ 128.062795] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:496865228worker2 # [ 128.069268] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:496865229worker2 # [ 128.075416] nix-daemon[841]: reaped child process 1340, status = succeeded5230client # copying 1 paths to 'lb:50051'...5231worker2 # [ 128.077996] nix-grpc-daemon[1300]: time=2026-09-24T11:49:36.693Z level=INFO msg="All 1 paths already cached"5232client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'...5233worker2 # [ 128.082523] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:496865234worker2 # [ 128.094995] nix-daemon[841]: accepted connection from pid 1240, user nix-grpc-daemon (trusted)5235worker2 # [ 128.240116] nix-daemon[841]: accepted connection from pid 1367, user nix-grpc-daemon (trusted)5236worker2 # [ 128.262111] nix-daemon[841]: reaped child process 1370, status = succeeded5237lb # [ 128.171546] niks3-server[1164]: 2026/09/24 11:49:36 INFO Received push request method=POST path=/api/pushes5238worker2 # [ 128.273602] nix-grpc-daemon[1300]: time=2026-09-24T11:49:36.892Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5239worker2 # [ 128.275131] nix-grpc-daemon[1300]: time=2026-09-24T11:49:36.892Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)"5240lb # [ 128.212348] niks3-server[1164]: 2026/09/24 11:49:36 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst5241worker2 # [ 128.316531] nix-grpc-daemon[1300]: time=2026-09-24T11:49:36.935Z level=INFO msg="Uploading 1 narinfos"5242lb # [ 128.226298] niks3-server[1164]: 2026/09/24 11:49:36 INFO Received sign narinfos request method=POST path=/api/pushes/38/sign5243lb # [ 128.233873] niks3-server[1164]: 2026/09/24 11:49:36 INFO Signed narinfos id=38 count=15244lb # [ 128.239094] niks3-server[1164]: 2026/09/24 11:49:36 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls5245lb # [ 128.254624] niks3-server[1164]: 2026/09/24 11:49:36 INFO Received complete push request method=POST path=/api/pushes/38/complete5246lb # [ 128.261526] niks3-server[1164]: 2026/09/24 11:49:36 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo5247worker2 # [ 128.357734] nix-grpc-daemon[1300]: time=2026-09-24T11:49:36.976Z level=INFO msg="Upload complete. (250ms)"5248worker2 # [ 128.367815] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49686 duration_s=0 paths=1 nar_bytes_in=6565249worker2 # [ 128.388805] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:496865250worker2 # [ 128.397897] nix-daemon[841]: accepted connection from pid 1240, user nix-grpc-daemon (trusted)5251worker2 # [ 128.404243] nix-daemon[841]: reaped child process 1365, status = succeeded5252client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051...5253worker2 # [ 128.418876] nix-daemon[841]: accepted connection from pid 1240, user nix-grpc-daemon (trusted)5254client # farm-b> worker2: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5255client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5256lb # [ 129.041387] envoy[907]: conn peer=192.168.1.3:44210 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2349 tx=1139 ms=820255257lb # [ 129.046498] envoy[907]: conn peer=[2001:db8:1::1]:33358 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1005 tx=305 ms=16045258client # farm-b> LOG-farm-b-after-niks3-restart5259worker2 # [ 129.322858] nix-daemon[841]: reaped child process 1374, status = succeeded5260worker2 # [ 129.390285] nix-daemon[841]: accepted connection from pid 1390, user nix-grpc-daemon (trusted)5261worker2 # [ 129.412421] nix-daemon[841]: reaped child process 1393, status = succeeded5262lb # [ 129.324857] niks3-server[1164]: 2026/09/24 11:49:37 INFO Received push request method=POST path=/api/pushes5263worker2 # [ 129.427240] nix-grpc-daemon[1300]: time=2026-09-24T11:49:38.045Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5264worker2 # [ 129.429562] nix-grpc-daemon[1300]: time=2026-09-24T11:49:38.045Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)"5265worker2 # [ 129.463472] nix-grpc-daemon[1300]: time=2026-09-24T11:49:38.082Z level=INFO msg="Uploading 1 narinfos"5266lb # [ 129.365134] niks3-server[1164]: 2026/09/24 11:49:37 INFO Received sign narinfos request method=POST path=/api/pushes/39/sign5267lb # [ 129.385057] niks3-server[1164]: 2026/09/24 11:49:37 INFO Signed narinfos id=39 count=15268lb # [ 129.392989] niks3-server[1164]: 2026/09/24 11:49:37 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst5269lb # [ 129.409003] niks3-server[1164]: 2026/09/24 11:49:37 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5270worker2 # [ 129.510583] nix-grpc-daemon[1300]: time=2026-09-24T11:49:38.129Z level=INFO msg="Upload complete. (277ms)"5271lb # [ 129.420616] niks3-server[1164]: 2026/09/24 11:49:38 INFO Received complete push request method=POST path=/api/pushes/39/complete5272lb # [ 129.425779] niks3-server[1164]: 2026/09/24 11:49:38 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.narinfo5273lb # [ 129.431282] niks3-server[1164]: 2026/09/24 11:49:38 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls5274worker2 # [ 129.525036] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49686 duration_s=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=15275worker2 # [ 129.548341] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:38Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=05276worker2 # [ 129.553035] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:38Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.4:50051 clients=15277worker2 # [ 129.559114] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:496865278worker2 # [ 129.564947] nix-daemon[841]: reaped child process 1372, status = succeeded5279worker2 # [ 129.568999] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:496865280worker2 # [ 129.581668] nix-grpc-daemon[1300]: time=2026-09-24T11:49:38.200Z level=INFO msg="All 2 paths already cached"5281client # copying 1 paths to 'lb:50051'...5282client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'...5283worker2 # [ 129.592322] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:496865284worker2 # [ 129.607403] nix-daemon[841]: accepted connection from pid 1240, user nix-grpc-daemon (trusted)5285worker2 # [ 129.753372] nix-daemon[841]: accepted connection from pid 1399, user nix-grpc-daemon (trusted)5286worker2 # [ 129.774699] nix-daemon[841]: reaped child process 1402, status = succeeded5287lb # [ 129.687394] niks3-server[1164]: 2026/09/24 11:49:38 INFO Received push request method=POST path=/api/pushes5288worker2 # [ 129.787940] nix-grpc-daemon[1300]: time=2026-09-24T11:49:38.406Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5289worker2 # [ 129.789729] nix-grpc-daemon[1300]: time=2026-09-24T11:49:38.406Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)"5290lb # [ 129.718449] niks3-server[1164]: 2026/09/24 11:49:38 INFO Received sign narinfos request method=POST path=/api/pushes/40/sign5291worker2 # [ 129.818694] nix-grpc-daemon[1300]: time=2026-09-24T11:49:38.437Z level=INFO msg="Uploading 1 narinfos"5292lb # [ 129.731711] niks3-server[1164]: 2026/09/24 11:49:38 INFO Signed narinfos id=40 count=15293lb # [ 129.740842] niks3-server[1164]: 2026/09/24 11:49:38 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls5294lb # [ 129.748749] niks3-server[1164]: 2026/09/24 11:49:38 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst5295lb # [ 129.760363] niks3-server[1164]: 2026/09/24 11:49:38 INFO Received complete push request method=POST path=/api/pushes/40/complete5296lb # [ 129.765004] niks3-server[1164]: 2026/09/24 11:49:38 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo5297worker2 # [ 129.855502] nix-grpc-daemon[1300]: time=2026-09-24T11:49:38.474Z level=INFO msg="Upload complete. (233ms)"5298worker2 # [ 129.866957] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:49686 duration_s=0 paths=1 nar_bytes_in=7845299worker2 # [ 129.882631] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:496865300worker2 # [ 129.890073] nix-daemon[841]: accepted connection from pid 1240, user nix-grpc-daemon (trusted)5301worker2 # [ 129.895427] nix-daemon[841]: reaped child process 1397, status = succeeded5302client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051...5303worker2 # [ 129.909591] nix-daemon[841]: accepted connection from pid 1240, user nix-grpc-daemon (trusted)5304client # farm-top> worker2: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5305client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5306client # farm-top> LOG-farm-top-after-niks3-restart5307worker2 # [ 130.963580] nix-daemon[841]: reaped child process 1406, status = succeeded5308worker2 # [ 131.071533] nix-daemon[841]: accepted connection from pid 1422, user nix-grpc-daemon (trusted)5309worker2 # [ 131.090746] nix-daemon[841]: reaped child process 1425, status = succeeded5310lb # [ 131.010229] niks3-server[1164]: 2026/09/24 11:49:39 INFO Received push request method=POST path=/api/pushes5311worker2 # [ 131.110373] nix-grpc-daemon[1300]: time=2026-09-24T11:49:39.728Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5312worker2 # [ 131.111898] nix-grpc-daemon[1300]: time=2026-09-24T11:49:39.728Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)"5313lb # [ 131.053958] niks3-server[1164]: 2026/09/24 11:49:39 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5314worker2 # [ 131.157000] nix-grpc-daemon[1300]: time=2026-09-24T11:49:39.775Z level=INFO msg="Uploading 1 narinfos"5315lb # [ 131.068088] niks3-server[1164]: 2026/09/24 11:49:39 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst5316lb # [ 131.076726] niks3-server[1164]: 2026/09/24 11:49:39 INFO Received sign narinfos request method=POST path=/api/pushes/41/sign5317lb # [ 131.079664] niks3-server[1164]: 2026/09/24 11:49:39 INFO Signed narinfos id=41 count=15318lb # [ 131.091925] niks3-server[1164]: 2026/09/24 11:49:39 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls5319worker2 # [ 131.183058] nix-grpc-daemon[1300]: time=2026-09-24T11:49:39.802Z level=INFO msg="Upload complete. (259ms)"5320lb # [ 131.095107] niks3-server[1164]: 2026/09/24 11:49:39 INFO Received complete push request method=POST path=/api/pushes/41/complete5321lb # [ 131.097956] niks3-server[1164]: 2026/09/24 11:49:39 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo5322worker2 # [ 131.196742] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:49686 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=15323worker2 # [ 131.217021] nix-daemon[841]: reaped child process 1404, status = succeeded5324client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2, in 6.96 seconds)5325client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath5326client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath, in 0.19 seconds)5327(finished: subtest: a niks3 restart does not cost the leader its role, in 7.65 seconds)5328subtest: scheduler down: the other node takes the lock and keeps it5329lb: must succeed: curl -sf localhost:9901/clusters5330lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5331lb: must succeed: curl -sf localhost:9901/clusters5332lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5333lb: must succeed: curl -sf localhost:9901/clusters5334lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5335lb: must succeed: curl -sf localhost:9901/clusters5336lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5337worker2 # [ 131.683577] systemd[1]: Stopping Nix worker-protocol over gRPC...5338worker2 # [ 132.114978] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:40Z level=info event=signal_exit5339lb # [ 132.829356] niks3-server[1164]: 2026/09/24 11:49:41 INFO lead: released remote=192.168.1.4:463425340worker2 # [ 133.921569] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:42Z level=info event=scheduler_restarting5341lb # [ 133.828386] envoy[907]: [2026-09-24 11:49:42.430][907][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream}5342worker1 # [ 133.861359] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:42Z level=info event=scheduler_restarting addr=lb:500515343worker2 # [ 133.932144] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:42Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:499145344worker2 # [ 133.943001] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:42Z level=info event=worker_gone addr=192.168.1.3:500515345worker2 # [ 133.947864] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:42Z level=info event=scheduler_restarting addr=lb:500515346worker2 # [ 133.952774] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:42Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:499145347worker2 # [ 133.958066] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:42Z level=info event=worker_gone addr=192.168.1.4:500515348worker2 # [ 133.983283] nix-daemon[841]: reaped child process 1292, status = succeeded5349worker1 # [ 134.066838] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:42Z level=info event=scheduler_connected addr=lb:500515350worker1 # [ 134.072420] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5351worker2 # [ 134.161807] nix-grpc-daemon[1240]: ts=2026-09-24T11:49:42Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5352worker1 # [ 134.273400] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:42Z level=info event=scheduler_connected addr=lb:500515353worker1 # [ 134.279058] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5354worker1 # [ 134.480140] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:43Z level=info event=scheduler_connected addr=lb:500515355worker1 # [ 134.486109] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5356worker1 # [ 134.686808] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:43Z level=info event=scheduler_connected addr=lb:500515357worker1 # [ 134.692409] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5358worker1 # [ 134.842841] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:43Z level=info event=scheduler_take_over5359lb # [ 134.822843] niks3-server[1164]: 2026/09/24 11:49:43 INFO lead: acquired remote=192.168.1.3:468745360worker1 # [ 134.895271] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:43Z level=info event=scheduler_connected addr=lb:500515361worker1 # [ 134.899543] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5362worker1 # [ 135.096083] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:43Z level=info event=scheduler_connected addr=lb:500515363worker1 # [ 135.101457] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5364worker1 # [ 135.302397] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:43Z level=info event=scheduler_connected addr=lb:500515365worker1 # [ 135.307924] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5366worker1 # [ 135.380067] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:43Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:595905367worker1 # [ 135.508905] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:44Z level=info event=scheduler_connected addr=lb:500515368worker1 # [ 135.517979] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:44Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:595945369worker1 # [ 135.527846] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:44Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05370worker2 # [ 136.192225] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5371worker2 # [ 136.197658] systemd[1]: Stopped Nix worker-protocol over gRPC.5372worker2 # [ 136.205375] systemd[1]: nix-grpc-daemon.service: Consumed 1.479s CPU time over 19.238s wall clock time, 39.4M memory peak, 202.6K incoming IP traffic, 128.1K outgoing IP traffic.5373worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5374worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.08 seconds)5375worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5376worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5377client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&25378client # don't know how to build these paths:5379client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5380worker1 # [ 136.510391] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:45Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:596025381worker1 # [ 136.518780] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596025382worker1 # [ 136.544088] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:45Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=05383worker1 # [ 136.554052] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:45Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.3:50051 clients=15384worker1 # [ 136.562890] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596025385worker1 # [ 136.570985] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596025386client # copying 1 paths to 'lb:50051'...5387client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'...5388worker1 # [ 136.581866] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596025389worker1 # [ 136.597175] nix-daemon[854]: accepted connection from pid 1302, user nix-grpc-daemon (trusted)5390worker1 # [ 136.625927] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:45Z level=info event=niks3_push_started pid=13755391worker1 # [ 136.762254] nix-grpc-daemon[1375]: time=2026-09-24T11:49:45.377Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""5392worker1 # [ 136.874230] nix-daemon[854]: accepted connection from pid 1388, user nix-grpc-daemon (trusted)5393worker1 # [ 136.894934] nix-daemon[854]: reaped child process 1391, status = succeeded5394lb # [ 136.882075] niks3-server[1164]: 2026/09/24 11:49:45 INFO Received push request method=POST path=/api/pushes5395worker1 # [ 136.913355] nix-grpc-daemon[1375]: time=2026-09-24T11:49:45.527Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5396worker1 # [ 136.914877] nix-grpc-daemon[1375]: time=2026-09-24T11:49:45.527Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)"5397lb # [ 136.928359] niks3-server[1164]: 2026/09/24 11:49:45 INFO Received sign narinfos request method=POST path=/api/pushes/42/sign5398worker1 # [ 136.958615] nix-grpc-daemon[1375]: time=2026-09-24T11:49:45.573Z level=INFO msg="Uploading 1 narinfos"5399lb # [ 136.940244] niks3-server[1164]: 2026/09/24 11:49:45 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst5400lb # [ 136.950246] niks3-server[1164]: 2026/09/24 11:49:45 INFO Signed narinfos id=42 count=15401lb # [ 136.954886] niks3-server[1164]: 2026/09/24 11:49:45 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls5402lb # [ 136.965643] niks3-server[1164]: 2026/09/24 11:49:45 INFO Received complete push request method=POST path=/api/pushes/42/complete5403worker1 # [ 136.990165] nix-grpc-daemon[1375]: time=2026-09-24T11:49:45.605Z level=INFO msg="Upload complete. (224ms)"5404lb # [ 136.970698] niks3-server[1164]: 2026/09/24 11:49:45 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo5405worker1 # [ 136.999946] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59602 duration_s=0 paths=1 nar_bytes_in=5205406worker1 # [ 137.014975] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596025407worker1 # [ 137.022448] nix-daemon[854]: accepted connection from pid 1302, user nix-grpc-daemon (trusted)5408worker1 # [ 137.027973] nix-daemon[854]: reaped child process 1373, status = succeeded5409client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051...5410worker1 # [ 137.041254] nix-daemon[854]: accepted connection from pid 1302, user nix-grpc-daemon (trusted)5411client # farm-a> node-a: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5412client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5413client # farm-a> LOG-farm-a-on-standby5414worker1 # [ 137.907974] nix-daemon[854]: reaped child process 1395, status = succeeded5415worker1 # [ 138.019383] nix-daemon[854]: accepted connection from pid 1411, user nix-grpc-daemon (trusted)5416worker1 # [ 138.042642] nix-daemon[854]: reaped child process 1414, status = succeeded5417lb # [ 138.026113] niks3-server[1164]: 2026/09/24 11:49:46 INFO Received push request method=POST path=/api/pushes5418worker1 # [ 138.059494] nix-grpc-daemon[1375]: time=2026-09-24T11:49:46.672Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5419worker1 # [ 138.062199] nix-grpc-daemon[1375]: time=2026-09-24T11:49:46.672Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)"5420lb # [ 138.063201] niks3-server[1164]: 2026/09/24 11:49:46 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5421lb # [ 138.086719] niks3-server[1164]: 2026/09/24 11:49:46 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5422worker1 # [ 138.117632] nix-grpc-daemon[1375]: time=2026-09-24T11:49:46.732Z level=INFO msg="Uploading 1 narinfos"5423lb # [ 138.098554] niks3-server[1164]: 2026/09/24 11:49:46 INFO Received sign narinfos request method=POST path=/api/pushes/43/sign5424lb # [ 138.104226] niks3-server[1164]: 2026/09/24 11:49:46 INFO Signed narinfos id=43 count=15425lb # [ 138.111454] niks3-server[1164]: 2026/09/24 11:49:46 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls5426lb # [ 138.120135] niks3-server[1164]: 2026/09/24 11:49:46 INFO Received complete push request method=POST path=/api/pushes/43/complete5427lb # [ 138.128697] niks3-server[1164]: 2026/09/24 11:49:46 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo5428worker1 # [ 138.154152] nix-grpc-daemon[1375]: time=2026-09-24T11:49:46.769Z level=INFO msg="Upload complete. (298ms)"5429worker1 # [ 138.176841] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59602 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=15430worker1 # [ 138.192369] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:46Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=05431worker1 # [ 138.202890] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:46Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.3:50051 clients=15432worker1 # [ 138.209488] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596025433worker1 # [ 138.215498] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596025434worker1 # [ 138.220287] nix-daemon[854]: reaped child process 1393, status = succeeded5435worker1 # [ 138.226676] nix-grpc-daemon[1375]: time=2026-09-24T11:49:46.842Z level=INFO msg="All 1 paths already cached"5436client # copying 1 paths to 'lb:50051'...5437client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'...5438worker1 # [ 138.236109] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596025439worker1 # [ 138.251000] nix-daemon[854]: accepted connection from pid 1302, user nix-grpc-daemon (trusted)5440worker1 # [ 138.396637] nix-daemon[854]: accepted connection from pid 1420, user nix-grpc-daemon (trusted)5441worker1 # [ 138.421786] nix-daemon[854]: reaped child process 1423, status = succeeded5442lb # [ 138.404515] niks3-server[1164]: 2026/09/24 11:49:47 INFO Received push request method=POST path=/api/pushes5443worker1 # [ 138.436116] nix-grpc-daemon[1375]: time=2026-09-24T11:49:47.051Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5444worker1 # [ 138.438357] nix-grpc-daemon[1375]: time=2026-09-24T11:49:47.051Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)"5445lb # [ 138.440878] niks3-server[1164]: 2026/09/24 11:49:47 INFO Received sign narinfos request method=POST path=/api/pushes/44/sign5446worker1 # [ 138.469364] nix-grpc-daemon[1375]: time=2026-09-24T11:49:47.084Z level=INFO msg="Uploading 1 narinfos"5447lb # [ 138.452788] niks3-server[1164]: 2026/09/24 11:49:47 INFO Signed narinfos id=44 count=15448lb # [ 138.461525] niks3-server[1164]: 2026/09/24 11:49:47 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst5449lb # [ 138.470501] niks3-server[1164]: 2026/09/24 11:49:47 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls5450lb # [ 138.481384] niks3-server[1164]: 2026/09/24 11:49:47 INFO Received complete push request method=POST path=/api/pushes/44/complete5451lb # [ 138.488291] niks3-server[1164]: 2026/09/24 11:49:47 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo5452worker1 # [ 138.508198] nix-grpc-daemon[1375]: time=2026-09-24T11:49:47.123Z level=INFO msg="Upload complete. (247ms)"5453worker1 # [ 138.518125] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59602 duration_s=0 paths=1 nar_bytes_in=6405454worker1 # [ 138.537089] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596025455worker1 # [ 138.543061] nix-daemon[854]: accepted connection from pid 1302, user nix-grpc-daemon (trusted)5456worker1 # [ 138.549233] nix-daemon[854]: reaped child process 1418, status = succeeded5457client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051...5458client # farm-b> node-a: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5459worker1 # [ 138.564279] nix-daemon[854]: accepted connection from pid 1302, user nix-grpc-daemon (trusted)5460lb # [ 139.043363] envoy[907]: conn peer=[2001:db8:1::1]:54714 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9693 tx=4458 ms=66925461lb # [ 139.049485] envoy[907]: conn peer=[2001:db8:1::4]:40130 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=14701 tx=1068 ms=169975462client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5463client # farm-b> LOG-farm-b-on-standby5464worker1 # [ 140.274367] nix-daemon[854]: reaped child process 1427, status = succeeded5465worker1 # [ 140.352584] nix-daemon[854]: accepted connection from pid 1443, user nix-grpc-daemon (trusted)5466worker1 # [ 140.376768] nix-daemon[854]: reaped child process 1446, status = succeeded5467lb # [ 140.354893] niks3-server[1164]: 2026/09/24 11:49:48 INFO Received push request method=POST path=/api/pushes5468worker1 # [ 140.386241] nix-grpc-daemon[1375]: time=2026-09-24T11:49:49.000Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5469worker1 # [ 140.387747] nix-grpc-daemon[1375]: time=2026-09-24T11:49:49.000Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)"5470lb # [ 140.394978] niks3-server[1164]: 2026/09/24 11:49:48 INFO Received sign narinfos request method=POST path=/api/pushes/45/sign5471worker1 # [ 140.427233] nix-grpc-daemon[1375]: time=2026-09-24T11:49:49.042Z level=INFO msg="Uploading 1 narinfos"5472lb # [ 140.410433] niks3-server[1164]: 2026/09/24 11:49:49 INFO Signed narinfos id=45 count=15473lb # [ 140.420378] niks3-server[1164]: 2026/09/24 11:49:49 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls5474lb # [ 140.430291] niks3-server[1164]: 2026/09/24 11:49:49 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5475lb # [ 140.443314] niks3-server[1164]: 2026/09/24 11:49:49 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst5476lb # [ 140.447826] niks3-server[1164]: 2026/09/24 11:49:49 INFO Received complete push request method=POST path=/api/pushes/45/complete5477lb # [ 140.452375] niks3-server[1164]: 2026/09/24 11:49:49 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo5478worker1 # [ 140.472391] nix-grpc-daemon[1375]: time=2026-09-24T11:49:49.087Z level=INFO msg="Upload complete. (295ms)"5479worker1 # [ 140.492792] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59602 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=15480worker1 # [ 140.512817] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:49Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=05481worker1 # [ 140.519108] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:49Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.3:50051 clients=15482worker1 # [ 140.526572] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596025483worker1 # [ 140.532622] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596025484worker1 # [ 140.538571] nix-daemon[854]: reaped child process 1425, status = succeeded5485worker1 # [ 140.541655] nix-grpc-daemon[1375]: time=2026-09-24T11:49:49.157Z level=INFO msg="All 2 paths already cached"5486client # copying 1 paths to 'lb:50051'...5487client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'...5488worker1 # [ 140.551421] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596025489worker1 # [ 140.565177] nix-daemon[854]: accepted connection from pid 1302, user nix-grpc-daemon (trusted)5490worker1 # [ 140.693266] nix-daemon[854]: accepted connection from pid 1452, user nix-grpc-daemon (trusted)5491worker1 # [ 140.712996] nix-daemon[854]: reaped child process 1455, status = succeeded5492lb # [ 140.697923] niks3-server[1164]: 2026/09/24 11:49:49 INFO Received push request method=POST path=/api/pushes5493worker1 # [ 140.729057] nix-grpc-daemon[1375]: time=2026-09-24T11:49:49.344Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5494worker1 # [ 140.730736] nix-grpc-daemon[1375]: time=2026-09-24T11:49:49.344Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)"5495worker1 # [ 140.749261] nix-grpc-daemon[1375]: time=2026-09-24T11:49:49.364Z level=INFO msg="Uploading 1 narinfos"5496lb # [ 140.728366] niks3-server[1164]: 2026/09/24 11:49:49 INFO Received sign narinfos request method=POST path=/api/pushes/46/sign5497lb # [ 140.741861] niks3-server[1164]: 2026/09/24 11:49:49 INFO Signed narinfos id=46 count=15498lb # [ 140.749919] niks3-server[1164]: 2026/09/24 11:49:49 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst5499lb # [ 140.766304] niks3-server[1164]: 2026/09/24 11:49:49 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls5500lb # [ 140.771522] niks3-server[1164]: 2026/09/24 11:49:49 INFO Received complete push request method=POST path=/api/pushes/46/complete5501worker1 # [ 140.796511] nix-grpc-daemon[1375]: time=2026-09-24T11:49:49.411Z level=INFO msg="Upload complete. (221ms)"5502lb # [ 140.777485] niks3-server[1164]: 2026/09/24 11:49:49 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo5503worker1 # [ 140.808104] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59602 duration_s=0 paths=1 nar_bytes_in=7685504worker1 # [ 140.824759] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596025505worker1 # [ 140.831284] nix-daemon[854]: accepted connection from pid 1302, user nix-grpc-daemon (trusted)5506worker1 # [ 140.837207] nix-daemon[854]: reaped child process 1450, status = succeeded5507client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051...5508client # farm-top> node-a: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5509worker1 # [ 140.851118] nix-daemon[854]: accepted connection from pid 1302, user nix-grpc-daemon (trusted)5510client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5511client # farm-top> LOG-farm-top-on-standby5512worker1 # [ 142.140759] nix-daemon[854]: reaped child process 1459, status = succeeded5513worker1 # [ 142.245844] nix-daemon[854]: accepted connection from pid 1475, user nix-grpc-daemon (trusted)5514worker1 # [ 142.268094] nix-daemon[854]: reaped child process 1478, status = succeeded5515lb # [ 142.246830] niks3-server[1164]: 2026/09/24 11:49:50 INFO Received push request method=POST path=/api/pushes5516worker1 # [ 142.278675] nix-grpc-daemon[1375]: time=2026-09-24T11:49:50.892Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5517worker1 # [ 142.280269] nix-grpc-daemon[1375]: time=2026-09-24T11:49:50.892Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)"5518lb # [ 142.288310] niks3-server[1164]: 2026/09/24 11:49:50 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5519worker1 # [ 142.322736] nix-grpc-daemon[1375]: time=2026-09-24T11:49:50.938Z level=INFO msg="Uploading 1 narinfos"5520lb # [ 142.303442] niks3-server[1164]: 2026/09/24 11:49:50 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst5521lb # [ 142.316829] niks3-server[1164]: 2026/09/24 11:49:50 INFO Received sign narinfos request method=POST path=/api/pushes/47/sign5522lb # [ 142.329226] niks3-server[1164]: 2026/09/24 11:49:50 INFO Signed narinfos id=47 count=15523lb # [ 142.331985] niks3-server[1164]: 2026/09/24 11:49:50 INFO Received complete push request method=POST path=/api/pushes/47/complete5524worker1 # [ 142.365652] nix-grpc-daemon[1375]: time=2026-09-24T11:49:50.981Z level=INFO msg="Upload complete. (302ms)"5525lb # [ 142.348905] niks3-server[1164]: 2026/09/24 11:49:50 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls5526worker1 # [ 142.381307] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59602 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=15527lb # [ 142.373188] niks3-server[1164]: 2026/09/24 11:49:50 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo5528worker1 # [ 142.400980] nix-daemon[854]: reaped child process 1457, status = succeeded5529client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2, in 6.15 seconds)5530client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath5531client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath, in 0.18 seconds)5532worker2 # [ 142.752778] systemd[1]: Starting scheduler-token.service...5533worker2 # [ 142.827822] systemd[1]: scheduler-token.service: Deactivated successfully.5534worker2 # [ 142.832867] systemd[1]: Finished scheduler-token.service.5535worker2 # [ 142.834454] systemd[1]: scheduler-token.service: Consumed 43ms CPU time over 78ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5536worker2 # [ 142.854693] systemd[1]: Starting Nix worker-protocol over gRPC...5537worker2 # [ 142.965501] nix-grpc-daemon[1446]: ts=2026-09-24T11:49:51Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5538worker2 # [ 142.967852] nix-grpc-daemon[1446]: ts=2026-09-24T11:49:51Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5539worker2 # [ 142.971823] nix-grpc-daemon[1446]: ts=2026-09-24T11:49:51Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15540worker2 # [ 142.978837] nix-grpc-daemon[1446]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5541lb: must succeed: curl -sf localhost:9901/clusters5542worker2 # [ 142.989801] nix-grpc-daemon[1446]: E0000 00:00:1790250591.595692 1446 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5543worker2 # [ 143.008324] nix-grpc-daemon[1446]: E0000 00:00:1790250591.597871 1446 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5544worker2 # [ 143.010635] nix-grpc-daemon[1446]: ts=2026-09-24T11:49:51Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket5545worker2 # [ 143.029551] nix-grpc-daemon[1446]: E0000 00:00:1790250591.613096 1486 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5546worker2 # [ 143.032636] systemd[1]: Started Nix worker-protocol over gRPC.5547worker2 # [ 143.033650] nix-grpc-daemon[1446]: E0000 00:00:1790250591.632948 1486 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5548worker2 # [ 143.035867] nix-grpc-daemon[1446]: E0000 00:00:1790250591.633143 1486 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5549worker2 # [ 143.041651] nix-grpc-daemon[1446]: E0000 00:00:1790250591.647747 1486 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5550worker2 # [ 143.046090] nix-grpc-daemon[1446]: ts=2026-09-24T11:49:51Z level=info event=scheduler_connected addr=lb:500515551worker1 # [ 142.987730] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:51Z level=info event=oidc_jwks provider=mock keys=15552worker1 # [ 142.993032] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:51Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:595945553worker1 # [ 143.002255] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:51Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05554lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds)5555worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5556worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5557lb: must succeed: curl -sf localhost:9901/clusters5558lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5559lb: must succeed: curl -sf localhost:9901/clusters5560lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5561worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5562worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5563(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 11.89 seconds)5564subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5565client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs5566client # [ 143.303300] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs.5567client # Running as unit: rs.service; invocation ID: 4d796c127ace47f4a4b7c06119dab96f5568client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs, in 0.09 seconds)5569worker2 # [ 143.604930] nix-grpc-daemon[1446]: ts=2026-09-24T11:49:52Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:436865570worker2 # [ 143.620485] nix-daemon[841]: accepted connection from pid 1446, user nix-grpc-daemon (trusted)5571client # [ 143.544938] nix[1381]: don't know how to build these paths:5572client # [ 143.546463] nix[1381]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5573worker1 # [ 143.586712] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:52Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=05574worker1 # [ 143.592929] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:52Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.3:50051 clients=15575worker1 # [ 143.597773] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596025576worker1 # [ 143.607028] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596025577client # [ 143.596525] nix[1381]: copying 1 paths to 'lb:50051'...5578worker1 # [ 143.620114] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596025579client # [ 143.600415] nix[1381]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'...5580worker1 # [ 143.638004] nix-daemon[854]: accepted connection from pid 1302, user nix-grpc-daemon (trusted)5581worker1 # [ 143.796144] nix-daemon[854]: accepted connection from pid 1491, user nix-grpc-daemon (trusted)5582worker1 # [ 143.818267] nix-daemon[854]: reaped child process 1494, status = succeeded5583lb # [ 143.802767] niks3-server[1164]: 2026/09/24 11:49:52 INFO Received push request method=POST path=/api/pushes5584worker1 # [ 143.838946] nix-grpc-daemon[1375]: time=2026-09-24T11:49:52.453Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5585worker1 # [ 143.840561] nix-grpc-daemon[1375]: time=2026-09-24T11:49:52.453Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)"5586worker1 # [ 143.880728] nix-grpc-daemon[1375]: time=2026-09-24T11:49:52.496Z level=INFO msg="Uploading 1 narinfos"5587lb # [ 143.847954] niks3-server[1164]: 2026/09/24 11:49:52 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst5588lb # [ 143.864948] niks3-server[1164]: 2026/09/24 11:49:52 INFO Received sign narinfos request method=POST path=/api/pushes/48/sign5589lb # [ 143.874273] niks3-server[1164]: 2026/09/24 11:49:52 INFO Signed narinfos id=48 count=15590lb # [ 143.882928] niks3-server[1164]: 2026/09/24 11:49:52 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls5591lb # [ 143.896345] niks3-server[1164]: 2026/09/24 11:49:52 INFO Received complete push request method=POST path=/api/pushes/48/complete5592worker1 # [ 143.924766] nix-grpc-daemon[1375]: time=2026-09-24T11:49:52.540Z level=INFO msg="Upload complete. (271ms)"5593lb # [ 143.905130] niks3-server[1164]: 2026/09/24 11:49:52 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo5594worker1 # [ 143.936954] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59602 duration_s=0 paths=1 nar_bytes_in=4165595worker1 # [ 143.952431] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596025596worker1 # [ 143.962588] nix-daemon[854]: accepted connection from pid 1302, user nix-grpc-daemon (trusted)5597worker1 # [ 143.968522] nix-daemon[854]: reaped child process 1489, status = succeeded5598client # [ 143.957604] nix[1381]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051...5599worker1 # [ 143.983677] nix-daemon[854]: accepted connection from pid 1302, user nix-grpc-daemon (trusted)5600lb: must succeed: curl -sf localhost:9901/clusters5601lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5602lb: must succeed: curl -sf localhost:9901/clusters5603lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5604lb: must succeed: curl -sf localhost:9901/clusters5605lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5606lb: must succeed: curl -sf localhost:9901/clusters5607lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5608worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5609worker2 # [ 146.276883] systemd-journald[270]: Received client request to rotate journal, rotating.5610worker2 # [ 146.297272] systemd-journald[270]: Vacuuming done, freed 0B of archived journals from /var/log/journal/73be792bd5fb4942b04d2918072afe0f.5611worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.09 seconds)5612worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service5613worker1 # [ 146.274135] systemd[1]: Stopping Nix worker-protocol over gRPC...5614worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds)5615client: must succeed: sleep 5.05616worker1 # [ 146.543046] nix-grpc-daemon[1302]: ts=2026-09-24T11:49:55Z level=info event=draining5617lb # [ 149.045035] envoy[907]: conn peer=[2001:db8:1::1]:37808 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9632 tx=4389 ms=58935618client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5619worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5620worker1 # [ 151.622218] nix-daemon[854]: reaped child process 1498, status = succeeded5621worker1 # [ 151.729666] nix-daemon[854]: accepted connection from pid 1539, user nix-grpc-daemon (trusted)5622worker1 # [ 151.752983] nix-daemon[854]: reaped child process 1542, status = succeeded5623lb # [ 151.730676] niks3-server[1164]: 2026/09/24 11:50:00 INFO Received push request method=POST path=/api/pushes5624worker1 # [ 151.764415] nix-grpc-daemon[1375]: time=2026-09-24T11:50:00.377Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5625worker1 # [ 151.765910] nix-grpc-daemon[1375]: time=2026-09-24T11:50:00.377Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)"5626lb # [ 151.781611] niks3-server[1164]: 2026/09/24 11:50:00 INFO Received sign narinfos request method=POST path=/api/pushes/49/sign5627worker1 # [ 151.813525] nix-grpc-daemon[1375]: time=2026-09-24T11:50:00.428Z level=INFO msg="Uploading 1 narinfos"5628lb # [ 151.797403] niks3-server[1164]: 2026/09/24 11:50:00 INFO Signed narinfos id=49 count=15629lb # [ 151.799879] niks3-server[1164]: 2026/09/24 11:50:00 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5630lb # [ 151.815666] niks3-server[1164]: 2026/09/24 11:50:00 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls5631lb # [ 151.823649] niks3-server[1164]: 2026/09/24 11:50:00 INFO Received complete push request method=POST path=/api/pushes/49/complete5632worker1 # [ 151.844133] nix-grpc-daemon[1375]: time=2026-09-24T11:50:00.459Z level=INFO msg="Upload complete. (244ms)"5633lb # [ 151.827055] niks3-server[1164]: 2026/09/24 11:50:00 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5634lb # [ 151.831957] niks3-server[1164]: 2026/09/24 11:50:00 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo5635worker1 # [ 151.857609] nix-grpc-daemon[1302]: ts=2026-09-24T11:50:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59602 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=15636worker1 # [ 151.876707] nix-daemon[854]: reaped child process 1496, status = succeeded5637client # [ 151.871034] systemd[1]: rs.service: Deactivated successfully.5638client # [ 151.876354] systemd[1]: rs.service: Consumed 200ms CPU time over 8.568s wall clock time, 41.2M memory peak, 7.3K incoming IP traffic, 8.4K outgoing IP traffic.5639worker1 # [ 152.544673] nix-grpc-daemon[1302]: ts=2026-09-24T11:50:01Z level=info event=signal_exit5640lb # [ 152.833384] niks3-server[1164]: 2026/09/24 11:50:01 INFO lead: released remote=192.168.1.3:468745641lb # [ 152.900556] niks3-server[1164]: 2026/09/24 11:50:01 INFO lead: acquired remote=192.168.1.4:351505642worker2 # [ 152.997811] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:01Z level=info event=scheduler_take_over5643worker2 # [ 153.921879] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:02Z level=info event=scheduler_restarting addr=lb:500515644worker1 # [ 153.852194] nix-grpc-daemon[1302]: ts=2026-09-24T11:50:02Z level=info event=scheduler_restarting5645lb # [ 153.827878] envoy[907]: [2026-09-24 11:50:02.429][907][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream}5646worker1 # [ 153.855945] nix-grpc-daemon[1302]: ts=2026-09-24T11:50:02Z level=info event=scheduler_restarting addr=lb:500515647worker1 # [ 153.862046] nix-grpc-daemon[1302]: ts=2026-09-24T11:50:02Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:595945648worker1 # [ 153.870908] nix-grpc-daemon[1302]: ts=2026-09-24T11:50:02Z level=info event=worker_gone addr=192.168.1.3:500515649worker1 # [ 153.875709] nix-grpc-daemon[1302]: ts=2026-09-24T11:50:02Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:595945650worker1 # [ 153.880736] nix-grpc-daemon[1302]: ts=2026-09-24T11:50:02Z level=info event=worker_gone addr=192.168.1.4:500515651worker1 # [ 153.906845] nix-daemon[854]: reaped child process 1354, status = succeeded5652worker2 # [ 154.125066] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:02Z level=info event=scheduler_connected addr=lb:500515653worker2 # [ 154.129975] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5654worker1 # [ 154.077288] nix-grpc-daemon[1302]: ts=2026-09-24T11:50:02Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5655worker2 # [ 154.331339] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:02Z level=info event=scheduler_connected addr=lb:500515656worker2 # [ 154.336903] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5657worker2 # [ 154.538115] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:03Z level=info event=scheduler_connected addr=lb:500515658worker2 # [ 154.556173] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:03Z level=info event=oidc_jwks provider=mock keys=15659worker2 # [ 154.561739] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:03Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:514405660worker2 # [ 154.567248] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:03Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05661worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.25 seconds)5662client: waiting for success: ! systemctl is-active rs5663client: (finished: waiting for success: ! systemctl is-active rs, in 0.07 seconds)5664client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5665client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds)5666client: must fail: journalctl -u rs | grep -q 'reconnecting'5667client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5668lb: must succeed: curl -sf localhost:9901/clusters5669lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5670worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5671worker1 # [ 155.118161] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5672worker1 # [ 155.124313] systemd[1]: Stopped Nix worker-protocol over gRPC.5673worker1 # [ 155.126959] systemd[1]: nix-grpc-daemon.service: Consumed 1.645s CPU time over 32.760s wall clock time, 39.7M memory peak, 389.6K incoming IP traffic, 140.5K outgoing IP traffic.5674worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5675worker1 # [ 155.150804] systemd[1]: Starting Nix worker-protocol over gRPC...5676worker2 # [ 155.286889] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:03Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:514505677worker1 # [ 155.312752] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:03Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5678worker1 # [ 155.314982] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:03Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5679worker1 # [ 155.320655] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:03Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15680worker1 # [ 155.326611] nix-grpc-daemon[1550]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5681worker1 # [ 155.330471] nix-grpc-daemon[1550]: E0000 00:00:1790250603.941533 1550 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5682worker1 # [ 155.340786] nix-grpc-daemon[1550]: E0000 00:00:1790250603.945701 1550 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5683worker1 # [ 155.352258] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:03Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket5684worker1 # [ 155.354428] systemd[1]: Started Nix worker-protocol over gRPC.5685worker1 # [ 155.385078] nix-grpc-daemon[1550]: E0000 00:00:1790250603.959672 1593 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5686worker1 # [ 155.388427] nix-grpc-daemon[1550]: E0000 00:00:1790250603.984383 1593 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5687worker1 # [ 155.394097] nix-grpc-daemon[1550]: E0000 00:00:1790250603.984604 1593 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5688worker1 # [ 155.396953] nix-grpc-daemon[1550]: E0000 00:00:1790250603.999456 1593 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5689worker1 # [ 155.402303] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:04Z level=info event=scheduler_connected addr=lb:500515690worker2 # [ 155.473269] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:04Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:514405691worker2 # [ 155.476170] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:04Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05692lb: must succeed: curl -sf localhost:9901/clusters5693lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5694worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5695worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5696(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 13.02 seconds)5697subtest: scheduler restart mid-build: build finishes, no double build5698client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid5699client # [ 156.325282] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid.5700client # Running as unit: mid.service; invocation ID: 711c5f785486418e9d9fea8d68af75c55701client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid, in 0.09 seconds)5702worker2 # [ 156.623447] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:05Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:436865703client # [ 156.539005] nix[1417]: don't know how to build these paths:5704client # [ 156.539984] nix[1417]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5705worker2 # [ 156.644873] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:05Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05706worker2 # [ 156.648314] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:05Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=15707worker2 # [ 156.655156] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436865708worker2 # [ 156.659628] nix-grpc-daemon[1446]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 3 ms (attempt 1/5)5709worker2 # [ 156.666917] nix-grpc-daemon[1446]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 143 ms (attempt 2/5)5710worker2 # [ 156.815392] nix-grpc-daemon[1446]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 68 ms (attempt 3/5)5711worker2 # [ 156.885882] nix-grpc-daemon[1446]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 750 ms (attempt 4/5)5712worker2 # [ 157.641414] nix-grpc-daemon[1446]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5713worker2 # [ 157.641707] nix-grpc-daemon[1446]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5714worker2 # [ 157.667139] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:436865715client # [ 157.589448] nix[1417]: copying 1 paths to 'lb:50051'...5716worker2 # [ 157.686106] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:436865717client # [ 157.594861] nix[1417]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'...5718worker2 # [ 157.706230] nix-daemon[841]: accepted connection from pid 1446, user nix-grpc-daemon (trusted)5719worker2 # [ 157.740000] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:06Z level=info event=niks3_push_started pid=15785720worker2 # [ 157.894090] nix-grpc-daemon[1578]: time=2026-09-24T11:50:06.512Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""5721worker2 # [ 158.009022] nix-daemon[841]: accepted connection from pid 1591, user nix-grpc-daemon (trusted)5722worker2 # [ 158.030140] nix-daemon[841]: reaped child process 1594, status = succeeded5723lb # [ 157.938087] niks3-server[1164]: 2026/09/24 11:50:06 INFO Received push request method=POST path=/api/pushes5724worker2 # [ 158.042708] nix-grpc-daemon[1578]: time=2026-09-24T11:50:06.660Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5725worker2 # [ 158.046760] nix-grpc-daemon[1578]: time=2026-09-24T11:50:06.660Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)"5726lb # [ 157.983308] niks3-server[1164]: 2026/09/24 11:50:06 INFO Received sign narinfos request method=POST path=/api/pushes/50/sign5727worker2 # [ 158.087117] nix-grpc-daemon[1578]: time=2026-09-24T11:50:06.705Z level=INFO msg="Uploading 1 narinfos"5728lb # [ 157.998816] niks3-server[1164]: 2026/09/24 11:50:06 INFO Signed narinfos id=50 count=15729lb # [ 158.005641] niks3-server[1164]: 2026/09/24 11:50:06 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls5730lb # [ 158.012309] niks3-server[1164]: 2026/09/24 11:50:06 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst5731lb # [ 158.024383] niks3-server[1164]: 2026/09/24 11:50:06 INFO Received complete push request method=POST path=/api/pushes/50/complete5732lb # [ 158.028848] niks3-server[1164]: 2026/09/24 11:50:06 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo5733worker2 # [ 158.124588] nix-grpc-daemon[1578]: time=2026-09-24T11:50:06.743Z level=INFO msg="Upload complete. (226ms)"5734worker2 # [ 158.132251] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43686 duration_s=0 paths=1 nar_bytes_in=4245735worker2 # [ 158.147280] nix-grpc-daemon[1446]: ts=2026-09-24T11:50:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:436865736worker2 # [ 158.155001] nix-daemon[841]: accepted connection from pid 1446, user nix-grpc-daemon (trusted)5737worker2 # [ 158.160129] nix-daemon[841]: reaped child process 1575, status = succeeded5738worker2 # [ 158.170842] nix-daemon[841]: accepted connection from pid 1446, user nix-grpc-daemon (trusted)5739client # [ 158.077576] nix[1417]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5740lb # [ 159.045090] envoy[907]: conn peer=[2001:db8:1::1]:34708 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2966 tx=1541 ms=83415741lb # [ 159.050144] envoy[907]: conn peer=[2001:db8:1::3]:45722 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2473 tx=870 ms=314815742lb: must succeed: curl -sf localhost:9901/clusters5743lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5744lb: must succeed: curl -sf localhost:9901/clusters5745lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5746worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5747worker2 # [ 160.420970] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1446 (nix-grpc-daemon) on client request.5748worker2 # [ 160.422590] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5749lb # [ 160.352942] niks3-server[1164]: 2026/09/24 11:50:08 INFO lead: released remote=192.168.1.4:351505750worker2 # [ 160.448876] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5751worker2 # [ 160.450464] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5752client # [ 160.356865] nix[1417]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again5753worker1 # [ 160.378580] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:08Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"5754lb # [ 160.360708] envoy[907]: [2026-09-24 11:50:08.962][907][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 13, upstream_reset_after_response_started{connection_termination}5755client # [ 160.363600] nix[1417]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5756worker2 # [ 160.466895] systemd[1]: nix-grpc-daemon.service: Consumed 523ms CPU time over 17.600s wall clock time, 34.7M memory peak, 217.3K incoming IP traffic, 52.8K outgoing IP traffic.5757worker2 # [ 160.477725] systemd[1]: Starting scheduler-token.service...5758worker2 # [ 160.506625] nix-daemon[841]: reaped child process 1505, status = succeeded5759worker2 # [ 160.507635] nix-daemon[841]: reaped child process 1596, status = succeeded5760worker2 # [ 160.516523] nix-daemon[1598]: unexpected Nix daemon error: error: interrupted by the user5761worker2 # [ 160.524124] nix-daemon[841]: reaped child process 1598, status = failed with exit code 15762worker2 # [ 160.552054] systemd[1]: nix-grpc-daemon.service: Scheduled restart job, restart counter is at 1.5763worker2 # [ 160.554943] systemd[1]: scheduler-token.service: Deactivated successfully.5764worker2 # [ 160.558859] systemd[1]: Finished scheduler-token.service.5765worker2 # [ 160.559680] systemd[1]: scheduler-token.service: Consumed 26ms CPU time over 91ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5766worker2 # [ 160.568275] systemd[1]: Starting Nix worker-protocol over gRPC...5767worker1 # [ 160.578794] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:09Z level=info event=scheduler_connected addr=lb:500515768worker2 # [ 160.667318] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:09Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5769worker2 # [ 160.669803] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:09Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5770worker2 # [ 160.673857] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:09Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15771worker2 # [ 160.679320] nix-grpc-daemon[1638]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5772worker2 # [ 160.682479] nix-grpc-daemon[1638]: E0000 00:00:1790250609.297703 1638 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5773lb # [ 160.603718] niks3-server[1164]: 2026/09/24 11:50:09 INFO lead: acquired remote=192.168.1.4:519085774worker2 # [ 160.693228] nix-grpc-daemon[1638]: E0000 00:00:1790250609.301437 1638 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5775worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.33 seconds)5776client: must succeed: sleep 5.05777worker2 # [ 160.740175] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:09Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket5778worker2 # [ 160.742054] nix-grpc-daemon[1638]: E0000 00:00:1790250609.319366 1681 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5779worker2 # [ 160.745619] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:09Z level=info event=scheduler_take_over5780worker2 # [ 160.747192] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:09Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:515065781worker2 # [ 160.749027] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:09Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05782worker2 # [ 160.750738] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:09Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:514945783worker2 # [ 160.754978] nix-grpc-daemon[1638]: E0000 00:00:1790250609.344575 1681 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5784worker2 # [ 160.760367] nix-grpc-daemon[1638]: E0000 00:00:1790250609.344738 1681 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5785worker2 # [ 160.763192] systemd[1]: Started Nix worker-protocol over gRPC.5786worker2 # [ 160.764487] nix-grpc-daemon[1638]: E0000 00:00:1790250609.363726 1681 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5787worker2 # [ 160.767123] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:09Z level=info event=scheduler_connected addr=lb:500515788worker2 # [ 160.770783] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:09Z level=info event=oidc_jwks provider=mock keys=15789worker2 # [ 160.772778] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:09Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:515065790worker2 # [ 160.777431] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:09Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05791worker2 # [ 160.961586] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:09Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05792worker2 # [ 160.967913] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:09Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.3:50051 clients=15793worker1 # [ 160.916963] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407125794worker1 # [ 160.932633] nix-daemon[854]: accepted connection from pid 1550, user nix-grpc-daemon (trusted)5795worker1 # [ 160.957520] nix-grpc-daemon[1550]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 20 ms (attempt 1/5)5796worker1 # [ 160.985687] nix-grpc-daemon[1550]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 28 ms (attempt 2/5)5797worker1 # [ 161.015629] nix-grpc-daemon[1550]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 132 ms (attempt 3/5)5798worker1 # [ 161.149899] nix-grpc-daemon[1550]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 103 ms (attempt 4/5)5799worker1 # [ 161.255819] nix-grpc-daemon[1550]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5800worker1 # [ 161.262671] nix-grpc-daemon[1550]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5801worker1 # [ 161.278755] nix-grpc-daemon[1550]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' from 'http://lb:5751'...5802worker1 # [ 161.305829] nix-daemon[854]: accepted connection from pid 1550, user nix-grpc-daemon (trusted)5803client # [ 161.306536] nix[1417]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5804worker1 # [ 161.330842] nix-daemon[854]: accepted connection from pid 1550, user nix-grpc-daemon (trusted)5805client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5806client: waiting for success: ! systemctl is-active mid5807worker1 # [ 165.970390] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:14Z level=info event=niks3_push_started pid=16565808worker1 # [ 166.001033] nix-daemon[854]: reaped child process 1631, status = succeeded5809worker1 # [ 166.111792] nix-grpc-daemon[1656]: time=2026-09-24T11:50:14.725Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""5810worker1 # [ 166.221750] nix-daemon[854]: accepted connection from pid 1668, user nix-grpc-daemon (trusted)5811worker1 # [ 166.239629] nix-daemon[854]: reaped child process 1671, status = succeeded5812lb # [ 166.230222] niks3-server[1164]: 2026/09/24 11:50:14 INFO Received push request method=POST path=/api/pushes5813worker1 # [ 166.260229] nix-grpc-daemon[1656]: time=2026-09-24T11:50:14.875Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5814worker1 # [ 166.265461] nix-grpc-daemon[1656]: time=2026-09-24T11:50:14.875Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)"5815lb # [ 166.284443] niks3-server[1164]: 2026/09/24 11:50:14 INFO Received sign narinfos request method=POST path=/api/pushes/51/sign5816lb # [ 166.305033] niks3-server[1164]: 2026/09/24 11:50:14 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls5817lb # [ 166.319751] niks3-server[1164]: 2026/09/24 11:50:14 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5818lb # [ 166.334473] niks3-server[1164]: 2026/09/24 11:50:14 INFO Signed narinfos id=51 count=15819worker1 # [ 166.354915] nix-grpc-daemon[1656]: time=2026-09-24T11:50:14.969Z level=INFO msg="Uploading 1 narinfos"5820lb # [ 166.347287] niks3-server[1164]: 2026/09/24 11:50:14 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst5821lb # [ 166.359013] niks3-server[1164]: 2026/09/24 11:50:14 INFO Received complete push request method=POST path=/api/pushes/51/complete5822lb # [ 166.376055] niks3-server[1164]: 2026/09/24 11:50:14 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo5823worker1 # [ 166.402151] nix-grpc-daemon[1656]: time=2026-09-24T11:50:15.017Z level=INFO msg="Upload complete. (291ms)"5824worker1 # [ 166.416353] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40712 duration_s=5 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=15825worker1 # [ 166.438789] nix-daemon[854]: reaped child process 1629, status = succeeded5826client # [ 166.429820] systemd[1]: mid.service: Deactivated successfully.5827client # [ 166.434198] systemd[1]: mid.service: Consumed 214ms CPU time over 10.108s wall clock time, 41.4M memory peak, 8.7K incoming IP traffic, 10.4K outgoing IP traffic.5828client: (finished: waiting for success: ! systemctl is-active mid, in 1.13 seconds)5829client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }5830client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.05 seconds)5831(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.85 seconds)5832subtest: interrupting the client stops the build on the worker5833client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr5834client # [ 167.143593] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr.5835client # Running as unit: intr.service; invocation ID: 678dc33b8f394599b094fb628fc09c3c5836client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr, in 0.05 seconds)5837worker1 # [ 167.350590] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:15Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:407125838client # [ 167.337453] nix[1453]: don't know how to build these paths:5839client # [ 167.338418] nix[1453]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5840worker2 # [ 167.441333] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:16Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=05841worker2 # [ 167.447632] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:16Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.4:50051 clients=15842worker2 # [ 167.472120] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379045843worker2 # [ 167.488614] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)5844worker2 # [ 167.509312] nix-grpc-daemon[1638]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 39 ms (attempt 1/5)5845worker2 # [ 167.555983] nix-grpc-daemon[1638]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 3 ms (attempt 2/5)5846worker2 # [ 167.565575] nix-grpc-daemon[1638]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 161 ms (attempt 3/5)5847worker2 # [ 167.736091] nix-grpc-daemon[1638]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 26 ms (attempt 4/5)5848worker2 # [ 167.764129] nix-grpc-daemon[1638]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5849worker2 # [ 167.768452] nix-grpc-daemon[1638]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5850worker2 # [ 167.790411] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:379045851client # [ 167.712218] nix[1453]: copying 1 paths to 'lb:50051'...5852worker2 # [ 167.806507] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:379045853client # [ 167.716171] nix[1453]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'...5854worker2 # [ 167.819697] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)5855worker2 # [ 167.851432] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:16Z level=info event=niks3_push_started pid=17145856worker2 # [ 167.998628] nix-grpc-daemon[1714]: time=2026-09-24T11:50:16.617Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""5857worker2 # [ 168.112126] nix-daemon[841]: accepted connection from pid 1727, user nix-grpc-daemon (trusted)5858worker2 # [ 168.130604] nix-daemon[841]: reaped child process 1730, status = succeeded5859lb # [ 168.045391] niks3-server[1164]: 2026/09/24 11:50:16 INFO Received push request method=POST path=/api/pushes5860worker2 # [ 168.151314] nix-grpc-daemon[1714]: time=2026-09-24T11:50:16.769Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5861worker2 # [ 168.155345] nix-grpc-daemon[1714]: time=2026-09-24T11:50:16.769Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)"5862worker2 # [ 168.191808] nix-grpc-daemon[1714]: time=2026-09-24T11:50:16.810Z level=INFO msg="Uploading 1 narinfos"5863lb # [ 168.091093] niks3-server[1164]: 2026/09/24 11:50:16 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls5864lb # [ 168.104568] niks3-server[1164]: 2026/09/24 11:50:16 INFO Received sign narinfos request method=POST path=/api/pushes/52/sign5865lb # [ 168.117627] niks3-server[1164]: 2026/09/24 11:50:16 INFO Signed narinfos id=52 count=15866lb # [ 168.123560] niks3-server[1164]: 2026/09/24 11:50:16 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst5867lb # [ 168.138354] niks3-server[1164]: 2026/09/24 11:50:16 INFO Received complete push request method=POST path=/api/pushes/52/complete5868lb # [ 168.145487] niks3-server[1164]: 2026/09/24 11:50:16 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo5869worker2 # [ 168.240970] nix-grpc-daemon[1714]: time=2026-09-24T11:50:16.860Z level=INFO msg="Upload complete. (238ms)"5870worker2 # [ 168.252057] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=0 paths=1 nar_bytes_in=4245871worker2 # [ 168.264820] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379045872worker2 # [ 168.277553] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)5873worker2 # [ 168.285916] nix-daemon[841]: reaped child process 1712, status = succeeded5874client # [ 168.201256] nix[1453]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051...5875worker2 # [ 168.297476] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)5876lb # [ 169.043638] envoy[907]: conn peer=[2001:db8:1::4]:42272 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=3470 tx=444 ms=174135877lb # [ 169.048793] envoy[907]: conn peer=[2001:db8:1::1]:58102 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3940 tx=1965 ms=98785878client: must succeed: systemctl kill -s INT intr5879worker2 # [ 170.910108] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:19Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5880client # [ 170.815184] systemd[1]: intr.service: Sent signal SIGINT to main process 1453 (nix) on client request.5881client # [ 170.822395] nix[1453]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5882client # [ 170.834618] nix[1453]: error: interrupted by the user5883client: (finished: must succeed: systemctl kill -s INT intr, in 0.08 seconds)5884client # [ 170.863395] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE5885client # [ 170.869518] systemd[1]: intr.service: Failed with result 'exit-code'.5886client # [ 170.872305] systemd[1]: intr.service: Consumed 180ms CPU time over 3.723s wall clock time, 41.5M memory peak, 6.5K incoming IP traffic, 8.3K outgoing IP traffic.5887worker2 # [ 171.125617] nix-daemon[1734]: unexpected Nix daemon error:5888worker2 # [ 171.126689] nix-daemon[841]: reaped child process 1732, status = succeeded5889worker2 # [ 171.132221] nix-daemon[1734]: error: interrupted by the user5890worker2 # [ 171.136332] nix-daemon[841]: reaped child process 1734, status = failed with exit code 15891(finished: subtest: interrupting the client stops the build on the worker, in 5.06 seconds)5892subtest: a deploy mid-build drains: the build finishes, the new generation takes over5893client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw5894client # [ 172.235868] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw.5895client # Running as unit: sw.service; invocation ID: da4a98fa2d434885bc8e2b08f856d8325896client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw, in 0.09 seconds)5897client # [ 172.445737] nix[1474]: don't know how to build these paths:5898client # [ 172.446794] nix[1474]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv5899worker2 # [ 172.538072] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:21Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:379045900worker2 # [ 172.552168] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:21Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=05901worker2 # [ 172.558275] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:21Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.3:50051 clients=15902worker1 # [ 172.491997] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407125903worker1 # [ 172.505553] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:407125904client # [ 172.496341] nix[1474]: copying 1 paths to 'lb:50051'...5905client # [ 172.500050] nix[1474]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'...5906worker1 # [ 172.522619] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:407125907worker1 # [ 172.538307] nix-daemon[854]: accepted connection from pid 1550, user nix-grpc-daemon (trusted)5908worker1 # [ 172.700673] nix-daemon[854]: accepted connection from pid 1703, user nix-grpc-daemon (trusted)5909worker1 # [ 172.723145] nix-daemon[854]: reaped child process 1706, status = succeeded5910lb # [ 172.708700] niks3-server[1164]: 2026/09/24 11:50:21 INFO Received push request method=POST path=/api/pushes5911worker1 # [ 172.747402] nix-grpc-daemon[1656]: time=2026-09-24T11:50:21.361Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5912worker1 # [ 172.749114] nix-grpc-daemon[1656]: time=2026-09-24T11:50:21.361Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)"5913lb # [ 172.747618] niks3-server[1164]: 2026/09/24 11:50:21 INFO Received sign narinfos request method=POST path=/api/pushes/53/sign5914worker1 # [ 172.777933] nix-grpc-daemon[1656]: time=2026-09-24T11:50:21.392Z level=INFO msg="Uploading 1 narinfos"5915lb # [ 172.761069] niks3-server[1164]: 2026/09/24 11:50:21 INFO Signed narinfos id=53 count=15916lb # [ 172.767110] niks3-server[1164]: 2026/09/24 11:50:21 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst5917lb # [ 172.780953] niks3-server[1164]: 2026/09/24 11:50:21 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls5918lb # [ 172.783676] niks3-server[1164]: 2026/09/24 11:50:21 INFO Received complete push request method=POST path=/api/pushes/53/complete5919lb # [ 172.796105] niks3-server[1164]: 2026/09/24 11:50:21 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo5920worker1 # [ 172.815622] nix-grpc-daemon[1656]: time=2026-09-24T11:50:21.431Z level=INFO msg="Upload complete. (261ms)"5921worker1 # [ 172.828373] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40712 duration_s=0 paths=1 nar_bytes_in=4165922worker1 # [ 172.844083] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:407125923worker1 # [ 172.850181] nix-daemon[854]: accepted connection from pid 1550, user nix-grpc-daemon (trusted)5924worker1 # [ 172.855190] nix-daemon[854]: reaped child process 1701, status = succeeded5925worker1 # [ 172.866271] nix-daemon[854]: accepted connection from pid 1550, user nix-grpc-daemon (trusted)5926client # [ 172.843628] nix[1474]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051...5927worker1: must succeed: systemctl reload nix-grpc-daemon.service5928worker1 # [ 174.910986] systemd[1]: Reloading Nix worker-protocol over gRPC...5929worker1 # [ 174.946845] systemd[1]: Reloaded Nix worker-protocol over gRPC.5930worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.08 seconds)5931worker1: must succeed: pgrep -f 'read -t [9]0 x'5932worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds)5933client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&25934worker1 # [ 175.224897] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:23Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:407125935client # don't know how to build these paths:5936client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5937worker2 # [ 175.303891] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:379045938worker2 # [ 175.329644] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:23Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=05939worker2 # [ 175.336519] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:23Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.4:50051 clients=15940worker2 # [ 175.346558] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379045941worker2 # [ 175.357632] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:379045942client # copying 1 paths to 'lb:50051'...5943client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'...5944worker2 # [ 175.373071] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:379045945worker2 # [ 175.390281] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)5946worker1 # [ 175.345026] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:23Z level=info event=draining5947worker2 # [ 175.551414] nix-daemon[841]: accepted connection from pid 1793, user nix-grpc-daemon (trusted)5948worker2 # [ 175.572358] nix-daemon[841]: reaped child process 1796, status = succeeded5949lb # [ 175.486587] niks3-server[1164]: 2026/09/24 11:50:24 INFO Received push request method=POST path=/api/pushes5950worker2 # [ 175.590612] nix-grpc-daemon[1714]: time=2026-09-24T11:50:24.208Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5951worker2 # [ 175.592333] nix-grpc-daemon[1714]: time=2026-09-24T11:50:24.208Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)"5952lb # [ 175.528643] niks3-server[1164]: 2026/09/24 11:50:24 INFO Received sign narinfos request method=POST path=/api/pushes/54/sign5953worker2 # [ 175.628987] nix-grpc-daemon[1714]: time=2026-09-24T11:50:24.247Z level=INFO msg="Uploading 1 narinfos"5954lb # [ 175.539681] niks3-server[1164]: 2026/09/24 11:50:24 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls5955lb # [ 175.551661] niks3-server[1164]: 2026/09/24 11:50:24 INFO Signed narinfos id=54 count=15956lb # [ 175.558071] niks3-server[1164]: 2026/09/24 11:50:24 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst5957lb # [ 175.571976] niks3-server[1164]: 2026/09/24 11:50:24 INFO Received complete push request method=POST path=/api/pushes/54/complete5958lb # [ 175.578866] niks3-server[1164]: 2026/09/24 11:50:24 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo5959worker2 # [ 175.669359] nix-grpc-daemon[1714]: time=2026-09-24T11:50:24.288Z level=INFO msg="Upload complete. (264ms)"5960worker2 # [ 175.680344] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=0 paths=1 nar_bytes_in=5285961worker2 # [ 175.705044] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379045962worker2 # [ 175.729982] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)5963worker2 # [ 175.733992] nix-daemon[841]: reaped child process 1791, status = succeeded5964worker2 # [ 175.742708] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)5965client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051...5966client # farm-a> worker2: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5967client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5968client # farm-a> LOG-farm-a-during-drain5969worker2 # [ 176.966503] nix-daemon[841]: reaped child process 1800, status = succeeded5970worker2 # [ 177.062780] nix-daemon[841]: accepted connection from pid 1816, user nix-grpc-daemon (trusted)5971worker2 # [ 177.081102] nix-daemon[841]: reaped child process 1819, status = succeeded5972lb # [ 176.994666] niks3-server[1164]: 2026/09/24 11:50:25 INFO Received push request method=POST path=/api/pushes5973worker2 # [ 177.096371] nix-grpc-daemon[1714]: time=2026-09-24T11:50:25.714Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5974worker2 # [ 177.097822] nix-grpc-daemon[1714]: time=2026-09-24T11:50:25.714Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)"5975lb # [ 177.042297] niks3-server[1164]: 2026/09/24 11:50:25 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst5976lb # [ 177.057055] niks3-server[1164]: 2026/09/24 11:50:25 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5977worker2 # [ 177.162223] nix-grpc-daemon[1714]: time=2026-09-24T11:50:25.781Z level=INFO msg="Uploading 1 narinfos"5978lb # [ 177.070772] niks3-server[1164]: 2026/09/24 11:50:25 INFO Received sign narinfos request method=POST path=/api/pushes/55/sign5979lb # [ 177.080622] niks3-server[1164]: 2026/09/24 11:50:25 INFO Signed narinfos id=55 count=15980lb # [ 177.086478] niks3-server[1164]: 2026/09/24 11:50:25 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls5981lb # [ 177.100834] niks3-server[1164]: 2026/09/24 11:50:25 INFO Received complete push request method=POST path=/api/pushes/55/complete5982worker2 # [ 177.197520] nix-grpc-daemon[1714]: time=2026-09-24T11:50:25.816Z level=INFO msg="Upload complete. (244ms)"5983lb # [ 177.108868] niks3-server[1164]: 2026/09/24 11:50:25 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo5984worker2 # [ 177.218060] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=15985worker2 # [ 177.235681] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:25Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=05986worker2 # [ 177.241194] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:25Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.4:50051 clients=15987worker2 # [ 177.245887] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379045988worker2 # [ 177.250180] nix-daemon[841]: reaped child process 1798, status = succeeded5989worker2 # [ 177.253837] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:379045990worker2 # [ 177.267168] nix-grpc-daemon[1714]: time=2026-09-24T11:50:25.886Z level=INFO msg="All 1 paths already cached"5991client # copying 1 paths to 'lb:50051'...5992client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'...5993worker2 # [ 177.277434] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:379045994worker2 # [ 177.289783] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)5995worker2 # [ 177.434610] nix-daemon[841]: accepted connection from pid 1825, user nix-grpc-daemon (trusted)5996worker2 # [ 177.455259] nix-daemon[841]: reaped child process 1828, status = succeeded5997lb # [ 177.362005] niks3-server[1164]: 2026/09/24 11:50:25 INFO Received push request method=POST path=/api/pushes5998worker2 # [ 177.462477] nix-grpc-daemon[1714]: time=2026-09-24T11:50:26.081Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5999worker2 # [ 177.464300] nix-grpc-daemon[1714]: time=2026-09-24T11:50:26.081Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)"6000lb # [ 177.388255] niks3-server[1164]: 2026/09/24 11:50:25 INFO Received sign narinfos request method=POST path=/api/pushes/56/sign6001worker2 # [ 177.489932] nix-grpc-daemon[1714]: time=2026-09-24T11:50:26.108Z level=INFO msg="Uploading 1 narinfos"6002lb # [ 177.403547] niks3-server[1164]: 2026/09/24 11:50:26 INFO Signed narinfos id=56 count=16003lb # [ 177.412600] niks3-server[1164]: 2026/09/24 11:50:26 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls6004lb # [ 177.426218] niks3-server[1164]: 2026/09/24 11:50:26 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst6005worker2 # [ 177.527303] nix-grpc-daemon[1714]: time=2026-09-24T11:50:26.146Z level=INFO msg="Upload complete. (223ms)"6006lb # [ 177.438504] niks3-server[1164]: 2026/09/24 11:50:26 INFO Received complete push request method=POST path=/api/pushes/56/complete6007lb # [ 177.443774] niks3-server[1164]: 2026/09/24 11:50:26 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo6008worker2 # [ 177.536273] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=0 paths=1 nar_bytes_in=6406009worker2 # [ 177.550253] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379046010worker2 # [ 177.557427] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6011worker2 # [ 177.562524] nix-daemon[841]: reaped child process 1823, status = succeeded6012client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051...6013client # farm-b> worker2: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv6014worker2 # [ 177.574151] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6015client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6016client # farm-b> LOG-farm-b-during-drain6017worker2 # [ 178.614885] nix-daemon[841]: reaped child process 1832, status = succeeded6018worker2 # [ 178.727349] nix-daemon[841]: accepted connection from pid 1848, user nix-grpc-daemon (trusted)6019worker2 # [ 178.749603] nix-daemon[841]: reaped child process 1851, status = succeeded6020lb # [ 178.657790] niks3-server[1164]: 2026/09/24 11:50:27 INFO Received push request method=POST path=/api/pushes6021worker2 # [ 178.759730] nix-grpc-daemon[1714]: time=2026-09-24T11:50:27.378Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6022worker2 # [ 178.762219] nix-grpc-daemon[1714]: time=2026-09-24T11:50:27.380Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)"6023lb # [ 178.700325] niks3-server[1164]: 2026/09/24 11:50:27 INFO Received sign narinfos request method=POST path=/api/pushes/57/sign6024worker2 # [ 178.801476] nix-grpc-daemon[1714]: time=2026-09-24T11:50:27.420Z level=INFO msg="Uploading 1 narinfos"6025lb # [ 178.715963] niks3-server[1164]: 2026/09/24 11:50:27 INFO Signed narinfos id=57 count=16026lb # [ 178.727083] niks3-server[1164]: 2026/09/24 11:50:27 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst6027lb # [ 178.739306] niks3-server[1164]: 2026/09/24 11:50:27 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls6028lb # [ 178.747108] niks3-server[1164]: 2026/09/24 11:50:27 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv6029lb # [ 178.758657] niks3-server[1164]: 2026/09/24 11:50:27 INFO Received complete push request method=POST path=/api/pushes/57/complete6030worker2 # [ 178.855557] nix-grpc-daemon[1714]: time=2026-09-24T11:50:27.474Z level=INFO msg="Upload complete. (269ms)"6031lb # [ 178.764933] niks3-server[1164]: 2026/09/24 11:50:27 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo6032worker2 # [ 178.875830] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=16033worker2 # [ 178.892282] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:27Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=06034worker2 # [ 178.902744] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:27Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.4:50051 clients=16035worker2 # [ 178.909147] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379046036worker2 # [ 178.914311] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:379046037worker2 # [ 178.919570] nix-daemon[841]: reaped child process 1830, status = succeeded6038client # copying 1 paths to 'lb:50051'...6039worker2 # [ 178.921751] nix-grpc-daemon[1714]: time=2026-09-24T11:50:27.537Z level=INFO msg="All 2 paths already cached"6040client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'...6041worker2 # [ 178.925343] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:379046042worker2 # [ 178.936113] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6043worker2 # [ 179.070238] nix-daemon[841]: accepted connection from pid 1857, user nix-grpc-daemon (trusted)6044worker2 # [ 179.090486] nix-daemon[841]: reaped child process 1860, status = succeeded6045lb # [ 179.002097] niks3-server[1164]: 2026/09/24 11:50:27 INFO Received push request method=POST path=/api/pushes6046worker2 # [ 179.102354] nix-grpc-daemon[1714]: time=2026-09-24T11:50:27.721Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6047worker2 # [ 179.103865] nix-grpc-daemon[1714]: time=2026-09-24T11:50:27.721Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)"6048lb # [ 179.029996] niks3-server[1164]: 2026/09/24 11:50:27 INFO Received sign narinfos request method=POST path=/api/pushes/58/sign6049worker2 # [ 179.127339] nix-grpc-daemon[1714]: time=2026-09-24T11:50:27.746Z level=INFO msg="Uploading 1 narinfos"6050lb # [ 179.042346] niks3-server[1164]: 2026/09/24 11:50:27 INFO Signed narinfos id=58 count=16051lb # [ 179.052796] niks3-server[1164]: 2026/09/24 11:50:27 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls6052lb # [ 179.065776] niks3-server[1164]: 2026/09/24 11:50:27 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst6053lb # [ 179.079644] envoy[907]: conn peer=[2001:db8:1::1]:48568 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3013 tx=829 ms=35156054worker2 # [ 179.175537] nix-grpc-daemon[1714]: time=2026-09-24T11:50:27.794Z level=INFO msg="Upload complete. (228ms)"6055lb # [ 179.086822] niks3-server[1164]: 2026/09/24 11:50:27 INFO Received complete push request method=POST path=/api/pushes/58/complete6056lb # [ 179.091149] niks3-server[1164]: 2026/09/24 11:50:27 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo6057worker2 # [ 179.187885] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=0 paths=1 nar_bytes_in=7686058worker2 # [ 179.202677] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379046059worker2 # [ 179.208899] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6060worker2 # [ 179.213979] nix-daemon[841]: reaped child process 1855, status = succeeded6061client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051...6062worker2 # [ 179.227294] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6063client # farm-top> worker2: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6064client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6065client # farm-top> LOG-farm-top-during-drain6066worker2 # [ 180.508850] nix-daemon[841]: reaped child process 1864, status = succeeded6067worker2 # [ 180.617344] nix-daemon[841]: accepted connection from pid 1880, user nix-grpc-daemon (trusted)6068worker2 # [ 180.639342] nix-daemon[841]: reaped child process 1883, status = succeeded6069lb # [ 180.553174] niks3-server[1164]: 2026/09/24 11:50:29 INFO Received push request method=POST path=/api/pushes6070worker2 # [ 180.655997] nix-grpc-daemon[1714]: time=2026-09-24T11:50:29.273Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6071worker2 # [ 180.657820] nix-grpc-daemon[1714]: time=2026-09-24T11:50:29.273Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)"6072lb # [ 180.590483] niks3-server[1164]: 2026/09/24 11:50:29 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst6073lb # [ 180.606816] niks3-server[1164]: 2026/09/24 11:50:29 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6074lb # [ 180.617490] niks3-server[1164]: 2026/09/24 11:50:29 INFO Received sign narinfos request method=POST path=/api/pushes/59/sign6075worker2 # [ 180.714991] nix-grpc-daemon[1714]: time=2026-09-24T11:50:29.333Z level=INFO msg="Uploading 1 narinfos"6076lb # [ 180.627268] niks3-server[1164]: 2026/09/24 11:50:29 INFO Signed narinfos id=59 count=16077lb # [ 180.633533] niks3-server[1164]: 2026/09/24 11:50:29 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls6078lb # [ 180.645140] niks3-server[1164]: 2026/09/24 11:50:29 INFO Received complete push request method=POST path=/api/pushes/59/complete6079lb # [ 180.654372] niks3-server[1164]: 2026/09/24 11:50:29 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo6080worker2 # [ 180.745597] nix-grpc-daemon[1714]: time=2026-09-24T11:50:29.364Z level=INFO msg="Upload complete. (306ms)"6081worker2 # [ 180.761343] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=16082worker2 # [ 180.778702] nix-daemon[841]: reaped child process 1862, status = succeeded6083client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2, in 5.73 seconds)6084client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath6085client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath, in 0.18 seconds)6086worker1: must succeed: pkill -f 'read -t [9]0 x'6087worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)6088client: waiting for success: ! systemctl is-active sw6089worker1 # [ 181.172058] nix-daemon[854]: reaped child process 1710, status = succeeded6090worker1 # [ 181.241342] nix-daemon[854]: accepted connection from pid 1752, user nix-grpc-daemon (trusted)6091worker1 # [ 181.264247] nix-daemon[854]: reaped child process 1755, status = succeeded6092lb # [ 181.247159] niks3-server[1164]: 2026/09/24 11:50:29 INFO Received push request method=POST path=/api/pushes6093worker1 # [ 181.278254] nix-grpc-daemon[1656]: time=2026-09-24T11:50:29.892Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6094worker1 # [ 181.279774] nix-grpc-daemon[1656]: time=2026-09-24T11:50:29.892Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)"6095lb # [ 181.279626] niks3-server[1164]: 2026/09/24 11:50:29 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls6096lb # [ 181.297769] niks3-server[1164]: 2026/09/24 11:50:29 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6097lb # [ 181.310559] niks3-server[1164]: 2026/09/24 11:50:29 INFO Received sign narinfos request method=POST path=/api/pushes/60/sign6098worker1 # [ 181.336361] nix-grpc-daemon[1656]: time=2026-09-24T11:50:29.951Z level=INFO msg="Uploading 1 narinfos"6099lb # [ 181.321256] niks3-server[1164]: 2026/09/24 11:50:29 INFO Signed narinfos id=60 count=16100lb # [ 181.327108] niks3-server[1164]: 2026/09/24 11:50:29 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv6101lb # [ 181.341100] niks3-server[1164]: 2026/09/24 11:50:29 INFO Received complete push request method=POST path=/api/pushes/60/complete6102worker1 # [ 181.366598] nix-grpc-daemon[1656]: time=2026-09-24T11:50:29.982Z level=INFO msg="Upload complete. (282ms)"6103lb # [ 181.347921] niks3-server[1164]: 2026/09/24 11:50:29 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo6104worker1 # [ 181.381287] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40712 duration_s=8 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=16105worker1 # [ 181.399441] nix-daemon[854]: reaped child process 1708, status = succeeded6106client # [ 181.391363] systemd[1]: sw.service: Deactivated successfully.6107client # [ 181.396758] systemd[1]: sw.service: Consumed 204ms CPU time over 9.159s wall clock time, 43.4M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.6108client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds)6109client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6110client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.05 seconds)6111worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6112worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 0.06 seconds)6113lb: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux6114lb: must succeed: curl -sf localhost:9901/clusters6115lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6116worker1 # [ 182.347764] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:30Z level=info event=signal_exit6117lb: must succeed: curl -sf localhost:9901/clusters6118lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6119lb: must succeed: curl -sf localhost:9901/clusters6120worker1 # [ 184.349291] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:32Z level=info event=scheduler_restarting6121worker2 # [ 184.435669] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:33Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:515066122worker2 # [ 184.436471] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:33Z level=info event=worker_gone addr=192.168.1.3:500516123worker1 # [ 184.384840] nix-grpc-daemon[1550]: ts=2026-09-24T11:50:32Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6124lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6125worker1 # [ 184.411719] nix-daemon[854]: reaped child process 1622, status = succeeded6126worker1 # [ 185.367980] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6127worker1 # [ 185.375178] systemd[1]: nix-grpc-daemon.service: Consumed 891ms CPU time over 30.218s wall clock time, 36M memory peak, 52.2K incoming IP traffic, 60.5K outgoing IP traffic.6128lb: must succeed: curl -sf localhost:9901/clusters6129lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6130worker1 # [ 185.560076] systemd[1]: nix-grpc-daemon.service: Scheduled restart job, restart counter is at 1.6131worker1 # [ 185.574133] systemd[1]: Starting Nix worker-protocol over gRPC...6132worker1 # [ 185.730384] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:34Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6133worker1 # [ 185.732723] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:34Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6134worker1 # [ 185.736638] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:34Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=16135worker1 # [ 185.743283] nix-grpc-daemon[1768]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6136worker1 # [ 185.751031] nix-grpc-daemon[1768]: E0000 00:00:1790250634.356677 1768 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6137worker1 # [ 185.759594] nix-grpc-daemon[1768]: E0000 00:00:1790250634.358788 1768 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6138worker1 # [ 185.786738] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:34Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket6139worker1 # [ 185.789393] nix-grpc-daemon[1768]: E0000 00:00:1790250634.369854 1808 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6140worker1 # [ 185.792405] nix-grpc-daemon[1768]: E0000 00:00:1790250634.387648 1808 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6141worker1 # [ 185.794886] nix-grpc-daemon[1768]: E0000 00:00:1790250634.387815 1808 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6142worker1 # [ 185.800681] nix-grpc-daemon[1768]: E0000 00:00:1790250634.402234 1808 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6143worker1 # [ 185.804873] systemd[1]: Started Nix worker-protocol over gRPC.6144worker1 # [ 185.805901] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:34Z level=info event=scheduler_connected addr=lb:500516145worker2 # [ 185.878158] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:34Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:515066146worker2 # [ 185.887013] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:34Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=06147lb: must succeed: curl -sf localhost:9901/clusters6148lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6149lb: (finished: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux, in 4.35 seconds)6150lb: must succeed: curl -sf localhost:9901/clusters6151lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6152worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6153worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6154(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 14.46 seconds)6155subtest: queue gauges show a burst while every build blocks6156lb: must succeed: curl -sf localhost:9901/clusters6157lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6158lb: must succeed: curl -sf localhost:9901/clusters6159lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6160worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6161worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6162client: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]'6163client # [ 186.863837] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto --impure --expr "map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ \"q0\" \"q1\" \"q2\" \"q3\" ]".6164client # Running as unit: burst.service; invocation ID: 192296f88bb648ed9b5d04a076becd7d6165client: (finished: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]', in 0.09 seconds)6166lb: must succeed: curl -sf localhost:9901/clusters6167lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6168lb: must succeed: curl -sf localhost:9901/clusters6169lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6170worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6171worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)6172worker1 # [ 187.113332] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:543646173worker1 # [ 187.128881] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6174client # [ 187.125245] nix[1530]: don't know how to build these paths:6175client # [ 187.126612] nix[1530]: /nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv6176client # [ 187.128326] nix[1530]: /nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv6177client # [ 187.130454] nix[1530]: /nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv6178client # [ 187.131902] nix[1530]: /nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv6179worker2 # [ 187.239732] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:35Z level=debug event=want client=5 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv cached=06180worker1 # [ 187.171975] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646181worker2 # [ 187.248483] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:35Z level=debug event=assigned drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv worker=192.168.1.4:50051 clients=16182worker2 # [ 187.254951] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:35Z level=debug event=want client=5 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv cached=06183worker1 # [ 187.187946] nix-grpc-daemon[1768]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 23 ms (attempt 1/5)6184worker2 # [ 187.262631] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:35Z level=debug event=assigned drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv worker=192.168.1.3:50051 clients=16185worker2 # [ 187.270931] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:35Z level=debug event=want client=5 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv cached=06186client # [ 187.188236] nix[1530]: copying 1 paths to 'lb:50051'...6187worker2 # [ 187.278047] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:35Z level=debug event=want client=5 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv cached=06188client # [ 187.192226] nix[1530]: copying path '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' to 'lb:50051'...6189worker2 # [ 187.285206] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379046190worker1 # [ 187.220493] nix-grpc-daemon[1768]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 149 ms (attempt 2/5)6191worker2 # [ 187.295198] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:379046192worker2 # [ 187.310473] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:379046193worker2 # [ 187.319306] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6194worker1 # [ 187.371741] nix-grpc-daemon[1768]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 33 ms (attempt 3/5)6195worker2 # [ 187.456053] nix-daemon[841]: accepted connection from pid 1898, user nix-grpc-daemon (trusted)6196worker2 # [ 187.477160] nix-daemon[841]: reaped child process 1901, status = succeeded6197worker1 # [ 187.406268] nix-grpc-daemon[1768]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 602 ms (attempt 4/5)6198lb # [ 187.387850] niks3-server[1164]: 2026/09/24 11:50:35 INFO Received push request method=POST path=/api/pushes6199worker2 # [ 187.490088] nix-grpc-daemon[1714]: time=2026-09-24T11:50:36.108Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6200worker2 # [ 187.491554] nix-grpc-daemon[1714]: time=2026-09-24T11:50:36.108Z level=INFO msg="Uploading d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv (416B)"6201lb # [ 187.426782] niks3-server[1164]: 2026/09/24 11:50:36 INFO Received sign narinfos request method=POST path=/api/pushes/61/sign6202worker2 # [ 187.525005] nix-grpc-daemon[1714]: time=2026-09-24T11:50:36.143Z level=INFO msg="Uploading 1 narinfos"6203lb # [ 187.438461] niks3-server[1164]: 2026/09/24 11:50:36 INFO Signed narinfos id=61 count=16204lb # [ 187.449037] niks3-server[1164]: 2026/09/24 11:50:36 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.ls6205lb # [ 187.457160] niks3-server[1164]: 2026/09/24 11:50:36 INFO Registered completed upload object_key=nar/06ismykzsix0w29371dm1hbzzf3ljf9jn7dyh8skp5wqfkkw0vlk.nar.zst6206lb # [ 187.476707] niks3-server[1164]: 2026/09/24 11:50:36 INFO Received complete push request method=POST path=/api/pushes/61/complete6207lb # [ 187.487253] niks3-server[1164]: 2026/09/24 11:50:36 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.narinfo6208worker2 # [ 187.584212] nix-grpc-daemon[1714]: time=2026-09-24T11:50:36.203Z level=INFO msg="Upload complete. (260ms)"6209worker2 # [ 187.594088] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=0 paths=1 nar_bytes_in=4166210worker2 # [ 187.612571] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379046211worker2 # [ 187.621944] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6212worker2 # [ 187.627570] nix-daemon[841]: reaped child process 1896, status = succeeded6213client # [ 187.547418] nix[1530]: building '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' on lb:50051...6214worker2 # [ 187.644379] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6215worker1 # [ 188.010933] nix-grpc-daemon[1768]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6216worker1 # [ 188.017637] nix-grpc-daemon[1768]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6217worker1 # [ 188.035833] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:543646218client # [ 188.028278] nix[1530]: copying 1 paths to 'lb:50051'...6219worker1 # [ 188.051462] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:543646220client # [ 188.032155] nix[1530]: copying path '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' to 'lb:50051'...6221worker1 # [ 188.069173] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6222worker1 # [ 188.098075] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:36Z level=info event=niks3_push_started pid=18286223lb: must succeed: curl -sf localhost:9901/clusters6224lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6225lb: must succeed: curl -sf localhost:9901/clusters6226lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6227worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6228worker1 # [ 188.251670] nix-grpc-daemon[1828]: time=2026-09-24T11:50:36.866Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""6229worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6230worker1 # [ 188.387894] nix-daemon[854]: accepted connection from pid 1841, user nix-grpc-daemon (trusted)6231worker1 # [ 188.420001] nix-daemon[854]: reaped child process 1848, status = succeeded6232lb # [ 188.397611] niks3-server[1164]: 2026/09/24 11:50:36 INFO Received push request method=POST path=/api/pushes6233worker1 # [ 188.429867] nix-grpc-daemon[1828]: time=2026-09-24T11:50:37.044Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6234worker1 # [ 188.431396] nix-grpc-daemon[1828]: time=2026-09-24T11:50:37.044Z level=INFO msg="Uploading dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv (416B)"6235lb # [ 188.441049] niks3-server[1164]: 2026/09/24 11:50:37 INFO Registered completed upload object_key=nar/06mdwsnf16qmfi86y8r33asi4vcrp5z7kdxqki87psgfpgvvpdki.nar.zst6236worker1 # [ 188.475318] nix-grpc-daemon[1828]: time=2026-09-24T11:50:37.089Z level=INFO msg="Uploading 1 narinfos"6237lb # [ 188.455918] niks3-server[1164]: 2026/09/24 11:50:37 INFO Received sign narinfos request method=POST path=/api/pushes/62/sign6238lb # [ 188.465846] niks3-server[1164]: 2026/09/24 11:50:37 INFO Signed narinfos id=62 count=16239lb # [ 188.473133] niks3-server[1164]: 2026/09/24 11:50:37 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.ls6240lb # [ 188.481706] niks3-server[1164]: 2026/09/24 11:50:37 INFO Received complete push request method=POST path=/api/pushes/62/complete6241lb # [ 188.495022] niks3-server[1164]: 2026/09/24 11:50:37 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.narinfo6242worker1 # [ 188.520648] nix-grpc-daemon[1828]: time=2026-09-24T11:50:37.136Z level=INFO msg="Upload complete. (265ms)"6243worker1 # [ 188.531253] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=0 paths=1 nar_bytes_in=4166244worker1 # [ 188.548743] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646245worker1 # [ 188.575984] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6246worker1 # [ 188.583810] nix-daemon[854]: reaped child process 1826, status = succeeded6247worker1 # [ 188.589791] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6248client # [ 188.568526] nix[1530]: building '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' on lb:50051...6249worker2 # [ 188.746912] nix-daemon[841]: reaped child process 1905, status = succeeded6250worker2 # [ 188.816450] nix-daemon[841]: accepted connection from pid 1929, user nix-grpc-daemon (trusted)6251worker2 # [ 188.836082] nix-daemon[841]: reaped child process 1932, status = succeeded6252lb # [ 188.747583] niks3-server[1164]: 2026/09/24 11:50:37 INFO Received push request method=POST path=/api/pushes6253worker2 # [ 188.852413] nix-grpc-daemon[1714]: time=2026-09-24T11:50:37.469Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6254worker2 # [ 188.854024] nix-grpc-daemon[1714]: time=2026-09-24T11:50:37.469Z level=INFO msg="Uploading k6zivhvzgmlyxfmy64fzd759bxlhf9b5-slow-q0 (120B)"6255lb # [ 188.795140] niks3-server[1164]: 2026/09/24 11:50:37 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.ls6256worker2 # [ 188.895819] nix-grpc-daemon[1714]: time=2026-09-24T11:50:37.514Z level=INFO msg="Uploading 1 narinfos"6257lb # [ 188.806359] niks3-server[1164]: 2026/09/24 11:50:37 INFO Received sign narinfos request method=POST path=/api/pushes/63/sign6258lb # [ 188.817699] niks3-server[1164]: 2026/09/24 11:50:37 INFO Signed narinfos id=63 count=16259lb # [ 188.823901] niks3-server[1164]: 2026/09/24 11:50:37 INFO Registered completed upload object_key=nar/0xr809jj32p5n6mnc6i7lqsz48wrm5f9jp14m77k724041711iq7.nar.zst6260lb # [ 188.835816] niks3-server[1164]: 2026/09/24 11:50:37 INFO Registered completed upload object_key=log/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv6261lb # [ 188.841942] niks3-server[1164]: 2026/09/24 11:50:37 INFO Received complete push request method=POST path=/api/pushes/63/complete6262worker2 # [ 188.937592] nix-grpc-daemon[1714]: time=2026-09-24T11:50:37.556Z level=INFO msg="Upload complete. (281ms)"6263lb # [ 188.847711] niks3-server[1164]: 2026/09/24 11:50:37 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.narinfo6264worker2 # [ 188.951954] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=1 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv assign_id=7 outputs=16265worker2 # [ 188.972875] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:37Z level=debug event=assigned drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv worker=192.168.1.4:50051 clients=16266worker2 # [ 188.979949] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379046267worker2 # [ 188.985171] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:379046268client # [ 188.896389] nix[1530]: copying 1 paths to 'lb:50051'...6269client # [ 188.898814] nix[1530]: copying path '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' to 'lb:50051'...6270worker2 # [ 188.998445] nix-daemon[841]: reaped child process 1903, status = succeeded6271worker2 # [ 189.004342] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6272worker2 # [ 189.006693] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:379046273worker2 # [ 189.139483] nix-daemon[841]: accepted connection from pid 1936, user nix-grpc-daemon (trusted)6274lb # [ 189.051492] envoy[907]: conn peer=[2001:db8:1::1]:43654 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9634 tx=4429 ms=54756275lb # [ 189.060660] envoy[907]: conn peer=[2001:db8:1::1]:43642 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2991 tx=1517 ms=89296276worker2 # [ 189.156055] nix-daemon[841]: reaped child process 1939, status = succeeded6277lb # [ 189.065010] envoy[907]: conn peer=[2001:db8:1::3]:53458 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1020 tx=482 ms=289756278lb # [ 189.072088] niks3-server[1164]: 2026/09/24 11:50:37 INFO Received push request method=POST path=/api/pushes6279worker2 # [ 189.173040] nix-grpc-daemon[1714]: time=2026-09-24T11:50:37.791Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6280worker2 # [ 189.174525] nix-grpc-daemon[1714]: time=2026-09-24T11:50:37.791Z level=INFO msg="Uploading jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv (416B)"6281lb # [ 189.095990] niks3-server[1164]: 2026/09/24 11:50:37 INFO Received sign narinfos request method=POST path=/api/pushes/64/sign6282worker2 # [ 189.192261] nix-grpc-daemon[1714]: time=2026-09-24T11:50:37.811Z level=INFO msg="Uploading 1 narinfos"6283lb # [ 189.104905] niks3-server[1164]: 2026/09/24 11:50:37 INFO Signed narinfos id=64 count=16284lb # [ 189.108942] niks3-server[1164]: 2026/09/24 11:50:37 INFO Registered completed upload object_key=nar/1fyfjzlx8vymsfmdv5cv8ymgjkr18fwm737qvys29plwwiwmyvwr.nar.zst6285lb # [ 189.113930] niks3-server[1164]: 2026/09/24 11:50:37 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.ls6286worker2 # [ 189.213931] nix-grpc-daemon[1714]: time=2026-09-24T11:50:37.833Z level=INFO msg="Upload complete. (201ms)"6287lb # [ 189.123746] niks3-server[1164]: 2026/09/24 11:50:37 INFO Received complete push request method=POST path=/api/pushes/64/complete6288lb # [ 189.127678] niks3-server[1164]: 2026/09/24 11:50:37 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.narinfo6289worker2 # [ 189.220195] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=0 paths=1 nar_bytes_in=4166290worker2 # [ 189.231902] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379046291worker2 # [ 189.237348] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6292worker2 # [ 189.241809] nix-daemon[841]: reaped child process 1934, status = succeeded6293client # [ 189.157941] nix[1530]: building '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' on lb:50051...6294worker2 # [ 189.253323] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6295worker2 # [ 191.120388] nix-daemon[841]: reaped child process 1943, status = succeeded6296worker1 # [ 191.098589] nix-daemon[854]: reaped child process 1852, status = succeeded6297worker2 # [ 191.219389] nix-daemon[841]: accepted connection from pid 1968, user nix-grpc-daemon (trusted)6298worker2 # [ 191.237459] nix-daemon[841]: reaped child process 1971, status = succeeded6299worker1 # [ 191.169444] nix-daemon[854]: accepted connection from pid 1877, user nix-grpc-daemon (trusted)6300lb # [ 191.153545] niks3-server[1164]: 2026/09/24 11:50:39 INFO Received push request method=POST path=/api/pushes6301worker2 # [ 191.256426] nix-grpc-daemon[1714]: time=2026-09-24T11:50:39.872Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6302worker2 # [ 191.258103] nix-grpc-daemon[1714]: time=2026-09-24T11:50:39.872Z level=INFO msg="Uploading gjyi8j90pk5flrsqqdhj6h132pcb7n33-slow-q2 (120B)"6303worker1 # [ 191.191882] nix-daemon[854]: reaped child process 1880, status = succeeded6304lb # [ 191.174958] niks3-server[1164]: 2026/09/24 11:50:39 INFO Received push request method=POST path=/api/pushes6305worker1 # [ 191.209964] nix-grpc-daemon[1828]: time=2026-09-24T11:50:39.823Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6306worker1 # [ 191.211628] nix-grpc-daemon[1828]: time=2026-09-24T11:50:39.823Z level=INFO msg="Uploading y3qkgrgq2zzpgvi2rf5chq94i627wkdb-slow-q3 (120B)"6307lb # [ 191.218304] niks3-server[1164]: 2026/09/24 11:50:39 INFO Received sign narinfos request method=POST path=/api/pushes/65/sign6308worker2 # [ 191.316329] nix-grpc-daemon[1714]: time=2026-09-24T11:50:39.935Z level=INFO msg="Uploading 1 narinfos"6309lb # [ 191.231788] niks3-server[1164]: 2026/09/24 11:50:39 INFO Signed narinfos id=65 count=16310lb # [ 191.237220] niks3-server[1164]: 2026/09/24 11:50:39 INFO Registered completed upload object_key=nar/15imdx7zyy6mzjjv6qvdzywh0d9w1n6ikc0rq4rq95aw9x6swfzj.nar.zst6311lb # [ 191.244837] niks3-server[1164]: 2026/09/24 11:50:39 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.ls6312lb # [ 191.247278] niks3-server[1164]: 2026/09/24 11:50:39 INFO Registered completed upload object_key=nar/1v1grzkvw3yj09x0dkdm2avihvqvnri156qzp768hk85zbw7lkn9.nar.zst6313worker1 # [ 191.277058] nix-grpc-daemon[1828]: time=2026-09-24T11:50:39.891Z level=INFO msg="Uploading 1 narinfos"6314lb # [ 191.260490] niks3-server[1164]: 2026/09/24 11:50:39 INFO Registered completed upload object_key=log/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv6315lb # [ 191.263002] niks3-server[1164]: 2026/09/24 11:50:39 INFO Registered completed upload object_key=log/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv6316lb # [ 191.273327] niks3-server[1164]: 2026/09/24 11:50:39 INFO Received sign narinfos request method=POST path=/api/pushes/66/sign6317lb # [ 191.275403] niks3-server[1164]: 2026/09/24 11:50:39 INFO Signed narinfos id=66 count=16318worker2 # [ 191.365046] nix-grpc-daemon[1714]: time=2026-09-24T11:50:39.984Z level=INFO msg="Upload complete. (301ms)"6319lb # [ 191.288288] niks3-server[1164]: 2026/09/24 11:50:39 INFO Received complete push request method=POST path=/api/pushes/65/complete6320worker1 # [ 191.308172] nix-grpc-daemon[1828]: time=2026-09-24T11:50:39.923Z level=INFO msg="Upload complete. (308ms)"6321lb # [ 191.290996] niks3-server[1164]: 2026/09/24 11:50:39 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.ls6322lb # [ 191.293361] niks3-server[1164]: 2026/09/24 11:50:39 INFO Received complete push request method=POST path=/api/pushes/66/complete6323worker2 # [ 191.380847] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=2 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv assign_id=9 outputs=16324lb # [ 191.296625] niks3-server[1164]: 2026/09/24 11:50:39 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.narinfo6325lb # [ 191.298928] niks3-server[1164]: 2026/09/24 11:50:39 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.narinfo6326worker1 # [ 191.321184] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=2 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv assign_id=8 outputs=16327worker2 # [ 191.400363] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:39Z level=debug event=assigned drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker=192.168.1.4:50051 clients=16328worker1 # [ 191.341348] nix-daemon[854]: reaped child process 1850, status = succeeded6329worker2 # [ 191.411205] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379046330worker2 # [ 191.418114] nix-daemon[841]: reaped child process 1941, status = succeeded6331worker2 # [ 191.422193] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:379046332client # [ 191.333298] nix[1530]: copying 1 paths to 'lb:50051'...6333worker2 # [ 191.427863] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:379046334client # [ 191.336672] nix[1530]: copying path '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' to 'lb:50051'...6335worker2 # [ 191.441874] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6336worker2 # [ 191.585712] nix-daemon[841]: accepted connection from pid 1975, user nix-grpc-daemon (trusted)6337worker2 # [ 191.631485] nix-daemon[841]: reaped child process 1978, status = succeeded6338lb # [ 191.538429] niks3-server[1164]: 2026/09/24 11:50:40 INFO Received push request method=POST path=/api/pushes6339worker2 # [ 191.638600] nix-grpc-daemon[1714]: time=2026-09-24T11:50:40.257Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6340worker2 # [ 191.640335] nix-grpc-daemon[1714]: time=2026-09-24T11:50:40.257Z level=INFO msg="Uploading pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv (416B)"6341lb # [ 191.565925] niks3-server[1164]: 2026/09/24 11:50:40 INFO Received sign narinfos request method=POST path=/api/pushes/67/sign6342worker2 # [ 191.663759] nix-grpc-daemon[1714]: time=2026-09-24T11:50:40.282Z level=INFO msg="Uploading 1 narinfos"6343lb # [ 191.576142] niks3-server[1164]: 2026/09/24 11:50:40 INFO Signed narinfos id=67 count=16344lb # [ 191.585168] niks3-server[1164]: 2026/09/24 11:50:40 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.ls6345lb # [ 191.595325] niks3-server[1164]: 2026/09/24 11:50:40 INFO Registered completed upload object_key=nar/1k21jrz411xc9gkca2pnr9mrjq1s3bg3zhsam3nnv4j066dgzgm7.nar.zst6346lb # [ 191.609504] niks3-server[1164]: 2026/09/24 11:50:40 INFO Received complete push request method=POST path=/api/pushes/67/complete6347worker2 # [ 191.708715] nix-grpc-daemon[1714]: time=2026-09-24T11:50:40.327Z level=INFO msg="Upload complete. (253ms)"6348lb # [ 191.617145] niks3-server[1164]: 2026/09/24 11:50:40 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.narinfo6349worker2 # [ 191.717624] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=0 paths=1 nar_bytes_in=4166350worker2 # [ 191.728708] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379046351worker2 # [ 191.739321] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6352worker2 # [ 191.745175] nix-daemon[841]: reaped child process 1973, status = succeeded6353client # [ 191.661283] nix[1530]: building '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' on lb:50051...6354worker2 # [ 191.756555] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6355worker2 # [ 193.388810] nix-daemon[841]: reaped child process 1982, status = succeeded6356worker2 # [ 193.477370] nix-daemon[841]: accepted connection from pid 2007, user nix-grpc-daemon (trusted)6357worker2 # [ 193.493548] nix-daemon[841]: reaped child process 2010, status = succeeded6358lb # [ 193.409334] niks3-server[1164]: 2026/09/24 11:50:42 INFO Received push request method=POST path=/api/pushes6359worker2 # [ 193.511023] nix-grpc-daemon[1714]: time=2026-09-24T11:50:42.128Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6360worker2 # [ 193.512590] nix-grpc-daemon[1714]: time=2026-09-24T11:50:42.129Z level=INFO msg="Uploading l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx-slow-q1 (120B)"6361lb # [ 193.450873] niks3-server[1164]: 2026/09/24 11:50:42 INFO Registered completed upload object_key=log/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv6362worker2 # [ 193.556393] nix-grpc-daemon[1714]: time=2026-09-24T11:50:42.175Z level=INFO msg="Uploading 1 narinfos"6363lb # [ 193.466229] niks3-server[1164]: 2026/09/24 11:50:42 INFO Registered completed upload object_key=nar/0c1c367r3msa4fz6imwp70zxi4fxvrkb86v3hv921wsrf13yd8v3.nar.zst6364lb # [ 193.477061] niks3-server[1164]: 2026/09/24 11:50:42 INFO Received sign narinfos request method=POST path=/api/pushes/68/sign6365lb # [ 193.479855] niks3-server[1164]: 2026/09/24 11:50:42 INFO Signed narinfos id=68 count=16366lb # [ 193.489662] niks3-server[1164]: 2026/09/24 11:50:42 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.ls6367worker2 # [ 193.584249] nix-grpc-daemon[1714]: time=2026-09-24T11:50:42.203Z level=INFO msg="Upload complete. (218ms)"6368lb # [ 193.493717] niks3-server[1164]: 2026/09/24 11:50:42 INFO Received complete push request method=POST path=/api/pushes/68/complete6369lb # [ 193.496964] niks3-server[1164]: 2026/09/24 11:50:42 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.narinfo6370worker2 # [ 193.593215] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=1 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv assign_id=10 outputs=16371worker2 # [ 193.609062] nix-daemon[841]: reaped child process 1980, status = succeeded6372client # [ 193.553462] systemd[1]: burst.service: Deactivated successfully.6373client # [ 193.558665] systemd[1]: burst.service: Consumed 251ms CPU time over 6.690s wall clock time, 53.6M memory peak, 14.1K incoming IP traffic, 18.8K outgoing IP traffic.6374client: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }6375client: (finished: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }, in 0.07 seconds)6376lb: must succeed: curl -sf localhost:9901/clusters6377lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6378lb: must succeed: curl -sf localhost:9901/clusters6379lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6380worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6381worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6382(finished: subtest: queue gauges show a burst while every build blocks, in 8.15 seconds)6383subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6384client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16385worker2 # [ 195.099782] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:43Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:379046386worker2 # [ 195.107508] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:379046387worker2 # [ 195.146023] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:43Z level=debug event=want client=6 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=06388worker2 # [ 195.152407] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:43Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=16389worker1 # [ 195.087650] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646390worker2 # [ 195.158728] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:43Z level=debug event=want client=6 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=06391worker2 # [ 195.164088] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:43Z level=debug event=want client=6 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=06392worker1 # [ 195.102598] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:543646393worker1 # [ 195.117255] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:543646394worker1 # [ 195.132432] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6395worker1 # [ 195.300217] nix-daemon[854]: accepted connection from pid 1896, user nix-grpc-daemon (trusted)6396worker1 # [ 195.319983] nix-daemon[854]: reaped child process 1899, status = succeeded6397lb # [ 195.305661] niks3-server[1164]: 2026/09/24 11:50:43 INFO Received push request method=POST path=/api/pushes6398worker1 # [ 195.335012] nix-grpc-daemon[1828]: time=2026-09-24T11:50:43.949Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6399worker1 # [ 195.336581] nix-grpc-daemon[1828]: time=2026-09-24T11:50:43.949Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)"6400lb # [ 195.338240] niks3-server[1164]: 2026/09/24 11:50:43 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst6401worker1 # [ 195.368523] nix-grpc-daemon[1828]: time=2026-09-24T11:50:43.983Z level=INFO msg="Uploading 1 narinfos"6402lb # [ 195.355080] niks3-server[1164]: 2026/09/24 11:50:43 INFO Received sign narinfos request method=POST path=/api/pushes/69/sign6403lb # [ 195.369796] niks3-server[1164]: 2026/09/24 11:50:43 INFO Signed narinfos id=69 count=16404worker1 # [ 195.395678] nix-grpc-daemon[1828]: time=2026-09-24T11:50:44.011Z level=INFO msg="Upload complete. (246ms)"6405lb # [ 195.374989] niks3-server[1164]: 2026/09/24 11:50:43 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls6406lb # [ 195.382895] niks3-server[1164]: 2026/09/24 11:50:43 INFO Received complete push request method=POST path=/api/pushes/69/complete6407worker1 # [ 195.404562] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=0 paths=1 nar_bytes_in=5126408lb # [ 195.388178] niks3-server[1164]: 2026/09/24 11:50:43 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo6409worker1 # [ 195.416657] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646410worker1 # [ 195.428236] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6411worker1 # [ 195.436314] nix-daemon[854]: reaped child process 1894, status = succeeded6412worker1 # [ 195.446414] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6413worker1 # [ 197.097244] nix-daemon[854]: reaped child process 1903, status = succeeded6414worker1 # [ 197.224916] nix-daemon[854]: accepted connection from pid 1919, user nix-grpc-daemon (trusted)6415worker1 # [ 197.246447] nix-daemon[854]: reaped child process 1922, status = succeeded6416lb # [ 197.233359] niks3-server[1164]: 2026/09/24 11:50:45 INFO Received push request method=POST path=/api/pushes6417worker1 # [ 197.263112] nix-grpc-daemon[1828]: time=2026-09-24T11:50:45.877Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6418worker1 # [ 197.264926] nix-grpc-daemon[1828]: time=2026-09-24T11:50:45.877Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)"6419worker1 # [ 197.290952] nix-grpc-daemon[1828]: time=2026-09-24T11:50:45.906Z level=INFO msg="Uploading 1 narinfos"6420lb # [ 197.269668] niks3-server[1164]: 2026/09/24 11:50:45 INFO Received sign narinfos request method=POST path=/api/pushes/70/sign6421lb # [ 197.284555] niks3-server[1164]: 2026/09/24 11:50:45 INFO Signed narinfos id=70 count=16422lb # [ 197.287809] niks3-server[1164]: 2026/09/24 11:50:45 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv6423lb # [ 197.308941] niks3-server[1164]: 2026/09/24 11:50:45 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6424lb # [ 197.316155] niks3-server[1164]: 2026/09/24 11:50:45 INFO Received complete push request method=POST path=/api/pushes/70/complete6425worker1 # [ 197.340964] nix-grpc-daemon[1828]: time=2026-09-24T11:50:45.956Z level=INFO msg="Upload complete. (283ms)"6426lb # [ 197.322927] niks3-server[1164]: 2026/09/24 11:50:45 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls6427lb # [ 197.328289] niks3-server[1164]: 2026/09/24 11:50:45 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo6428worker2 # [ 197.421689] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:46Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=16429worker1 # [ 197.354509] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=11 outputs=16430worker2 # [ 197.435514] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:46Z level=debug event=want client=6 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=06431worker1 # [ 197.372404] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646432worker1 # [ 197.382164] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:543646433worker1 # [ 197.387489] nix-daemon[854]: reaped child process 1901, status = succeeded6434worker1 # [ 197.395034] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:543646435worker1 # [ 197.408253] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6436worker1 # [ 197.567849] nix-daemon[854]: accepted connection from pid 1926, user nix-grpc-daemon (trusted)6437worker1 # [ 197.588548] nix-daemon[854]: reaped child process 1929, status = succeeded6438lb # [ 197.566358] niks3-server[1164]: 2026/09/24 11:50:46 INFO Received push request method=POST path=/api/pushes6439worker1 # [ 197.595687] nix-grpc-daemon[1828]: time=2026-09-24T11:50:46.210Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6440worker1 # [ 197.597620] nix-grpc-daemon[1828]: time=2026-09-24T11:50:46.210Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)"6441lb # [ 197.596658] niks3-server[1164]: 2026/09/24 11:50:46 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls6442worker1 # [ 197.626156] nix-grpc-daemon[1828]: time=2026-09-24T11:50:46.241Z level=INFO msg="Uploading 1 narinfos"6443lb # [ 197.612247] niks3-server[1164]: 2026/09/24 11:50:46 INFO Received sign narinfos request method=POST path=/api/pushes/71/sign6444lb # [ 197.621087] niks3-server[1164]: 2026/09/24 11:50:46 INFO Signed narinfos id=71 count=16445lb # [ 197.623691] niks3-server[1164]: 2026/09/24 11:50:46 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst6446worker1 # [ 197.658874] nix-grpc-daemon[1828]: time=2026-09-24T11:50:46.274Z level=INFO msg="Upload complete. (239ms)"6447lb # [ 197.638391] niks3-server[1164]: 2026/09/24 11:50:46 INFO Received complete push request method=POST path=/api/pushes/71/complete6448lb # [ 197.642563] niks3-server[1164]: 2026/09/24 11:50:46 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo6449worker1 # [ 197.668283] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=0 paths=1 nar_bytes_in=5126450worker1 # [ 197.684426] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646451worker1 # [ 197.690666] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6452worker1 # [ 197.695628] nix-daemon[854]: reaped child process 1924, status = succeeded6453worker1 # [ 197.709795] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6454worker1 # [ 199.000912] nix-daemon[854]: reaped child process 1933, status = succeeded6455lb # [ 199.049591] envoy[907]: conn peer=[2001:db8:1::1]:39028 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9684 tx=5219 ms=64456456worker1 # [ 199.095713] nix-daemon[854]: accepted connection from pid 1949, user nix-grpc-daemon (trusted)6457worker1 # [ 199.116585] nix-daemon[854]: reaped child process 1952, status = succeeded6458lb # [ 199.094438] niks3-server[1164]: 2026/09/24 11:50:47 INFO Received push request method=POST path=/api/pushes6459worker1 # [ 199.124715] nix-grpc-daemon[1828]: time=2026-09-24T11:50:47.740Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6460worker1 # [ 199.127838] nix-grpc-daemon[1828]: time=2026-09-24T11:50:47.740Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)"6461lb # [ 199.138461] niks3-server[1164]: 2026/09/24 11:50:47 INFO Received sign narinfos request method=POST path=/api/pushes/72/sign6462worker1 # [ 199.167276] nix-grpc-daemon[1828]: time=2026-09-24T11:50:47.782Z level=INFO msg="Uploading 1 narinfos"6463lb # [ 199.152892] niks3-server[1164]: 2026/09/24 11:50:47 INFO Signed narinfos id=72 count=16464lb # [ 199.161796] niks3-server[1164]: 2026/09/24 11:50:47 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6465lb # [ 199.176194] niks3-server[1164]: 2026/09/24 11:50:47 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv6466worker1 # [ 199.198158] nix-grpc-daemon[1828]: time=2026-09-24T11:50:47.813Z level=INFO msg="Upload complete. (277ms)"6467lb # [ 199.181127] niks3-server[1164]: 2026/09/24 11:50:47 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls6468lb # [ 199.189153] niks3-server[1164]: 2026/09/24 11:50:47 INFO Received complete push request method=POST path=/api/pushes/72/complete6469worker2 # [ 199.281542] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:47Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=16470worker1 # [ 199.212339] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=12 outputs=16471lb # [ 199.195304] niks3-server[1164]: 2026/09/24 11:50:47 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo6472worker2 # [ 199.293781] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:47Z level=debug event=want client=6 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=06473worker1 # [ 199.231604] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646474worker1 # [ 199.237787] nix-daemon[854]: reaped child process 1931, status = succeeded6475worker1 # [ 199.245767] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:543646476worker1 # [ 199.258285] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:543646477worker1 # [ 199.269430] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6478worker1 # [ 199.410669] nix-daemon[854]: accepted connection from pid 1956, user nix-grpc-daemon (trusted)6479worker1 # [ 199.433371] nix-daemon[854]: reaped child process 1959, status = succeeded6480lb # [ 199.411286] niks3-server[1164]: 2026/09/24 11:50:48 INFO Received push request method=POST path=/api/pushes6481worker1 # [ 199.440307] nix-grpc-daemon[1828]: time=2026-09-24T11:50:48.055Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6482worker1 # [ 199.442305] nix-grpc-daemon[1828]: time=2026-09-24T11:50:48.057Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)"6483lb # [ 199.441564] niks3-server[1164]: 2026/09/24 11:50:48 INFO Received sign narinfos request method=POST path=/api/pushes/73/sign6484worker1 # [ 199.469757] nix-grpc-daemon[1828]: time=2026-09-24T11:50:48.085Z level=INFO msg="Uploading 1 narinfos"6485lb # [ 199.455292] niks3-server[1164]: 2026/09/24 11:50:48 INFO Signed narinfos id=73 count=16486lb # [ 199.464695] niks3-server[1164]: 2026/09/24 11:50:48 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls6487lb # [ 199.481580] niks3-server[1164]: 2026/09/24 11:50:48 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst6488worker1 # [ 199.513765] nix-grpc-daemon[1828]: time=2026-09-24T11:50:48.129Z level=INFO msg="Upload complete. (231ms)"6489lb # [ 199.493431] niks3-server[1164]: 2026/09/24 11:50:48 INFO Received complete push request method=POST path=/api/pushes/73/complete6490lb # [ 199.501488] niks3-server[1164]: 2026/09/24 11:50:48 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo6491worker1 # [ 199.522638] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=0 paths=1 nar_bytes_in=5126492worker1 # [ 199.540667] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646493worker1 # [ 199.546998] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6494worker1 # [ 199.553070] nix-daemon[854]: reaped child process 1954, status = succeeded6495worker1 # [ 199.564159] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6496worker1 # [ 200.818113] nix-daemon[854]: reaped child process 1963, status = succeeded6497worker1 # [ 200.924538] nix-daemon[854]: accepted connection from pid 1979, user nix-grpc-daemon (trusted)6498worker1 # [ 200.945053] nix-daemon[854]: reaped child process 1982, status = succeeded6499lb # [ 200.926542] niks3-server[1164]: 2026/09/24 11:50:49 INFO Received push request method=POST path=/api/pushes6500worker1 # [ 200.958969] nix-grpc-daemon[1828]: time=2026-09-24T11:50:49.572Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6501worker1 # [ 200.960554] nix-grpc-daemon[1828]: time=2026-09-24T11:50:49.572Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)"6502lb # [ 200.970305] niks3-server[1164]: 2026/09/24 11:50:49 INFO Received sign narinfos request method=POST path=/api/pushes/74/sign6503worker1 # [ 200.999344] nix-grpc-daemon[1828]: time=2026-09-24T11:50:49.614Z level=INFO msg="Uploading 1 narinfos"6504lb # [ 200.983056] niks3-server[1164]: 2026/09/24 11:50:49 INFO Signed narinfos id=74 count=16505lb # [ 200.990589] niks3-server[1164]: 2026/09/24 11:50:49 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls6506lb # [ 200.997934] niks3-server[1164]: 2026/09/24 11:50:49 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv6507lb # [ 201.008125] niks3-server[1164]: 2026/09/24 11:50:49 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6508lb # [ 201.012328] niks3-server[1164]: 2026/09/24 11:50:49 INFO Received complete push request method=POST path=/api/pushes/74/complete6509worker1 # [ 201.032813] nix-grpc-daemon[1828]: time=2026-09-24T11:50:49.648Z level=INFO msg="Upload complete. (255ms)"6510lb # [ 201.016309] niks3-server[1164]: 2026/09/24 11:50:49 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo6511worker2 # [ 201.117646] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:49Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=16512worker1 # [ 201.046205] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=13 outputs=16513worker2 # [ 201.125484] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:49Z level=debug event=want client=6 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=06514worker1 # [ 201.068299] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646515worker1 # [ 201.072166] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:543646516worker1 # [ 201.077301] nix-daemon[854]: reaped child process 1961, status = succeeded6517worker1 # [ 201.083192] nix-grpc-daemon[1828]: time=2026-09-24T11:50:49.698Z level=INFO msg="All 1 paths already cached"6518worker1 # [ 201.091676] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:543646519worker1 # [ 201.103494] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6520worker1 # [ 201.242362] nix-daemon[854]: accepted connection from pid 1986, user nix-grpc-daemon (trusted)6521worker1 # [ 201.263594] nix-daemon[854]: reaped child process 1989, status = succeeded6522lb # [ 201.246293] niks3-server[1164]: 2026/09/24 11:50:49 INFO Received push request method=POST path=/api/pushes6523worker1 # [ 201.275360] nix-grpc-daemon[1828]: time=2026-09-24T11:50:49.890Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6524worker1 # [ 201.277271] nix-grpc-daemon[1828]: time=2026-09-24T11:50:49.890Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)"6525lb # [ 201.275286] niks3-server[1164]: 2026/09/24 11:50:49 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst6526worker1 # [ 201.306601] nix-grpc-daemon[1828]: time=2026-09-24T11:50:49.922Z level=INFO msg="Uploading 1 narinfos"6527lb # [ 201.289503] niks3-server[1164]: 2026/09/24 11:50:49 INFO Received sign narinfos request method=POST path=/api/pushes/75/sign6528lb # [ 201.300128] niks3-server[1164]: 2026/09/24 11:50:49 INFO Signed narinfos id=75 count=16529lb # [ 201.311348] niks3-server[1164]: 2026/09/24 11:50:49 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls6530worker1 # [ 201.338318] nix-grpc-daemon[1828]: time=2026-09-24T11:50:49.953Z level=INFO msg="Upload complete. (222ms)"6531lb # [ 201.318132] niks3-server[1164]: 2026/09/24 11:50:49 INFO Received complete push request method=POST path=/api/pushes/75/complete6532lb # [ 201.321622] niks3-server[1164]: 2026/09/24 11:50:49 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo6533worker1 # [ 201.346315] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=0 paths=1 nar_bytes_in=6246534worker1 # [ 201.356635] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646535worker1 # [ 201.367964] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6536worker1 # [ 201.373486] nix-daemon[854]: reaped child process 1984, status = succeeded6537worker1 # [ 201.387116] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6538worker1 # [ 202.267149] nix-daemon[854]: reaped child process 1993, status = succeeded6539worker1 # [ 202.390468] nix-daemon[854]: accepted connection from pid 2009, user nix-grpc-daemon (trusted)6540worker1 # [ 202.412129] nix-daemon[854]: reaped child process 2012, status = succeeded6541lb # [ 202.390028] niks3-server[1164]: 2026/09/24 11:50:50 INFO Received push request method=POST path=/api/pushes6542worker1 # [ 202.420585] nix-grpc-daemon[1828]: time=2026-09-24T11:50:51.035Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6543worker1 # [ 202.422157] nix-grpc-daemon[1828]: time=2026-09-24T11:50:51.035Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)"6544lb # [ 202.435424] niks3-server[1164]: 2026/09/24 11:50:51 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst6545lb # [ 202.449250] niks3-server[1164]: 2026/09/24 11:50:51 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv6546worker1 # [ 202.479126] nix-grpc-daemon[1828]: time=2026-09-24T11:50:51.094Z level=INFO msg="Uploading 1 narinfos"6547lb # [ 202.462383] niks3-server[1164]: 2026/09/24 11:50:51 INFO Received sign narinfos request method=POST path=/api/pushes/76/sign6548lb # [ 202.472481] niks3-server[1164]: 2026/09/24 11:50:51 INFO Signed narinfos id=76 count=16549lb # [ 202.475275] niks3-server[1164]: 2026/09/24 11:50:51 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls6550lb # [ 202.486342] niks3-server[1164]: 2026/09/24 11:50:51 INFO Received complete push request method=POST path=/api/pushes/76/complete6551worker1 # [ 202.515954] nix-grpc-daemon[1828]: time=2026-09-24T11:50:51.131Z level=INFO msg="Upload complete. (318ms)"6552lb # [ 202.496278] niks3-server[1164]: 2026/09/24 11:50:51 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo6553worker2 # [ 202.596834] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:51Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=16554worker1 # [ 202.529002] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=14 outputs=16555worker2 # [ 202.609475] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:51Z level=debug event=want client=6 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=06556worker1 # [ 202.548648] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646557worker1 # [ 202.561205] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:543646558worker1 # [ 202.567138] nix-daemon[854]: reaped child process 1991, status = succeeded6559worker1 # [ 202.574229] nix-grpc-daemon[1828]: time=2026-09-24T11:50:51.189Z level=INFO msg="All 1 paths already cached"6560worker1 # [ 202.582567] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:543646561worker1 # [ 202.596387] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6562worker1 # [ 202.732925] nix-daemon[854]: accepted connection from pid 2016, user nix-grpc-daemon (trusted)6563worker1 # [ 202.752861] nix-daemon[854]: reaped child process 2019, status = succeeded6564lb # [ 202.736074] niks3-server[1164]: 2026/09/24 11:50:51 INFO Received push request method=POST path=/api/pushes6565worker1 # [ 202.765216] nix-grpc-daemon[1828]: time=2026-09-24T11:50:51.380Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6566worker1 # [ 202.766741] nix-grpc-daemon[1828]: time=2026-09-24T11:50:51.380Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)"6567lb # [ 202.764487] niks3-server[1164]: 2026/09/24 11:50:51 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls6568worker1 # [ 202.805005] nix-grpc-daemon[1828]: time=2026-09-24T11:50:51.420Z level=INFO msg="Uploading 1 narinfos"6569lb # [ 202.778289] niks3-server[1164]: 2026/09/24 11:50:51 INFO Received sign narinfos request method=POST path=/api/pushes/77/sign6570lb # [ 202.790677] niks3-server[1164]: 2026/09/24 11:50:51 INFO Signed narinfos id=77 count=16571lb # [ 202.796285] niks3-server[1164]: 2026/09/24 11:50:51 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst6572lb # [ 202.804529] niks3-server[1164]: 2026/09/24 11:50:51 INFO Received complete push request method=POST path=/api/pushes/77/complete6573worker1 # [ 202.833957] nix-grpc-daemon[1828]: time=2026-09-24T11:50:51.449Z level=INFO msg="Upload complete. (227ms)"6574lb # [ 202.814772] niks3-server[1164]: 2026/09/24 11:50:51 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo6575worker1 # [ 202.843614] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=0 paths=1 nar_bytes_in=6246576worker1 # [ 202.857326] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646577worker1 # [ 202.866159] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6578worker1 # [ 202.871648] nix-daemon[854]: reaped child process 2014, status = succeeded6579worker1 # [ 202.881520] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6580worker1 # [ 204.346620] nix-daemon[854]: reaped child process 2023, status = succeeded6581worker1 # [ 204.473457] nix-daemon[854]: accepted connection from pid 2039, user nix-grpc-daemon (trusted)6582worker1 # [ 204.495322] nix-daemon[854]: reaped child process 2042, status = succeeded6583lb # [ 204.478561] niks3-server[1164]: 2026/09/24 11:50:53 INFO Received push request method=POST path=/api/pushes6584worker1 # [ 204.512479] nix-grpc-daemon[1828]: time=2026-09-24T11:50:53.127Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6585worker1 # [ 204.514380] nix-grpc-daemon[1828]: time=2026-09-24T11:50:53.127Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)"6586lb # [ 204.527331] niks3-server[1164]: 2026/09/24 11:50:53 INFO Received sign narinfos request method=POST path=/api/pushes/78/sign6587worker1 # [ 204.556392] nix-grpc-daemon[1828]: time=2026-09-24T11:50:53.171Z level=INFO msg="Uploading 1 narinfos"6588lb # [ 204.542009] niks3-server[1164]: 2026/09/24 11:50:53 INFO Signed narinfos id=78 count=16589lb # [ 204.549796] niks3-server[1164]: 2026/09/24 11:50:53 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv6590lb # [ 204.557015] niks3-server[1164]: 2026/09/24 11:50:53 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls6591lb # [ 204.559857] niks3-server[1164]: 2026/09/24 11:50:53 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst6592lb # [ 204.569462] niks3-server[1164]: 2026/09/24 11:50:53 INFO Received complete push request method=POST path=/api/pushes/78/complete6593lb # [ 204.578327] niks3-server[1164]: 2026/09/24 11:50:53 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo6594worker1 # [ 204.598062] nix-grpc-daemon[1828]: time=2026-09-24T11:50:53.213Z level=INFO msg="Upload complete. (304ms)"6595worker2 # [ 204.678437] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:53Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=16596worker1 # [ 204.613217] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=15 outputs=16597worker2 # [ 204.690968] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:53Z level=debug event=want client=6 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=06598worker1 # [ 204.631201] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646599worker1 # [ 204.645236] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:543646600worker1 # [ 204.651859] nix-grpc-daemon[1828]: time=2026-09-24T11:50:53.265Z level=INFO msg="All 1 paths already cached"6601worker1 # [ 204.663904] nix-daemon[854]: reaped child process 2021, status = succeeded6602worker1 # [ 204.671416] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6603worker1 # [ 204.675702] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:543646604worker1 # [ 204.808157] nix-daemon[854]: accepted connection from pid 2046, user nix-grpc-daemon (trusted)6605worker1 # [ 204.828678] nix-daemon[854]: reaped child process 2049, status = succeeded6606lb # [ 204.810498] niks3-server[1164]: 2026/09/24 11:50:53 INFO Received push request method=POST path=/api/pushes6607worker1 # [ 204.840098] nix-grpc-daemon[1828]: time=2026-09-24T11:50:53.455Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6608worker1 # [ 204.841589] nix-grpc-daemon[1828]: time=2026-09-24T11:50:53.455Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)"6609lb # [ 204.839766] niks3-server[1164]: 2026/09/24 11:50:53 INFO Received sign narinfos request method=POST path=/api/pushes/79/sign6610worker1 # [ 204.866338] nix-grpc-daemon[1828]: time=2026-09-24T11:50:53.481Z level=INFO msg="Uploading 1 narinfos"6611lb # [ 204.852317] niks3-server[1164]: 2026/09/24 11:50:53 INFO Signed narinfos id=79 count=16612lb # [ 204.861352] niks3-server[1164]: 2026/09/24 11:50:53 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst6613lb # [ 204.873507] niks3-server[1164]: 2026/09/24 11:50:53 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls6614lb # [ 204.878357] niks3-server[1164]: 2026/09/24 11:50:53 INFO Received complete push request method=POST path=/api/pushes/79/complete6615worker1 # [ 204.904168] nix-grpc-daemon[1828]: time=2026-09-24T11:50:53.519Z level=INFO msg="Upload complete. (222ms)"6616lb # [ 204.884353] niks3-server[1164]: 2026/09/24 11:50:53 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo6617worker1 # [ 204.912303] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=0 paths=1 nar_bytes_in=6246618worker1 # [ 204.924905] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646619worker1 # [ 204.936666] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6620worker1 # [ 204.942180] nix-daemon[854]: reaped child process 2044, status = succeeded6621worker1 # [ 204.955397] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6622worker1 # [ 206.265492] nix-daemon[854]: reaped child process 2053, status = succeeded6623worker1 # [ 206.336320] nix-daemon[854]: accepted connection from pid 2069, user nix-grpc-daemon (trusted)6624worker1 # [ 206.358384] nix-daemon[854]: reaped child process 2072, status = succeeded6625lb # [ 206.342993] niks3-server[1164]: 2026/09/24 11:50:54 INFO Received push request method=POST path=/api/pushes6626worker1 # [ 206.376449] nix-grpc-daemon[1828]: time=2026-09-24T11:50:54.989Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6627worker1 # [ 206.378172] nix-grpc-daemon[1828]: time=2026-09-24T11:50:54.989Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)"6628lb # [ 206.393106] niks3-server[1164]: 2026/09/24 11:50:54 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls6629worker1 # [ 206.424907] nix-grpc-daemon[1828]: time=2026-09-24T11:50:55.040Z level=INFO msg="Uploading 1 narinfos"6630lb # [ 206.404810] niks3-server[1164]: 2026/09/24 11:50:55 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst6631lb # [ 206.415913] niks3-server[1164]: 2026/09/24 11:50:55 INFO Received sign narinfos request method=POST path=/api/pushes/80/sign6632lb # [ 206.426637] niks3-server[1164]: 2026/09/24 11:50:55 INFO Signed narinfos id=80 count=16633lb # [ 206.432921] niks3-server[1164]: 2026/09/24 11:50:55 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv6634worker1 # [ 206.453682] nix-grpc-daemon[1828]: time=2026-09-24T11:50:55.069Z level=INFO msg="Upload complete. (287ms)"6635lb # [ 206.438629] niks3-server[1164]: 2026/09/24 11:50:55 INFO Received complete push request method=POST path=/api/pushes/80/complete6636lb # [ 206.443669] niks3-server[1164]: 2026/09/24 11:50:55 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo6637worker2 # [ 206.539076] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:55Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=16638worker1 # [ 206.468631] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=16 outputs=16639worker2 # [ 206.548767] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:55Z level=debug event=want client=6 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=06640worker1 # [ 206.488331] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646641worker1 # [ 206.496658] nix-daemon[854]: reaped child process 2051, status = succeeded6642worker1 # [ 206.505936] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:543646643worker1 # [ 206.520579] nix-grpc-daemon[1828]: time=2026-09-24T11:50:55.136Z level=INFO msg="All 2 paths already cached"6644worker1 # [ 206.530170] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:543646645worker1 # [ 206.545588] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6646worker1 # [ 206.704239] nix-daemon[854]: accepted connection from pid 2076, user nix-grpc-daemon (trusted)6647lb # [ 206.706520] niks3-server[1164]: 2026/09/24 11:50:55 INFO Received push request method=POST path=/api/pushes6648worker1 # [ 206.728471] nix-daemon[854]: reaped child process 2079, status = succeeded6649worker1 # [ 206.733319] nix-grpc-daemon[1828]: time=2026-09-24T11:50:55.348Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6650worker1 # [ 206.734844] nix-grpc-daemon[1828]: time=2026-09-24T11:50:55.348Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)"6651lb # [ 206.725465] niks3-server[1164]: 2026/09/24 11:50:55 INFO Received sign narinfos request method=POST path=/api/pushes/81/sign6652worker1 # [ 206.749700] nix-grpc-daemon[1828]: time=2026-09-24T11:50:55.365Z level=INFO msg="Uploading 1 narinfos"6653lb # [ 206.731513] niks3-server[1164]: 2026/09/24 11:50:55 INFO Signed narinfos id=81 count=16654lb # [ 206.736974] niks3-server[1164]: 2026/09/24 11:50:55 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst6655lb # [ 206.739420] niks3-server[1164]: 2026/09/24 11:50:55 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls6656lb # [ 206.747271] niks3-server[1164]: 2026/09/24 11:50:55 INFO Received complete push request method=POST path=/api/pushes/81/complete6657worker1 # [ 206.775132] nix-grpc-daemon[1828]: time=2026-09-24T11:50:55.390Z level=INFO msg="Upload complete. (217ms)"6658lb # [ 206.755816] niks3-server[1164]: 2026/09/24 11:50:55 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo6659worker1 # [ 206.783766] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=0 paths=1 nar_bytes_in=7526660worker1 # [ 206.794294] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646661worker1 # [ 206.798823] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6662worker1 # [ 206.802708] nix-daemon[854]: reaped child process 2074, status = succeeded6663worker1 # [ 206.814447] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6664worker1 # [ 208.688771] nix-daemon[854]: reaped child process 2083, status = succeeded6665worker1 # [ 208.797215] nix-daemon[854]: accepted connection from pid 2099, user nix-grpc-daemon (trusted)6666worker1 # [ 208.819448] nix-daemon[854]: reaped child process 2102, status = succeeded6667lb # [ 208.803197] niks3-server[1164]: 2026/09/24 11:50:57 INFO Received push request method=POST path=/api/pushes6668worker1 # [ 208.834967] nix-grpc-daemon[1828]: time=2026-09-24T11:50:57.449Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6669worker1 # [ 208.836690] nix-grpc-daemon[1828]: time=2026-09-24T11:50:57.449Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)"6670worker1 # [ 208.882135] nix-grpc-daemon[1828]: time=2026-09-24T11:50:57.497Z level=INFO msg="Uploading 1 narinfos"6671lb # [ 208.849756] niks3-server[1164]: 2026/09/24 11:50:57 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls6672lb # [ 208.867368] niks3-server[1164]: 2026/09/24 11:50:57 INFO Received sign narinfos request method=POST path=/api/pushes/82/sign6673lb # [ 208.876325] niks3-server[1164]: 2026/09/24 11:50:57 INFO Signed narinfos id=82 count=16674lb # [ 208.886188] niks3-server[1164]: 2026/09/24 11:50:57 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv6675worker1 # [ 208.915564] nix-grpc-daemon[1828]: time=2026-09-24T11:50:57.531Z level=INFO msg="Upload complete. (291ms)"6676lb # [ 208.896595] niks3-server[1164]: 2026/09/24 11:50:57 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst6677lb # [ 208.901041] niks3-server[1164]: 2026/09/24 11:50:57 INFO Received complete push request method=POST path=/api/pushes/82/complete6678lb # [ 208.906017] niks3-server[1164]: 2026/09/24 11:50:57 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo6679worker2 # [ 208.999158] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:57Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=16680worker1 # [ 208.929266] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=2 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=17 outputs=16681worker1 # [ 208.952474] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646682worker1 # [ 208.959481] nix-daemon[854]: reaped child process 2081, status = succeeded6683worker1 # [ 208.964291] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:543646684worker1 # [ 208.973218] nix-grpc-daemon[1828]: time=2026-09-24T11:50:57.588Z level=INFO msg="All 2 paths already cached"6685worker1 # [ 208.981752] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:543646686worker1 # [ 208.994694] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6687worker1 # [ 209.133296] nix-daemon[854]: accepted connection from pid 2106, user nix-grpc-daemon (trusted)6688worker1 # [ 209.153583] nix-daemon[854]: reaped child process 2109, status = succeeded6689lb # [ 209.138363] niks3-server[1164]: 2026/09/24 11:50:57 INFO Received push request method=POST path=/api/pushes6690worker1 # [ 209.168833] nix-grpc-daemon[1828]: time=2026-09-24T11:50:57.784Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6691worker1 # [ 209.170332] nix-grpc-daemon[1828]: time=2026-09-24T11:50:57.784Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)"6692worker1 # [ 209.189121] nix-grpc-daemon[1828]: time=2026-09-24T11:50:57.804Z level=INFO msg="Uploading 1 narinfos"6693lb # [ 209.168695] niks3-server[1164]: 2026/09/24 11:50:57 INFO Received sign narinfos request method=POST path=/api/pushes/83/sign6694lb # [ 209.180105] niks3-server[1164]: 2026/09/24 11:50:57 INFO Signed narinfos id=83 count=16695lb # [ 209.189338] niks3-server[1164]: 2026/09/24 11:50:57 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst6696lb # [ 209.206306] niks3-server[1164]: 2026/09/24 11:50:57 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls6697lb # [ 209.211816] niks3-server[1164]: 2026/09/24 11:50:57 INFO Received complete push request method=POST path=/api/pushes/83/complete6698worker1 # [ 209.238808] nix-grpc-daemon[1828]: time=2026-09-24T11:50:57.854Z level=INFO msg="Upload complete. (232ms)"6699lb # [ 209.219777] niks3-server[1164]: 2026/09/24 11:50:57 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo6700worker1 # [ 209.249029] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=0 paths=1 nar_bytes_in=7526701worker1 # [ 209.264378] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646702worker1 # [ 209.270588] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6703worker1 # [ 209.275409] nix-daemon[854]: reaped child process 2104, status = succeeded6704worker1 # [ 209.286287] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6705worker1 # [ 210.334829] nix-daemon[854]: reaped child process 2113, status = succeeded6706worker1 # [ 210.435839] nix-daemon[854]: accepted connection from pid 2129, user nix-grpc-daemon (trusted)6707worker1 # [ 210.457166] nix-daemon[854]: reaped child process 2132, status = succeeded6708lb # [ 210.441784] niks3-server[1164]: 2026/09/24 11:50:59 INFO Received push request method=POST path=/api/pushes6709worker1 # [ 210.474115] nix-grpc-daemon[1828]: time=2026-09-24T11:50:59.087Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6710worker1 # [ 210.475616] nix-grpc-daemon[1828]: time=2026-09-24T11:50:59.087Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)"6711lb # [ 210.485136] niks3-server[1164]: 2026/09/24 11:50:59 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls6712worker1 # [ 210.517534] nix-grpc-daemon[1828]: time=2026-09-24T11:50:59.132Z level=INFO msg="Uploading 1 narinfos"6713lb # [ 210.498313] niks3-server[1164]: 2026/09/24 11:50:59 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv6714lb # [ 210.509011] niks3-server[1164]: 2026/09/24 11:50:59 INFO Received sign narinfos request method=POST path=/api/pushes/84/sign6715lb # [ 210.515568] niks3-server[1164]: 2026/09/24 11:50:59 INFO Signed narinfos id=84 count=16716worker1 # [ 210.546453] nix-grpc-daemon[1828]: time=2026-09-24T11:50:59.161Z level=INFO msg="Upload complete. (248ms)"6717lb # [ 210.526632] niks3-server[1164]: 2026/09/24 11:50:59 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst6718lb # [ 210.537507] niks3-server[1164]: 2026/09/24 11:50:59 INFO Received complete push request method=POST path=/api/pushes/84/complete6719worker2 # [ 210.627655] nix-grpc-daemon[1638]: ts=2026-09-24T11:50:59Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=16720worker1 # [ 210.559036] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=18 outputs=16721lb # [ 210.542595] niks3-server[1164]: 2026/09/24 11:50:59 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo6722worker1 # [ 210.578285] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646723worker1 # [ 210.584150] nix-daemon[854]: reaped child process 2111, status = succeeded6724worker1 # [ 210.591609] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:543646725worker1 # [ 210.603634] nix-grpc-daemon[1828]: time=2026-09-24T11:50:59.219Z level=INFO msg="All 2 paths already cached"6726worker1 # [ 210.612631] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:543646727worker1 # [ 210.625541] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6728worker1 # [ 210.767300] nix-daemon[854]: accepted connection from pid 2136, user nix-grpc-daemon (trusted)6729worker1 # [ 210.789193] nix-daemon[854]: reaped child process 2139, status = succeeded6730lb # [ 210.766163] niks3-server[1164]: 2026/09/24 11:50:59 INFO Received push request method=POST path=/api/pushes6731worker1 # [ 210.795259] nix-grpc-daemon[1828]: time=2026-09-24T11:50:59.410Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6732worker1 # [ 210.796948] nix-grpc-daemon[1828]: time=2026-09-24T11:50:59.410Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)"6733lb # [ 210.799536] niks3-server[1164]: 2026/09/24 11:50:59 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls6734worker1 # [ 210.830174] nix-grpc-daemon[1828]: time=2026-09-24T11:50:59.445Z level=INFO msg="Uploading 1 narinfos"6735lb # [ 210.813056] niks3-server[1164]: 2026/09/24 11:50:59 INFO Received sign narinfos request method=POST path=/api/pushes/85/sign6736lb # [ 210.821999] niks3-server[1164]: 2026/09/24 11:50:59 INFO Signed narinfos id=85 count=16737lb # [ 210.830489] niks3-server[1164]: 2026/09/24 11:50:59 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst6738worker1 # [ 210.857691] nix-grpc-daemon[1828]: time=2026-09-24T11:50:59.473Z level=INFO msg="Upload complete. (221ms)"6739lb # [ 210.837634] niks3-server[1164]: 2026/09/24 11:50:59 INFO Received complete push request method=POST path=/api/pushes/85/complete6740lb # [ 210.842174] niks3-server[1164]: 2026/09/24 11:50:59 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo6741worker1 # [ 210.866369] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=0 paths=1 nar_bytes_in=7526742worker1 # [ 210.879403] nix-grpc-daemon[1768]: ts=2026-09-24T11:50:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:543646743worker1 # [ 210.886330] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6744worker1 # [ 210.892342] nix-daemon[854]: reaped child process 2134, status = succeeded6745worker1 # [ 210.905470] nix-daemon[854]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6746worker1 # [ 211.534562] nix-daemon[854]: reaped child process 2143, status = succeeded6747worker1 # [ 211.640366] nix-daemon[854]: accepted connection from pid 2159, user nix-grpc-daemon (trusted)6748worker1 # [ 211.662032] nix-daemon[854]: reaped child process 2162, status = succeeded6749lb # [ 211.643842] niks3-server[1164]: 2026/09/24 11:51:00 INFO Received push request method=POST path=/api/pushes6750worker1 # [ 211.676403] nix-grpc-daemon[1828]: time=2026-09-24T11:51:00.290Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6751worker1 # [ 211.678201] nix-grpc-daemon[1828]: time=2026-09-24T11:51:00.290Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)"6752lb # [ 211.692485] niks3-server[1164]: 2026/09/24 11:51:00 INFO Received sign narinfos request method=POST path=/api/pushes/86/sign6753worker1 # [ 211.723465] nix-grpc-daemon[1828]: time=2026-09-24T11:51:00.338Z level=INFO msg="Uploading 1 narinfos"6754lb # [ 211.705265] niks3-server[1164]: 2026/09/24 11:51:00 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv6755lb # [ 211.714466] niks3-server[1164]: 2026/09/24 11:51:00 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst6756lb # [ 211.725936] niks3-server[1164]: 2026/09/24 11:51:00 INFO Signed narinfos id=86 count=16757lb # [ 211.727699] niks3-server[1164]: 2026/09/24 11:51:00 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls6758lb # [ 211.731962] niks3-server[1164]: 2026/09/24 11:51:00 INFO Received complete push request method=POST path=/api/pushes/86/complete6759lb # [ 211.735880] niks3-server[1164]: 2026/09/24 11:51:00 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo6760worker1 # [ 211.755919] nix-grpc-daemon[1828]: time=2026-09-24T11:51:00.371Z level=INFO msg="Upload complete. (242ms)"6761worker1 # [ 211.768884] nix-grpc-daemon[1768]: ts=2026-09-24T11:51:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54364 duration_s=0 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=19 outputs=16762worker1 # [ 211.786204] nix-daemon[854]: reaped child process 2141, status = succeeded6763client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 17.04 seconds)6764worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6765worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6766client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]'6767client # [ 211.950582] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6768client # Running as unit: gpu.service; invocation ID: 806c8770f99a42759f593e28cfc033e66769client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds)6770lb: must succeed: curl -sf localhost:9901/clusters6771lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6772lb: must succeed: curl -sf localhost:9901/clusters6773lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6774worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6775worker1 # [ 212.177384] nix-grpc-daemon[1768]: ts=2026-09-24T11:51:00Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:543646776client # [ 212.163035] nix[1604]: don't know how to build these paths:6777client # [ 212.164376] nix[1604]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv6778worker2 # [ 212.262768] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:379046779worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6780worker2 # [ 212.286313] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:00Z level=debug event=want client=7 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=06781client # [ 212.201146] nix[1604]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip}, waiting up to 30s for a worker6782lb: must succeed: curl -sf localhost:9901/clusters6783lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6784lb: must succeed: curl -sf localhost:9901/clusters6785lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6786worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6787worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6788client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6789lb # [ 219.052618] envoy[907]: conn peer=[2001:db8:1::1]:43146 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27222 tx=12373 ms=167586790client # [ 242.207791] nix[1604]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip}6791client # [ 242.220825] nix[1604]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed6792client # [ 242.262631] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6793client # [ 242.270777] systemd[1]: gpu.service: Failed with result 'exit-code'.6794client # [ 242.274448] systemd[1]: gpu.service: Consumed 210ms CPU time over 30.317s wall clock time, 37.4M memory peak, 5.6K incoming IP traffic, 5.2K outgoing IP traffic.6795client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.17 seconds)6796client: must succeed: journalctl -u gpu -o cat6797client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)6798lb: must succeed: curl -sf localhost:9901/clusters6799lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6800lb: must succeed: curl -sf localhost:9901/clusters6801lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6802worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6803worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6804(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 47.97 seconds)6805subtest: low disk drains a worker and builds go to the other6806worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6807worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.22 seconds)6808worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6809worker1 # [ 244.782244] nix-grpc-daemon[1768]: ts=2026-09-24T11:51:33Z level=info event=unhealthy reason=min_free6810worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.10 seconds)6811client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&26812client # don't know how to build these paths:6813client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6814worker2 # [ 245.384257] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:34Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:379046815worker2 # [ 245.391137] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:379046816worker2 # [ 245.417804] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:34Z level=debug event=want client=8 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=06817worker2 # [ 245.425023] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:34Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=16818worker2 # [ 245.434200] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379046819worker2 # [ 245.445405] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:379046820client # copying 1 paths to 'lb:50051'...6821client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'...6822worker2 # [ 245.461311] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:379046823worker2 # [ 245.477034] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6824worker2 # [ 245.626898] nix-daemon[841]: accepted connection from pid 2048, user nix-grpc-daemon (trusted)6825worker2 # [ 245.650065] nix-daemon[841]: reaped child process 2051, status = succeeded6826lb # [ 245.558756] niks3-server[1164]: 2026/09/24 11:51:34 INFO Received push request method=POST path=/api/pushes6827worker2 # [ 245.664110] nix-grpc-daemon[1714]: time=2026-09-24T11:51:34.282Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6828worker2 # [ 245.665729] nix-grpc-daemon[1714]: time=2026-09-24T11:51:34.282Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)"6829worker2 # [ 245.708876] nix-grpc-daemon[1714]: time=2026-09-24T11:51:34.327Z level=INFO msg="Uploading 1 narinfos"6830lb # [ 245.610435] niks3-server[1164]: 2026/09/24 11:51:34 INFO Received sign narinfos request method=POST path=/api/pushes/87/sign6831lb # [ 245.621785] niks3-server[1164]: 2026/09/24 11:51:34 INFO Signed narinfos id=87 count=16832lb # [ 245.637204] niks3-server[1164]: 2026/09/24 11:51:34 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst6833lb # [ 245.644249] niks3-server[1164]: 2026/09/24 11:51:34 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls6834lb # [ 245.652614] niks3-server[1164]: 2026/09/24 11:51:34 INFO Received complete push request method=POST path=/api/pushes/87/complete6835lb # [ 245.661307] niks3-server[1164]: 2026/09/24 11:51:34 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo6836worker2 # [ 245.752523] nix-grpc-daemon[1714]: time=2026-09-24T11:51:34.371Z level=INFO msg="Upload complete. (259ms)"6837worker2 # [ 245.764533] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=0 paths=1 nar_bytes_in=5126838worker2 # [ 245.783686] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379046839worker2 # [ 245.793693] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6840worker2 # [ 245.798836] nix-daemon[841]: reaped child process 2046, status = succeeded6841client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051...6842worker2 # [ 245.811863] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6843client # farm-a> worker2: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6844client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6845client # farm-a> LOG-farm-a-drain6846worker2 # [ 247.086832] nix-daemon[841]: reaped child process 2055, status = succeeded6847worker2 # [ 247.212911] nix-daemon[841]: accepted connection from pid 2071, user nix-grpc-daemon (trusted)6848worker2 # [ 247.234090] nix-daemon[841]: reaped child process 2074, status = succeeded6849lb # [ 247.150429] niks3-server[1164]: 2026/09/24 11:51:35 INFO Received push request method=POST path=/api/pushes6850worker2 # [ 247.252472] nix-grpc-daemon[1714]: time=2026-09-24T11:51:35.870Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6851worker2 # [ 247.254239] nix-grpc-daemon[1714]: time=2026-09-24T11:51:35.870Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)"6852lb # [ 247.191798] niks3-server[1164]: 2026/09/24 11:51:35 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6853worker2 # [ 247.297897] nix-grpc-daemon[1714]: time=2026-09-24T11:51:35.916Z level=INFO msg="Uploading 1 narinfos"6854lb # [ 247.207202] niks3-server[1164]: 2026/09/24 11:51:35 INFO Received sign narinfos request method=POST path=/api/pushes/88/sign6855lb # [ 247.216985] niks3-server[1164]: 2026/09/24 11:51:35 INFO Signed narinfos id=88 count=16856lb # [ 247.222317] niks3-server[1164]: 2026/09/24 11:51:35 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls6857lb # [ 247.232492] niks3-server[1164]: 2026/09/24 11:51:35 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6858lb # [ 247.235485] niks3-server[1164]: 2026/09/24 11:51:35 INFO Received complete push request method=POST path=/api/pushes/88/complete6859worker2 # [ 247.333810] nix-grpc-daemon[1714]: time=2026-09-24T11:51:35.952Z level=INFO msg="Upload complete. (304ms)"6860lb # [ 247.243321] niks3-server[1164]: 2026/09/24 11:51:35 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo6861worker2 # [ 247.351538] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=20 outputs=16862worker2 # [ 247.372364] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:35Z level=debug event=want client=8 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=06863worker2 # [ 247.379857] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:35Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=16864worker2 # [ 247.387284] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379046865worker2 # [ 247.394028] nix-daemon[841]: reaped child process 2053, status = succeeded6866worker2 # [ 247.398185] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:379046867client # copying 1 paths to 'lb:50051'...6868worker2 # [ 247.404190] nix-grpc-daemon[1714]: time=2026-09-24T11:51:36.022Z level=INFO msg="All 1 paths already cached"6869client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'...6870worker2 # [ 247.410202] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:379046871worker2 # [ 247.421582] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6872worker2 # [ 247.563885] nix-daemon[841]: accepted connection from pid 2080, user nix-grpc-daemon (trusted)6873worker2 # [ 247.588237] nix-daemon[841]: reaped child process 2083, status = succeeded6874lb # [ 247.502488] niks3-server[1164]: 2026/09/24 11:51:36 INFO Received push request method=POST path=/api/pushes6875worker2 # [ 247.603802] nix-grpc-daemon[1714]: time=2026-09-24T11:51:36.222Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6876worker2 # [ 247.605510] nix-grpc-daemon[1714]: time=2026-09-24T11:51:36.222Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)"6877worker2 # [ 247.626837] nix-grpc-daemon[1714]: time=2026-09-24T11:51:36.245Z level=INFO msg="Uploading 1 narinfos"6878lb # [ 247.535066] niks3-server[1164]: 2026/09/24 11:51:36 INFO Received sign narinfos request method=POST path=/api/pushes/89/sign6879lb # [ 247.549639] niks3-server[1164]: 2026/09/24 11:51:36 INFO Signed narinfos id=89 count=16880lb # [ 247.557577] niks3-server[1164]: 2026/09/24 11:51:36 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst6881worker2 # [ 247.663944] nix-grpc-daemon[1714]: time=2026-09-24T11:51:36.283Z level=INFO msg="Upload complete. (229ms)"6882lb # [ 247.572963] niks3-server[1164]: 2026/09/24 11:51:36 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls6883lb # [ 247.580342] niks3-server[1164]: 2026/09/24 11:51:36 INFO Received complete push request method=POST path=/api/pushes/89/complete6884worker2 # [ 247.672255] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=0 paths=1 nar_bytes_in=6326885lb # [ 247.583658] niks3-server[1164]: 2026/09/24 11:51:36 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo6886worker2 # [ 247.686787] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379046887worker2 # [ 247.693098] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6888worker2 # [ 247.697510] nix-daemon[841]: reaped child process 2078, status = succeeded6889client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051...6890worker2 # [ 247.710519] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6891client # farm-b> worker2: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6892client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6893client # farm-b> LOG-farm-b-drain6894worker2 # [ 248.800507] nix-daemon[841]: reaped child process 2087, status = succeeded6895worker2 # [ 248.895136] nix-daemon[841]: accepted connection from pid 2103, user nix-grpc-daemon (trusted)6896worker2 # [ 248.911956] nix-daemon[841]: reaped child process 2106, status = succeeded6897lb # [ 248.829927] niks3-server[1164]: 2026/09/24 11:51:37 INFO Received push request method=POST path=/api/pushes6898worker2 # [ 248.932470] nix-grpc-daemon[1714]: time=2026-09-24T11:51:37.550Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6899worker2 # [ 248.934299] nix-grpc-daemon[1714]: time=2026-09-24T11:51:37.550Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)"6900lb # [ 248.863500] niks3-server[1164]: 2026/09/24 11:51:37 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst6901worker2 # [ 248.975412] nix-grpc-daemon[1714]: time=2026-09-24T11:51:37.594Z level=INFO msg="Uploading 1 narinfos"6902lb # [ 248.883669] niks3-server[1164]: 2026/09/24 11:51:37 INFO Received sign narinfos request method=POST path=/api/pushes/90/sign6903lb # [ 248.895002] niks3-server[1164]: 2026/09/24 11:51:37 INFO Signed narinfos id=90 count=16904lb # [ 248.901252] niks3-server[1164]: 2026/09/24 11:51:37 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls6905lb # [ 248.910091] niks3-server[1164]: 2026/09/24 11:51:37 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6906lb # [ 248.924302] niks3-server[1164]: 2026/09/24 11:51:37 INFO Received complete push request method=POST path=/api/pushes/90/complete6907lb # [ 248.927828] niks3-server[1164]: 2026/09/24 11:51:37 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo6908worker2 # [ 249.024955] nix-grpc-daemon[1714]: time=2026-09-24T11:51:37.643Z level=INFO msg="Upload complete. (304ms)"6909worker2 # [ 249.045686] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=21 outputs=16910worker2 # [ 249.060644] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:37Z level=debug event=want client=8 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=06911worker2 # [ 249.071477] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:37Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=16912worker2 # [ 249.078030] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379046913worker2 # [ 249.084473] nix-daemon[841]: reaped child process 2085, status = succeeded6914worker2 # [ 249.088346] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:379046915client # copying 1 paths to 'lb:50051'...6916worker2 # [ 249.094321] nix-grpc-daemon[1714]: time=2026-09-24T11:51:37.711Z level=INFO msg="All 2 paths already cached"6917client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'...6918worker2 # [ 249.099663] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:379046919worker2 # [ 249.116294] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6920lb # [ 249.053920] envoy[907]: conn peer=[2001:db8:1::1]:45808 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=931 tx=613 ms=300826921worker2 # [ 249.252513] nix-daemon[841]: accepted connection from pid 2112, user nix-grpc-daemon (trusted)6922worker2 # [ 249.273567] nix-daemon[841]: reaped child process 2115, status = succeeded6923lb # [ 249.180734] niks3-server[1164]: 2026/09/24 11:51:37 INFO Received push request method=POST path=/api/pushes6924worker2 # [ 249.280830] nix-grpc-daemon[1714]: time=2026-09-24T11:51:37.899Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6925worker2 # [ 249.282345] nix-grpc-daemon[1714]: time=2026-09-24T11:51:37.899Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)"6926lb # [ 249.212076] niks3-server[1164]: 2026/09/24 11:51:37 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst6927worker2 # [ 249.313212] nix-grpc-daemon[1714]: time=2026-09-24T11:51:37.932Z level=INFO msg="Uploading 1 narinfos"6928lb # [ 249.225793] niks3-server[1164]: 2026/09/24 11:51:37 INFO Received sign narinfos request method=POST path=/api/pushes/91/sign6929lb # [ 249.235983] niks3-server[1164]: 2026/09/24 11:51:37 INFO Signed narinfos id=91 count=16930lb # [ 249.239540] niks3-server[1164]: 2026/09/24 11:51:37 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls6931lb # [ 249.250673] niks3-server[1164]: 2026/09/24 11:51:37 INFO Received complete push request method=POST path=/api/pushes/91/complete6932worker2 # [ 249.352977] nix-grpc-daemon[1714]: time=2026-09-24T11:51:37.972Z level=INFO msg="Upload complete. (229ms)"6933lb # [ 249.262406] niks3-server[1164]: 2026/09/24 11:51:37 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo6934worker2 # [ 249.360398] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=0 paths=1 nar_bytes_in=7606935worker2 # [ 249.375184] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:379046936worker2 # [ 249.384193] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6937worker2 # [ 249.387939] nix-daemon[841]: reaped child process 2110, status = succeeded6938client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051...6939worker2 # [ 249.401261] nix-daemon[841]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6940client # farm-top> worker2: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6941client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6942client # farm-top> LOG-farm-top-drain6943worker2 # [ 250.676832] nix-daemon[841]: reaped child process 2119, status = succeeded6944worker2 # [ 250.789533] nix-daemon[841]: accepted connection from pid 2135, user nix-grpc-daemon (trusted)6945worker2 # [ 250.808355] nix-daemon[841]: reaped child process 2138, status = succeeded6946lb # [ 250.723111] niks3-server[1164]: 2026/09/24 11:51:39 INFO Received push request method=POST path=/api/pushes6947worker2 # [ 250.826012] nix-grpc-daemon[1714]: time=2026-09-24T11:51:39.442Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6948worker2 # [ 250.827768] nix-grpc-daemon[1714]: time=2026-09-24T11:51:39.442Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)"6949worker2 # [ 250.861588] nix-grpc-daemon[1714]: time=2026-09-24T11:51:39.480Z level=INFO msg="Uploading 1 narinfos"6950lb # [ 250.762314] niks3-server[1164]: 2026/09/24 11:51:39 INFO Received sign narinfos request method=POST path=/api/pushes/92/sign6951lb # [ 250.777707] niks3-server[1164]: 2026/09/24 11:51:39 INFO Signed narinfos id=92 count=16952lb # [ 250.782925] niks3-server[1164]: 2026/09/24 11:51:39 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst6953lb # [ 250.794402] niks3-server[1164]: 2026/09/24 11:51:39 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls6954lb # [ 250.804679] niks3-server[1164]: 2026/09/24 11:51:39 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6955lb # [ 250.807912] niks3-server[1164]: 2026/09/24 11:51:39 INFO Received complete push request method=POST path=/api/pushes/92/complete6956worker2 # [ 250.902876] nix-grpc-daemon[1714]: time=2026-09-24T11:51:39.521Z level=INFO msg="Upload complete. (286ms)"6957lb # [ 250.814637] niks3-server[1164]: 2026/09/24 11:51:39 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo6958worker2 # [ 250.917180] nix-grpc-daemon[1638]: ts=2026-09-24T11:51:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37904 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=22 outputs=16959worker2 # [ 250.935193] nix-daemon[841]: reaped child process 2117, status = succeeded6960client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2, in 5.79 seconds)6961client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath6962client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath, in 0.18 seconds)6963worker1: must succeed: rm /nix/.rw-store/fill6964worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.16 seconds)6965worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'6966worker1 # [ 251.799622] nix-grpc-daemon[1768]: ts=2026-09-24T11:51:40Z level=info event=healthy reason=min_free6967worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.14 seconds)6968(finished: subtest: low disk drains a worker and builds go to the other, in 9.66 seconds)6969(finished: run the VM test script, in 253.36 seconds)6970test script finished in 253.67s6971cleanup6972kill QemuMachine (pid 49)6973client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6974client # [2026-09-24T11:51:41Z INFO virtiofsd] Client disconnected, shutting down6975client # [2026-09-24T11:51:41Z INFO virtiofsd] Client disconnected, shutting down6976client # [2026-09-24T11:51:41Z INFO virtiofsd] Client disconnected, shutting down6977kill QemuMachine (pid 50)6978lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6979lb # [2026-09-24T11:51:42Z INFO virtiofsd] Client disconnected, shutting down6980lb # [2026-09-24T11:51:42Z INFO virtiofsd] Client disconnected, shutting down6981lb # [2026-09-24T11:51:42Z INFO virtiofsd] Client disconnected, shutting down6982kill QemuMachine (pid 51)6983worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6984worker1 # [2026-09-24T11:51:42Z INFO virtiofsd] Client disconnected, shutting down6985worker1 # [2026-09-24T11:51:42Z INFO virtiofsd] Client disconnected, shutting down6986worker1 # [2026-09-24T11:51:42Z INFO virtiofsd] Client disconnected, shutting down6987kill QemuMachine (pid 52)6988worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6989worker2 # [2026-09-24T11:51:42Z INFO virtiofsd] Client disconnected, shutting down6990worker2 # [2026-09-24T11:51:42Z INFO virtiofsd] Client disconnected, shutting down6991worker2 # [2026-09-24T11:51:42Z INFO virtiofsd] Client disconnected, shutting down6992(finished: cleanup, in 1.47 seconds)6993additionally exposed symbols:6994 client, lb, worker1, worker2,6995 vlan1,6996 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh