vm-test-run-nix-grpc-farm
checks.aarch64-linux.farm
· build #547
· raw
1tribuchet: building on eliza2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12lb # Disk image does not exist, creating the virtualisation disk image...13worker1 # Disk image does not exist, creating the virtualisation disk image...14worker1: QEMU running (pid 51)15lb: QEMU running (pid 50)16client # Disk image does not exist, creating the virtualisation disk image...17client: QEMU running (pid 49)18lb # Formatting '/build/vm-state-lb/tmp.1GSJkwPrrR', fmt=raw size=107374182419client # Formatting '/build/vm-state-client/tmp.fiobgkBmEO', fmt=raw size=107374182420worker2: QEMU running (pid 52)21lb # mke2fs 1.47.4 (6-Mar-2025)22client # mke2fs 1.47.4 (6-Mar-2025)23worker1 # Formatting '/build/vm-state-worker1/tmp.oZ3eC9pqCe', fmt=raw size=107374182424worker2 # Disk image does not exist, creating the virtualisation disk image...25(finished: start all VMs, in 0.51 seconds)26worker1 # mke2fs 1.47.4 (6-Mar-2025)27worker2 # Formatting '/build/vm-state-worker2/tmp.VozQ3rtJI1', fmt=raw size=107374182428client: waiting for unit network-addresses-eth1.service29worker1 # Discarding device blocks: 0/262144 done30worker2 # mke2fs 1.47.4 (6-Mar-2025)31client: waiting for the VM to finish booting32worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes33worker2 # Discarding device blocks: 0/262144 done34client # Discarding device blocks: 0/262144 done35lb # Discarding device blocks: 0/262144 done36worker1 # Filesystem UUID: 63ca42a6-0a78-4a40-aed9-62f78a28294737worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes38client # Creating filesystem with 262144 4k blocks and 65536 inodes39lb # Creating filesystem with 262144 4k blocks and 65536 inodes40worker1 # Superblock backups stored on blocks:41worker2 # Filesystem UUID: 9e950184-7788-4e15-b451-aa57d7ad5f0042client # Filesystem UUID: 39e36b9e-d9a6-41e9-be3e-c0c8076db10543lb # Filesystem UUID: 0369716f-fabe-48ec-a9df-ad7d213a749844worker1 # 32768, 98304, 163840, 22937645worker2 # Superblock backups stored on blocks:46client # Superblock backups stored on blocks:47lb # Superblock backups stored on blocks:48worker1 # 49worker2 # 32768, 98304, 163840, 22937650client # 32768, 98304, 163840, 22937651lb # 32768, 98304, 163840, 22937652worker1 # Allocating group tables: 0/8 done53worker2 # 54client # 55lb # 56worker1 # Writing inode tables: 0/8 done57worker2 # Allocating group tables: 0/8 done58client # Allocating group tables: 0/8 done59lb # Allocating group tables: 0/8 done60worker1 # Creating journal (8192 blocks): done61worker2 # Writing inode tables: 0/8 done62client # Writing inode tables: 0/8 done63lb # Writing inode tables: 0/8 done64worker1 # Writing superblocks and filesystem accounting information: 0/8 done65worker2 # Creating journal (8192 blocks): done66client # Creating journal (8192 blocks): done67lb # Creating journal (8192 blocks): done68worker1 # 69worker2 # Writing superblocks and filesystem accounting information: 0/8 done70client # Writing superblocks and filesystem accounting information: 0/8 done71lb # Writing superblocks and filesystem accounting information: 0/8 done72worker1 # Virtualisation disk image created.73worker2 # 74client # 75lb # 76worker1 # Starting virtiofs daemons...77worker2 # Virtualisation disk image created.78client # Virtualisation disk image created.79lb # Virtualisation disk image created.80worker1 # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)81worker2 # Starting virtiofs daemons...82client # Starting virtiofs daemons...83lb # Starting virtiofs daemons...84worker1 # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether85worker2 # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)86client # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)87lb # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)88worker1 # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)89worker2 # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether90client # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether91lb # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether92worker1 # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether93worker2 # [2026-09-23T15:59:28Z INFO virtiofsd] Waiting for vhost-user socket connection...94client # [2026-09-23T15:59:28Z INFO virtiofsd] Waiting for vhost-user socket connection...95lb # [2026-09-23T15:59:28Z INFO virtiofsd] Waiting for vhost-user socket connection...96worker1 # [2026-09-23T15:59:28Z INFO virtiofsd] Waiting for vhost-user socket connection...97worker2 # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)98client # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)99lb # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)100worker1 # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)101worker2 # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether102client # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether103lb # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether104worker1 # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether105worker2 # [2026-09-23T15:59:28Z INFO virtiofsd] Waiting for vhost-user socket connection...106client # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)107lb # [2026-09-23T15:59:28Z INFO virtiofsd] Waiting for vhost-user socket connection...108worker1 # [2026-09-23T15:59:28Z INFO virtiofsd] Waiting for vhost-user socket connection...109worker2 # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)110client # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether111lb # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)112worker1 # [2026-09-23T15:59:28Z INFO virtiofsd] Waiting for vhost-user socket connection...113worker2 # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether114client # [2026-09-23T15:59:28Z INFO virtiofsd] Waiting for vhost-user socket connection...115lb # [2026-09-23T15:59:28Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether116worker1 # [2026-09-23T15:59:28Z INFO virtiofsd] Client connected, servicing requests117worker2 # [2026-09-23T15:59:28Z INFO virtiofsd] Waiting for vhost-user socket connection...118client # [2026-09-23T15:59:28Z INFO virtiofsd] Waiting for vhost-user socket connection...119lb # [2026-09-23T15:59:28Z INFO virtiofsd] Waiting for vhost-user socket connection...120worker1 # [2026-09-23T15:59:28Z INFO virtiofsd] Client connected, servicing requests121worker2 # [2026-09-23T15:59:28Z INFO virtiofsd] Client connected, servicing requests122client # [2026-09-23T15:59:28Z INFO virtiofsd] Client connected, servicing requests123lb # [2026-09-23T15:59:28Z INFO virtiofsd] Client connected, servicing requests124worker1 # [2026-09-23T15:59:28Z INFO virtiofsd] Client connected, servicing requests125worker2 # [2026-09-23T15:59:28Z INFO virtiofsd] Client connected, servicing requests126client # [2026-09-23T15:59:28Z INFO virtiofsd] Client connected, servicing requests127worker2 # [2026-09-23T15:59:28Z INFO virtiofsd] Client connected, servicing requests128client # [2026-09-23T15:59:28Z INFO virtiofsd] Client connected, servicing requests129lb # [2026-09-23T15:59:28Z INFO virtiofsd] Client connected, servicing requests130lb # [2026-09-23T15:59:28Z INFO virtiofsd] Client connected, servicing requests131worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]132worker1 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026133worker1 # [ 0.000000] KASLR enabled134worker1 # [ 0.000000] random: crng init done135worker1 # [ 0.000000] Machine model: linux,dummy-virt136worker1 # [ 0.000000] efi: UEFI not found.137worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT138worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]139worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]140worker1 # [ 0.000000] Zone ranges:141worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]142worker1 # [ 0.000000] DMA32 empty143worker1 # [ 0.000000] Normal empty144worker1 # [ 0.000000] Device empty145worker1 # [ 0.000000] Movable zone start for each node146worker1 # [ 0.000000] Early memory node ranges147worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]148worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]149worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000150worker1 # [ 0.000000] psci: probing for conduit method from DT.151worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware.152worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs153worker1 # [ 0.000000] psci: Trusted OS migration not required154worker1 # [ 0.000000] psci: SMC Calling Convention v1.1155worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)156worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296157worker1 # [ 0.000000] Detected PIPT I-cache on CPU0158worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)159worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface160worker1 # [ 0.000000] CPU features: detected: Spectre-v4161worker1 # [ 0.000000] CPU features: detected: Spectre-BHB162worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38163worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23164worker1 # [ 0.000000] alternatives: applying boot alternatives165worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/8ywmcs4wqfsykii45m0wrnjpayzliy1j-nixos-system-worker1-test/init regInfo=/nix/store/2xnyxk9qqas7nahgj7fiqv6zy78vllkd-closure-info/registration console=ttyAMA0,115200n8 console=tty0166worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/2xnyxk9qqas7nahgj7fiqv6zy78vllkd-closure-info/registration", will be passed to user space.167worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes168worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)169worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)170worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB171worker1 # [ 0.000000] software IO TLB: area num 1.172worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB173worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)174worker1 # [ 0.000000] Fallback order for Node 0: 0175worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216176worker1 # [ 0.000000] Policy zone: DMA177worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off178worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1179worker1 # [ 0.000000] allocated 3145728 bytes of page_ext180worker1 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages181worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups182worker1 # [ 0.000000] rcu: Hierarchical RCU implementation.183worker1 # [ 0.000000] rcu: RCU event tracing is enabled.184worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.185worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled.186worker1 # [ 0.000000] Rude variant of Tasks RCU enabled.187worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled.188worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.189worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1190worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.191worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.192worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.193worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0194worker1 # [ 0.000000] GICv3: 256 SPIs implemented195worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented196worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq197worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI198worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0199worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000200worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]201worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)202worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)203worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000204worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000205worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.206worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns207worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).208worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns209worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns210worker1 # [ 0.000034] arm-pv: using stolen time PV211worker1 # [ 0.000650] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)212worker1 # [ 0.000862] Console: colour dummy device 80x25213worker1 # [ 0.000870] printk: legacy console [tty0] enabled214worker1 # [ 0.001060] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)215worker1 # [ 0.001066] pid_max: default: 32768 minimum: 301216worker1 # [ 0.001144] LSM: initializing lsm=capability,landlock,yama,bpf,ima217worker1 # [ 0.001298] landlock: Up and running.218worker1 # [ 0.001300] Yama: becoming mindful.219worker1 # [ 0.001925] LSM support for eBPF active220worker1 # [ 0.002104] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)221worker1 # [ 0.002141] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)222worker1 # [ 0.003927] rcu: Hierarchical SRCU implementation.223worker1 # [ 0.003932] rcu: Max phase no-delay instances is 1000.224worker1 # [ 0.005189] fsl-mc MSI: its@8080000 domain created225worker1 # [ 0.005277] EFI services will not be available.226worker1 # [ 0.005342] smp: Bringing up secondary CPUs ...227worker1 # [ 0.005350] smp: Brought up 1 node, 1 CPU228worker1 # [ 0.005353] SMP: Total of 1 processors activated.229worker1 # [ 0.005359] CPU: All CPU(s) started at EL1230worker1 # [ 0.005369] CPU features: detected: Branch Target Identification231worker1 # [ 0.005373] CPU features: detected: ARMv8.4 Translation Table Level232worker1 # [ 0.005377] CPU features: detected: Instruction cache invalidation not required for I/D coherence233worker1 # [ 0.005380] CPU features: detected: Data cache clean to the PoU not required for I/D coherence234worker1 # [ 0.005384] CPU features: detected: Common not Private translations235worker1 # [ 0.005387] CPU features: detected: CRC32 instructions236worker1 # [ 0.005390] CPU features: detected: Data cache clean to Point of Deep Persistence237worker1 # [ 0.005393] CPU features: detected: Data cache clean to Point of Persistence238worker1 # [ 0.005396] CPU features: detected: Data independent timing control (DIT)239lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]240worker1 # [ 0.005399] CPU features: detected: E0PD241worker1 # [ 0.005402] CPU features: detected: Enhanced Counter Virtualization242lb # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026243worker1 # [ 0.005405] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)244lb # [ 0.000000] KASLR enabled245lb # [ 0.000000] random: crng init done246worker1 # [ 0.005408] CPU features: detected: Enhanced Virtualization Traps247lb # [ 0.000000] Machine model: linux,dummy-virt248worker1 # [ 0.005412] CPU features: detected: Fine Grained Traps249lb # [ 0.000000] efi: UEFI not found.250client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]251worker1 # [ 0.005415] CPU features: detected: Generic authentication (architected QARMA5 algorithm)252lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT253worker1 # [ 0.005419] CPU features: detected: RCpc load-acquire (LDAPR)254lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]255client # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026256worker1 # [ 0.005422] CPU features: detected: LSE atomic instructions257client # [ 0.000000] KASLR enabled258lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]259client # [ 0.000000] random: crng init done260worker1 # [ 0.005425] CPU features: detected: Privileged Access Never261lb # [ 0.000000] Zone ranges:262client # [ 0.000000] Machine model: linux,dummy-virt263worker1 # [ 0.005428] CPU features: detected: PMUv3264client # [ 0.000000] efi: UEFI not found.265lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]266lb # [ 0.000000] DMA32 empty267worker1 # [ 0.005431] CPU features: detected: RAS Extension Support268lb # [ 0.000000] Normal empty269client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT270worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]271lb # [ 0.000000] Device empty272lb # [ 0.000000] Movable zone start for each node273client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]274worker1 # [ 0.005434] CPU features: detected: RASv1p1 Extension Support275lb # [ 0.000000] Early memory node ranges276client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]277worker2 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026278worker1 # [ 0.005436] CPU features: detected: Random Number Generator279client # [ 0.000000] Zone ranges:280lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]281worker2 # [ 0.000000] KASLR enabled282worker2 # [ 0.000000] random: crng init done283worker1 # [ 0.005439] CPU features: detected: Speculation barrier (SB)284client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]285lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]286worker2 # [ 0.000000] Machine model: linux,dummy-virt287worker1 # [ 0.005441] CPU features: detected: Stage-2 Force Write-Back288client # [ 0.000000] DMA32 empty289worker2 # [ 0.000000] efi: UEFI not found.290client # [ 0.000000] Normal empty291lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000292client # [ 0.000000] Device empty293worker1 # [ 0.005444] CPU features: detected: TLB range maintenance instructions294lb # [ 0.000000] psci: probing for conduit method from DT.295client # [ 0.000000] Movable zone start for each node296worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT297lb # [ 0.000000] psci: PSCIv1.3 detected in firmware.298worker1 # [ 0.005449] CPU features: detected: Speculative Store Bypassing Safe (SSBS)299client # [ 0.000000] Early memory node ranges300worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]301lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs302worker1 # [ 0.005490] alternatives: applying system-wide alternatives303client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]304worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]305lb # [ 0.000000] psci: Trusted OS migration not required306worker2 # [ 0.000000] Zone ranges:307worker1 # [ 0.008410] CPU features: detected: BBM Level 2 without TLB conflict abort308client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]309lb # [ 0.000000] psci: SMC Calling Convention v1.1310worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]311client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000312worker2 # [ 0.000000] DMA32 empty313lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)314worker2 # [ 0.000000] Normal empty315client # [ 0.000000] psci: probing for conduit method from DT.316worker1 # [ 0.008593] Memory: 1405840K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124664K reserved, 32768K cma-reserved)317worker2 # [ 0.000000] Device empty318client # [ 0.000000] psci: PSCIv1.3 detected in firmware.319lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296320worker1 # [ 0.009016] devtmpfs: initialized321worker2 # [ 0.000000] Movable zone start for each node322lb # [ 0.000000] Detected PIPT I-cache on CPU0323client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs324worker2 # [ 0.000000] Early memory node ranges325worker1 # [ 0.010879] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)326client # [ 0.000000] psci: Trusted OS migration not required327worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]328lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)329client # [ 0.000000] psci: SMC Calling Convention v1.1330worker1 # [ 0.010900] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).331lb # [ 0.000000] CPU features: detected: GICv3 CPU interface332worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]333worker1 # [ 0.011086] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL334client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)335worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000336lb # [ 0.000000] CPU features: detected: Spectre-v4337worker1 # [ 0.011090] 0 pages in range for non-PLT usage338lb # [ 0.000000] CPU features: detected: Spectre-BHB339worker2 # [ 0.000000] psci: probing for conduit method from DT.340client # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296341worker1 # [ 0.011091] 508272 pages in range for PLT usage342worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware.343client # [ 0.000000] Detected PIPT I-cache on CPU0344lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38345worker1 # [ 0.011197] pinctrl core: initialized pinctrl subsystem346lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23347worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs348worker1 # [ 0.011996] DMI not present or invalid.349client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)350worker2 # [ 0.000000] psci: Trusted OS migration not required351lb # [ 0.000000] alternatives: applying boot alternatives352worker1 # [ 0.015676] NET: Registered PF_NETLINK/PF_ROUTE protocol family353client # [ 0.000000] CPU features: detected: GICv3 CPU interface354worker2 # [ 0.000000] psci: SMC Calling Convention v1.1355client # [ 0.000000] CPU features: detected: Spectre-v4356worker1 # [ 0.017839] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations357client # [ 0.000000] CPU features: detected: Spectre-BHB358worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)359worker1 # [ 0.018010] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations360client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38361worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296362worker2 # [ 0.000000] Detected PIPT I-cache on CPU0363worker1 # [ 0.018207] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations364client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23365client # [ 0.000000] alternatives: applying boot alternatives366worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)367worker1 # [ 0.018228] audit: initializing netlink subsys (disabled)368lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/5hmvppg1fs8xkiccvl0m50n4dxjz92fr-nixos-system-lb-test/init regInfo=/nix/store/4s18yqhkjka9yjskv51fw7r9nw16dgfd-closure-info/registration console=ttyAMA0,115200n8 console=tty0369worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface370worker1 # [ 0.018765] thermal_sys: Registered thermal governor 'fair_share'371worker2 # [ 0.000000] CPU features: detected: Spectre-v4372worker1 # [ 0.018767] thermal_sys: Registered thermal governor 'bang_bang'373worker2 # [ 0.000000] CPU features: detected: Spectre-BHB374lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/4s18yqhkjka9yjskv51fw7r9nw16dgfd-closure-info/registration", will be passed to user space.375worker1 # [ 0.018771] thermal_sys: Registered thermal governor 'step_wise'376worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38377worker1 # [ 0.018774] thermal_sys: Registered thermal governor 'user_space'378worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23379lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes380worker1 # [ 0.018779] thermal_sys: Registered thermal governor 'power_allocator'381worker2 # [ 0.000000] alternatives: applying boot alternatives382client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/ra6r7v42kn1phhf50aiw8qw3p1s4g3i7-nixos-system-client-test/init regInfo=/nix/store/lgfd5irrpwicr30w9sv4rcyvvpw4p2ws-closure-info/registration console=ttyAMA0,115200n8 console=tty0383lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)384worker1 # [ 0.018811] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1385worker1 # [ 0.018820] cpuidle: using governor ladder386lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)387worker1 # [ 0.018826] cpuidle: using governor menu388client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/lgfd5irrpwicr30w9sv4rcyvvpw4p2ws-closure-info/registration", will be passed to user space.389lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB390worker1 # [ 0.019022] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.391client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes392lb # [ 0.000000] software IO TLB: area num 1.393worker1 # [ 0.019038] ASID allocator initialised with 65536 entries394lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB395client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)396worker1 # [ 0.020330] Serial: AMBA PL011 UART driver397worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/bxvmlpjg1y5fh7n09mwrfkkffnsyixlj-nixos-system-worker2-test/init regInfo=/nix/store/85g4acyh62j9yqhpzlfzw6d69c8rs2px-closure-info/registration console=ttyAMA0,115200n8 console=tty0398lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)399client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)400worker1 # [ 0.025804] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1401lb # [ 0.000000] Fallback order for Node 0: 0402worker1 # [ 0.025958] printk: console [ttyAMA0] enabled403worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/85g4acyh62j9yqhpzlfzw6d69c8rs2px-closure-info/registration", will be passed to user space.404lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216405client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB406lb # [ 0.000000] Policy zone: DMA407client # [ 0.000000] software IO TLB: area num 1.408worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes409lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off410client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB411worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)412lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1413client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)414lb # [ 0.000000] allocated 3145728 bytes of page_ext415worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)416client # [ 0.000000] Fallback order for Node 0: 0417lb # [ 0.000000] ftrace: allocating 74950 entries in 294 pages418worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB419lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups420client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216421worker2 # [ 0.000000] software IO TLB: area num 1.422client # [ 0.000000] Policy zone: DMA423lb # [ 0.000000] rcu: Hierarchical RCU implementation.424worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB425lb # [ 0.000000] rcu: RCU event tracing is enabled.426client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off427worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)428lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.429client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1430worker2 # [ 0.000000] Fallback order for Node 0: 0431client # [ 0.000000] allocated 3145728 bytes of page_ext432lb # [ 0.000000] Trampoline variant of Tasks RCU enabled.433worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216434lb # [ 0.000000] Rude variant of Tasks RCU enabled.435client # [ 0.000000] ftrace: allocating 74950 entries in 294 pages436worker2 # [ 0.000000] Policy zone: DMA437lb # [ 0.000000] Tracing variant of Tasks RCU enabled.438client # [ 0.000000] ftrace: allocated 294 pages with 4 groups439worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off440client # [ 0.000000] rcu: Hierarchical RCU implementation.441lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.442client # [ 0.000000] rcu: RCU event tracing is enabled.443worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1444lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1445worker1 # [ 0.147234] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages446worker2 # [ 0.000000] allocated 3145728 bytes of page_ext447client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.448worker2 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages449worker1 # [ 0.147250] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page450client # [ 0.000000] Trampoline variant of Tasks RCU enabled.451lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.452worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups453client # [ 0.000000] Rude variant of Tasks RCU enabled.454worker1 # [ 0.147255] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages455worker2 # [ 0.000000] rcu: Hierarchical RCU implementation.456client # [ 0.000000] Tracing variant of Tasks RCU enabled.457lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.458worker1 # [ 0.147260] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page459worker2 # [ 0.000000] rcu: RCU event tracing is enabled.460client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.461worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.462worker1 # [ 0.147264] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages463lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.464client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1465worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled.466lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0467worker1 # [ 0.147269] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page468lb # [ 0.000000] GICv3: 256 SPIs implemented469worker2 # [ 0.000000] Rude variant of Tasks RCU enabled.470client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.471worker1 # [ 0.147274] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages472worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled.473lb # [ 0.000000] GICv3: 0 Extended SPIs implemented474lb # [ 0.000000] Root IRQ handler: gic_handle_irq475worker1 # [ 0.147278] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page476worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.477client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.478lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI479lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0480worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1481client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.482lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000483client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0484worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.485worker1 # [ 0.154398] fbcon: Taking over console486lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]487client # [ 0.000000] GICv3: 256 SPIs implemented488worker1 # [ 0.154411] ACPI: Interpreter disabled.489client # [ 0.000000] GICv3: 0 Extended SPIs implemented490worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.491lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1)492worker1 # [ 0.156239] iommu: Default domain type: Translated493client # [ 0.000000] Root IRQ handler: gic_handle_irq494worker1 # [ 0.156248] iommu: DMA domain TLB invalidation policy: strict mode495client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI496worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.497lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1)498client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0499worker1 # [ 0.157826] SCSI subsystem initialized500worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0501lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000502worker2 # [ 0.000000] GICv3: 256 SPIs implemented503client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000504worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented505client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]506lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000507worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq508lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.509worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI510client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)511worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0512lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns513worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000514client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)515lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).516worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]517client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000518client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000519worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1)520lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns521client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.522lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns523worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1)524worker1 # [ 0.163441] usbcore: registered new interface driver usbfs525lb # [ 0.000030] arm-pv: using stolen time PV526worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a00000527client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns528worker1 # [ 0.163473] usbcore: registered new interface driver hub529worker1 # [ 0.163490] usbcore: registered new device driver usb530client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).531worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a10000532lb # [ 0.000406] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)533worker1 # [ 0.163774] pps_core: LinuxPPS API ver. 1 registered534lb # [ 0.000586] Console: colour dummy device 80x25535worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.536lb # [ 0.000593] printk: legacy console [tty0] enabled537client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns538worker1 # [ 0.163780] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>539worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns540worker1 # [ 0.163797] PTP clock support registered541client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns542worker1 # [ 0.163846] EDAC MC: Ver: 3.0.0543lb # [ 0.000783] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)544client # [ 0.000030] arm-pv: using stolen time PV545worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).546lb # [ 0.000790] pid_max: default: 32768 minimum: 301547worker1 # [ 0.168263] scmi_core: SCMI protocol bus registered548lb # [ 0.000870] LSM: initializing lsm=capability,landlock,yama,bpf,ima549client # [ 0.000413] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)550worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns551lb # [ 0.001046] landlock: Up and running.552worker1 # [ 0.169199] FPGA manager framework553client # [ 0.000582] Console: colour dummy device 80x25554lb # [ 0.001049] Yama: becoming mindful.555worker1 # [ 0.170082] vgaarb: loaded556worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns557client # [ 0.000590] printk: legacy console [tty0] enabled558lb # [ 0.001503] LSM support for eBPF active559worker2 # [ 0.000030] arm-pv: using stolen time PV560worker1 # [ 0.170682] clocksource: Switched to clocksource arch_sys_counter561lb # [ 0.001651] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)562client # [ 0.000783] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)563worker2 # [ 0.000389] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)564client # [ 0.000790] pid_max: default: 32768 minimum: 301565lb # [ 0.001684] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)566worker2 # [ 0.000557] Console: colour dummy device 80x25567lb # [ 0.003413] rcu: Hierarchical SRCU implementation.568client # [ 0.000870] LSM: initializing lsm=capability,landlock,yama,bpf,ima569worker2 # [ 0.000564] printk: legacy console [tty0] enabled570client # [ 0.001033] landlock: Up and running.571lb # [ 0.003418] rcu: Max phase no-delay instances is 1000.572client # [ 0.001036] Yama: becoming mindful.573lb # [ 0.004701] fsl-mc MSI: its@8080000 domain created574client # [ 0.001511] LSM support for eBPF active575worker2 # [ 0.000756] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)576lb # [ 0.004800] EFI services will not be available.577worker1 # [ 0.174224] VFS: Disk quotas dquot_6.6.0578worker2 # [ 0.000762] pid_max: default: 32768 minimum: 301579client # [ 0.001651] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)580lb # [ 0.004918] smp: Bringing up secondary CPUs ...581worker1 # [ 0.174257] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)582lb # [ 0.004926] smp: Brought up 1 node, 1 CPU583worker2 # [ 0.000843] LSM: initializing lsm=capability,landlock,yama,bpf,ima584client # [ 0.001683] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)585worker2 # [ 0.000972] landlock: Up and running.586lb # [ 0.004929] SMP: Total of 1 processors activated.587worker2 # [ 0.000975] Yama: becoming mindful.588client # [ 0.003551] rcu: Hierarchical SRCU implementation.589lb # [ 0.004935] CPU: All CPU(s) started at EL1590worker2 # [ 0.001417] LSM support for eBPF active591client # [ 0.003556] rcu: Max phase no-delay instances is 1000.592lb # [ 0.004945] CPU features: detected: Branch Target Identification593client # [ 0.004802] fsl-mc MSI: its@8080000 domain created594worker2 # [ 0.001543] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)595lb # [ 0.004949] CPU features: detected: ARMv8.4 Translation Table Level596worker1 # [ 0.177825] netfs: FS-Cache loaded597client # [ 0.004892] EFI services will not be available.598worker1 # [ 0.177929] pnp: PnP ACPI: disabled599worker2 # [ 0.001576] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)600client # [ 0.004964] smp: Bringing up secondary CPUs ...601lb # [ 0.004953] CPU features: detected: Instruction cache invalidation not required for I/D coherence602worker2 # [ 0.003420] rcu: Hierarchical SRCU implementation.603client # [ 0.004972] smp: Brought up 1 node, 1 CPU604worker2 # [ 0.003424] rcu: Max phase no-delay instances is 1000.605client # [ 0.004975] SMP: Total of 1 processors activated.606lb # [ 0.004956] CPU features: detected: Data cache clean to the PoU not required for I/D coherence607client # [ 0.004981] CPU: All CPU(s) started at EL1608worker2 # [ 0.004663] fsl-mc MSI: its@8080000 domain created609lb # [ 0.004960] CPU features: detected: Common not Private translations610worker2 # [ 0.004752] EFI services will not be available.611client # [ 0.004991] CPU features: detected: Branch Target Identification612worker2 # [ 0.004817] smp: Bringing up secondary CPUs ...613lb # [ 0.004964] CPU features: detected: CRC32 instructions614client # [ 0.004996] CPU features: detected: ARMv8.4 Translation Table Level615worker2 # [ 0.004825] smp: Brought up 1 node, 1 CPU616lb # [ 0.004967] CPU features: detected: Data cache clean to Point of Deep Persistence617worker2 # [ 0.004828] SMP: Total of 1 processors activated.618client # [ 0.004999] CPU features: detected: Instruction cache invalidation not required for I/D coherence619worker2 # [ 0.004833] CPU: All CPU(s) started at EL1620worker1 # [ 0.181696] NET: Registered PF_INET protocol family621lb # [ 0.004970] CPU features: detected: Data cache clean to Point of Persistence622worker2 # [ 0.004844] CPU features: detected: Branch Target Identification623lb # [ 0.004973] CPU features: detected: Data independent timing control (DIT)624worker1 # [ 0.181957] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)625client # [ 0.005003] CPU features: detected: Data cache clean to the PoU not required for I/D coherence626worker2 # [ 0.004849] CPU features: detected: ARMv8.4 Translation Table Level627lb # [ 0.004976] CPU features: detected: E0PD628client # [ 0.005006] CPU features: detected: Common not Private translations629lb # [ 0.004979] CPU features: detected: Enhanced Counter Virtualization630client # [ 0.005010] CPU features: detected: CRC32 instructions631worker2 # [ 0.004852] CPU features: detected: Instruction cache invalidation not required for I/D coherence632lb # [ 0.004982] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)633client # [ 0.005012] CPU features: detected: Data cache clean to Point of Deep Persistence634worker2 # [ 0.004855] CPU features: detected: Data cache clean to the PoU not required for I/D coherence635lb # [ 0.004985] CPU features: detected: Enhanced Virtualization Traps636client # [ 0.005016] CPU features: detected: Data cache clean to Point of Persistence637worker2 # [ 0.004859] CPU features: detected: Common not Private translations638lb # [ 0.004989] CPU features: detected: Fine Grained Traps639client # [ 0.005019] CPU features: detected: Data independent timing control (DIT)640worker2 # [ 0.004863] CPU features: detected: CRC32 instructions641client # [ 0.005022] CPU features: detected: E0PD642lb # [ 0.004992] CPU features: detected: Generic authentication (architected QARMA5 algorithm)643worker2 # [ 0.004865] CPU features: detected: Data cache clean to Point of Deep Persistence644client # [ 0.005025] CPU features: detected: Enhanced Counter Virtualization645lb # [ 0.004997] CPU features: detected: RCpc load-acquire (LDAPR)646worker2 # [ 0.004869] CPU features: detected: Data cache clean to Point of Persistence647lb # [ 0.005000] CPU features: detected: LSE atomic instructions648client # [ 0.005028] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)649worker2 # [ 0.004872] CPU features: detected: Data independent timing control (DIT)650lb # [ 0.005003] CPU features: detected: Privileged Access Never651client # [ 0.005031] CPU features: detected: Enhanced Virtualization Traps652worker2 # [ 0.004875] CPU features: detected: E0PD653lb # [ 0.005005] CPU features: detected: PMUv3654client # [ 0.005035] CPU features: detected: Fine Grained Traps655lb # [ 0.005009] CPU features: detected: RAS Extension Support656worker2 # [ 0.004878] CPU features: detected: Enhanced Counter Virtualization657lb # [ 0.005011] CPU features: detected: RASv1p1 Extension Support658client # [ 0.005038] CPU features: detected: Generic authentication (architected QARMA5 algorithm)659worker2 # [ 0.004881] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)660lb # [ 0.005014] CPU features: detected: Random Number Generator661client # [ 0.005043] CPU features: detected: RCpc load-acquire (LDAPR)662worker2 # [ 0.004884] CPU features: detected: Enhanced Virtualization Traps663client # [ 0.005046] CPU features: detected: LSE atomic instructions664lb # [ 0.005017] CPU features: detected: Speculation barrier (SB)665worker2 # [ 0.004887] CPU features: detected: Fine Grained Traps666client # [ 0.005049] CPU features: detected: Privileged Access Never667client # [ 0.005051] CPU features: detected: PMUv3668worker2 # [ 0.004890] CPU features: detected: Generic authentication (architected QARMA5 algorithm)669client # [ 0.005055] CPU features: detected: RAS Extension Support670worker2 # [ 0.004896] CPU features: detected: RCpc load-acquire (LDAPR)671client # [ 0.005058] CPU features: detected: RASv1p1 Extension Support672worker2 # [ 0.004899] CPU features: detected: LSE atomic instructions673client # [ 0.005060] CPU features: detected: Random Number Generator674worker2 # [ 0.004901] CPU features: detected: Privileged Access Never675worker2 # [ 0.004904] CPU features: detected: PMUv3676client # [ 0.005063] CPU features: detected: Speculation barrier (SB)677worker2 # [ 0.004907] CPU features: detected: RAS Extension Support678client # [ 0.005065] CPU features: detected: Stage-2 Force Write-Back679worker2 # [ 0.004910] CPU features: detected: RASv1p1 Extension Support680client # [ 0.005068] CPU features: detected: TLB range maintenance instructions681worker2 # [ 0.004912] CPU features: detected: Random Number Generator682client # [ 0.005073] CPU features: detected: Speculative Store Bypassing Safe (SSBS)683worker2 # [ 0.004915] CPU features: detected: Speculation barrier (SB)684worker2 # [ 0.004918] CPU features: detected: Stage-2 Force Write-Back685worker2 # [ 0.004921] CPU features: detected: TLB range maintenance instructions686worker2 # [ 0.004925] CPU features: detected: Speculative Store Bypassing Safe (SSBS)687lb # [ 0.005020] CPU features: detected: Stage-2 Force Write-Back688worker2 # [ 0.004963] alternatives: applying system-wide alternatives689lb # [ 0.005022] CPU features: detected: TLB range maintenance instructions690worker2 # [ 0.007847] CPU features: detected: BBM Level 2 without TLB conflict abort691lb # [ 0.005027] CPU features: detected: Speculative Store Bypassing Safe (SSBS)692lb # [ 0.005067] alternatives: applying system-wide alternatives693worker2 # [ 0.008069] Memory: 1405708K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124632K reserved, 32768K cma-reserved)694lb # [ 0.008058] CPU features: detected: BBM Level 2 without TLB conflict abort695worker2 # [ 0.008453] devtmpfs: initialized696worker2 # [ 0.010206] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)697lb # [ 0.008249] Memory: 1405692K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124636K reserved, 32768K cma-reserved)698lb # [ 0.008621] devtmpfs: initialized699worker2 # [ 0.010232] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).700client # [ 0.005110] alternatives: applying system-wide alternatives701worker2 # [ 0.010407] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL702lb # [ 0.010426] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)703worker2 # [ 0.010411] 0 pages in range for non-PLT usage704client # [ 0.008065] CPU features: detected: BBM Level 2 without TLB conflict abort705lb # [ 0.010448] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).706worker2 # [ 0.010412] 508272 pages in range for PLT usage707worker2 # [ 0.010517] pinctrl core: initialized pinctrl subsystem708lb # [ 0.010624] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL709worker2 # [ 0.011261] DMI not present or invalid.710lb # [ 0.010628] 0 pages in range for non-PLT usage711client # [ 0.008222] Memory: 1405640K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124660K reserved, 32768K cma-reserved)712client # [ 0.008603] devtmpfs: initialized713worker2 # [ 0.014394] NET: Registered PF_NETLINK/PF_ROUTE protocol family714lb # [ 0.010629] 508272 pages in range for PLT usage715lb # [ 0.010744] pinctrl core: initialized pinctrl subsystem716client # [ 0.010390] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)717worker2 # [ 0.016559] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations718lb # [ 0.011503] DMI not present or invalid.719worker2 # [ 0.016745] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations720client # [ 0.010411] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).721lb # [ 0.014678] NET: Registered PF_NETLINK/PF_ROUTE protocol family722client # [ 0.010589] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL723lb # [ 0.016881] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations724worker2 # [ 0.016938] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations725client # [ 0.010593] 0 pages in range for non-PLT usage726worker2 # [ 0.016959] audit: initializing netlink subsys (disabled)727lb # [ 0.017073] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations728client # [ 0.010594] 508272 pages in range for PLT usage729worker2 # [ 0.017516] thermal_sys: Registered thermal governor 'fair_share'730client # [ 0.010699] pinctrl core: initialized pinctrl subsystem731lb # [ 0.017270] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations732client # [ 0.011427] DMI not present or invalid.733worker2 # [ 0.017518] thermal_sys: Registered thermal governor 'bang_bang'734lb # [ 0.017292] audit: initializing netlink subsys (disabled)735client # [ 0.014522] NET: Registered PF_NETLINK/PF_ROUTE protocol family736worker2 # [ 0.017521] thermal_sys: Registered thermal governor 'step_wise'737lb # [ 0.017861] thermal_sys: Registered thermal governor 'fair_share'738worker2 # [ 0.017524] thermal_sys: Registered thermal governor 'user_space'739client # [ 0.016683] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations740lb # [ 0.017863] thermal_sys: Registered thermal governor 'bang_bang'741worker2 # [ 0.017529] thermal_sys: Registered thermal governor 'power_allocator'742client # [ 0.016856] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations743lb # [ 0.017866] thermal_sys: Registered thermal governor 'step_wise'744worker2 # [ 0.017552] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1745lb # [ 0.017869] thermal_sys: Registered thermal governor 'user_space'746client # [ 0.017052] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations747worker2 # [ 0.017562] cpuidle: using governor ladder748client # [ 0.017071] audit: initializing netlink subsys (disabled)749worker1 # [ 0.209945] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)750lb # [ 0.017874] thermal_sys: Registered thermal governor 'power_allocator'751worker2 # [ 0.017567] cpuidle: using governor menu752client # [ 0.017605] thermal_sys: Registered thermal governor 'fair_share'753worker1 # [ 0.209987] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)754worker2 # [ 0.017744] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.755lb # [ 0.017907] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1756client # [ 0.017607] thermal_sys: Registered thermal governor 'bang_bang'757lb # [ 0.017915] cpuidle: using governor ladder758worker1 # [ 0.210014] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)759lb # [ 0.017921] cpuidle: using governor menu760worker2 # [ 0.017759] ASID allocator initialised with 65536 entries761client # [ 0.017610] thermal_sys: Registered thermal governor 'step_wise'762worker1 # [ 0.210081] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)763worker2 # [ 0.018933] Serial: AMBA PL011 UART driver764lb # [ 0.018114] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.765client # [ 0.017613] thermal_sys: Registered thermal governor 'user_space'766lb # [ 0.018129] ASID allocator initialised with 65536 entries767worker1 # [ 0.210223] TCP: Hash tables configured (established 16384 bind 16384)768client # [ 0.017618] thermal_sys: Registered thermal governor 'power_allocator'769worker2 # [ 0.024049] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1770lb # [ 0.019342] Serial: AMBA PL011 UART driver771worker2 # [ 0.024218] printk: console [ttyAMA0] enabled772worker1 # [ 0.210301] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)773client # [ 0.017650] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1774lb # [ 0.024634] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1775client # [ 0.017658] cpuidle: using governor ladder776worker1 # [ 0.210335] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)777lb # [ 0.024782] printk: console [ttyAMA0] enabled778client # [ 0.017663] cpuidle: using governor menu779worker1 # [ 0.210370] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)780client # [ 0.017851] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.781worker1 # [ 0.210476] NET: Registered PF_UNIX/PF_LOCAL protocol family782lb # [ 0.144692] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages783client # [ 0.017866] ASID allocator initialised with 65536 entries784worker1 # [ 0.210504] NET: Registered PF_XDP protocol family785client # [ 0.019055] Serial: AMBA PL011 UART driver786lb # [ 0.144714] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page787worker1 # [ 0.210519] PCI: CLS 0 bytes, default 64788worker1 # [ 0.210761] Trying to unpack rootfs image as initramfs...789lb # [ 0.144719] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages790client # [ 0.024256] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1791client # [ 0.024379] printk: console [ttyAMA0] enabled792lb # [ 0.144724] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page793lb # [ 0.144728] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages794worker2 # [ 0.140164] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages795client # [ 0.144472] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages796lb # [ 0.144732] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page797worker2 # [ 0.140179] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page798client # [ 0.144490] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page799lb # [ 0.144736] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages800worker2 # [ 0.140184] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages801client # [ 0.144495] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages802lb # [ 0.144741] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page803worker2 # [ 0.140189] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page804client # [ 0.144499] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page805worker2 # [ 0.140193] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages806client # [ 0.144504] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages807worker2 # [ 0.140197] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page808client # [ 0.144508] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page809lb # [ 0.152048] fbcon: Taking over console810worker2 # [ 0.140202] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages811lb # [ 0.152062] ACPI: Interpreter disabled.812client # [ 0.144512] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages813worker1 # [ 0.228032] kvm [1]: HYP mode not available814lb # [ 0.153944] iommu: Default domain type: Translated815worker2 # [ 0.140206] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page816client # [ 0.144516] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page817lb # [ 0.153954] iommu: DMA domain TLB invalidation policy: strict mode818lb # [ 0.155602] SCSI subsystem initialized819worker2 # [ 0.147554] fbcon: Taking over console820client # [ 0.151921] fbcon: Taking over console821worker2 # [ 0.147569] ACPI: Interpreter disabled.822client # [ 0.151936] ACPI: Interpreter disabled.823worker2 # [ 0.149426] iommu: Default domain type: Translated824worker2 # [ 0.149436] iommu: DMA domain TLB invalidation policy: strict mode825worker2 # [ 0.151102] SCSI subsystem initialized826client # [ 0.158205] iommu: Default domain type: Translated827client # [ 0.158214] iommu: DMA domain TLB invalidation policy: strict mode828client # [ 0.159888] SCSI subsystem initialized829lb # [ 0.163546] usbcore: registered new interface driver usbfs830lb # [ 0.163581] usbcore: registered new interface driver hub831client # [ 0.160563] usbcore: registered new interface driver usbfs832worker2 # [ 0.156481] usbcore: registered new interface driver usbfs833lb # [ 0.163598] usbcore: registered new device driver usb834client # [ 0.160596] usbcore: registered new interface driver hub835worker2 # [ 0.156514] usbcore: registered new interface driver hub836lb # [ 0.163892] pps_core: LinuxPPS API ver. 1 registered837client # [ 0.160620] usbcore: registered new device driver usb838worker2 # [ 0.156530] usbcore: registered new device driver usb839client # [ 0.160883] pps_core: LinuxPPS API ver. 1 registered840worker2 # [ 0.156804] pps_core: LinuxPPS API ver. 1 registered841lb # [ 0.163899] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>842lb # [ 0.163916] PTP clock support registered843client # [ 0.160889] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>844worker2 # [ 0.156811] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>845lb # [ 0.163973] EDAC MC: Ver: 3.0.0846client # [ 0.160906] PTP clock support registered847worker2 # [ 0.156820] PTP clock support registered848client # [ 0.160957] EDAC MC: Ver: 3.0.0849worker2 # [ 0.156877] EDAC MC: Ver: 3.0.0850lb # [ 0.168553] scmi_core: SCMI protocol bus registered851client # [ 0.165463] scmi_core: SCMI protocol bus registered852worker2 # [ 0.161305] scmi_core: SCMI protocol bus registered853lb # [ 0.169531] FPGA manager framework854lb # [ 0.170545] vgaarb: loaded855client # [ 0.166455] FPGA manager framework856worker2 # [ 0.162273] FPGA manager framework857client # [ 0.167402] vgaarb: loaded858worker2 # [ 0.163230] vgaarb: loaded859lb # [ 0.171167] clocksource: Switched to clocksource arch_sys_counter860client # [ 0.168036] clocksource: Switched to clocksource arch_sys_counter861worker2 # [ 0.163877] clocksource: Switched to clocksource arch_sys_counter862lb # [ 0.171778] VFS: Disk quotas dquot_6.6.0863lb # [ 0.171811] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)864lb # [ 0.174128] netfs: FS-Cache loaded865lb # [ 0.174240] pnp: PnP ACPI: disabled866client # [ 0.172015] VFS: Disk quotas dquot_6.6.0867client # [ 0.174588] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)868worker2 # [ 0.170965] VFS: Disk quotas dquot_6.6.0869client # [ 0.175731] netfs: FS-Cache loaded870worker2 # [ 0.171001] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)871client # [ 0.175855] pnp: PnP ACPI: disabled872worker2 # [ 0.174679] netfs: FS-Cache loaded873worker2 # [ 0.174805] pnp: PnP ACPI: disabled874lb # [ 0.180213] NET: Registered PF_INET protocol family875client # [ 0.179771] NET: Registered PF_INET protocol family876lb # [ 0.180493] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)877worker2 # [ 0.178566] NET: Registered PF_INET protocol family878client # [ 0.180699] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)879worker2 # [ 0.178841] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)880lb # [ 0.209639] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)881worker2 # [ 0.207570] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)882lb # [ 0.209678] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)883worker2 # [ 0.207612] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)884lb # [ 0.209703] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)885worker2 # [ 0.207638] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)886lb # [ 0.209772] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)887worker2 # [ 0.207714] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)888lb # [ 0.209916] TCP: Hash tables configured (established 16384 bind 16384)889worker2 # [ 0.207868] TCP: Hash tables configured (established 16384 bind 16384)890lb # [ 0.209996] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)891client # [ 0.209097] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)892lb # [ 0.210030] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)893client # [ 0.209139] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)894lb # [ 0.210065] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)895client # [ 0.209166] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)896lb # [ 0.210156] NET: Registered PF_UNIX/PF_LOCAL protocol family897client # [ 0.209236] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)898lb # [ 0.210181] NET: Registered PF_XDP protocol family899lb # [ 0.210198] PCI: CLS 0 bytes, default 64900client # [ 0.209379] TCP: Hash tables configured (established 16384 bind 16384)901lb # [ 0.210437] Trying to unpack rootfs image as initramfs...902client # [ 0.209461] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)903client # [ 0.209497] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)904client # [ 0.209543] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)905client # [ 0.209652] NET: Registered PF_UNIX/PF_LOCAL protocol family906client # [ 0.209672] NET: Registered PF_XDP protocol family907client # [ 0.209688] PCI: CLS 0 bytes, default 64908worker2 # [ 0.212720] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)909client # [ 0.209915] Trying to unpack rootfs image as initramfs...910worker2 # [ 0.212757] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)911worker2 # [ 0.212792] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)912worker2 # [ 0.212883] NET: Registered PF_UNIX/PF_LOCAL protocol family913lb # [ 0.225055] kvm [1]: HYP mode not available914worker2 # [ 0.212908] NET: Registered PF_XDP protocol family915worker1 # [ 0.311787] Initialise system trusted keyrings916worker2 # [ 0.212926] PCI: CLS 0 bytes, default 64917worker1 # [ 0.312514] workingset: timestamp_bits=42 max_order=19 bucket_order=0918worker2 # [ 0.213149] Trying to unpack rootfs image as initramfs...919worker1 # [ 0.313774] squashfs: version 4.0 (2009/01/31) Phillip Lougher920client # [ 0.225719] kvm [1]: HYP mode not available921worker2 # [ 0.224914] kvm [1]: HYP mode not available922worker1 # [ 0.318717] 9p: Installing v9fs 9p2000 file system support923worker1 # [ 0.338678] Key type asymmetric registered924worker1 # [ 0.346725] Asymmetric key parser 'x509' registered925worker1 # [ 0.346796] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)926worker1 # [ 0.348389] io scheduler mq-deadline registered927worker1 # [ 0.348400] io scheduler kyber registered928worker1 # [ 0.353375] pl061_gpio 9030000.pl061: PL061 GPIO chip registered929worker1 # [ 0.358713] ledtrig-cpu: registered to indicate activity on CPUs930worker1 # [ 0.359172] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:931worker1 # [ 0.359192] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000932worker1 # [ 0.359206] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000933worker1 # [ 0.359215] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000934worker1 # [ 0.359242] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits935worker1 # [ 0.359275] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]936worker1 # [ 0.359376] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00937worker1 # [ 0.359385] pci_bus 0000:00: root bus resource [bus 00-ff]938worker1 # [ 0.359391] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]939worker1 # [ 0.359397] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]940worker1 # [ 0.359402] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]941worker1 # [ 0.359478] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint942worker1 # [ 0.359929] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint943worker1 # [ 0.360121] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]944worker1 # [ 0.360139] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]945worker1 # [ 0.360170] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]946worker1 # [ 0.360188] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]947worker1 # [ 0.360660] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint948worker1 # [ 0.360849] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]949worker1 # [ 0.360866] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]950worker1 # [ 0.360897] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]951worker1 # [ 0.361367] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint952worker1 # [ 0.361554] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]953worker1 # [ 0.361572] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]954worker1 # [ 0.361603] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]955worker1 # [ 0.362062] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint956worker1 # [ 0.362251] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]957worker1 # [ 0.362268] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]958worker1 # [ 0.362299] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]959worker1 # [ 0.362319] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]960lb # [ 0.315849] Initialise system trusted keyrings961lb # [ 0.316599] workingset: timestamp_bits=42 max_order=19 bucket_order=0962lb # [ 0.317874] squashfs: version 4.0 (2009/01/31) Phillip Lougher963lb # [ 0.318652] 9p: Installing v9fs 9p2000 file system support964client # [ 0.316547] Initialise system trusted keyrings965client # [ 0.317289] workingset: timestamp_bits=42 max_order=19 bucket_order=0966worker1 # [ 0.394862] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint967client # [ 0.318536] squashfs: version 4.0 (2009/01/31) Phillip Lougher968worker1 # [ 0.395082] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]969worker1 # [ 0.395116] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]970client # [ 0.319311] 9p: Installing v9fs 9p2000 file system support971worker2 # [ 0.316408] Initialise system trusted keyrings972worker1 # [ 0.395664] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint973worker2 # [ 0.317152] workingset: timestamp_bits=42 max_order=19 bucket_order=0974worker1 # [ 0.395857] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]975worker1 # [ 0.395888] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]976worker2 # [ 0.318406] squashfs: version 4.0 (2009/01/31) Phillip Lougher977worker1 # [ 0.396286] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint978worker2 # [ 0.319210] 9p: Installing v9fs 9p2000 file system support979worker1 # [ 0.396471] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]980worker1 # [ 0.396750] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint981worker1 # [ 0.396944] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]982worker1 # [ 0.396976] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]983worker1 # [ 0.397437] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint984worker1 # [ 0.397632] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]985worker1 # [ 0.397664] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]986worker1 # [ 0.398128] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint987worker1 # [ 0.398323] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]988worker1 # [ 0.398355] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]989worker1 # [ 0.398832] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint990worker1 # [ 0.399108] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]991worker1 # [ 0.399125] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]992worker1 # [ 0.399155] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]993worker1 # [ 0.399604] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint994worker1 # [ 0.399795] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]995worker1 # [ 0.399811] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]996worker1 # [ 0.399841] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]997worker1 # [ 0.400407] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned998client # [ 0.339950] Key type asymmetric registered999worker1 # [ 0.400418] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1000client # [ 0.339971] Asymmetric key parser 'x509' registered1001worker1 # [ 0.400424] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1002worker1 # [ 0.400470] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1003worker1 # [ 0.400518] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1004lb # [ 0.347354] Key type asymmetric registered1005worker1 # [ 0.400566] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1006lb # [ 0.347380] Asymmetric key parser 'x509' registered1007worker1 # [ 0.400624] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1008lb # [ 0.347445] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)1009worker1 # [ 0.400669] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1010lb # [ 0.349625] io scheduler mq-deadline registered1011lb # [ 0.349637] io scheduler kyber registered1012worker1 # [ 0.400715] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1013worker1 # [ 0.400760] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1014client # [ 0.348073] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)1015worker1 # [ 0.400805] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1016client # [ 0.349127] io scheduler mq-deadline registered1017client # [ 0.349139] io scheduler kyber registered1018worker1 # [ 0.400850] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1019worker1 # [ 0.400922] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1020worker1 # [ 0.400967] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1021worker1 # [ 0.400989] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1022worker2 # [ 0.348095] Key type asymmetric registered1023worker2 # [ 0.348118] Asymmetric key parser 'x509' registered1024worker1 # [ 0.401010] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1025worker2 # [ 0.348177] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)1026worker1 # [ 0.401032] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1027worker2 # [ 0.350310] io scheduler mq-deadline registered1028worker1 # [ 0.401053] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1029worker2 # [ 0.350321] io scheduler kyber registered1030worker1 # [ 0.401074] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1031lb # [ 0.359425] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1032worker1 # [ 0.401096] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1033worker1 # [ 0.401120] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1034worker1 # [ 0.401142] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1035worker1 # [ 0.401164] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1036lb # [ 0.360833] ledtrig-cpu: registered to indicate activity on CPUs1037worker1 # [ 0.401186] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1038lb # [ 0.361251] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1039worker1 # [ 0.401208] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1040client # [ 0.360172] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1041worker1 # [ 0.401230] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1042lb # [ 0.361268] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001043worker1 # [ 0.401250] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1044lb # [ 0.361280] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001045worker1 # [ 0.401271] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1046worker1 # [ 0.401292] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1047lb # [ 0.361288] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001048client # [ 0.361544] ledtrig-cpu: registered to indicate activity on CPUs1049worker1 # [ 0.401313] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1050lb # [ 0.361319] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1051worker1 # [ 0.401335] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1052worker2 # [ 0.360065] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1053client # [ 0.361951] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1054worker1 # [ 0.401360] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1055lb # [ 0.361343] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1056client # [ 0.361968] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001057worker1 # [ 0.401370] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1058lb # [ 0.361417] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001059client # [ 0.361980] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001060worker1 # [ 0.401374] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1061lb # [ 0.361427] pci_bus 0000:00: root bus resource [bus 00-ff]1062worker1 # [ 0.402195] pci 0000:00:07.0: enabling device (0000 -> 0002)1063lb # [ 0.361433] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1064client # [ 0.361988] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001065worker2 # [ 0.361450] ledtrig-cpu: registered to indicate activity on CPUs1066lb # [ 0.361438] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1067worker2 # [ 0.361854] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1068client # [ 0.362008] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1069lb # [ 0.361443] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1070worker2 # [ 0.361872] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001071client # [ 0.362034] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1072lb # [ 0.361536] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1073client # [ 0.362106] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001074worker2 # [ 0.361885] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001075lb # [ 0.361974] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1076client # [ 0.362116] pci_bus 0000:00: root bus resource [bus 00-ff]1077lb # [ 0.362160] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1078worker2 # [ 0.361893] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001079client # [ 0.362121] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1080lb # [ 0.362176] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1081client # [ 0.362126] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1082worker2 # [ 0.361914] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1083lb # [ 0.362206] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1084client # [ 0.362131] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1085worker2 # [ 0.361937] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1086lb # [ 0.362222] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1087client # [ 0.362186] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1088worker2 # [ 0.362019] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001089lb # [ 0.362677] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1090worker2 # [ 0.362029] pci_bus 0000:00: root bus resource [bus 00-ff]1091client # [ 0.362630] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1092lb # [ 0.362860] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1093worker1 # [ 0.458874] pci 0000:00:07.0: quirk_usb_early_handoff+0x0/0xa60 took 55350 usecs1094worker2 # [ 0.362035] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1095client # [ 0.362817] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1096lb # [ 0.362876] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1097worker2 # [ 0.362041] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1098client # [ 0.362833] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1099lb # [ 0.362905] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1100client # [ 0.362863] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1101worker2 # [ 0.362046] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1102client # [ 0.362879] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1103worker2 # [ 0.362100] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1104lb # [ 0.382269] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1105client # [ 0.363333] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1106worker2 # [ 0.362544] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1107lb # [ 0.382456] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1108client # [ 0.363519] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1109worker2 # [ 0.362734] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1110lb # [ 0.382472] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1111client # [ 0.363535] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1112worker2 # [ 0.362752] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1113lb # [ 0.382501] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1114client # [ 0.363564] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1115worker2 # [ 0.362782] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1116lb # [ 0.382951] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1117client # [ 0.364026] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1118worker2 # [ 0.362799] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1119lb # [ 0.383132] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1120worker2 # [ 0.363261] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1121client # [ 0.383710] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1122lb # [ 0.383148] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1123worker2 # [ 0.363448] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1124client # [ 0.383729] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1125worker2 # [ 0.363464] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1126client # [ 0.383759] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1127worker2 # [ 0.363494] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1128client # [ 0.386354] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1129worker2 # [ 0.382670] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1130client # [ 0.386540] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1131worker2 # [ 0.382858] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1132client # [ 0.386556] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1133worker2 # [ 0.382875] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1134client # [ 0.386585] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1135worker2 # [ 0.382906] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1136client # [ 0.386603] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1137worker2 # [ 0.383369] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1138worker2 # [ 0.383553] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1139client # [ 0.387052] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1140worker2 # [ 0.383570] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1141client # [ 0.387238] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1142worker2 # [ 0.383600] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1143client # [ 0.387267] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1144worker2 # [ 0.383619] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1145client # [ 0.387748] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1146client # [ 0.387934] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1147client # [ 0.387964] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1148worker1 # [ 0.480771] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1149lb # [ 0.395227] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1150lb # [ 0.395258] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1151lb # [ 0.395759] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1152lb # [ 0.395962] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1153lb # [ 0.395992] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1154lb # [ 0.396451] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1155client # [ 0.400436] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1156lb # [ 0.396637] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1157client # [ 0.400632] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1158lb # [ 0.396667] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1159client # [ 0.400885] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1160lb # [ 0.397057] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1161client # [ 0.401073] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1162lb # [ 0.397237] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1163client # [ 0.401103] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1164worker2 # [ 0.396183] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1165lb # [ 0.397520] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1166client # [ 0.401550] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1167lb # [ 0.397706] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1168worker2 # [ 0.396390] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1169client # [ 0.401738] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1170worker2 # [ 0.396421] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1171lb # [ 0.397736] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1172worker1 # [ 0.490995] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1173client # [ 0.401768] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1174worker2 # [ 0.396913] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1175lb # [ 0.398185] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1176client # [ 0.402218] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1177worker2 # [ 0.397105] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1178lb # [ 0.398370] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1179client # [ 0.402405] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1180worker2 # [ 0.397135] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1181lb # [ 0.398400] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1182client # [ 0.402435] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1183worker2 # [ 0.397544] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1184lb # [ 0.398848] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1185client # [ 0.402888] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1186worker2 # [ 0.397729] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1187lb # [ 0.399034] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1188client # [ 0.403172] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1189client # [ 0.403190] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1190lb # [ 0.399064] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1191worker2 # [ 0.397977] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1192client # [ 0.403219] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1193worker2 # [ 0.398168] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1194lb # [ 0.399547] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1195lb # [ 0.399812] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1196worker2 # [ 0.398198] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1197client # [ 0.403677] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1198lb # [ 0.399839] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1199client # [ 0.403865] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1200worker2 # [ 0.398653] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1201client # [ 0.403881] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1202lb # [ 0.399868] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1203worker2 # [ 0.398846] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1204worker1 # [ 0.498721] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1205client # [ 0.403911] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1206lb # [ 0.400345] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1207worker2 # [ 0.398877] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1208worker1 # [ 0.500642] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1209lb # [ 0.400528] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1210worker1 # [ 0.502529] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1211worker2 # [ 0.399335] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1212lb # [ 0.400544] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1213worker2 # [ 0.399527] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1214lb # [ 0.400574] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1215worker2 # [ 0.399557] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1216lb # [ 0.401138] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1217worker2 # [ 0.400048] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1218lb # [ 0.401149] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1219worker2 # [ 0.400316] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1220worker2 # [ 0.400334] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1221lb # [ 0.401154] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1222worker2 # [ 0.400364] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1223lb # [ 0.401198] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1224worker2 # [ 0.400831] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1225lb # [ 0.401245] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1226worker2 # [ 0.401027] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1227lb # [ 0.401292] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1228worker2 # [ 0.401043] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1229client # [ 0.424575] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1230lb # [ 0.401339] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1231worker2 # [ 0.401076] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1232client # [ 0.424595] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1233worker2 # [ 0.401654] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1234lb # [ 0.401384] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1235worker2 # [ 0.401666] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1236client # [ 0.424600] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1237worker1 # [ 0.512318] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1238lb # [ 0.401429] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1239worker1 # [ 0.513993] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1240worker2 # [ 0.401672] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1241client # [ 0.424647] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1242lb # [ 0.401474] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1243worker2 # [ 0.401717] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1244client # [ 0.424696] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1245lb # [ 0.401520] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1246worker2 # [ 0.401764] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1247client # [ 0.424743] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1248lb # [ 0.401565] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1249worker2 # [ 0.401816] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1250client # [ 0.424792] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1251lb # [ 0.401637] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1252worker2 # [ 0.401866] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1253client # [ 0.424839] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1254lb # [ 0.401682] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1255worker2 # [ 0.401914] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1256lb # [ 0.401704] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1257client # [ 0.424888] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1258lb # [ 0.401725] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1259worker2 # [ 0.401965] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1260client # [ 0.424935] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1261lb # [ 0.401746] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1262worker2 # [ 0.402017] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1263client # [ 0.424982] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1264lb # [ 0.401767] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1265worker2 # [ 0.402064] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1266client # [ 0.425029] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1267lb # [ 0.401788] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1268worker2 # [ 0.402112] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1269client # [ 0.425134] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1270lb # [ 0.401810] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1271client # [ 0.425189] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1272worker2 # [ 0.402188] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1273lb # [ 0.401834] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1274client # [ 0.425210] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1275worker2 # [ 0.402238] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1276lb # [ 0.401855] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1277client # [ 0.425232] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1278lb # [ 0.401877] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1279worker2 # [ 0.402263] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1280worker1 # [ 0.524751] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1281client # [ 0.425253] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1282worker2 # [ 0.402285] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1283lb # [ 0.401899] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1284worker1 # [ 0.526516] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1285client # [ 0.425275] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1286worker2 # [ 0.402309] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1287lb # [ 0.401921] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1288client # [ 0.425296] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1289worker2 # [ 0.402332] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1290lb # [ 0.401943] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1291client # [ 0.425317] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1292lb # [ 0.401964] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1293worker2 # [ 0.402354] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1294client # [ 0.425343] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1295lb # [ 0.401985] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1296worker2 # [ 0.402376] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1297worker1 # [ 0.529811] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1298lb # [ 0.402006] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1299client # [ 0.425365] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1300worker2 # [ 0.402403] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1301lb # [ 0.402027] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1302client # [ 0.425392] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1303worker2 # [ 0.402426] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1304lb # [ 0.402048] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1305client # [ 0.425415] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1306worker2 # [ 0.402448] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1307lb # [ 0.402074] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1308client # [ 0.425437] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1309worker2 # [ 0.402471] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1310lb # [ 0.402083] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1311client # [ 0.425460] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1312worker2 # [ 0.402495] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1313lb # [ 0.402088] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1314client # [ 0.425481] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1315worker2 # [ 0.402518] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1316lb # [ 0.402894] pci 0000:00:07.0: enabling device (0000 -> 0002)1317client # [ 0.425502] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1318worker2 # [ 0.402539] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1319client # [ 0.425525] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1320worker2 # [ 0.402563] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1321client # [ 0.425549] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1322worker2 # [ 0.402585] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1323client # [ 0.425570] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1324worker2 # [ 0.402607] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1325client # [ 0.425598] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1326worker2 # [ 0.402630] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1327worker1 # [ 0.538088] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1328client # [ 0.425607] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1329worker2 # [ 0.402657] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1330client # [ 0.425612] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1331worker2 # [ 0.402666] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1332client # [ 0.426435] pci 0000:00:07.0: enabling device (0000 -> 0002)1333worker2 # [ 0.402671] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1334lb # [ 0.459283] pci 0000:00:07.0: quirk_usb_early_handoff+0x0/0xa60 took 55073 usecs1335worker2 # [ 0.403509] pci 0000:00:07.0: enabling device (0000 -> 0002)1336worker2 # [ 0.460007] pci 0000:00:07.0: quirk_usb_early_handoff+0x0/0xa60 took 55179 usecs1337worker1 # [ 0.550581] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1338worker1 # [ 0.553153] msm_serial: driver initialized1339worker1 # [ 0.553288] SuperH (H)SCI(F) driver initialized1340worker1 # [ 0.553342] STM32 USART driver initialized1341lb # [ 0.480400] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1342lb # [ 0.482481] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1343client # [ 0.482488] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1344worker2 # [ 0.480608] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1345worker2 # [ 0.482744] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1346lb # [ 0.493613] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1347client # [ 0.492667] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1348client # [ 0.496199] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1349worker2 # [ 0.493815] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1350client # [ 0.498215] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1351worker2 # [ 0.495770] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1352worker1 # [ 0.585472] loop: module loaded1353worker1 # [ 0.585644] virtio_blk virtio2: 1/0/0 default/read/poll queues1354worker1 # [ 0.586457] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1355lb # [ 0.503348] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1356lb # [ 0.506880] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1357lb # [ 0.509020] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1358lb # [ 0.510671] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1359worker2 # [ 0.505646] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1360client # [ 0.508233] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1361client # [ 0.510980] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1362worker2 # [ 0.512060] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1363worker2 # [ 0.514005] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1364worker1 # [ 0.599362] megasas: 07.734.00.00-rc11365worker2 # [ 0.515868] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1366worker1 # [ 0.600072] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1367worker1 # [ 0.602025] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001368worker1 # [ 0.602052] Intel/Sharp Extended Query Table at 0x00311369worker1 # [ 0.603606] Using buffer write method1370worker1 # [ 0.603668] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1371lb # [ 0.528615] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1372lb # [ 0.530592] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1373client # [ 0.520595] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1374worker1 # [ 0.605496] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001375worker1 # [ 0.605545] Intel/Sharp Extended Query Table at 0x00311376client # [ 0.522773] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1377client # [ 0.524633] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1378lb # [ 0.533978] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1379client # [ 0.526501] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1380worker2 # [ 0.529754] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1381lb # [ 0.537058] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1382worker2 # [ 0.531567] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1383worker1 # [ 0.623545] Using buffer write method1384worker1 # [ 0.623591] Concatenating MTD devices:1385worker1 # [ 0.623595] (0): "0.flash"1386worker1 # [ 0.623605] (1): "0.flash"1387client # [ 0.536604] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1388worker1 # [ 0.623608] into device "0.flash"1389worker2 # [ 0.535519] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1390lb # [ 0.550424] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1391client # [ 0.553596] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1392lb # [ 0.556204] msm_serial: driver initialized1393lb # [ 0.556406] SuperH (H)SCI(F) driver initialized1394lb # [ 0.556462] STM32 USART driver initialized1395client # [ 0.556185] msm_serial: driver initialized1396client # [ 0.556336] SuperH (H)SCI(F) driver initialized1397client # [ 0.556387] STM32 USART driver initialized1398worker2 # [ 0.556208] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1399worker2 # [ 0.557991] msm_serial: driver initialized1400worker2 # [ 0.558118] SuperH (H)SCI(F) driver initialized1401worker2 # [ 0.558171] STM32 USART driver initialized1402lb # [ 0.588892] loop: module loaded1403client # [ 0.585825] loop: module loaded1404lb # [ 0.589071] virtio_blk virtio2: 1/0/0 default/read/poll queues1405client # [ 0.586000] virtio_blk virtio2: 1/0/0 default/read/poll queues1406lb # [ 0.589824] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1407client # [ 0.586739] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1408worker2 # [ 0.586594] loop: module loaded1409worker2 # [ 0.586778] virtio_blk virtio2: 1/0/0 default/read/poll queues1410worker2 # [ 0.587536] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1411client # [ 0.592664] megasas: 07.734.00.00-rc11412lb # [ 0.595800] megasas: 07.734.00.00-rc11413client # [ 0.593365] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1414lb # [ 0.596588] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1415client # [ 0.595251] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001416client # [ 0.595276] Intel/Sharp Extended Query Table at 0x00311417worker2 # [ 0.592464] megasas: 07.734.00.00-rc11418worker2 # [ 0.593124] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1419lb # [ 0.608243] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001420client # [ 0.604876] Using buffer write method1421lb # [ 0.608276] Intel/Sharp Extended Query Table at 0x00311422lb # [ 0.609843] Using buffer write method1423client # [ 0.604946] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1424lb # [ 0.609925] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1425client # [ 0.606477] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001426client # [ 0.606523] Intel/Sharp Extended Query Table at 0x00311427worker2 # [ 0.604906] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001428worker2 # [ 0.604967] Intel/Sharp Extended Query Table at 0x00311429worker2 # [ 0.606551] Using buffer write method1430worker2 # [ 0.606621] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1431client # [ 0.620079] Using buffer write method1432client # [ 0.620107] Concatenating MTD devices:1433client # [ 0.620111] (0): "0.flash"1434client # [ 0.620115] (1): "0.flash"1435client # [ 0.620122] into device "0.flash"1436lb # [ 0.623792] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001437lb # [ 0.623865] Intel/Sharp Extended Query Table at 0x00311438lb # [ 0.625428] Using buffer write method1439lb # [ 0.625456] Concatenating MTD devices:1440lb # [ 0.625460] (0): "0.flash"1441lb # [ 0.625472] (1): "0.flash"1442lb # [ 0.625476] into device "0.flash"1443worker2 # [ 0.621811] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001444worker2 # [ 0.621844] Intel/Sharp Extended Query Table at 0x00311445worker2 # [ 0.623392] Using buffer write method1446worker2 # [ 0.623420] Concatenating MTD devices:1447worker2 # [ 0.623424] (0): "0.flash"1448worker2 # [ 0.623435] (1): "0.flash"1449worker2 # [ 0.623439] into device "0.flash"1450worker1 # [ 0.864288] Freeing initrd memory: 26976K1451worker1 # [ 0.870570] tun: Universal TUN/TAP device driver, 1.61452worker1 # [ 0.874375] thunder_xcv, ver 1.01453worker1 # [ 0.874427] thunder_bgx, ver 1.01454worker1 # [ 0.874449] nicpf, ver 1.01455worker1 # [ 0.876266] e1000: Intel(R) PRO/1000 Network Driver1456worker1 # [ 0.876279] e1000: Copyright (c) 1999-2006 Intel Corporation.1457worker1 # [ 0.876304] e1000e: Intel(R) PRO/1000 Network Driver1458worker1 # [ 0.876313] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1459worker1 # [ 0.876342] igb: Intel(R) Gigabit Ethernet Network Driver1460worker1 # [ 0.876348] igb: Copyright (c) 2007-2014 Intel Corporation.1461worker1 # [ 0.876371] igbvf: Intel(R) Gigabit Virtual Function Network Driver1462worker1 # [ 0.876377] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1463worker1 # [ 0.876509] sky2: driver version 1.301464worker1 # [ 0.878105] usbcore: registered new interface driver usb-storage1465worker1 # [ 0.878240] usbcore: registered new interface driver usbserial_generic1466worker1 # [ 0.878257] usbserial: USB Serial support registered for generic1467worker1 # [ 0.879458] ehci-pci 0000:00:07.0: EHCI Host Controller1468worker1 # [ 0.879490] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11469worker1 # [ 0.879776] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001470worker1 # [ 0.890687] hv_vmbus: registering driver hyperv_keyboard1471worker1 # [ 0.891447] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001472worker1 # [ 0.891760] hub 1-0:1.0: USB hub found1473worker1 # [ 0.891782] hub 1-0:1.0: 6 ports detected1474worker1 # [ 0.894659] rtc-pl031 9010000.pl031: registered as rtc01475worker1 # [ 0.894688] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T15:59:30 UTC (1790179170)1476worker1 # [ 0.896737] i2c_dev: i2c /dev entries driver1477worker1 # [ 0.900368] sdhci: Secure Digital Host Controller Interface driver1478worker1 # [ 0.900381] sdhci: Copyright(c) Pierre Ossman1479worker1 # [ 0.900660] Synopsys Designware Multimedia Card Interface Driver1480worker1 # [ 0.901022] sdhci-pltfm: SDHCI platform and OF driver helper1481worker1 # [ 0.902543] hid: raw HID events driver (C) Jiri Kosina1482worker1 # [ 0.906094] usbcore: registered new interface driver usbhid1483worker1 # [ 0.906104] usbhid: USB HID core driver1484client # [ 0.859888] Freeing initrd memory: 26972K1485worker1 # [ 0.941076] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1486worker1 # [ 0.943414] drop_monitor: Initializing network drop monitor service1487worker1 # [ 0.943661] NET: Registered PF_INET6 protocol family1488worker1 # [ 0.946979] Segment Routing with IPv61489worker1 # [ 0.947009] In-situ OAM (IOAM) with IPv61490worker1 # [ 0.947038] NET: Registered PF_PACKET protocol family1491worker1 # [ 0.948665] 9pnet: Installing 9P2000 support1492client # [ 0.866005] tun: Universal TUN/TAP device driver, 1.61493worker1 # [ 0.948721] Key type dns_resolver registered1494lb # [ 0.869833] Freeing initrd memory: 26948K1495client # [ 0.869894] thunder_xcv, ver 1.01496worker2 # [ 0.865022] Freeing initrd memory: 26944K1497client # [ 0.869940] thunder_bgx, ver 1.01498lb # [ 0.876088] tun: Universal TUN/TAP device driver, 1.61499client # [ 0.869962] nicpf, ver 1.01500client # [ 0.870504] e1000: Intel(R) PRO/1000 Network Driver1501client # [ 0.870511] e1000: Copyright (c) 1999-2006 Intel Corporation.1502worker1 # [ 0.955509] registered taskstats version 11503client # [ 0.870535] e1000e: Intel(R) PRO/1000 Network Driver1504worker1 # [ 0.955654] Loading compiled-in X.509 certificates1505client # [ 0.870543] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1506worker2 # [ 0.871023] tun: Universal TUN/TAP device driver, 1.61507client # [ 0.870574] igb: Intel(R) Gigabit Ethernet Network Driver1508client # [ 0.870580] igb: Copyright (c) 2007-2014 Intel Corporation.1509client # [ 0.870603] igbvf: Intel(R) Gigabit Virtual Function Network Driver1510client # [ 0.870608] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1511client # [ 0.870739] sky2: driver version 1.301512worker2 # [ 0.874736] thunder_xcv, ver 1.01513lb # [ 0.879683] thunder_xcv, ver 1.01514client # [ 0.879770] ehci-pci 0000:00:07.0: EHCI Host Controller1515worker2 # [ 0.874779] thunder_bgx, ver 1.01516worker1 # [ 0.964334] Demotion targets for Node 0: null1517lb # [ 0.879726] thunder_bgx, ver 1.01518worker1 # [ 0.964449] Key type .fscrypt registered1519worker2 # [ 0.874801] nicpf, ver 1.01520lb # [ 0.879750] nicpf, ver 1.01521client # [ 0.879801] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11522worker2 # [ 0.875339] e1000: Intel(R) PRO/1000 Network Driver1523worker1 # [ 0.964456] Key type fscrypt-provisioning registered1524lb # [ 0.880360] e1000: Intel(R) PRO/1000 Network Driver1525client # [ 0.880129] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001526worker1 # [ 0.964557] ima: No TPM chip found, activating TPM-bypass!1527worker2 # [ 0.875346] e1000: Copyright (c) 1999-2006 Intel Corporation.1528lb # [ 0.880367] e1000: Copyright (c) 1999-2006 Intel Corporation.1529worker1 # [ 0.964587] ima: Allocated hash algorithm: sha11530worker2 # [ 0.875369] e1000e: Intel(R) PRO/1000 Network Driver1531lb # [ 0.880391] e1000e: Intel(R) PRO/1000 Network Driver1532worker1 # [ 0.964609] ima: No architecture policies found1533worker2 # [ 0.875377] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1534lb # [ 0.880399] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1535worker2 # [ 0.875406] igb: Intel(R) Gigabit Ethernet Network Driver1536lb # [ 0.880445] igb: Intel(R) Gigabit Ethernet Network Driver1537client # [ 0.882707] usbcore: registered new interface driver usb-storage1538worker1 # [ 0.968549] input: gpio-keys as /devices/platform/gpio-keys/input/input01539worker2 # [ 0.875411] igb: Copyright (c) 2007-2014 Intel Corporation.1540lb # [ 0.880451] igb: Copyright (c) 2007-2014 Intel Corporation.1541client # [ 0.882860] usbcore: registered new interface driver usbserial_generic1542worker2 # [ 0.875450] igbvf: Intel(R) Gigabit Virtual Function Network Driver1543lb # [ 0.880476] igbvf: Intel(R) Gigabit Virtual Function Network Driver1544client # [ 0.882874] usbserial: USB Serial support registered for generic1545worker2 # [ 0.875456] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1546lb # [ 0.880482] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1547client # [ 0.883461] hv_vmbus: registering driver hyperv_keyboard1548worker2 # [ 0.875589] sky2: driver version 1.301549lb # [ 0.880620] sky2: driver version 1.301550lb # [ 0.882230] usbcore: registered new interface driver usb-storage1551client # [ 0.887089] rtc-pl031 9010000.pl031: registered as rtc01552lb # [ 0.882326] usbcore: registered new interface driver usbserial_generic1553lb # [ 0.882342] usbserial: USB Serial support registered for generic1554client # [ 0.887115] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T15:59:30 UTC (1790179170)1555worker2 # [ 0.884573] ehci-pci 0000:00:07.0: EHCI Host Controller1556lb # [ 0.882953] hv_vmbus: registering driver hyperv_keyboard1557client # [ 0.887419] i2c_dev: i2c /dev entries driver1558worker2 # [ 0.884603] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11559lb # [ 0.883880] ehci-pci 0000:00:07.0: EHCI Host Controller1560worker2 # [ 0.884877] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001561lb # [ 0.883905] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11562lb # [ 0.884105] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001563worker2 # [ 0.887427] usbcore: registered new interface driver usb-storage1564worker2 # [ 0.887505] usbcore: registered new interface driver usbserial_generic1565lb # [ 0.896291] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001566lb # [ 0.896605] hub 1-0:1.0: USB hub found1567worker2 # [ 0.887518] usbserial: USB Serial support registered for generic1568lb # [ 0.896626] hub 1-0:1.0: 6 ports detected1569worker2 # [ 0.890394] hv_vmbus: registering driver hyperv_keyboard1570client # [ 0.892795] sdhci: Secure Digital Host Controller Interface driver1571client # [ 0.892808] sdhci: Copyright(c) Pierre Ossman1572lb # [ 0.899395] rtc-pl031 9010000.pl031: registered as rtc01573worker2 # [ 0.891956] rtc-pl031 9010000.pl031: registered as rtc01574client # [ 0.893065] Synopsys Designware Multimedia Card Interface Driver1575lb # [ 0.899421] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T15:59:30 UTC (1790179170)1576client # [ 0.893437] sdhci-pltfm: SDHCI platform and OF driver helper1577lb # [ 0.899728] i2c_dev: i2c /dev entries driver1578worker2 # [ 0.891981] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T15:59:30 UTC (1790179170)1579client # [ 0.894913] hid: raw HID events driver (C) Jiri Kosina1580worker2 # [ 0.892280] i2c_dev: i2c /dev entries driver1581client # [ 0.895153] usbcore: registered new interface driver usbhid1582client # [ 0.895159] usbhid: USB HID core driver1583client # [ 0.900503] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001584client # [ 0.900807] hub 1-0:1.0: USB hub found1585worker1 # [ 0.986664] clk: Disabling unused clocks1586client # [ 0.900827] hub 1-0:1.0: 6 ports detected1587lb # [ 0.905071] sdhci: Secure Digital Host Controller Interface driver1588lb # [ 0.905083] sdhci: Copyright(c) Pierre Ossman1589client # [ 0.903056] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1590worker1 # [ 0.987258] PM: genpd: Disabling unused power domains1591lb # [ 0.905358] Synopsys Designware Multimedia Card Interface Driver1592worker2 # [ 0.897593] sdhci: Secure Digital Host Controller Interface driver1593lb # [ 0.905749] sdhci-pltfm: SDHCI platform and OF driver helper1594worker2 # [ 0.897606] sdhci: Copyright(c) Pierre Ossman1595client # [ 0.905680] drop_monitor: Initializing network drop monitor service1596worker2 # [ 0.897867] Synopsys Designware Multimedia Card Interface Driver1597client # [ 0.905865] NET: Registered PF_INET6 protocol family1598worker1 # [ 0.990909] Freeing unused kernel memory: 4736K1599worker2 # [ 0.898234] sdhci-pltfm: SDHCI platform and OF driver helper1600worker1 # [ 0.991124] Run /init as init process1601lb # [ 0.910011] hid: raw HID events driver (C) Jiri Kosina1602worker2 # [ 0.899762] hid: raw HID events driver (C) Jiri Kosina1603client # [ 0.907762] Segment Routing with IPv61604lb # [ 0.910291] usbcore: registered new interface driver usbhid1605worker2 # [ 0.899914] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001606client # [ 0.907780] In-situ OAM (IOAM) with IPv61607lb # [ 0.910298] usbhid: USB HID core driver1608worker2 # [ 0.900214] hub 1-0:1.0: USB hub found1609client # [ 0.907808] NET: Registered PF_PACKET protocol family1610worker2 # [ 0.900237] hub 1-0:1.0: 6 ports detected1611worker2 # [ 0.905669] usbcore: registered new interface driver usbhid1612worker2 # [ 0.905680] usbhid: USB HID core driver1613lb # [ 0.913154] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1614client # [ 0.909462] 9pnet: Installing 9P2000 support1615client # [ 0.909513] Key type dns_resolver registered1616lb # [ 0.914750] drop_monitor: Initializing network drop monitor service1617lb # [ 0.914925] NET: Registered PF_INET6 protocol family1618lb # [ 0.918173] Segment Routing with IPv61619worker2 # [ 0.907982] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1620lb # [ 0.918194] In-situ OAM (IOAM) with IPv61621worker2 # [ 0.909494] drop_monitor: Initializing network drop monitor service1622lb # [ 0.918223] NET: Registered PF_PACKET protocol family1623worker2 # [ 0.909643] NET: Registered PF_INET6 protocol family1624worker2 # [ 0.912718] Segment Routing with IPv61625worker2 # [ 0.912736] In-situ OAM (IOAM) with IPv61626worker2 # [ 0.912764] NET: Registered PF_PACKET protocol family1627lb # [ 0.920091] 9pnet: Installing 9P2000 support1628lb # [ 0.920138] Key type dns_resolver registered1629client # [ 0.916096] registered taskstats version 11630worker2 # [ 0.914464] 9pnet: Installing 9P2000 support1631client # [ 0.916254] Loading compiled-in X.509 certificates1632worker2 # [ 0.914511] Key type dns_resolver registered1633lb # [ 0.927011] registered taskstats version 11634lb # [ 0.927153] Loading compiled-in X.509 certificates1635worker1 # [ 1.009045] systemd[1]: Successfully made /usr/ read-only.1636client # [ 0.924966] Demotion targets for Node 0: null1637client # [ 0.925072] Key type .fscrypt registered1638client # [ 0.925078] Key type fscrypt-provisioning registered1639client # [ 0.925173] ima: No TPM chip found, activating TPM-bypass!1640client # [ 0.925192] ima: Allocated hash algorithm: sha11641client # [ 0.925213] ima: No architecture policies found1642worker2 # [ 0.921129] registered taskstats version 11643worker2 # [ 0.921285] Loading compiled-in X.509 certificates1644client # [ 0.929453] input: gpio-keys as /devices/platform/gpio-keys/input/input01645lb # [ 0.935826] Demotion targets for Node 0: null1646lb # [ 0.935930] Key type .fscrypt registered1647lb # [ 0.935937] Key type fscrypt-provisioning registered1648lb # [ 0.936035] ima: No TPM chip found, activating TPM-bypass!1649lb # [ 0.936055] ima: Allocated hash algorithm: sha11650worker2 # [ 0.930273] Demotion targets for Node 0: null1651lb # [ 0.936077] ima: No architecture policies found1652worker2 # [ 0.930374] Key type .fscrypt registered1653worker2 # [ 0.930381] Key type fscrypt-provisioning registered1654worker2 # [ 0.930474] ima: No TPM chip found, activating TPM-bypass!1655lb # [ 0.940392] input: gpio-keys as /devices/platform/gpio-keys/input/input01656worker2 # [ 0.930493] ima: Allocated hash algorithm: sha11657worker2 # [ 0.930515] ima: No architecture policies found1658worker2 # [ 0.934599] input: gpio-keys as /devices/platform/gpio-keys/input/input01659client # [ 0.947666] clk: Disabling unused clocks1660client # [ 0.947697] PM: genpd: Disabling unused power domains1661client # [ 0.951848] Freeing unused kernel memory: 4736K1662client # [ 0.952657] Run /init as init process1663lb # [ 0.958798] clk: Disabling unused clocks1664lb # [ 0.958830] PM: genpd: Disabling unused power domains1665worker2 # [ 0.952549] clk: Disabling unused clocks1666lb # [ 0.963064] Freeing unused kernel memory: 4736K1667worker2 # [ 0.952581] PM: genpd: Disabling unused power domains1668lb # [ 0.963879] Run /init as init process1669worker2 # [ 0.956688] Freeing unused kernel memory: 4736K1670worker2 # [ 0.956874] Run /init as init process1671client # [ 0.966279] systemd[1]: Successfully made /usr/ read-only.1672lb # [ 0.977374] systemd[1]: Successfully made /usr/ read-only.1673worker2 # [ 0.974079] systemd[1]: Successfully made /usr/ read-only.1674worker1 # [ 1.162763] usb 1-1: new high-speed USB device number 2 using ehci-pci1675lb # [ 1.143280] usb 1-1: new high-speed USB device number 2 using ehci-pci1676client # [ 1.148114] usb 1-1: new high-speed USB device number 2 using ehci-pci1677worker2 # [ 1.147958] usb 1-1: new high-speed USB device number 2 using ehci-pci1678worker1 # [ 1.315313] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11679worker1 # [ 1.343935] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1680worker1 # [ 1.355945] systemd[1]: Detected virtualization qemu.1681worker1 # [ 1.358002] systemd[1]: Detected architecture arm64.1682worker1 # [ 1.359950] systemd[1]: Running in initrd.1683worker1 # [ 1.362478] systemd[1]: Initializing machine ID from random generator.1684worker1 # [ 1.365632] systemd[1]: Hostname set to <worker1>.1685lb # [ 1.297854] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11686worker2 # [ 1.300561] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11687client # [ 1.301088] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1688client # [ 1.314036] systemd[1]: Detected virtualization qemu.1689client # [ 1.316371] systemd[1]: Detected architecture arm64.1690lb # [ 1.312207] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1691lb # [ 1.324467] systemd[1]: Detected virtualization qemu.1692worker1 # [ 1.403078] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01693lb # [ 1.326539] systemd[1]: Detected architecture arm64.1694client # [ 1.319281] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11695worker2 # [ 1.308859] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1696lb # [ 1.328632] systemd[1]: Running in initrd.1697client # [ 1.325273] systemd[1]: Running in initrd.1698worker2 # [ 1.321279] systemd[1]: Detected virtualization qemu.1699worker2 # [ 1.323367] systemd[1]: Detected architecture arm64.1700lb # [ 1.331162] systemd[1]: Initializing machine ID from random generator.1701client # [ 1.327884] systemd[1]: Initializing machine ID from random generator.1702worker2 # [ 1.325373] systemd[1]: Running in initrd.1703lb # [ 1.333995] systemd[1]: Hostname set to <lb>.1704client # [ 1.330989] systemd[1]: Hostname set to <client>.1705worker2 # [ 1.328055] systemd[1]: Initializing machine ID from random generator.1706worker2 # [ 1.331056] systemd[1]: Hostname set to <worker2>.1707lb # [ 1.383488] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01708worker2 # [ 1.380156] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01709client # [ 1.400318] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01710worker1 # [ 1.526767] usb 1-2: new high-speed USB device number 3 using ehci-pci1711lb # [ 1.507264] usb 1-2: new high-speed USB device number 3 using ehci-pci1712worker2 # [ 1.503953] usb 1-2: new high-speed USB device number 3 using ehci-pci1713client # [ 1.520104] usb 1-2: new high-speed USB device number 3 using ehci-pci1714worker1 # [ 1.677916] systemd[1]: bpf-restrict-fs: LSM BPF program attached1715worker1 # [ 1.695124] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21716worker1 # [ 1.700774] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01717client # [ 1.642243] systemd[1]: bpf-restrict-fs: LSM BPF program attached1718lb # [ 1.646517] systemd[1]: bpf-restrict-fs: LSM BPF program attached1719worker2 # [ 1.665573] systemd[1]: bpf-restrict-fs: LSM BPF program attached1720lb # [ 1.671431] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21721worker2 # [ 1.668925] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21722lb # [ 1.679419] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01723client # [ 1.678629] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21724worker2 # [ 1.669349] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01725client # [ 1.684166] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01726worker1 # [ 1.791585] systemd[1]: Queued start job for default target Initrd Default Target.1727worker1 # [ 1.801067] systemd[1]: Created slice Slice /system/modprobe.1728worker1 # [ 1.802214] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1729worker1 # [ 1.803556] systemd[1]: Expecting device /dev/disk/by-label/nixos...1730worker1 # [ 1.804578] systemd[1]: Reached target Path Units.1731worker1 # [ 1.805357] systemd[1]: Reached target Slice Units.1732worker1 # [ 1.806149] systemd[1]: Reached target Swaps.1733worker1 # [ 1.806933] systemd[1]: Reached target Timer Units.1734worker1 # [ 1.807934] systemd[1]: Listening on D-Bus System Message Bus Socket.1735worker1 # [ 1.809118] systemd[1]: Listening on Journal Socket (/dev/log).1736worker1 # [ 1.810206] systemd[1]: Listening on Journal Sockets.1737worker1 # [ 1.811202] systemd[1]: Listening on udev Control Socket.1738worker1 # [ 1.812188] systemd[1]: Listening on udev Kernel Socket.1739worker1 # [ 1.813061] systemd[1]: Reached target Socket Units.1740worker1 # [ 1.815586] systemd[1]: Starting Create List of Static Device Nodes...1741worker1 # [ 1.816713] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1742worker1 # [ 1.824087] systemd[1]: Mounting Kernel Configuration File System...1743worker1 # [ 1.834995] systemd[1]: Starting Journal Service...1744client # [ 1.752896] systemd[1]: Queued start job for default target Initrd Default Target.1745lb # [ 1.759830] systemd[1]: Queued start job for default target Initrd Default Target.1746client # [ 1.762470] systemd[1]: Created slice Slice /system/modprobe.1747client # [ 1.763660] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1748client # [ 1.765089] systemd[1]: Expecting device /dev/disk/by-label/nixos...1749lb # [ 1.769535] systemd[1]: Created slice Slice /system/modprobe.1750client # [ 1.766147] systemd[1]: Reached target Path Units.1751client # [ 1.766959] systemd[1]: Reached target Slice Units.1752lb # [ 1.770737] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1753client # [ 1.767798] systemd[1]: Reached target Swaps.1754lb # [ 1.772211] systemd[1]: Expecting device /dev/disk/by-label/nixos...1755client # [ 1.768623] systemd[1]: Reached target Timer Units.1756lb # [ 1.773286] systemd[1]: Reached target Path Units.1757client # [ 1.769641] systemd[1]: Listening on D-Bus System Message Bus Socket.1758lb # [ 1.774111] systemd[1]: Reached target Slice Units.1759worker2 # [ 1.766052] systemd[1]: Queued start job for default target Initrd Default Target.1760client # [ 1.770872] systemd[1]: Listening on Journal Socket (/dev/log).1761lb # [ 1.774937] systemd[1]: Reached target Swaps.1762lb # [ 1.775753] systemd[1]: Reached target Timer Units.1763client # [ 1.772015] systemd[1]: Listening on Journal Sockets.1764lb # [ 1.776784] systemd[1]: Listening on D-Bus System Message Bus Socket.1765lb # [ 1.778110] systemd[1]: Listening on Journal Socket (/dev/log).1766worker1 # [ 1.859082] systemd[1]: Starting Load Kernel Modules...1767client # [ 1.773098] systemd[1]: Listening on udev Control Socket.1768client # [ 1.773233] systemd[1]: Listening on udev Kernel Socket.1769lb # [ 1.779274] systemd[1]: Listening on Journal Sockets.1770worker1 # [ 1.860032] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1771client # [ 1.773257] systemd[1]: Reached target Socket Units.1772lb # [ 1.780287] systemd[1]: Listening on udev Control Socket.1773lb # [ 1.781329] systemd[1]: Listening on udev Kernel Socket.1774client # [ 1.777459] systemd[1]: Starting Create List of Static Device Nodes...1775lb # [ 1.782244] systemd[1]: Reached target Socket Units.1776worker2 # [ 1.774323] systemd[1]: Created slice Slice /system/modprobe.1777client # [ 1.778604] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1778worker2 # [ 1.775494] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1779lb # [ 1.784740] systemd[1]: Starting Create List of Static Device Nodes...1780worker2 # [ 1.776906] systemd[1]: Expecting device /dev/disk/by-label/nixos...1781worker2 # [ 1.777928] systemd[1]: Reached target Path Units.1782worker2 # [ 1.778728] systemd[1]: Reached target Slice Units.1783lb # [ 1.785916] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1784worker2 # [ 1.779545] systemd[1]: Reached target Swaps.1785worker2 # [ 1.780324] systemd[1]: Reached target Timer Units.1786worker2 # [ 1.781317] systemd[1]: Listening on D-Bus System Message Bus Socket.1787worker2 # [ 1.782619] systemd[1]: Listening on Journal Socket (/dev/log).1788worker2 # [ 1.783707] systemd[1]: Listening on Journal Sockets.1789client # [ 1.788221] systemd[1]: Mounting Kernel Configuration File System...1790worker2 # [ 1.784753] systemd[1]: Listening on udev Control Socket.1791worker2 # [ 1.785745] systemd[1]: Listening on udev Kernel Socket.1792worker2 # [ 1.786620] systemd[1]: Reached target Socket Units.1793worker2 # [ 1.789080] systemd[1]: Starting Create List of Static Device Nodes...1794lb # [ 1.795379] systemd[1]: Mounting Kernel Configuration File System...1795worker2 # [ 1.790227] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1796client # [ 1.796350] systemd[1]: Starting Journal Service...1797lb # [ 1.803455] systemd[1]: Starting Journal Service...1798worker1 # [ 1.883095] systemd[1]: Starting Coldplug All udev Devices...1799worker2 # [ 1.800093] systemd[1]: Mounting Kernel Configuration File System...1800worker1 # [ 1.893972] systemd-journald[72]: Collecting audit messages is disabled.1801worker2 # [ 1.808192] systemd[1]: Starting Journal Service...1802client # [ 1.820424] systemd[1]: Starting Load Kernel Modules...1803lb # [ 1.828531] systemd[1]: Starting Load Kernel Modules...1804client # [ 1.824316] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1805worker1 # [ 1.906810] systemd[1]: Finished Create List of Static Device Nodes.1806lb # [ 1.829489] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1807worker1 # [ 1.911950] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1808worker1 # [ 1.912224] systemd[1]: Mounted Kernel Configuration File System.1809client # [ 1.836230] systemd[1]: Starting Coldplug All udev Devices...1810worker2 # [ 1.832296] systemd[1]: Starting Load Kernel Modules...1811worker2 # [ 1.833243] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1812lb # [ 1.847261] systemd[1]: Starting Coldplug All udev Devices...1813client # [ 1.845854] systemd-journald[72]: Collecting audit messages is disabled.1814client # [ 1.852217] systemd[1]: Finished Create List of Static Device Nodes.1815client # [ 1.853255] systemd[1]: Mounted Kernel Configuration File System.1816lb # [ 1.861700] systemd-journald[72]: Collecting audit messages is disabled.1817worker2 # [ 1.855436] systemd-journald[72]: Collecting audit messages is disabled.1818worker1 # [ 1.944883] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1819worker2 # [ 1.856597] systemd[1]: Starting Coldplug All udev Devices...1820client # [ 1.864310] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1821lb # [ 1.875422] systemd[1]: Finished Create List of Static Device Nodes.1822lb # [ 1.876116] systemd[1]: Mounted Kernel Configuration File System.1823lb # [ 1.878903] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1824worker1 # [ 1.959374] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1825worker1 # [ 1.961648] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1826worker2 # [ 1.872040] systemd[1]: Finished Create List of Static Device Nodes.1827worker2 # [ 1.872742] systemd[1]: Mounted Kernel Configuration File System.1828worker1 # [ 1.964160] systemd[1]: Starting Create Static Device Nodes in /dev...1829worker1 # [ 1.972046] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01830worker1 # [ 1.972285] [drm] features: -virgl +edid -resource_blob -host_visible1831worker1 # [ 1.972295] [drm] features: -context_init1832worker1 # [ 1.973008] [drm] number of scanouts: 11833worker1 # [ 1.973026] [drm] number of cap sets: 01834worker2 # [ 1.884270] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1835client # [ 1.904685] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1836lb # [ 1.910016] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1837client # [ 1.908427] systemd[1]: Starting Create Static Device Nodes in /dev...1838lb # [ 1.923226] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1839worker1 # [ 1.999092] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1840worker1 # [ 1.999111] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01841client # [ 1.922957] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1842lb # [ 1.932718] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01843worker2 # [ 1.924591] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1844lb # [ 1.932956] [drm] features: -virgl +edid -resource_blob -host_visible1845worker2 # [ 1.927107] systemd[1]: Starting Create Static Device Nodes in /dev...1846lb # [ 1.932966] [drm] features: -context_init1847lb # [ 1.933690] [drm] number of scanouts: 11848lb # [ 1.933708] [drm] number of cap sets: 01849worker1 # [ 2.023316] systemd[1]: Finished Create Static Device Nodes in /dev.1850worker2 # [ 1.935027] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1851client # [ 1.940098] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1852worker1 # [ 2.023513] systemd[1]: Reached target Preparation for Local File Systems.1853worker1 # [ 2.023539] systemd[1]: Reached target Local File Systems.1854lb # [ 1.947839] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1855worker1 # [ 2.027474] systemd[1]: Starting Rule-based Manager for Device Events and Files...1856lb # [ 1.950340] systemd[1]: Starting Create Static Device Nodes in /dev...1857client # [ 1.953538] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01858worker1 # [ 2.031974] Console: switching to colour frame buffer device 160x501859client # [ 1.953779] [drm] features: -virgl +edid -resource_blob -host_visible1860worker1 # [ 2.038602] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1861client # [ 1.953789] [drm] features: -context_init1862client # [ 1.954503] [drm] number of scanouts: 11863client # [ 1.954521] [drm] number of cap sets: 01864lb # [ 1.959659] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1865lb # [ 1.959677] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01866worker2 # [ 1.951999] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1867worker2 # [ 1.958141] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01868worker2 # [ 1.958373] [drm] features: -virgl +edid -resource_blob -host_visible1869worker2 # [ 1.958383] [drm] features: -context_init1870worker2 # [ 1.959070] [drm] number of scanouts: 11871worker2 # [ 1.959088] [drm] number of cap sets: 01872client # [ 1.972679] systemd[1]: Finished Create Static Device Nodes in /dev.1873client # [ 1.972877] systemd[1]: Reached target Preparation for Local File Systems.1874client # [ 1.972902] systemd[1]: Reached target Local File Systems.1875client # [ 1.976791] systemd[1]: Starting Rule-based Manager for Device Events and Files...1876worker2 # [ 1.976489] systemd[1]: Finished Create Static Device Nodes in /dev.1877worker2 # [ 1.976822] systemd[1]: Reached target Preparation for Local File Systems.1878worker2 # [ 1.976851] systemd[1]: Reached target Local File Systems.1879client # [ 1.981429] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1880worker1 # [ 2.066971] systemd[1]: Finished Load Kernel Modules.1881client # [ 1.981447] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01882worker1 # [ 2.070228] systemd[1]: Starting Apply Kernel Variables...1883worker2 # [ 1.984260] systemd[1]: Starting Rule-based Manager for Device Events and Files...1884worker1 # [ 2.077538] systemd[1]: Started Journal Service.1885worker2 # [ 1.989308] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1886worker2 # [ 1.989327] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01887lb # [ 1.996176] Console: switching to colour frame buffer device 160x501888lb # [ 2.002967] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1889worker1 # [ 2.072329] systemd-modules-load[74]: Inserted module 'dm_mod'1890worker1 # [ 2.073639] systemd-modules-load[74]: Module 'virtio_balloon' is built in1891lb # [ 2.006523] systemd[1]: Finished Create Static Device Nodes in /dev.1892lb # [ 2.006891] systemd[1]: Reached target Preparation for Local File Systems.1893lb # [ 2.006926] systemd[1]: Reached target Local File Systems.1894worker1 # [ 2.084316] systemd-modules-load[74]: Module 'virtio_console' is built in1895lb # [ 2.019919] systemd[1]: Starting Rule-based Manager for Device Events and Files...1896worker2 # [ 2.008338] Console: switching to colour frame buffer device 160x501897client # [ 2.011382] Console: switching to colour frame buffer device 160x501898client # [ 2.017864] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1899client # [ 2.020936] systemd[1]: Started Journal Service.1900worker1 # [ 2.093098] systemd-modules-load[74]: Inserted module 'virtio_gpu'1901client # [ 2.007984] systemd-modules-load[74]: Inserted module 'dm_mod'1902worker1 # [ 2.094196] systemd-modules-load[74]: Module 'virtio_rng' is built in1903client # [ 2.009355] systemd-modules-load[74]: Module 'virtio_balloon' is built in1904client # [ 2.010757] systemd-modules-load[74]: Module 'virtio_console' is built in1905worker1 # [ 2.100298] systemd[1]: Starting Create System Files and Directories...1906client # [ 2.024131] systemd[1]: Starting Create System Files and Directories...1907worker2 # [ 2.035930] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1908worker2 # [ 2.040070] systemd[1]: Started Journal Service.1909lb # [ 2.047417] systemd[1]: Finished Load Kernel Modules.1910worker1 # [ 2.117389] systemd[1]: Finished Apply Kernel Variables.1911client # [ 2.032337] systemd-modules-load[74]: Inserted module 'virtio_gpu'1912client # [ 2.033302] systemd-modules-load[74]: Module 'virtio_rng' is built in1913lb # [ 2.050818] systemd[1]: Starting Apply Kernel Variables...1914lb # [ 2.044445] systemd-modules-load[74]: Inserted module 'dm_mod'1915lb # [ 2.045725] systemd-modules-load[74]: Module 'virtio_balloon' is built in1916client # [ 2.041334] systemd[1]: Finished Load Kernel Modules.1917lb # [ 2.046792] systemd-modules-load[74]: Module 'virtio_console' is built in1918worker2 # [ 2.037935] systemd-modules-load[74]: Inserted module 'dm_mod'1919lb # [ 2.047850] systemd-modules-load[74]: Inserted module 'virtio_gpu'1920worker1 # [ 2.129987] systemd-udevd[79]: Using default interface naming scheme 'v261'.1921worker2 # [ 2.040715] systemd-modules-load[74]: Module 'virtio_balloon' is built in1922worker2 # [ 2.041846] systemd-modules-load[74]: Module 'virtio_console' is built in1923worker2 # [ 2.042877] systemd-modules-load[74]: Inserted module 'virtio_gpu'1924worker2 # [ 2.043830] systemd-modules-load[74]: Module 'virtio_rng' is built in1925lb # [ 2.075455] systemd[1]: Started Journal Service.1926worker1 # [ 2.144981] systemd[1]: Finished Create System Files and Directories.1927lb # [ 2.064375] systemd-modules-load[74]: Module 'virtio_rng' is built in1928worker2 # [ 2.057283] systemd[1]: Starting Create System Files and Directories...1929worker2 # [ 2.058347] systemd[1]: Finished Load Kernel Modules.1930client # [ 2.064193] systemd[1]: Starting Apply Kernel Variables...1931worker2 # [ 2.067442] systemd-udevd[79]: Using default interface naming scheme 'v261'.1932lb # [ 2.076383] systemd[1]: Starting Create System Files and Directories...1933client # [ 2.075230] systemd-udevd[79]: Using default interface naming scheme 'v261'.1934client # [ 2.077364] systemd[1]: Finished Create System Files and Directories.1935worker1 # [ 2.164773] systemd[1]: Started Rule-based Manager for Device Events and Files.1936lb # [ 2.089859] systemd[1]: Finished Apply Kernel Variables.1937worker2 # [ 2.093188] systemd[1]: Starting Apply Kernel Variables...1938lb # [ 2.105634] systemd-udevd[78]: Using default interface naming scheme 'v261'.1939client # [ 2.101265] systemd[1]: Finished Apply Kernel Variables.1940worker2 # [ 2.105030] systemd[1]: Finished Create System Files and Directories.1941worker2 # [ 2.110030] systemd[1]: Finished Apply Kernel Variables.1942lb # [ 2.120837] systemd[1]: Finished Create System Files and Directories.1943client # [ 2.117759] systemd[1]: Started Rule-based Manager for Device Events and Files.1944worker2 # [ 2.126128] systemd[1]: Started Rule-based Manager for Device Events and Files.1945lb # [ 2.140317] systemd[1]: Started Rule-based Manager for Device Events and Files.1946worker1 # [ 2.224174] systemd[1]: Starting Virtual Console Setup...1947client # [ 2.172081] systemd[1]: Starting Virtual Console Setup...1948worker2 # [ 2.176271] systemd[1]: Starting Virtual Console Setup...1949worker1 # [ 2.272434] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1950worker1 # [ 2.275816] systemd[1]: Finished Virtual Console Setup.1951lb # [ 2.196165] systemd[1]: Starting Virtual Console Setup...1952client # [ 2.220415] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1953client # [ 2.223748] systemd[1]: Finished Virtual Console Setup.1954worker2 # [ 2.232449] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1955worker2 # [ 2.235805] systemd[1]: Finished Virtual Console Setup.1956lb # [ 2.244457] systemd-vconsole-setup[102]: Configuration of first virtual console was skipped, ignoring remaining ones.1957lb # [ 2.247821] systemd[1]: Finished Virtual Console Setup.1958worker1 # [ 2.883598] systemd[1]: Finished Coldplug All udev Devices.1959worker1 # [ 2.884640] systemd[1]: Reached target System Initialization.1960worker1 # [ 2.888105] systemd[1]: Reached target Basic System.1961client # [ 2.833892] systemd[1]: Finished Coldplug All udev Devices.1962worker2 # [ 2.830219] systemd[1]: Finished Coldplug All udev Devices.1963worker2 # [ 2.831191] systemd[1]: Reached target System Initialization.1964worker2 # [ 2.836102] systemd[1]: Reached target Basic System.1965client # [ 2.840125] systemd[1]: Reached target System Initialization.1966client # [ 2.841021] systemd[1]: Reached target Basic System.1967lb # [ 2.858052] systemd[1]: Finished Coldplug All udev Devices.1968lb # [ 2.858986] systemd[1]: Reached target System Initialization.1969lb # [ 2.864169] systemd[1]: Reached target Basic System.1970worker1 # [ 3.041689] (udev-worker)[91]: Network interface NamePolicy= disabled on kernel command line.1971worker1 # [ 3.060934] (udev-worker)[100]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1972worker1 # [ 3.068715] (udev-worker)[100]: Network interface NamePolicy= disabled on kernel command line.1973client # [ 2.996091] (udev-worker)[96]: Network interface NamePolicy= disabled on kernel command line.1974worker2 # [ 2.992372] (udev-worker)[94]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1975lb # [ 3.008078] (udev-worker)[98]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1976worker2 # [ 3.006033] (udev-worker)[96]: Network interface NamePolicy= disabled on kernel command line.1977worker2 # [ 3.009748] (udev-worker)[94]: Network interface NamePolicy= disabled on kernel command line.1978client # [ 3.017469] (udev-worker)[91]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1979client # [ 3.021520] (udev-worker)[91]: Network interface NamePolicy= disabled on kernel command line.1980lb # [ 3.042815] (udev-worker)[98]: Network interface NamePolicy= disabled on kernel command line.1981lb # [ 3.045998] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line.1982worker2 # [ 3.063527] systemd[1]: Found device /dev/disk/by-label/nixos.1983worker1 # [ 3.154194] systemd[1]: Found device /dev/disk/by-label/nixos.1984worker1 # [ 3.155858] systemd[1]: Reached target Initrd Root Device.1985worker2 # [ 3.066645] systemd[1]: Reached target Initrd Root Device.1986worker1 # [ 3.160801] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1987worker2 # [ 3.075431] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1988client # [ 3.096312] systemd[1]: Found device /dev/disk/by-label/nixos.1989client # [ 3.102838] systemd[1]: Reached target Initrd Root Device.1990client # [ 3.108169] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1991lb # [ 3.113380] systemd[1]: Found device /dev/disk/by-label/nixos.1992worker1 # [ 3.200458] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks1993lb # [ 3.119422] systemd[1]: Reached target Initrd Root Device.1994worker2 # [ 3.115902] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks1995lb # [ 3.126103] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1996worker1 # [ 3.209387] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1997worker1 # [ 3.211435] systemd[1]: Mounting /sysroot...1998worker2 # [ 3.122397] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1999worker2 # [ 3.128898] systemd[1]: Mounting /sysroot...2000client # [ 3.152233] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks
loading 2,005 hidden lines…
4006lb # [ 48.691277] niks3-server[953]: 2026/09/23 16:00:18 INFO Received complete push request method=POST path=/api/pushes/7/complete4007worker2 # [ 48.691114] nix-grpc-daemon[854]: time=2026-09-23T16:00:18.314Z level=INFO msg="Upload complete. (257ms)"4008lb # [ 48.702134] niks3-server[953]: 2026/09/23 16:00:18 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo4009worker2 # [ 48.703517] nix-grpc-daemon[785]: ts=2026-09-23T16:00:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=0 paths=1 nar_bytes_in=7524010worker2 # [ 48.720958] nix-grpc-daemon[785]: ts=2026-09-23T16:00:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563944011worker2 # [ 48.730702] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4012worker2 # [ 48.735200] nix-daemon[842]: reaped child process 970, status = succeeded4013client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051...4014worker2 # [ 48.748498] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4015client # farm-top> worker2: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4016client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4017client # farm-top> LOG-farm-top-t1b4018worker2 # [ 50.218401] nix-daemon[842]: reaped child process 979, status = succeeded4019worker2 # [ 50.380142] nix-daemon[842]: accepted connection from pid 995, user nix-grpc-daemon (trusted)4020worker2 # [ 50.404684] nix-daemon[842]: reaped child process 998, status = succeeded4021lb # [ 50.413884] niks3-server[953]: 2026/09/23 16:00:20 INFO Received push request method=POST path=/api/pushes4022worker2 # [ 50.420433] nix-grpc-daemon[854]: time=2026-09-23T16:00:20.042Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4023worker2 # [ 50.421901] nix-grpc-daemon[854]: time=2026-09-23T16:00:20.042Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)"4024lb # [ 50.488512] niks3-server[953]: 2026/09/23 16:00:20 INFO Received sign narinfos request method=POST path=/api/pushes/8/sign4025lb # [ 50.498641] niks3-server[953]: 2026/09/23 16:00:20 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4026worker2 # [ 50.490577] nix-grpc-daemon[854]: time=2026-09-23T16:00:20.113Z level=INFO msg="Uploading 1 narinfos"4027lb # [ 50.507027] niks3-server[953]: 2026/09/23 16:00:20 INFO Signed narinfos id=8 count=14028lb # [ 50.511573] niks3-server[953]: 2026/09/23 16:00:20 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst4029lb # [ 50.518666] niks3-server[953]: 2026/09/23 16:00:20 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls4030lb # [ 50.530011] niks3-server[953]: 2026/09/23 16:00:20 INFO Received complete push request method=POST path=/api/pushes/8/complete4031lb # [ 50.535944] niks3-server[953]: 2026/09/23 16:00:20 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo4032worker2 # [ 50.535286] nix-grpc-daemon[854]: time=2026-09-23T16:00:20.158Z level=INFO msg="Upload complete. (371ms)"4033worker2 # [ 50.549894] nix-grpc-daemon[785]: ts=2026-09-23T16:00:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=14034worker2 # [ 50.582589] nix-daemon[842]: reaped child process 977, status = succeeded4035client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.50 seconds)4036client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath4037client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.20 seconds)4038lb: must succeed: curl -sf localhost:9901/clusters4039lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4040lb: must succeed: curl -sf localhost:9901/clusters4041lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4042worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4043worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.07 seconds)4044(finished: subtest: inputs already in the cache are not sent to the scheduler, in 3.06 seconds)4045subtest: outputs read back through any worker carry the cache signature4046client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&24047worker1 # [ 51.306835] nix-grpc-daemon[485]: ts=2026-09-23T16:00:20Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:596604048client # don't know how to build these paths:4049client # /nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv4050worker1 # [ 51.317426] nix-grpc-daemon[485]: ts=2026-09-23T16:00:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596604051worker1 # [ 51.343283] nix-grpc-daemon[485]: ts=2026-09-23T16:00:20Z level=debug event=want client=4 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv cached=04052worker1 # [ 51.350357] nix-grpc-daemon[485]: ts=2026-09-23T16:00:20Z level=debug event=assigned drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker=192.168.1.3:50051 clients=14053worker1 # [ 51.360256] nix-grpc-daemon[485]: ts=2026-09-23T16:00:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596604054worker1 # [ 51.371315] nix-grpc-daemon[485]: ts=2026-09-23T16:00:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596604055client # copying 1 paths to 'lb:50051'...4056client # copying path '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' to 'lb:50051'...4057worker1 # [ 51.386992] nix-grpc-daemon[485]: ts=2026-09-23T16:00:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596604058worker1 # [ 51.401152] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4059worker1 # [ 51.442508] nix-grpc-daemon[485]: ts=2026-09-23T16:00:21Z level=info event=niks3_push_started pid=9014060worker1 # [ 51.609349] nix-grpc-daemon[901]: time=2026-09-23T16:00:21.228Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""4061worker1 # [ 51.730252] nix-daemon[849]: accepted connection from pid 914, user nix-grpc-daemon (trusted)4062worker1 # [ 51.754818] nix-daemon[849]: reaped child process 917, status = succeeded4063lb # [ 51.681140] niks3-server[953]: 2026/09/23 16:00:21 INFO Received push request method=POST path=/api/pushes4064worker1 # [ 51.773923] nix-grpc-daemon[901]: time=2026-09-23T16:00:21.392Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4065worker1 # [ 51.775540] nix-grpc-daemon[901]: time=2026-09-23T16:00:21.392Z level=INFO msg="Uploading 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv (512B)"4066lb # [ 51.731400] niks3-server[953]: 2026/09/23 16:00:21 INFO Received sign narinfos request method=POST path=/api/pushes/9/sign4067worker1 # [ 51.824073] nix-grpc-daemon[901]: time=2026-09-23T16:00:21.442Z level=INFO msg="Uploading 1 narinfos"4068lb # [ 51.743897] niks3-server[953]: 2026/09/23 16:00:21 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.ls4069lb # [ 51.751733] niks3-server[953]: 2026/09/23 16:00:21 INFO Signed narinfos id=9 count=14070lb # [ 51.757186] niks3-server[953]: 2026/09/23 16:00:21 INFO Registered completed upload object_key=nar/13mhmmy2qay13imbbdi287vhryz1hwnyg0w6agd9s0ph37wnsqll.nar.zst4071lb # [ 51.768347] niks3-server[953]: 2026/09/23 16:00:21 INFO Received complete push request method=POST path=/api/pushes/9/complete4072lb # [ 51.782605] niks3-server[953]: 2026/09/23 16:00:21 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.narinfo4073worker1 # [ 51.870698] nix-grpc-daemon[901]: time=2026-09-23T16:00:21.490Z level=INFO msg="Upload complete. (255ms)"4074worker1 # [ 51.883774] nix-grpc-daemon[485]: ts=2026-09-23T16:00:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=0 paths=1 nar_bytes_in=5124075worker1 # [ 51.896721] nix-grpc-daemon[485]: ts=2026-09-23T16:00:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596604076worker1 # [ 51.911670] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4077worker1 # [ 51.924441] nix-daemon[849]: reaped child process 899, status = succeeded4078client # building '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' on lb:50051...4079client # farm-a> node-a: building 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv4080worker1 # [ 51.935579] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4081client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4082client # farm-a> LOG-farm-a-sig4083worker1 # [ 52.564653] nix-daemon[849]: reaped child process 921, status = succeeded4084worker1 # [ 52.662367] nix-daemon[849]: accepted connection from pid 937, user nix-grpc-daemon (trusted)4085worker1 # [ 52.678337] nix-daemon[849]: reaped child process 940, status = succeeded4086lb # [ 52.605062] niks3-server[953]: 2026/09/23 16:00:22 INFO Received push request method=POST path=/api/pushes4087worker1 # [ 52.698557] nix-grpc-daemon[901]: time=2026-09-23T16:00:22.316Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4088worker1 # [ 52.700575] nix-grpc-daemon[901]: time=2026-09-23T16:00:22.316Z level=INFO msg="Uploading 2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a (128B)"4089lb # [ 52.647580] niks3-server[953]: 2026/09/23 16:00:22 INFO Registered completed upload object_key=log/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv4090worker1 # [ 52.744752] nix-grpc-daemon[901]: time=2026-09-23T16:00:22.364Z level=INFO msg="Uploading 1 narinfos"4091lb # [ 52.664631] niks3-server[953]: 2026/09/23 16:00:22 INFO Received sign narinfos request method=POST path=/api/pushes/10/sign4092lb # [ 52.675465] niks3-server[953]: 2026/09/23 16:00:22 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst4093lb # [ 52.689645] niks3-server[953]: 2026/09/23 16:00:22 INFO Signed narinfos id=10 count=14094worker1 # [ 52.774515] nix-grpc-daemon[901]: time=2026-09-23T16:00:22.394Z level=INFO msg="Upload complete. (228ms)"4095lb # [ 52.695862] niks3-server[953]: 2026/09/23 16:00:22 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.ls4096lb # [ 52.701149] niks3-server[953]: 2026/09/23 16:00:22 INFO Received complete push request method=POST path=/api/pushes/10/complete4097lb # [ 52.707894] niks3-server[953]: 2026/09/23 16:00:22 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.narinfo4098worker1 # [ 52.790529] nix-grpc-daemon[485]: ts=2026-09-23T16:00:22Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=0 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv assign_id=5 outputs=14099worker1 # [ 52.805269] nix-grpc-daemon[485]: ts=2026-09-23T16:00:22Z level=debug event=want client=4 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv cached=04100worker1 # [ 52.813825] nix-grpc-daemon[485]: ts=2026-09-23T16:00:22Z level=debug event=assigned drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker=192.168.1.3:50051 clients=14101worker1 # [ 52.819456] nix-grpc-daemon[485]: ts=2026-09-23T16:00:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596604102worker1 # [ 52.824932] nix-daemon[849]: reaped child process 919, status = succeeded4103worker1 # [ 52.828245] nix-grpc-daemon[485]: ts=2026-09-23T16:00:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596604104worker1 # [ 52.840467] nix-grpc-daemon[901]: time=2026-09-23T16:00:22.460Z level=INFO msg="All 1 paths already cached"4105client # copying 1 paths to 'lb:50051'...4106worker1 # [ 52.849637] nix-grpc-daemon[485]: ts=2026-09-23T16:00:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596604107client # copying path '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' to 'lb:50051'...4108worker1 # [ 52.862891] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4109worker1 # [ 53.007388] nix-daemon[849]: accepted connection from pid 946, user nix-grpc-daemon (trusted)4110worker1 # [ 53.026531] nix-daemon[849]: reaped child process 949, status = succeeded4111lb # [ 52.949820] niks3-server[953]: 2026/09/23 16:00:22 INFO Received push request method=POST path=/api/pushes4112worker1 # [ 53.039929] nix-grpc-daemon[901]: time=2026-09-23T16:00:22.659Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4113worker1 # [ 53.041725] nix-grpc-daemon[901]: time=2026-09-23T16:00:22.659Z level=INFO msg="Uploading z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv (624B)"4114lb # [ 52.978841] niks3-server[953]: 2026/09/23 16:00:22 INFO Received sign narinfos request method=POST path=/api/pushes/11/sign4115worker1 # [ 53.069038] nix-grpc-daemon[901]: time=2026-09-23T16:00:22.688Z level=INFO msg="Uploading 1 narinfos"4116lb # [ 52.993788] niks3-server[953]: 2026/09/23 16:00:22 INFO Signed narinfos id=11 count=14117lb # [ 53.001489] niks3-server[953]: 2026/09/23 16:00:22 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.ls4118lb # [ 53.017000] niks3-server[953]: 2026/09/23 16:00:22 INFO Registered completed upload object_key=nar/0cnhkgsbmxv5hj7zw3fa59qfxikw6251mlhz2gz1v6pgn8c8shwp.nar.zst4119worker1 # [ 53.109616] nix-grpc-daemon[901]: time=2026-09-23T16:00:22.729Z level=INFO msg="Upload complete. (233ms)"4120lb # [ 53.028162] niks3-server[953]: 2026/09/23 16:00:22 INFO Received complete push request method=POST path=/api/pushes/11/complete4121lb # [ 53.032056] niks3-server[953]: 2026/09/23 16:00:22 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.narinfo4122worker1 # [ 53.121192] nix-grpc-daemon[485]: ts=2026-09-23T16:00:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=0 paths=1 nar_bytes_in=6244123worker1 # [ 53.132360] nix-grpc-daemon[485]: ts=2026-09-23T16:00:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596604124worker1 # [ 53.140186] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4125worker1 # [ 53.145365] nix-daemon[849]: reaped child process 944, status = succeeded4126client # building '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' on lb:50051...4127worker1 # [ 53.157054] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4128client # farm-b> node-a: building z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv4129client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4130client # farm-b> LOG-farm-b-sig4131worker1 # [ 53.810444] nix-daemon[849]: reaped child process 953, status = succeeded4132worker1 # [ 53.927855] nix-daemon[849]: accepted connection from pid 969, user nix-grpc-daemon (trusted)4133worker1 # [ 53.947348] nix-daemon[849]: reaped child process 972, status = succeeded4134lb # [ 53.870099] niks3-server[953]: 2026/09/23 16:00:23 INFO Received push request method=POST path=/api/pushes4135worker1 # [ 53.963451] nix-grpc-daemon[901]: time=2026-09-23T16:00:23.582Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4136worker1 # [ 53.965598] nix-grpc-daemon[901]: time=2026-09-23T16:00:23.582Z level=INFO msg="Uploading h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b (176B)"4137lb # [ 53.916993] niks3-server[953]: 2026/09/23 16:00:23 INFO Received sign narinfos request method=POST path=/api/pushes/12/sign4138worker1 # [ 54.006074] nix-grpc-daemon[901]: time=2026-09-23T16:00:23.625Z level=INFO msg="Uploading 1 narinfos"4139lb # [ 53.926174] niks3-server[953]: 2026/09/23 16:00:23 INFO Registered completed upload object_key=log/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv4140lb # [ 53.933992] niks3-server[953]: 2026/09/23 16:00:23 INFO Signed narinfos id=12 count=14141lb # [ 53.939206] niks3-server[953]: 2026/09/23 16:00:23 INFO Registered completed upload object_key=nar/0f75gpfkjj0g9xi2113yvvpdw0l16spc2f1xik1ifnzfbrvx4pfk.nar.zst4142lb # [ 53.949170] niks3-server[953]: 2026/09/23 16:00:23 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.ls4143lb # [ 53.954561] niks3-server[953]: 2026/09/23 16:00:23 INFO Received complete push request method=POST path=/api/pushes/12/complete4144worker1 # [ 54.034929] nix-grpc-daemon[901]: time=2026-09-23T16:00:23.654Z level=INFO msg="Upload complete. (265ms)"4145lb # [ 53.957920] niks3-server[953]: 2026/09/23 16:00:23 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.narinfo4146worker1 # [ 54.053558] nix-grpc-daemon[485]: ts=2026-09-23T16:00:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=0 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv assign_id=6 outputs=14147worker1 # [ 54.068364] nix-grpc-daemon[485]: ts=2026-09-23T16:00:23Z level=debug event=want client=4 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv cached=04148worker1 # [ 54.071696] nix-grpc-daemon[485]: ts=2026-09-23T16:00:23Z level=debug event=assigned drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker=192.168.1.3:50051 clients=14149worker1 # [ 54.078097] nix-grpc-daemon[485]: ts=2026-09-23T16:00:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596604150worker1 # [ 54.082695] nix-daemon[849]: reaped child process 951, status = succeeded4151worker1 # [ 54.086049] nix-grpc-daemon[485]: ts=2026-09-23T16:00:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596604152worker1 # [ 54.096659] nix-grpc-daemon[901]: time=2026-09-23T16:00:23.716Z level=INFO msg="All 2 paths already cached"4153client # copying 1 paths to 'lb:50051'...4154client # copying path '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' to 'lb:50051'...4155worker1 # [ 54.106805] nix-grpc-daemon[485]: ts=2026-09-23T16:00:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596604156worker1 # [ 54.120182] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4157worker1 # [ 54.258897] nix-daemon[849]: accepted connection from pid 978, user nix-grpc-daemon (trusted)4158worker1 # [ 54.280244] nix-daemon[849]: reaped child process 981, status = succeeded4159lb # [ 54.196509] niks3-server[953]: 2026/09/23 16:00:23 INFO Received push request method=POST path=/api/pushes4160worker1 # [ 54.286570] nix-grpc-daemon[901]: time=2026-09-23T16:00:23.906Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4161worker1 # [ 54.288393] nix-grpc-daemon[901]: time=2026-09-23T16:00:23.906Z level=INFO msg="Uploading knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv (752B)"4162lb # [ 54.229252] niks3-server[953]: 2026/09/23 16:00:23 INFO Received sign narinfos request method=POST path=/api/pushes/13/sign4163worker1 # [ 54.318850] nix-grpc-daemon[901]: time=2026-09-23T16:00:23.938Z level=INFO msg="Uploading 1 narinfos"4164lb # [ 54.242264] niks3-server[953]: 2026/09/23 16:00:23 INFO Registered completed upload object_key=nar/0lpagyrngav67x80d1hq0njjlkmqdkvqawraz8dsp36ag50v6iwf.nar.zst4165worker1 # [ 54.345549] nix-grpc-daemon[901]: time=2026-09-23T16:00:23.965Z level=INFO msg="Upload complete. (214ms)"4166lb # [ 54.257725] niks3-server[953]: 2026/09/23 16:00:23 INFO Signed narinfos id=13 count=14167lb # [ 54.269780] niks3-server[953]: 2026/09/23 16:00:23 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.ls4168worker1 # [ 54.352270] nix-grpc-daemon[485]: ts=2026-09-23T16:00:23Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=0 paths=1 nar_bytes_in=7524169lb # [ 54.274823] niks3-server[953]: 2026/09/23 16:00:23 INFO Received complete push request method=POST path=/api/pushes/13/complete4170worker1 # [ 54.364786] nix-grpc-daemon[485]: ts=2026-09-23T16:00:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596604171lb # [ 54.286707] niks3-server[953]: 2026/09/23 16:00:23 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.narinfo4172worker1 # [ 54.372366] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4173worker1 # [ 54.377501] nix-daemon[849]: reaped child process 976, status = succeeded4174client # building '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' on lb:50051...4175client # farm-top> node-a: building knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv4176worker1 # [ 54.391392] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4177client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4178client # farm-top> LOG-farm-top-sig4179worker1 # [ 55.289031] nix-daemon[849]: reaped child process 985, status = succeeded4180worker1 # [ 55.372619] nix-daemon[849]: accepted connection from pid 1001, user nix-grpc-daemon (trusted)4181worker1 # [ 55.394027] nix-daemon[849]: reaped child process 1004, status = succeeded4182lb # [ 55.317191] niks3-server[953]: 2026/09/23 16:00:24 INFO Received push request method=POST path=/api/pushes4183worker1 # [ 55.409584] nix-grpc-daemon[901]: time=2026-09-23T16:00:25.028Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4184worker1 # [ 55.411252] nix-grpc-daemon[901]: time=2026-09-23T16:00:25.028Z level=INFO msg="Uploading 4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top (232B)"4185lb # [ 55.358541] niks3-server[953]: 2026/09/23 16:00:24 INFO Received sign narinfos request method=POST path=/api/pushes/14/sign4186worker1 # [ 55.449007] nix-grpc-daemon[901]: time=2026-09-23T16:00:25.068Z level=INFO msg="Uploading 1 narinfos"4187lb # [ 55.372069] niks3-server[953]: 2026/09/23 16:00:24 INFO Signed narinfos id=14 count=14188lb # [ 55.374887] niks3-server[953]: 2026/09/23 16:00:24 INFO Registered completed upload object_key=log/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv4189lb # [ 55.389099] niks3-server[953]: 2026/09/23 16:00:25 INFO Registered completed upload object_key=nar/07ar364rwqqniwi2nkdpqlgv7ws7702n5lqz703ackfmssrh40v8.nar.zst4190lb # [ 55.400954] niks3-server[953]: 2026/09/23 16:00:25 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.ls4191lb # [ 55.403844] niks3-server[953]: 2026/09/23 16:00:25 INFO Received complete push request method=POST path=/api/pushes/14/complete4192worker1 # [ 55.491186] nix-grpc-daemon[901]: time=2026-09-23T16:00:25.111Z level=INFO msg="Upload complete. (289ms)"4193lb # [ 55.410522] niks3-server[953]: 2026/09/23 16:00:25 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.narinfo4194worker1 # [ 55.505170] nix-grpc-daemon[485]: ts=2026-09-23T16:00:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=1 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv assign_id=7 outputs=14195worker1 # [ 55.523715] nix-daemon[849]: reaped child process 983, status = succeeded4196client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2, in 4.47 seconds)4197client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath4198client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath, in 0.18 seconds)4199client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top4200worker1 # [ 55.856994] nix-grpc-daemon[485]: ts=2026-09-23T16:00:25Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:483424201worker1 # [ 55.865350] nix-grpc-daemon[485]: ts=2026-09-23T16:00:25Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:483424202client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.15 seconds)4203client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top4204worker2 # [ 55.919210] nix-grpc-daemon[785]: ts=2026-09-23T16:00:25Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:371444205client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked):4206client # /nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a4207client # /nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b4208client # /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top4209worker2 # [ 55.949941] nix-grpc-daemon[785]: ts=2026-09-23T16:00:25Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:371444210worker2 # [ 55.958975] nix-grpc-daemon[785]: copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'http://lb:5751'...4211worker2 # [ 55.981052] nix-grpc-daemon[785]: copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'http://lb:5751'...4212worker2 # [ 55.995339] nix-grpc-daemon[785]: copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'http://lb:5751'...4213client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.25 seconds)4214client: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top4215client # finding garbage collector roots...4216client # removing stale temporary roots file "/nix/var/nix/temproots/927"4217client # removing stale temporary roots file "/nix/var/nix/temproots/898"4218client # removing stale temporary roots file "/nix/var/nix/temproots/886"4219client # removing stale temporary roots file "/nix/var/nix/temproots/848"4220client # removing stale temporary roots file "/nix/var/nix/temproots/942"4221client # removing stale temporary roots file "/nix/var/nix/temproots/865"4222client # removing stale temporary roots file "/nix/var/nix/temproots/919"4223client # removing stale temporary roots file "/nix/var/nix/temproots/906"4224client # deleting specified paths...4225client # deleting '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top'4226client # deleting unused links...4227client # note: hard linking is currently saving 0.0 KiB4228worker2 # [ 56.370107] nix-grpc-daemon[785]: ts=2026-09-23T16:00:25Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:563944229worker2 # [ 56.380079] nix-grpc-daemon[785]: ts=2026-09-23T16:00:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563944230worker2 # [ 56.387836] nix-grpc-daemon[854]: time=2026-09-23T16:00:26.009Z level=INFO msg="All 1 paths already cached"4231worker2 # [ 56.393884] nix-grpc-daemon[785]: ts=2026-09-23T16:00:26Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:563944232client # copying 3 paths...4233worker1 # [ 56.489622] nix-grpc-daemon[485]: ts=2026-09-23T16:00:26Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:596604234worker1 # [ 56.497184] nix-grpc-daemon[485]: ts=2026-09-23T16:00:26Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:596604235client # copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'grpc://lb:50051'...4236worker1 # [ 56.522422] nix-grpc-daemon[485]: ts=2026-09-23T16:00:26Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:596604237worker1 # [ 56.530293] nix-grpc-daemon[485]: ts=2026-09-23T16:00:26Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=0 paths=1 nar_bytes_out=2324238worker2 # [ 56.445841] nix-grpc-daemon[785]: ts=2026-09-23T16:00:26Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:563944239worker2 # [ 56.453792] nix-grpc-daemon[785]: ts=2026-09-23T16:00:26Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=0 paths=1 nar_bytes_out=1764240worker1 # [ 56.551541] nix-grpc-daemon[485]: ts=2026-09-23T16:00:26Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:596604241worker1 # [ 56.558405] nix-grpc-daemon[485]: ts=2026-09-23T16:00:26Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=0 paths=1 nar_bytes_out=1284242client # copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'grpc://lb:50051'...4243client # copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'grpc://lb:50051'...4244client: (finished: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.47 seconds)4245(finished: subtest: outputs read back through any worker carry the cache signature, in 5.52 seconds)4246subtest: two clients wanting the same drv build it once4247client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup4248client # [ 56.553762] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup.4249client # Running as unit: dedup1.service; invocation ID: 2894a8ddfe8545e0bca386f107e6f8db4250client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 0.05 seconds)4251client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup4252worker1 # [ 56.813290] nix-grpc-daemon[485]: ts=2026-09-23T16:00:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:596604253client # [ 56.734506] nix[995]: don't know how to build these paths:4254client # [ 56.736056] nix[995]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4255worker1 # [ 56.833230] nix-grpc-daemon[485]: ts=2026-09-23T16:00:26Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04256worker1 # [ 56.839415] nix-grpc-daemon[485]: ts=2026-09-23T16:00:26Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.4:50051 clients=14257worker2 # [ 56.753106] nix-grpc-daemon[785]: ts=2026-09-23T16:00:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563944258worker2 # [ 56.769722] nix-grpc-daemon[785]: ts=2026-09-23T16:00:26Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563944259client # [ 56.788102] nix[995]: copying 1 paths to 'lb:50051'...4260client # [ 56.789445] nix[995]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'...4261worker2 # [ 56.785098] nix-grpc-daemon[785]: ts=2026-09-23T16:00:26Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563944262worker2 # [ 56.801056] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4263worker2 # [ 56.977830] nix-daemon[842]: accepted connection from pid 1005, user nix-grpc-daemon (trusted)4264worker2 # [ 56.997594] nix-daemon[842]: reaped child process 1008, status = succeeded4265lb # [ 57.014578] niks3-server[953]: 2026/09/23 16:00:26 INFO Received push request method=POST path=/api/pushes4266worker2 # [ 57.013973] nix-grpc-daemon[854]: time=2026-09-23T16:00:26.636Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4267worker2 # [ 57.015988] nix-grpc-daemon[854]: time=2026-09-23T16:00:26.636Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)"4268lb # [ 57.048547] niks3-server[953]: 2026/09/23 16:00:26 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls4269worker2 # [ 57.055164] nix-grpc-daemon[854]: time=2026-09-23T16:00:26.678Z level=INFO msg="Uploading 1 narinfos"4270lb # [ 57.063164] niks3-server[953]: 2026/09/23 16:00:26 INFO Received sign narinfos request method=POST path=/api/pushes/15/sign4271lb # [ 57.075067] niks3-server[953]: 2026/09/23 16:00:26 INFO Signed narinfos id=15 count=14272lb # [ 57.086119] niks3-server[953]: 2026/09/23 16:00:26 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst4273lb # [ 57.096504] niks3-server[953]: 2026/09/23 16:00:26 INFO Received complete push request method=POST path=/api/pushes/15/complete4274lb # [ 57.104142] niks3-server[953]: 2026/09/23 16:00:26 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo4275worker2 # [ 57.095174] nix-grpc-daemon[854]: time=2026-09-23T16:00:26.718Z level=INFO msg="Upload complete. (274ms)"4276worker2 # [ 57.109123] nix-grpc-daemon[785]: ts=2026-09-23T16:00:26Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=0 paths=1 nar_bytes_in=4324277worker2 # [ 57.124764] nix-grpc-daemon[785]: ts=2026-09-23T16:00:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563944278worker2 # [ 57.133790] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4279worker2 # [ 57.137838] nix-daemon[842]: reaped child process 1003, status = succeeded4280worker2 # [ 57.149307] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4281client # [ 57.151860] nix[995]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4282client # [ 57.636943] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup.4283client # Running as unit: dedup2.service; invocation ID: 45a0a226402c4af4954d175656f6b2bd4284client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 1.10 seconds)4285client: must succeed: sleep 24286worker2 # [ 57.880934] nix-grpc-daemon[785]: ts=2026-09-23T16:00:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:563944287client # [ 57.893407] nix[1011]: this derivation will be built:4288worker2 # [ 57.891991] nix-grpc-daemon[785]: ts=2026-09-23T16:00:27Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:563944289client # [ 57.900073] nix[1011]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4290worker2 # [ 57.909899] nix-grpc-daemon[785]: ts=2026-09-23T16:00:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563944291worker1 # [ 57.996304] nix-grpc-daemon[485]: ts=2026-09-23T16:00:27Z level=debug event=want client=6 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04292client # [ 57.916983] nix[1011]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4293lb # [ 58.081246] envoy[897]: conn peer=[2001:db8:1::1]:53558 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4185 tx=1920 ms=22464294lb # [ 58.086189] envoy[897]: conn peer=[2001:db8:1::1]:39240 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9498 tx=4352 ms=42114295lb # [ 58.091354] envoy[897]: conn peer=[2001:db8:1::1]:39254 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1051 tx=2891 ms=1294296lb # [ 58.098410] envoy[897]: conn peer=[2001:db8:1::1]:39276 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=329 ms=414297lb # [ 58.103198] envoy[897]: conn peer=[2001:db8:1::1]:39258 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=384 tx=410 ms=684298lb # [ 58.107921] envoy[897]: conn peer=[2001:db8:1::1]:39262 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=375 ms=564299client: (finished: must succeed: sleep 2, in 2.07 seconds)4300client: waiting for success: ! systemctl is-active dedup1 dedup24301worker2 # [ 60.236656] nix-daemon[842]: reaped child process 1012, status = succeeded4302worker2 # [ 60.338196] nix-daemon[842]: accepted connection from pid 1038, user nix-grpc-daemon (trusted)4303worker2 # [ 60.358577] nix-daemon[842]: reaped child process 1041, status = succeeded4304lb # [ 60.366301] niks3-server[953]: 2026/09/23 16:00:29 INFO Received push request method=POST path=/api/pushes4305worker2 # [ 60.368477] nix-grpc-daemon[854]: time=2026-09-23T16:00:29.990Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4306worker2 # [ 60.369940] nix-grpc-daemon[854]: time=2026-09-23T16:00:29.990Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)"4307lb # [ 60.406289] niks3-server[953]: 2026/09/23 16:00:30 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4308lb # [ 60.430196] niks3-server[953]: 2026/09/23 16:00:30 INFO Received sign narinfos request method=POST path=/api/pushes/16/sign4309worker2 # [ 60.427227] nix-grpc-daemon[854]: time=2026-09-23T16:00:30.050Z level=INFO msg="Uploading 1 narinfos"4310lb # [ 60.438353] niks3-server[953]: 2026/09/23 16:00:30 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4311lb # [ 60.451168] niks3-server[953]: 2026/09/23 16:00:30 INFO Signed narinfos id=16 count=14312lb # [ 60.456824] niks3-server[953]: 2026/09/23 16:00:30 INFO Received complete push request method=POST path=/api/pushes/16/complete4313lb # [ 60.459351] niks3-server[953]: 2026/09/23 16:00:30 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls4314worker2 # [ 60.455124] nix-grpc-daemon[854]: time=2026-09-23T16:00:30.078Z level=INFO msg="Upload complete. (288ms)"4315lb # [ 60.466373] niks3-server[953]: 2026/09/23 16:00:30 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo4316worker2 # [ 60.469203] nix-grpc-daemon[785]: ts=2026-09-23T16:00:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=14317worker2 # [ 60.488312] nix-grpc-daemon[785]: ts=2026-09-23T16:00:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=2 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=14318worker2 # [ 60.495488] nix-daemon[842]: reaped child process 1010, status = succeeded4319client # [ 60.529338] systemd[1]: dedup1.service: Deactivated successfully.4320client # [ 60.532897] systemd[1]: dedup1.service: Consumed 162ms CPU time over 3.978s wall clock time, 41.5M memory peak, 7.1K incoming IP traffic, 8.4K outgoing IP traffic.4321client # [ 60.541567] systemd[1]: dedup2.service: Deactivated successfully.4322client # [ 60.546074] systemd[1]: dedup2.service: Consumed 211ms CPU time over 2.904s wall clock time, 33.2M memory peak, 6.5K incoming IP traffic, 5.8K outgoing IP traffic.4323client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.13 seconds)4324client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4325client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.07 seconds)4326worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4327worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4328worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4329worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4330worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4331worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4332worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4333worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4334(finished: subtest: two clients wanting the same drv build it once, in 4.96 seconds)4335subtest: --timeout and --max-silent-time reach the builder4336worker2 # [ 61.692675] nix-grpc-daemon[785]: ts=2026-09-23T16:00:31Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:563944337worker1 # [ 61.801442] nix-grpc-daemon[485]: ts=2026-09-23T16:00:31Z level=debug event=want client=7 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv cached=04338worker1 # [ 61.810959] nix-grpc-daemon[485]: ts=2026-09-23T16:00:31Z level=debug event=assigned drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv worker=192.168.1.3:50051 clients=14339worker1 # [ 61.811225] nix-grpc-daemon[485]: ts=2026-09-23T16:00:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596604340worker1 # [ 61.817198] nix-grpc-daemon[485]: ts=2026-09-23T16:00:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596604341worker1 # [ 61.843589] nix-grpc-daemon[485]: ts=2026-09-23T16:00:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596604342worker1 # [ 61.858881] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4343worker1 # [ 62.014883] nix-daemon[849]: accepted connection from pid 1031, user nix-grpc-daemon (trusted)4344worker1 # [ 62.037032] nix-daemon[849]: reaped child process 1034, status = succeeded4345lb # [ 61.961095] niks3-server[953]: 2026/09/23 16:00:31 INFO Received push request method=POST path=/api/pushes4346worker1 # [ 62.051652] nix-grpc-daemon[901]: time=2026-09-23T16:00:31.670Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4347worker1 # [ 62.053663] nix-grpc-daemon[901]: time=2026-09-23T16:00:31.670Z level=INFO msg="Uploading sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv (424B)"4348lb # [ 61.992822] niks3-server[953]: 2026/09/23 16:00:31 INFO Registered completed upload object_key=nar/1m12xmj6yzmb9y2j8ly0yvh5fyzy1nqk7ajhqwr4wn3b5h0bsch5.nar.zst4349worker1 # [ 62.085332] nix-grpc-daemon[901]: time=2026-09-23T16:00:31.705Z level=INFO msg="Uploading 1 narinfos"4350lb # [ 62.009331] niks3-server[953]: 2026/09/23 16:00:31 INFO Received sign narinfos request method=POST path=/api/pushes/17/sign4351lb # [ 62.018536] niks3-server[953]: 2026/09/23 16:00:31 INFO Signed narinfos id=17 count=14352lb # [ 62.023769] niks3-server[953]: 2026/09/23 16:00:31 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.ls4353lb # [ 62.034676] niks3-server[953]: 2026/09/23 16:00:31 INFO Received complete push request method=POST path=/api/pushes/17/complete4354lb # [ 62.038749] niks3-server[953]: 2026/09/23 16:00:31 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.narinfo4355worker1 # [ 62.124377] nix-grpc-daemon[901]: time=2026-09-23T16:00:31.744Z level=INFO msg="Upload complete. (249ms)"4356worker1 # [ 62.136300] nix-grpc-daemon[485]: ts=2026-09-23T16:00:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=0 paths=1 nar_bytes_in=4244357worker1 # [ 62.152784] nix-grpc-daemon[485]: ts=2026-09-23T16:00:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596604358worker1 # [ 62.159640] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4359worker1 # [ 62.171192] nix-daemon[849]: reaped child process 1029, status = succeeded4360worker1 # [ 62.177794] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4361lb # [ 68.080499] envoy[897]: conn peer=[2001:db8:1::1]:39300 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1579 tx=1597 ms=26034362lb # [ 68.080862] envoy[897]: conn peer=[2001:db8:1::1]:39284 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3005 tx=1541 ms=37614363worker1 # [ 68.196174] nix-grpc-daemon[485]: ts=2026-09-23T16:00:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=6 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv assign_id=9 outputs=04364worker1 # [ 68.220158] nix-daemon[849]: reaped child process 1036, status = succeeded4365worker1 # [ 68.221726] nix-daemon[849]: reaped child process 1038, status = succeeded4366worker2 # [ 68.341123] nix-grpc-daemon[785]: ts=2026-09-23T16:00:37Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:563944367worker1 # [ 68.440143] nix-grpc-daemon[485]: ts=2026-09-23T16:00:38Z level=debug event=want client=8 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv cached=04368worker1 # [ 68.446930] nix-grpc-daemon[485]: ts=2026-09-23T16:00:38Z level=debug event=assigned drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv worker=192.168.1.4:50051 clients=14369worker2 # [ 68.359486] nix-grpc-daemon[785]: ts=2026-09-23T16:00:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563944370worker2 # [ 68.373883] nix-grpc-daemon[785]: ts=2026-09-23T16:00:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563944371worker2 # [ 68.388515] nix-grpc-daemon[785]: ts=2026-09-23T16:00:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563944372worker2 # [ 68.403685] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4373worker2 # [ 68.565142] nix-daemon[842]: accepted connection from pid 1054, user nix-grpc-daemon (trusted)4374worker2 # [ 68.585239] nix-daemon[842]: reaped child process 1057, status = succeeded4375lb # [ 68.602584] niks3-server[953]: 2026/09/23 16:00:38 INFO Received push request method=POST path=/api/pushes4376worker2 # [ 68.603040] nix-grpc-daemon[854]: time=2026-09-23T16:00:38.226Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4377worker2 # [ 68.604788] nix-grpc-daemon[854]: time=2026-09-23T16:00:38.226Z level=INFO msg="Uploading pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv (424B)"4378lb # [ 68.640263] niks3-server[953]: 2026/09/23 16:00:38 INFO Registered completed upload object_key=nar/14f5yrnllfav3svmw4v4svw15r4zlf5yv1p972jy14mmq6gdw943.nar.zst4379worker2 # [ 68.644801] nix-grpc-daemon[854]: time=2026-09-23T16:00:38.268Z level=INFO msg="Uploading 1 narinfos"4380lb # [ 68.657513] niks3-server[953]: 2026/09/23 16:00:38 INFO Received sign narinfos request method=POST path=/api/pushes/18/sign4381lb # [ 68.666906] niks3-server[953]: 2026/09/23 16:00:38 INFO Signed narinfos id=18 count=14382lb # [ 68.680540] niks3-server[953]: 2026/09/23 16:00:38 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.ls4383worker2 # [ 68.681137] nix-grpc-daemon[854]: time=2026-09-23T16:00:38.304Z level=INFO msg="Upload complete. (259ms)"4384lb # [ 68.690387] niks3-server[953]: 2026/09/23 16:00:38 INFO Received complete push request method=POST path=/api/pushes/18/complete4385lb # [ 68.698748] niks3-server[953]: 2026/09/23 16:00:38 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.narinfo4386worker2 # [ 68.691212] nix-grpc-daemon[785]: ts=2026-09-23T16:00:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=0 paths=1 nar_bytes_in=4244387worker2 # [ 68.708727] nix-grpc-daemon[785]: ts=2026-09-23T16:00:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563944388worker2 # [ 68.716519] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4389worker2 # [ 68.722064] nix-daemon[842]: reaped child process 1052, status = succeeded4390worker2 # [ 68.733434] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4391worker2 # [ 74.550781] nix-grpc-daemon[785]: ts=2026-09-23T16:00:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=5 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv assign_id=10 outputs=04392worker2 # [ 74.578541] nix-daemon[842]: reaped child process 1059, status = succeeded4393worker2 # [ 74.580728] nix-daemon[842]: reaped child process 1061, status = succeeded4394(finished: subtest: --timeout and --max-silent-time reach the builder, in 13.27 seconds)4395subtest: the first client leaving does not fail the second4396client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share4397client # [ 74.805140] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share.4398client # Running as unit: share1.service; invocation ID: 7671abb398144c5aa89e8f72df244e204399client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 0.09 seconds)4400client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share4401worker1 # [ 75.099524] nix-grpc-daemon[485]: ts=2026-09-23T16:00:44Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:596604402client # [ 75.021778] nix[1084]: don't know how to build these paths:4403client # [ 75.022978] nix[1084]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4404worker1 # [ 75.119762] nix-grpc-daemon[485]: ts=2026-09-23T16:00:44Z level=debug event=want client=9 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=04405worker1 # [ 75.129435] nix-grpc-daemon[485]: ts=2026-09-23T16:00:44Z level=debug event=assigned drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker=192.168.1.3:50051 clients=14406worker1 # [ 75.138930] nix-grpc-daemon[485]: ts=2026-09-23T16:00:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596604407worker1 # [ 75.146492] nix-grpc-daemon[485]: ts=2026-09-23T16:00:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596604408client # [ 75.068618] nix[1084]: copying 1 paths to 'lb:50051'...4409client # [ 75.071931] nix[1084]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' to 'lb:50051'...4410worker1 # [ 75.159214] nix-grpc-daemon[485]: ts=2026-09-23T16:00:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596604411worker1 # [ 75.174577] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4412worker1 # [ 75.330553] nix-daemon[849]: accepted connection from pid 1065, user nix-grpc-daemon (trusted)4413worker1 # [ 75.351179] nix-daemon[849]: reaped child process 1068, status = succeeded4414lb # [ 75.275115] niks3-server[953]: 2026/09/23 16:00:44 INFO Received push request method=POST path=/api/pushes4415worker1 # [ 75.366453] nix-grpc-daemon[901]: time=2026-09-23T16:00:44.985Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4416worker1 # [ 75.368561] nix-grpc-daemon[901]: time=2026-09-23T16:00:44.985Z level=INFO msg="Uploading rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv (432B)"4417lb # [ 75.310650] niks3-server[953]: 2026/09/23 16:00:44 INFO Registered completed upload object_key=nar/0chdjpc9an2ckdblh08lnyrmzvw3h8qs76cn5ydwjwvgwkrm731l.nar.zst4418worker1 # [ 75.404871] nix-grpc-daemon[901]: time=2026-09-23T16:00:45.024Z level=INFO msg="Uploading 1 narinfos"4419lb # [ 75.325220] niks3-server[953]: 2026/09/23 16:00:44 INFO Received sign narinfos request method=POST path=/api/pushes/19/sign4420lb # [ 75.334607] niks3-server[953]: 2026/09/23 16:00:44 INFO Signed narinfos id=19 count=14421lb # [ 75.340326] niks3-server[953]: 2026/09/23 16:00:44 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.ls4422lb # [ 75.352765] niks3-server[953]: 2026/09/23 16:00:44 INFO Received complete push request method=POST path=/api/pushes/19/complete4423lb # [ 75.359871] niks3-server[953]: 2026/09/23 16:00:44 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.narinfo4424worker1 # [ 75.441027] nix-grpc-daemon[901]: time=2026-09-23T16:00:45.060Z level=INFO msg="Upload complete. (251ms)"4425worker1 # [ 75.451296] nix-grpc-daemon[485]: ts=2026-09-23T16:00:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=0 paths=1 nar_bytes_in=4324426worker1 # [ 75.468674] nix-grpc-daemon[485]: ts=2026-09-23T16:00:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596604427worker1 # [ 75.477550] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4428worker1 # [ 75.483276] nix-daemon[849]: reaped child process 1063, status = succeeded4429worker1 # [ 75.496560] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4430client # [ 75.409376] nix[1084]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051...4431client # [ 75.918245] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share.4432client # Running as unit: share2.service; invocation ID: 1a7d02e87daa4edbbd11c65f485571664433client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 1.12 seconds)4434worker1 # [ 76.226750] nix-grpc-daemon[485]: ts=2026-09-23T16:00:45Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:596604435client # [ 76.149393] nix[1100]: this derivation will be built:4436worker2 # [ 76.150601] nix-grpc-daemon[785]: ts=2026-09-23T16:00:45Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:563944437worker2 # [ 76.164639] nix-grpc-daemon[785]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' from 'http://lb:5751'...4438client # [ 76.186731] nix[1100]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4439worker1 # [ 76.284447] nix-grpc-daemon[485]: ts=2026-09-23T16:00:45Z level=debug event=want client=10 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=04440worker1 # [ 76.287678] nix-grpc-daemon[485]: ts=2026-09-23T16:00:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596604441client # [ 76.205218] nix[1100]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051...4442client: must succeed: sleep 2; systemctl kill -s INT share14443lb # [ 78.085262] envoy[897]: conn peer=[2001:db8:1::1]:40074 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3001 tx=952 ms=64354444lb # [ 78.085651] envoy[897]: conn peer=[2001:db8:1::1]:40090 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2989 tx=977 ms=62354445client # [ 79.281394] systemd[1]: share1.service: Sent signal SIGINT to main process 1084 (nix) on client request.4446client # [ 79.290907] nix[1084]: shutting downrxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)4447client # [ 79.303962] nix[1084]: error: interrupted by the user4448client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.09 seconds)4449client: waiting for success: ! systemctl is-active share14450client # [ 79.360930] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE4451client # [ 79.364630] systemd[1]: share1.service: Failed with result 'exit-code'.4452client # [ 79.366269] systemd[1]: share1.service: Consumed 176ms CPU time over 4.558s wall clock time, 43.5M memory peak, 6.2K incoming IP traffic, 8.2K outgoing IP traffic.4453client: (finished: waiting for success: ! systemctl is-active share1, in 1.11 seconds)4454worker1 # [ 80.729899] nix-daemon[849]: reaped child process 1072, status = succeeded4455client: waiting for success: ! systemctl is-active share24456worker1 # [ 80.821146] nix-daemon[849]: accepted connection from pid 1108, user nix-grpc-daemon (trusted)4457worker1 # [ 80.841842] nix-daemon[849]: reaped child process 1111, status = succeeded4458lb # [ 80.765987] niks3-server[953]: 2026/09/23 16:00:50 INFO Received push request method=POST path=/api/pushes4459worker1 # [ 80.860382] nix-grpc-daemon[901]: time=2026-09-23T16:00:50.477Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4460worker1 # [ 80.861910] nix-grpc-daemon[901]: time=2026-09-23T16:00:50.477Z level=INFO msg="Uploading fjww0fd4rwk1s35jav3cyal62l24n356-slow-share (120B)"4461lb # [ 80.816596] niks3-server[953]: 2026/09/23 16:00:50 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.ls4462worker1 # [ 80.906874] nix-grpc-daemon[901]: time=2026-09-23T16:00:50.526Z level=INFO msg="Uploading 1 narinfos"4463lb # [ 80.826476] niks3-server[953]: 2026/09/23 16:00:50 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst4464lb # [ 80.835861] niks3-server[953]: 2026/09/23 16:00:50 INFO Received sign narinfos request method=POST path=/api/pushes/20/sign4465lb # [ 80.845313] niks3-server[953]: 2026/09/23 16:00:50 INFO Signed narinfos id=20 count=14466lb # [ 80.847162] niks3-server[953]: 2026/09/23 16:00:50 INFO Registered completed upload object_key=log/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4467worker1 # [ 80.935663] nix-grpc-daemon[901]: time=2026-09-23T16:00:50.555Z level=INFO msg="Upload complete. (223ms)"4468lb # [ 80.854671] niks3-server[953]: 2026/09/23 16:00:50 INFO Received complete push request method=POST path=/api/pushes/20/complete4469lb # [ 80.858646] niks3-server[953]: 2026/09/23 16:00:50 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.narinfo4470worker1 # [ 80.948865] nix-grpc-daemon[485]: ts=2026-09-23T16:00:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=5 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=14471worker1 # [ 80.965317] nix-grpc-daemon[485]: ts=2026-09-23T16:00:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=4 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=14472worker1 # [ 80.971344] nix-daemon[849]: reaped child process 1070, status = succeeded4473client # [ 80.903082] systemd[1]: share2.service: Deactivated successfully.4474client # [ 80.907449] systemd[1]: share2.service: Consumed 199ms CPU time over 4.989s wall clock time, 35.3M memory peak, 6.5K incoming IP traffic, 5.9K outgoing IP traffic.4475client: (finished: waiting for success: ! systemctl is-active share2, in 1.12 seconds)4476client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }4477client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.05 seconds)4478(finished: subtest: the first client leaving does not fail the second, in 7.10 seconds)4479subtest: build hook: nix-daemon with builders = grpc://lb4480client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log4481client # [ 81.975386] systemd[1]: Started Nix Daemon.4482client # [ 82.092415] nix-daemon[1152]: accepted connection from pid 1149, user root (trusted)4483worker1 # [ 82.329596] nix-grpc-daemon[485]: ts=2026-09-23T16:00:51Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:596604484worker2 # [ 82.243852] nix-grpc-daemon[785]: ts=2026-09-23T16:00:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563944485worker1 # [ 82.345512] nix-grpc-daemon[485]: ts=2026-09-23T16:00:51Z level=debug event=want client=11 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=04486worker1 # [ 82.352172] nix-grpc-daemon[485]: ts=2026-09-23T16:00:51Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.4:50051 clients=14487worker2 # [ 82.264894] nix-grpc-daemon[785]: ts=2026-09-23T16:00:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563944488worker2 # [ 82.281077] nix-grpc-daemon[785]: ts=2026-09-23T16:00:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563944489worker2 # [ 82.296899] nix-grpc-daemon[785]: ts=2026-09-23T16:00:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563944490worker2 # [ 82.311409] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4491worker2 # [ 82.475927] nix-daemon[842]: accepted connection from pid 1101, user nix-grpc-daemon (trusted)4492worker2 # [ 82.495918] nix-daemon[842]: reaped child process 1104, status = succeeded4493lb # [ 82.511704] niks3-server[953]: 2026/09/23 16:00:52 INFO Received push request method=POST path=/api/pushes4494worker2 # [ 82.513104] nix-grpc-daemon[854]: time=2026-09-23T16:00:52.136Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4495worker2 # [ 82.515004] nix-grpc-daemon[854]: time=2026-09-23T16:00:52.136Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)"4496lb # [ 82.540538] niks3-server[953]: 2026/09/23 16:00:52 INFO Received sign narinfos request method=POST path=/api/pushes/21/sign4497worker2 # [ 82.539214] nix-grpc-daemon[854]: time=2026-09-23T16:00:52.162Z level=INFO msg="Uploading 1 narinfos"4498lb # [ 82.552226] niks3-server[953]: 2026/09/23 16:00:52 INFO Signed narinfos id=21 count=14499lb # [ 82.561552] niks3-server[953]: 2026/09/23 16:00:52 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst4500lb # [ 82.577752] niks3-server[953]: 2026/09/23 16:00:52 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls4501lb # [ 82.585574] niks3-server[953]: 2026/09/23 16:00:52 INFO Received complete push request method=POST path=/api/pushes/21/complete4502worker2 # [ 82.582898] nix-grpc-daemon[854]: time=2026-09-23T16:00:52.206Z level=INFO msg="Upload complete. (253ms)"4503lb # [ 82.591909] niks3-server[953]: 2026/09/23 16:00:52 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo4504worker2 # [ 82.592600] nix-grpc-daemon[785]: ts=2026-09-23T16:00:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=0 paths=1 nar_bytes_in=5124505worker2 # [ 82.607334] nix-grpc-daemon[785]: ts=2026-09-23T16:00:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563944506worker2 # [ 82.613836] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4507worker2 # [ 82.618605] nix-daemon[842]: reaped child process 1099, status = succeeded4508worker2 # [ 82.629334] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4509worker2 # [ 84.268748] nix-daemon[842]: reaped child process 1108, status = succeeded4510worker2 # [ 84.400070] nix-daemon[842]: accepted connection from pid 1124, user nix-grpc-daemon (trusted)4511worker2 # [ 84.422436] nix-daemon[842]: reaped child process 1127, status = succeeded4512lb # [ 84.430261] niks3-server[953]: 2026/09/23 16:00:54 INFO Received push request method=POST path=/api/pushes4513worker2 # [ 84.432408] nix-grpc-daemon[854]: time=2026-09-23T16:00:54.055Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4514worker2 # [ 84.433910] nix-grpc-daemon[854]: time=2026-09-23T16:00:54.055Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)"4515lb # [ 84.482308] niks3-server[953]: 2026/09/23 16:00:54 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv4516worker2 # [ 84.485007] nix-grpc-daemon[854]: time=2026-09-23T16:00:54.108Z level=INFO msg="Uploading 1 narinfos"4517lb # [ 84.497207] niks3-server[953]: 2026/09/23 16:00:54 INFO Received sign narinfos request method=POST path=/api/pushes/22/sign4518lb # [ 84.505339] niks3-server[953]: 2026/09/23 16:00:54 INFO Signed narinfos id=22 count=14519lb # [ 84.509692] niks3-server[953]: 2026/09/23 16:00:54 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4520lb # [ 84.522822] niks3-server[953]: 2026/09/23 16:00:54 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls4521lb # [ 84.526333] niks3-server[953]: 2026/09/23 16:00:54 INFO Received complete push request method=POST path=/api/pushes/22/complete4522lb # [ 84.531454] niks3-server[953]: 2026/09/23 16:00:54 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo4523worker2 # [ 84.521695] nix-grpc-daemon[854]: time=2026-09-23T16:00:54.145Z level=INFO msg="Upload complete. (280ms)"4524worker2 # [ 84.538681] nix-grpc-daemon[785]: ts=2026-09-23T16:00:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=12 outputs=14525worker2 # [ 84.554828] nix-grpc-daemon[785]: ts=2026-09-23T16:00:54Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:563944526worker2 # [ 84.558986] nix-daemon[842]: reaped child process 1106, status = succeeded4527worker2 # [ 84.579478] nix-grpc-daemon[785]: ts=2026-09-23T16:00:54Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:563944528worker2 # [ 84.586778] nix-grpc-daemon[785]: ts=2026-09-23T16:00:54Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=0 paths=1 nar_bytes_out=1284529worker1 # [ 84.935178] nix-grpc-daemon[485]: ts=2026-09-23T16:00:54Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:596604530worker2 # [ 84.849171] nix-grpc-daemon[785]: ts=2026-09-23T16:00:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563944531worker2 # [ 84.860465] nix-grpc-daemon[854]: time=2026-09-23T16:00:54.484Z level=INFO msg="All 1 paths already cached"4532worker1 # [ 84.971024] nix-grpc-daemon[485]: ts=2026-09-23T16:00:54Z level=debug event=want client=12 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=04533worker1 # [ 84.978260] nix-grpc-daemon[485]: ts=2026-09-23T16:00:54Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.4:50051 clients=14534worker2 # [ 84.891941] nix-grpc-daemon[785]: ts=2026-09-23T16:00:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563944535worker2 # [ 84.903998] nix-grpc-daemon[785]: ts=2026-09-23T16:00:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563944536worker2 # [ 84.917042] nix-grpc-daemon[854]: time=2026-09-23T16:00:54.540Z level=INFO msg="All 1 paths already cached"4537worker2 # [ 84.928136] nix-grpc-daemon[785]: ts=2026-09-23T16:00:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563944538worker2 # [ 84.942885] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4539worker2 # [ 85.107194] nix-daemon[842]: accepted connection from pid 1131, user nix-grpc-daemon (trusted)4540worker2 # [ 85.131059] nix-daemon[842]: reaped child process 1134, status = succeeded4541lb # [ 85.145117] niks3-server[953]: 2026/09/23 16:00:54 INFO Received push request method=POST path=/api/pushes4542worker2 # [ 85.147573] nix-grpc-daemon[854]: time=2026-09-23T16:00:54.770Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4543worker2 # [ 85.149294] nix-grpc-daemon[854]: time=2026-09-23T16:00:54.771Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)"4544lb # [ 85.182254] niks3-server[953]: 2026/09/23 16:00:54 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst4545worker2 # [ 85.186887] nix-grpc-daemon[854]: time=2026-09-23T16:00:54.810Z level=INFO msg="Uploading 1 narinfos"4546lb # [ 85.198720] niks3-server[953]: 2026/09/23 16:00:54 INFO Received sign narinfos request method=POST path=/api/pushes/23/sign4547lb # [ 85.207640] niks3-server[953]: 2026/09/23 16:00:54 INFO Signed narinfos id=23 count=14548lb # [ 85.213496] niks3-server[953]: 2026/09/23 16:00:54 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls4549lb # [ 85.227957] niks3-server[953]: 2026/09/23 16:00:54 INFO Received complete push request method=POST path=/api/pushes/23/complete4550lb # [ 85.240468] niks3-server[953]: 2026/09/23 16:00:54 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo4551worker2 # [ 85.237825] nix-grpc-daemon[854]: time=2026-09-23T16:00:54.861Z level=INFO msg="Upload complete. (277ms)"4552worker2 # [ 85.249620] nix-grpc-daemon[785]: ts=2026-09-23T16:00:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=0 paths=1 nar_bytes_in=6244553worker2 # [ 85.268848] nix-grpc-daemon[785]: ts=2026-09-23T16:00:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563944554worker2 # [ 85.279936] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4555worker2 # [ 85.287916] nix-daemon[842]: reaped child process 1129, status = succeeded4556worker2 # [ 85.302974] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4557worker2 # [ 86.542230] nix-daemon[842]: reaped child process 1138, status = succeeded4558worker2 # [ 86.651061] nix-daemon[842]: accepted connection from pid 1154, user nix-grpc-daemon (trusted)4559worker2 # [ 86.670953] nix-daemon[842]: reaped child process 1157, status = succeeded4560lb # [ 86.681925] niks3-server[953]: 2026/09/23 16:00:56 INFO Received push request method=POST path=/api/pushes4561worker2 # [ 86.684610] nix-grpc-daemon[854]: time=2026-09-23T16:00:56.307Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4562worker2 # [ 86.686078] nix-grpc-daemon[854]: time=2026-09-23T16:00:56.307Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)"4563lb # [ 86.730387] niks3-server[953]: 2026/09/23 16:00:56 INFO Received sign narinfos request method=POST path=/api/pushes/24/sign4564worker2 # [ 86.732480] nix-grpc-daemon[854]: time=2026-09-23T16:00:56.355Z level=INFO msg="Uploading 1 narinfos"4565lb # [ 86.744322] niks3-server[953]: 2026/09/23 16:00:56 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst4566lb # [ 86.755321] niks3-server[953]: 2026/09/23 16:00:56 INFO Signed narinfos id=24 count=14567lb # [ 86.765354] niks3-server[953]: 2026/09/23 16:00:56 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv4568lb # [ 86.770953] niks3-server[953]: 2026/09/23 16:00:56 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls4569lb # [ 86.774013] niks3-server[953]: 2026/09/23 16:00:56 INFO Received complete push request method=POST path=/api/pushes/24/complete4570lb # [ 86.777411] niks3-server[953]: 2026/09/23 16:00:56 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo4571worker2 # [ 86.768539] nix-grpc-daemon[854]: time=2026-09-23T16:00:56.392Z level=INFO msg="Upload complete. (251ms)"4572worker2 # [ 86.781205] nix-grpc-daemon[785]: ts=2026-09-23T16:00:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=13 outputs=14573worker2 # [ 86.802620] nix-grpc-daemon[785]: ts=2026-09-23T16:00:56Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:563944574worker2 # [ 86.807361] nix-daemon[842]: reaped child process 1136, status = succeeded4575worker2 # [ 86.822146] nix-grpc-daemon[785]: ts=2026-09-23T16:00:56Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:563944576worker2 # [ 86.828953] nix-grpc-daemon[785]: ts=2026-09-23T16:00:56Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=0 paths=1 nar_bytes_out=1764577worker2 # [ 87.063108] nix-grpc-daemon[785]: ts=2026-09-23T16:00:56Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:563944578worker2 # [ 87.071287] nix-grpc-daemon[785]: ts=2026-09-23T16:00:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563944579worker2 # [ 87.079643] nix-grpc-daemon[854]: time=2026-09-23T16:00:56.701Z level=INFO msg="All 2 paths already cached"4580worker1 # [ 87.185317] nix-grpc-daemon[485]: ts=2026-09-23T16:00:56Z level=debug event=want client=13 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=04581worker1 # [ 87.191694] nix-grpc-daemon[485]: ts=2026-09-23T16:00:56Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.4:50051 clients=14582worker2 # [ 87.104790] nix-grpc-daemon[785]: ts=2026-09-23T16:00:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563944583worker2 # [ 87.120263] nix-grpc-daemon[785]: ts=2026-09-23T16:00:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563944584worker2 # [ 87.134383] nix-grpc-daemon[854]: time=2026-09-23T16:00:56.758Z level=INFO msg="All 2 paths already cached"4585worker2 # [ 87.144615] nix-grpc-daemon[785]: ts=2026-09-23T16:00:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563944586worker2 # [ 87.158477] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4587worker2 # [ 87.318364] nix-daemon[842]: accepted connection from pid 1161, user nix-grpc-daemon (trusted)4588worker2 # [ 87.339842] nix-daemon[842]: reaped child process 1164, status = succeeded4589lb # [ 87.351962] niks3-server[953]: 2026/09/23 16:00:56 INFO Received push request method=POST path=/api/pushes4590worker2 # [ 87.352776] nix-grpc-daemon[854]: time=2026-09-23T16:00:56.976Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4591worker2 # [ 87.354791] nix-grpc-daemon[854]: time=2026-09-23T16:00:56.976Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)"4592lb # [ 87.382558] niks3-server[953]: 2026/09/23 16:00:56 INFO Received sign narinfos request method=POST path=/api/pushes/25/sign4593worker2 # [ 87.382493] nix-grpc-daemon[854]: time=2026-09-23T16:00:57.006Z level=INFO msg="Uploading 1 narinfos"4594lb # [ 87.398040] niks3-server[953]: 2026/09/23 16:00:57 INFO Signed narinfos id=25 count=14595lb # [ 87.405483] niks3-server[953]: 2026/09/23 16:00:57 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst4596lb # [ 87.421107] niks3-server[953]: 2026/09/23 16:00:57 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls4597worker2 # [ 87.420409] nix-grpc-daemon[854]: time=2026-09-23T16:00:57.044Z level=INFO msg="Upload complete. (244ms)"4598lb # [ 87.429438] niks3-server[953]: 2026/09/23 16:00:57 INFO Received complete push request method=POST path=/api/pushes/25/complete4599lb # [ 87.434615] niks3-server[953]: 2026/09/23 16:00:57 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo4600worker2 # [ 87.432998] nix-grpc-daemon[785]: ts=2026-09-23T16:00:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=0 paths=1 nar_bytes_in=7524601worker2 # [ 87.448554] nix-grpc-daemon[785]: ts=2026-09-23T16:00:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563944602worker2 # [ 87.455676] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4603worker2 # [ 87.462161] nix-daemon[842]: reaped child process 1159, status = succeeded4604worker2 # [ 87.474744] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4605lb # [ 88.086675] envoy[897]: conn peer=[2001:db8:1::1]:36386 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3003 tx=788 ms=42994606lb # [ 88.091706] envoy[897]: conn peer=[2001:db8:1::1]:36400 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1592 tx=1598 ms=47354607lb # [ 88.096810] envoy[897]: conn peer=[2001:db8:1::1]:42766 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=333 ms=424608lb # [ 88.102581] envoy[897]: conn peer=[2001:db8:1::1]:42754 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3445 tx=1940 ms=23754609lb # [ 88.108813] envoy[897]: conn peer=[2001:db8:1::1]:42782 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3887 tx=2157 ms=20044610lb # [ 88.114507] envoy[897]: conn peer=[2001:db8:1::1]:42784 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=375 ms=304611worker2 # [ 88.736892] nix-daemon[842]: reaped child process 1168, status = succeeded4612worker2 # [ 88.827158] nix-daemon[842]: accepted connection from pid 1184, user nix-grpc-daemon (trusted)4613worker2 # [ 88.846333] nix-daemon[842]: reaped child process 1187, status = succeeded4614lb # [ 88.861992] niks3-server[953]: 2026/09/23 16:00:58 INFO Received push request method=POST path=/api/pushes4615worker2 # [ 88.864402] nix-grpc-daemon[854]: time=2026-09-23T16:00:58.486Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4616worker2 # [ 88.865936] nix-grpc-daemon[854]: time=2026-09-23T16:00:58.486Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)"4617worker2 # [ 88.907485] nix-grpc-daemon[854]: time=2026-09-23T16:00:58.531Z level=INFO msg="Uploading 1 narinfos"4618lb # [ 88.906679] niks3-server[953]: 2026/09/23 16:00:58 INFO Received sign narinfos request method=POST path=/api/pushes/26/sign4619lb # [ 88.920569] niks3-server[953]: 2026/09/23 16:00:58 INFO Signed narinfos id=26 count=14620lb # [ 88.931578] niks3-server[953]: 2026/09/23 16:00:58 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls4621lb # [ 88.941085] niks3-server[953]: 2026/09/23 16:00:58 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv4622lb # [ 88.956524] niks3-server[953]: 2026/09/23 16:00:58 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst4623lb # [ 88.965231] niks3-server[953]: 2026/09/23 16:00:58 INFO Received complete push request method=POST path=/api/pushes/26/complete4624worker2 # [ 88.956836] nix-grpc-daemon[854]: time=2026-09-23T16:00:58.580Z level=INFO msg="Upload complete. (267ms)"4625lb # [ 88.968719] niks3-server[953]: 2026/09/23 16:00:58 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo4626worker2 # [ 88.969297] nix-grpc-daemon[785]: ts=2026-09-23T16:00:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=14 outputs=14627worker1 # [ 89.063414] nix-grpc-daemon[485]: ts=2026-09-23T16:00:58Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:596604628worker2 # [ 88.988890] nix-daemon[842]: reaped child process 1166, status = succeeded4629worker1 # [ 89.084207] nix-grpc-daemon[485]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'...4630worker1 # [ 89.101190] nix-grpc-daemon[485]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'...4631worker1 # [ 89.115829] nix-grpc-daemon[485]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'...4632worker2 # [ 89.064299] nix-grpc-daemon[785]: ts=2026-09-23T16:00:58Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:563944633worker2 # [ 89.071233] nix-grpc-daemon[785]: ts=2026-09-23T16:00:58Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=0 paths=1 nar_bytes_out=2324634client # [ 89.160673] nix-daemon[1152]: reaped child process 1154, status = succeeded4635client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.34 seconds)4636client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top4637client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.04 seconds)4638client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4639client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds)4640client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4641client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)4642(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.45 seconds)4643subtest: an input only one worker has still reaches the builder4644client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only4645client # this derivation will be built:4646client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv4647client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'...4648client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only, in 0.30 seconds)4649client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure4650client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.12 seconds)4651worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4652worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.20 seconds)4653worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only4654worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.03 seconds)4655client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&24656client # [ 90.127009] nix-daemon[1152]: accepted connection from pid 1234, user root (trusted)4657client # this derivation will be built:4658client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4659client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4660worker2 # [ 90.271697] nix-grpc-daemon[785]: ts=2026-09-23T16:00:59Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:563944661worker2 # [ 90.279526] nix-grpc-daemon[785]: ts=2026-09-23T16:00:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563944662client # copying 1 paths...4663client # copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' to 'grpc://lb:50051'...4664worker1 # [ 90.381829] nix-grpc-daemon[485]: ts=2026-09-23T16:01:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596604665worker1 # [ 90.397084] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4666worker1 # [ 90.574449] nix-daemon[849]: accepted connection from pid 1129, user nix-grpc-daemon (trusted)4667worker1 # [ 90.593922] nix-daemon[849]: reaped child process 1132, status = succeeded4668lb # [ 90.523354] niks3-server[953]: 2026/09/23 16:01:00 INFO Received push request method=POST path=/api/pushes4669worker1 # [ 90.612561] nix-grpc-daemon[901]: time=2026-09-23T16:01:00.231Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4670worker1 # [ 90.614246] nix-grpc-daemon[901]: time=2026-09-23T16:01:00.231Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)"4671lb # [ 90.554978] niks3-server[953]: 2026/09/23 16:01:00 INFO Received sign narinfos request method=POST path=/api/pushes/27/sign4672worker1 # [ 90.646766] nix-grpc-daemon[901]: time=2026-09-23T16:01:00.265Z level=INFO msg="Uploading 1 narinfos"4673lb # [ 90.569619] niks3-server[953]: 2026/09/23 16:01:00 INFO Signed narinfos id=27 count=14674lb # [ 90.576562] niks3-server[953]: 2026/09/23 16:01:00 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4675lb # [ 90.587159] niks3-server[953]: 2026/09/23 16:01:00 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls4676lb # [ 90.600951] niks3-server[953]: 2026/09/23 16:01:00 INFO Received complete push request method=POST path=/api/pushes/27/complete4677worker1 # [ 90.687597] nix-grpc-daemon[901]: time=2026-09-23T16:01:00.307Z level=INFO msg="Upload complete. (275ms)"4678lb # [ 90.607757] niks3-server[953]: 2026/09/23 16:01:00 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo4679worker1 # [ 90.701204] nix-grpc-daemon[485]: ts=2026-09-23T16:01:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=0 paths=1 nar_bytes_in=1204680worker1 # [ 90.724384] nix-grpc-daemon[485]: ts=2026-09-23T16:01:00Z level=debug event=want client=14 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=04681worker1 # [ 90.729485] nix-grpc-daemon[485]: ts=2026-09-23T16:01:00Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.3:50051 clients=14682worker1 # [ 90.736657] nix-grpc-daemon[485]: ts=2026-09-23T16:01:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596604683worker1 # [ 90.742751] nix-daemon[849]: reaped child process 1127, status = succeeded4684worker1 # [ 90.746298] nix-grpc-daemon[485]: ts=2026-09-23T16:01:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596604685client # copying 2 paths to 'lb:50051'...4686client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'...4687worker1 # [ 90.762911] nix-grpc-daemon[485]: ts=2026-09-23T16:01:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596604688client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'...4689worker1 # [ 90.776191] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4690worker1 # [ 90.920438] nix-daemon[849]: accepted connection from pid 1138, user nix-grpc-daemon (trusted)4691worker1 # [ 90.943273] nix-daemon[849]: reaped child process 1141, status = succeeded4692lb # [ 90.860963] niks3-server[953]: 2026/09/23 16:01:00 INFO Received push request method=POST path=/api/pushes4693worker1 # [ 90.952184] nix-grpc-daemon[901]: time=2026-09-23T16:01:00.571Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4694worker1 # [ 90.953681] nix-grpc-daemon[901]: time=2026-09-23T16:01:00.571Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)"4695worker1 # [ 90.955449] nix-grpc-daemon[901]: time=2026-09-23T16:01:00.571Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)"4696lb # [ 90.902878] niks3-server[953]: 2026/09/23 16:01:00 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst4697lb # [ 90.917875] niks3-server[953]: 2026/09/23 16:01:00 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst4698worker1 # [ 91.013077] nix-grpc-daemon[901]: time=2026-09-23T16:01:00.632Z level=INFO msg="Uploading 2 narinfos"4699lb # [ 90.931567] niks3-server[953]: 2026/09/23 16:01:00 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls4700lb # [ 90.939819] niks3-server[953]: 2026/09/23 16:01:00 INFO Received sign narinfos request method=POST path=/api/pushes/28/sign4701lb # [ 90.946396] niks3-server[953]: 2026/09/23 16:01:00 INFO Signed narinfos id=28 count=24702lb # [ 90.951126] niks3-server[953]: 2026/09/23 16:01:00 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls4703lb # [ 90.964809] niks3-server[953]: 2026/09/23 16:01:00 INFO Received complete push request method=POST path=/api/pushes/28/complete4704worker1 # [ 91.054120] nix-grpc-daemon[901]: time=2026-09-23T16:01:00.673Z level=INFO msg="Upload complete. (262ms)"4705lb # [ 90.973917] niks3-server[953]: 2026/09/23 16:01:00 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo4706lb # [ 90.980385] niks3-server[953]: 2026/09/23 16:01:00 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4707worker1 # [ 91.064390] nix-grpc-daemon[485]: ts=2026-09-23T16:01:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=0 paths=2 nar_bytes_in=9844708worker1 # [ 91.084845] nix-grpc-daemon[485]: ts=2026-09-23T16:01:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596604709worker1 # [ 91.093358] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4710worker1 # [ 91.100367] nix-daemon[849]: reaped child process 1136, status = succeeded4711client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051...4712worker1 # [ 91.114595] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4713client # uses-input-w1onlya> node-a: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4714client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds4715worker1 # [ 92.188612] nix-daemon[849]: reaped child process 1145, status = succeeded4716worker1 # [ 92.271861] nix-daemon[849]: accepted connection from pid 1161, user nix-grpc-daemon (trusted)4717worker1 # [ 92.288996] nix-daemon[849]: reaped child process 1164, status = succeeded4718lb # [ 92.214286] niks3-server[953]: 2026/09/23 16:01:01 INFO Received push request method=POST path=/api/pushes4719worker1 # [ 92.306640] nix-grpc-daemon[901]: time=2026-09-23T16:01:01.924Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4720lb # [ 92.245024] niks3-server[953]: 2026/09/23 16:01:01 INFO Received sign narinfos request method=POST path=/api/pushes/29/sign4721worker1 # [ 92.335158] nix-grpc-daemon[901]: time=2026-09-23T16:01:01.954Z level=INFO msg="Uploading 1 narinfos"4722lb # [ 92.257472] niks3-server[953]: 2026/09/23 16:01:01 INFO Signed narinfos id=29 count=14723lb # [ 92.268279] niks3-server[953]: 2026/09/23 16:01:01 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4724lb # [ 92.280479] niks3-server[953]: 2026/09/23 16:01:01 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls4725lb # [ 92.292171] niks3-server[953]: 2026/09/23 16:01:01 INFO Received complete push request method=POST path=/api/pushes/29/complete4726worker1 # [ 92.377812] nix-grpc-daemon[901]: time=2026-09-23T16:01:01.997Z level=INFO msg="Upload complete. (251ms)"4727lb # [ 92.299316] niks3-server[953]: 2026/09/23 16:01:01 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo4728client # copying 1 paths...4729worker1 # [ 92.393172] nix-grpc-daemon[485]: ts=2026-09-23T16:01:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=15 outputs=14730worker1 # [ 92.411189] nix-grpc-daemon[485]: ts=2026-09-23T16:01:02Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:596604731worker1 # [ 92.417033] nix-daemon[849]: reaped child process 1143, status = succeeded4732client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'...4733worker2 # [ 92.354573] nix-grpc-daemon[785]: ts=2026-09-23T16:01:01Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:563944734worker2 # [ 92.367671] nix-grpc-daemon[785]: copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'http://lb:5751'...4735worker2 # [ 92.385377] nix-grpc-daemon[785]: ts=2026-09-23T16:01:02Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=0 paths=1 nar_bytes_out=1204736client # [ 92.472940] nix-daemon[1152]: reaped child process 1237, status = succeeded4737client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.54 seconds)4738client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&24739client # [ 92.642794] nix-daemon[1152]: accepted connection from pid 1256, user root (trusted)4740client # this derivation will be built:4741client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4742worker1 # [ 92.875725] nix-grpc-daemon[485]: ts=2026-09-23T16:01:02Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:596604743client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4744worker2 # [ 92.789319] nix-grpc-daemon[785]: ts=2026-09-23T16:01:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563944745worker1 # [ 92.904878] nix-grpc-daemon[485]: ts=2026-09-23T16:01:02Z level=debug event=want client=15 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=04746worker1 # [ 92.911463] nix-grpc-daemon[485]: ts=2026-09-23T16:01:02Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.4:50051 clients=14747worker2 # [ 92.824739] nix-grpc-daemon[785]: ts=2026-09-23T16:01:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563944748worker2 # [ 92.839687] nix-grpc-daemon[785]: ts=2026-09-23T16:01:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563944749client # copying 1 paths to 'lb:50051'...4750client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'...4751worker2 # [ 92.858816] nix-grpc-daemon[785]: ts=2026-09-23T16:01:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563944752worker2 # [ 92.874225] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4753worker2 # [ 92.896954] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 57 ms (attempt 1/5)4754worker2 # [ 92.961921] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 140 ms (attempt 2/5)4755worker2 # [ 93.103995] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 156 ms (attempt 3/5)4756worker2 # [ 93.261998] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 775 ms (attempt 4/5)4757worker2 # [ 94.040786] nix-grpc-daemon[785]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4758worker2 # [ 94.045783] nix-grpc-daemon[785]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4759worker2 # [ 94.055520] nix-grpc-daemon[785]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'...4760worker2 # [ 94.238088] nix-daemon[842]: accepted connection from pid 1200, user nix-grpc-daemon (trusted)4761worker2 # [ 94.260328] nix-daemon[842]: reaped child process 1203, status = succeeded4762lb # [ 94.273448] niks3-server[953]: 2026/09/23 16:01:03 INFO Received push request method=POST path=/api/pushes4763worker2 # [ 94.276715] nix-grpc-daemon[854]: time=2026-09-23T16:01:03.899Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4764worker2 # [ 94.278428] nix-grpc-daemon[854]: time=2026-09-23T16:01:03.899Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)"4765lb # [ 94.306451] niks3-server[953]: 2026/09/23 16:01:03 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst4766worker2 # [ 94.309402] nix-grpc-daemon[854]: time=2026-09-23T16:01:03.932Z level=INFO msg="Uploading 1 narinfos"4767lb # [ 94.318814] niks3-server[953]: 2026/09/23 16:01:03 INFO Received sign narinfos request method=POST path=/api/pushes/30/sign4768lb # [ 94.331229] niks3-server[953]: 2026/09/23 16:01:03 INFO Signed narinfos id=30 count=14769lb # [ 94.337189] niks3-server[953]: 2026/09/23 16:01:03 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls4770lb # [ 94.349528] niks3-server[953]: 2026/09/23 16:01:03 INFO Received complete push request method=POST path=/api/pushes/30/complete4771worker2 # [ 94.345767] nix-grpc-daemon[854]: time=2026-09-23T16:01:03.969Z level=INFO msg="Upload complete. (270ms)"4772lb # [ 94.356254] niks3-server[953]: 2026/09/23 16:01:03 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo4773worker2 # [ 94.357043] nix-grpc-daemon[785]: ts=2026-09-23T16:01:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=1 paths=1 nar_bytes_in=5684774worker2 # [ 94.369949] nix-grpc-daemon[785]: ts=2026-09-23T16:01:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563944775worker2 # [ 94.382509] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4776worker2 # [ 94.392326] nix-daemon[842]: reaped child process 1193, status = succeeded4777worker2 # [ 94.397410] nix-grpc-daemon[785]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'...4778worker2 # [ 94.411512] nix-daemon[842]: accepted connection from pid 785, user nix-grpc-daemon (trusted)4779client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051...4780client # uses-input-w1onlyb> worker2: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4781client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds4782worker2 # [ 95.276378] nix-daemon[842]: reaped child process 1208, status = succeeded4783worker2 # [ 95.366472] nix-daemon[842]: accepted connection from pid 1224, user nix-grpc-daemon (trusted)4784worker2 # [ 95.386674] nix-daemon[842]: reaped child process 1227, status = succeeded4785lb # [ 95.394562] niks3-server[953]: 2026/09/23 16:01:05 INFO Received push request method=POST path=/api/pushes4786worker2 # [ 95.396476] nix-grpc-daemon[854]: time=2026-09-23T16:01:05.018Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4787lb # [ 95.425400] niks3-server[953]: 2026/09/23 16:01:05 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls4788worker2 # [ 95.429361] nix-grpc-daemon[854]: time=2026-09-23T16:01:05.052Z level=INFO msg="Uploading 1 narinfos"4789lb # [ 95.438590] niks3-server[953]: 2026/09/23 16:01:05 INFO Received sign narinfos request method=POST path=/api/pushes/31/sign4790lb # [ 95.449556] niks3-server[953]: 2026/09/23 16:01:05 INFO Signed narinfos id=31 count=14791lb # [ 95.460158] niks3-server[953]: 2026/09/23 16:01:05 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4792worker2 # [ 95.459183] nix-grpc-daemon[854]: time=2026-09-23T16:01:05.082Z level=INFO msg="Upload complete. (234ms)"4793lb # [ 95.468735] niks3-server[953]: 2026/09/23 16:01:05 INFO Received complete push request method=POST path=/api/pushes/31/complete4794lb # [ 95.472141] niks3-server[953]: 2026/09/23 16:01:05 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo4795client # copying 1 paths...4796worker2 # [ 95.468860] nix-grpc-daemon[785]: ts=2026-09-23T16:01:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56394 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=16 outputs=14797worker1 # [ 95.567855] nix-grpc-daemon[485]: ts=2026-09-23T16:01:05Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:596604798worker2 # [ 95.483372] nix-daemon[842]: reaped child process 1205, status = succeeded4799worker1 # [ 95.580970] nix-grpc-daemon[485]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'...4800client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'...4801worker1 # [ 95.627634] nix-grpc-daemon[485]: ts=2026-09-23T16:01:05Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:596604802worker1 # [ 95.634888] nix-grpc-daemon[485]: ts=2026-09-23T16:01:05Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=0 paths=1 nar_bytes_out=1204803client # [ 95.632299] nix-daemon[1152]: reaped child process 1259, status = succeeded4804client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.16 seconds)4805client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&24806client # [ 95.808124] nix-daemon[1152]: accepted connection from pid 1278, user root (trusted)4807client # this derivation will be built:4808client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4809worker1 # [ 96.041783] nix-grpc-daemon[485]: ts=2026-09-23T16:01:05Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:596604810client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4811worker2 # [ 95.955435] nix-grpc-daemon[785]: ts=2026-09-23T16:01:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563944812worker2 # [ 95.967608] nix-grpc-daemon[854]: time=2026-09-23T16:01:05.591Z level=INFO msg="All 1 paths already cached"4813worker1 # [ 96.074407] nix-grpc-daemon[485]: ts=2026-09-23T16:01:05Z level=debug event=want client=16 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=04814worker1 # [ 96.082178] nix-grpc-daemon[485]: ts=2026-09-23T16:01:05Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.3:50051 clients=14815worker1 # [ 96.091518] nix-grpc-daemon[485]: ts=2026-09-23T16:01:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596604816worker1 # [ 96.103026] nix-grpc-daemon[485]: ts=2026-09-23T16:01:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:596604817client # copying 1 paths to 'lb:50051'...4818worker1 # [ 96.119514] nix-grpc-daemon[901]: time=2026-09-23T16:01:05.736Z level=INFO msg="All 1 paths already cached"4819client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'...4820worker1 # [ 96.125221] nix-grpc-daemon[485]: ts=2026-09-23T16:01:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:596604821worker1 # [ 96.140647] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4822worker1 # [ 96.294336] nix-daemon[849]: accepted connection from pid 1173, user nix-grpc-daemon (trusted)4823worker1 # [ 96.315430] nix-daemon[849]: reaped child process 1176, status = succeeded4824lb # [ 96.232447] niks3-server[953]: 2026/09/23 16:01:05 INFO Received push request method=POST path=/api/pushes4825worker1 # [ 96.325074] nix-grpc-daemon[901]: time=2026-09-23T16:01:05.944Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4826worker1 # [ 96.326559] nix-grpc-daemon[901]: time=2026-09-23T16:01:05.944Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)"4827worker1 # [ 96.363476] nix-grpc-daemon[901]: time=2026-09-23T16:01:05.982Z level=INFO msg="Uploading 1 narinfos"4828lb # [ 96.271924] niks3-server[953]: 2026/09/23 16:01:05 INFO Received sign narinfos request method=POST path=/api/pushes/32/sign4829lb # [ 96.288657] niks3-server[953]: 2026/09/23 16:01:05 INFO Signed narinfos id=32 count=14830lb # [ 96.294803] niks3-server[953]: 2026/09/23 16:01:05 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls4831lb # [ 96.306186] niks3-server[953]: 2026/09/23 16:01:05 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst4832lb # [ 96.316275] niks3-server[953]: 2026/09/23 16:01:05 INFO Received complete push request method=POST path=/api/pushes/32/complete4833worker1 # [ 96.401938] nix-grpc-daemon[901]: time=2026-09-23T16:01:06.021Z level=INFO msg="Upload complete. (244ms)"4834lb # [ 96.322809] niks3-server[953]: 2026/09/23 16:01:05 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo4835worker1 # [ 96.412874] nix-grpc-daemon[485]: ts=2026-09-23T16:01:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=0 paths=1 nar_bytes_in=5684836worker1 # [ 96.428538] nix-grpc-daemon[485]: ts=2026-09-23T16:01:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:596604837worker1 # [ 96.437481] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4838worker1 # [ 96.447466] nix-daemon[849]: reaped child process 1171, status = succeeded4839worker1 # [ 96.454137] nix-daemon[849]: accepted connection from pid 485, user nix-grpc-daemon (trusted)4840client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051...4841client # uses-input-w1onlyc> node-a: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4842client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds4843worker1 # [ 97.320255] nix-daemon[849]: reaped child process 1180, status = succeeded4844worker1 # [ 97.425733] nix-daemon[849]: accepted connection from pid 1196, user nix-grpc-daemon (trusted)4845worker1 # [ 97.443474] nix-daemon[849]: reaped child process 1199, status = succeeded4846lb # [ 97.370045] niks3-server[953]: 2026/09/23 16:01:06 INFO Received push request method=POST path=/api/pushes4847worker1 # [ 97.462330] nix-grpc-daemon[901]: time=2026-09-23T16:01:07.080Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4848lb # [ 97.403185] niks3-server[953]: 2026/09/23 16:01:07 INFO Received sign narinfos request method=POST path=/api/pushes/33/sign4849worker1 # [ 97.493292] nix-grpc-daemon[901]: time=2026-09-23T16:01:07.112Z level=INFO msg="Uploading 1 narinfos"4850lb # [ 97.417209] niks3-server[953]: 2026/09/23 16:01:07 INFO Signed narinfos id=33 count=14851lb # [ 97.424058] niks3-server[953]: 2026/09/23 16:01:07 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls4852lb # [ 97.427894] niks3-server[953]: 2026/09/23 16:01:07 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4853lb # [ 97.442909] niks3-server[953]: 2026/09/23 16:01:07 INFO Received complete push request method=POST path=/api/pushes/33/complete4854lb # [ 97.451576] niks3-server[953]: 2026/09/23 16:01:07 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo4855worker1 # [ 97.538630] nix-grpc-daemon[901]: time=2026-09-23T16:01:07.158Z level=INFO msg="Upload complete. (272ms)"4856client # copying 1 paths...4857worker1 # [ 97.553174] nix-grpc-daemon[485]: ts=2026-09-23T16:01:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=17 outputs=14858worker2 # [ 97.466079] nix-grpc-daemon[785]: ts=2026-09-23T16:01:07Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:563944859worker2 # [ 97.480630] nix-grpc-daemon[785]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'...4860worker1 # [ 97.573772] nix-daemon[849]: reaped child process 1178, status = succeeded4861client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'...4862worker1 # [ 97.614301] nix-grpc-daemon[485]: ts=2026-09-23T16:01:07Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:596604863worker1 # [ 97.621423] nix-grpc-daemon[485]: ts=2026-09-23T16:01:07Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59660 duration_s=0 paths=1 nar_bytes_out=1204864client # [ 97.621315] nix-daemon[1152]: reaped child process 1281, status = succeeded4865client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2, in 1.99 seconds)4866(finished: subtest: an input only one worker has still reaches the builder, in 8.34 seconds)4867subtest: upload whose reference the worker lacks is completed from the cache4868client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache4869client # this derivation will be built:4870client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv4871client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'...4872client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache, in 0.33 seconds)4873client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache4874lb # [ 98.087692] envoy[897]: conn peer=[2001:db8:1::1]:42802 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=412 tx=406 ms=274875client # this derivation will be built:4876client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv4877lb # [ 98.092606] envoy[897]: conn peer=[2001:db8:1::1]:42792 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4378 tx=2392 ms=20264878lb # [ 98.097701] envoy[897]: conn peer=[2001:db8:1::1]:42818 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4250 tx=2073 ms=21314879lb # [ 98.104144] envoy[897]: conn peer=[2001:db8:1::1]:39070 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=329 ms=544880lb # [ 98.109610] envoy[897]: conn peer=[2001:db8:1::1]:39076 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3757 tx=2127 ms=27754881lb # [ 98.114513] envoy[897]: conn peer=[2001:db8:1::1]:39082 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=328 ms=314882lb # [ 98.119256] envoy[897]: conn peer=[2001:db8:1::1]:39088 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3743 tx=2099 ms=15944883lb # [ 98.124144] envoy[897]: conn peer=[2001:db8:1::1]:39094 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=422 tx=328 ms=314884client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'...4885client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache, in 0.29 seconds)4886client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure4887client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.12 seconds)4888worker1: must succeed: nix-store --import < /tmp/shared/ref.closure4889worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.20 seconds)4890worker2 # [ 98.632935] systemd[1]: Stopping Nix worker-protocol over gRPC...4891worker2 # [ 98.897867] nix-grpc-daemon[785]: ts=2026-09-23T16:01:08Z level=info event=signal_exit4892worker2 # [ 100.870588] nix-grpc-daemon[785]: ts=2026-09-23T16:01:10Z level=info event=scheduler_restarting4893worker1 # [ 100.972686] nix-grpc-daemon[485]: ts=2026-09-23T16:01:10Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:596464894worker1 # [ 100.981080] nix-grpc-daemon[485]: ts=2026-09-23T16:01:10Z level=info event=worker_gone addr=192.168.1.4:500514895worker2 # [ 100.908653] nix-grpc-daemon[785]: ts=2026-09-23T16:01:10Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4896worker2 # [ 100.938396] nix-daemon[842]: reaped child process 844, status = succeeded4897worker2 # [ 101.543374] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4898worker2 # [ 101.551297] systemd[1]: Stopped Nix worker-protocol over gRPC.4899worker2 # [ 101.556809] systemd[1]: nix-grpc-daemon.service: Consumed 3.417s CPU time over 1min 7.869s wall clock time, 62.4M memory peak, 178.5K incoming IP traffic, 239.9K outgoing IP traffic.4900lb: waiting until exactly ['worker1'] are healthy members of aarch64-linux4901lb: must succeed: curl -sf localhost:9901/clusters4902lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4903lb: (finished: waiting until exactly ['worker1'] are healthy members of aarch64-linux, in 0.08 seconds)4904client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&24905worker1 # [ 101.959421] nix-grpc-daemon[485]: ts=2026-09-23T16:01:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:596604906worker1 # [ 101.968957] nix-grpc-daemon[485]: ts=2026-09-23T16:01:11Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:596604907client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4908client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.25 seconds)4909worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4910worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.03 seconds)4911worker2 # [ 101.996247] systemd[1]: Starting scheduler-token.service...4912worker2 # [ 102.063112] systemd[1]: scheduler-token.service: Deactivated successfully.4913worker2 # [ 102.067211] systemd[1]: Finished scheduler-token.service.4914worker2 # [ 102.082471] systemd[1]: scheduler-token.service: Consumed 36ms CPU time over 69ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4915worker2 # [ 102.088997] systemd[1]: Starting Nix worker-protocol over gRPC...4916worker2 # [ 102.202457] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:11Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4917worker2 # [ 102.204824] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:11Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4918worker2 # [ 102.208687] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:11Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=14919worker2 # [ 102.215176] nix-grpc-daemon[1252]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR4920worker2 # [ 102.236140] nix-grpc-daemon[1252]: E0000 00:00:1790179271.836836 1252 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4921worker2 # [ 102.238587] nix-grpc-daemon[1252]: E0000 00:00:1790179271.838849 1252 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4922worker2 # [ 102.264352] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:11Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket4923worker2 # [ 102.266601] nix-grpc-daemon[1252]: E0000 00:00:1790179271.848913 1292 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4924worker2 # [ 102.269312] nix-grpc-daemon[1252]: E0000 00:00:1790179271.871304 1292 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4925worker2 # [ 102.274675] nix-grpc-daemon[1252]: E0000 00:00:1790179271.871496 1292 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4926worker2 # [ 102.281589] nix-grpc-daemon[1252]: E0000 00:00:1790179271.887082 1292 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4927worker2 # [ 102.284342] systemd[1]: Started Nix worker-protocol over gRPC.4928worker2 # [ 102.285197] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:11Z level=info event=scheduler_connected addr=lb:500514929worker1 # [ 102.373478] nix-grpc-daemon[485]: ts=2026-09-23T16:01:11Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:596464930worker1 # [ 102.391776] nix-grpc-daemon[485]: ts=2026-09-23T16:01:11Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=04931worker1 # [ 102.427992] systemd[1]: Stopping Nix worker-protocol over gRPC...4932worker1 # [ 102.567762] nix-grpc-daemon[485]: ts=2026-09-23T16:01:12Z level=info event=signal_exit4933lb # [ 102.912826] niks3-server[953]: 2026/09/23 16:01:12 INFO lead: released remote=192.168.1.3:552444934worker1 # [ 103.996692] nix-grpc-daemon[485]: ts=2026-09-23T16:01:13Z level=info event=scheduler_restarting4935worker2 # [ 103.907516] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:13Z level=info event=scheduler_restarting addr=lb:500514936lb # [ 103.914680] envoy[897]: [2026-09-23 16:01:13.526][897][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream}4937worker1 # [ 104.001609] nix-grpc-daemon[485]: ts=2026-09-23T16:01:13Z level=info event=scheduler_restarting addr=lb:500514938worker1 # [ 104.001769] nix-grpc-daemon[485]: ts=2026-09-23T16:01:13Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:596464939worker1 # [ 104.001881] nix-grpc-daemon[485]: ts=2026-09-23T16:01:13Z level=info event=worker_gone addr=192.168.1.3:500514940worker1 # [ 104.006069] nix-grpc-daemon[485]: ts=2026-09-23T16:01:13Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:596464941worker1 # [ 104.027060] nix-grpc-daemon[485]: ts=2026-09-23T16:01:13Z level=info event=worker_gone addr=192.168.1.4:500514942worker1 # [ 104.057871] nix-daemon[849]: reaped child process 851, status = succeeded4943worker2 # [ 104.107974] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:13Z level=info event=scheduler_connected addr=lb:500514944worker2 # [ 104.113641] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4945worker1 # [ 104.222246] nix-grpc-daemon[485]: ts=2026-09-23T16:01:13Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4946worker2 # [ 104.314534] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:13Z level=info event=scheduler_connected addr=lb:500514947worker2 # [ 104.320159] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4948worker2 # [ 104.521269] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:14Z level=info event=scheduler_connected addr=lb:500514949worker2 # [ 104.526718] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4950worker2 # [ 104.727947] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:14Z level=info event=scheduler_connected addr=lb:500514951worker2 # [ 104.733323] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4952worker2 # [ 104.936593] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:14Z level=info event=scheduler_connected addr=lb:500514953worker2 # [ 104.940749] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4954worker2 # [ 105.137341] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:14Z level=info event=scheduler_connected addr=lb:500514955worker2 # [ 105.142712] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4956worker2 # [ 105.343919] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:14Z level=info event=scheduler_connected addr=lb:500514957worker2 # [ 105.349156] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:14Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4958worker1 # [ 105.566450] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4959worker1 # [ 105.571417] systemd[1]: Stopped Nix worker-protocol over gRPC.4960worker1 # [ 105.580742] systemd[1]: nix-grpc-daemon.service: Consumed 2.902s CPU time over 1min 35.402s wall clock time, 67M memory peak, 539.6K incoming IP traffic, 262.1K outgoing IP traffic.4961lb: waiting until exactly ['worker2'] are healthy members of aarch64-linux4962lb: must succeed: curl -sf localhost:9901/clusters4963worker2 # [ 105.550287] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:15Z level=info event=scheduler_connected addr=lb:500514964worker2 # [ 105.555682] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4965lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4966lb: (finished: waiting until exactly ['worker2'] are healthy members of aarch64-linux, in 0.08 seconds)4967client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&24968worker2 # [ 105.756834] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:15Z level=info event=scheduler_connected addr=lb:500514969worker2 # [ 105.762068] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4970worker2 # [ 105.815268] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:487484971worker2 # [ 105.830496] nix-daemon[842]: accepted connection from pid 1252, user nix-grpc-daemon (trusted)4972worker2 # [ 105.856932] nix-grpc-daemon[1252]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 88 ms (attempt 1/5)4973worker2 # [ 105.954015] nix-grpc-daemon[1252]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 17 ms (attempt 2/5)4974worker2 # [ 105.963318] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:15Z level=info event=scheduler_connected addr=lb:500514975worker2 # [ 105.971838] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4976worker2 # [ 105.976979] nix-grpc-daemon[1252]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 63 ms (attempt 3/5)4977worker2 # [ 106.036853] nix-grpc-daemon[1252]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 224 ms (attempt 4/5)4978worker2 # [ 106.170303] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:15Z level=info event=scheduler_connected addr=lb:500514979worker2 # [ 106.175494] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4980worker2 # [ 106.263370] nix-grpc-daemon[1252]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4981worker2 # [ 106.269665] nix-grpc-daemon[1252]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4982client # copying 2 paths...4983client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'...4984worker2 # [ 106.294260] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:487484985client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'...4986worker2 # [ 106.312814] nix-daemon[842]: accepted connection from pid 1252, user nix-grpc-daemon (trusted)4987worker2 # [ 106.340865] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:15Z level=info event=niks3_push_started pid=13124988worker2 # [ 106.376660] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:16Z level=info event=scheduler_connected addr=lb:500514989worker2 # [ 106.381642] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4990worker2 # [ 106.510668] nix-grpc-daemon[1312]: time=2026-09-23T16:01:16.132Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""4991worker2 # [ 106.584502] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:16Z level=info event=scheduler_connected addr=lb:500514992worker2 # [ 106.604197] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4993worker2 # [ 106.622682] nix-daemon[842]: accepted connection from pid 1325, user nix-grpc-daemon (trusted)4994worker2 # [ 106.638539] nix-daemon[842]: reaped child process 1328, status = succeeded4995lb # [ 106.656915] niks3-server[953]: 2026/09/23 16:01:16 INFO Received push request method=POST path=/api/pushes4996worker2 # [ 106.663251] nix-grpc-daemon[1312]: time=2026-09-23T16:01:16.285Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4997worker2 # [ 106.666820] nix-grpc-daemon[1312]: time=2026-09-23T16:01:16.285Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)"4998worker2 # [ 106.669313] nix-grpc-daemon[1312]: time=2026-09-23T16:01:16.285Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)"4999lb # [ 106.720186] niks3-server[953]: 2026/09/23 16:01:16 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls5000worker2 # [ 106.723063] nix-grpc-daemon[1312]: time=2026-09-23T16:01:16.345Z level=INFO msg="Uploading 2 narinfos"5001lb # [ 106.733413] niks3-server[953]: 2026/09/23 16:01:16 INFO Received sign narinfos request method=POST path=/api/pushes/34/sign5002lb # [ 106.741642] niks3-server[953]: 2026/09/23 16:01:16 INFO Signed narinfos id=34 count=25003lb # [ 106.746363] niks3-server[953]: 2026/09/23 16:01:16 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls5004lb # [ 106.752614] niks3-server[953]: 2026/09/23 16:01:16 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5005lb # [ 106.755518] niks3-server[953]: 2026/09/23 16:01:16 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst5006lb # [ 106.769266] niks3-server[953]: 2026/09/23 16:01:16 INFO Received complete push request method=POST path=/api/pushes/34/complete5007worker2 # [ 106.760000] nix-grpc-daemon[1312]: time=2026-09-23T16:01:16.383Z level=INFO msg="Upload complete. (250ms)"5008lb # [ 106.771600] niks3-server[953]: 2026/09/23 16:01:16 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo5009lb # [ 106.774632] niks3-server[953]: 2026/09/23 16:01:16 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5010worker2 # [ 106.772139] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48748 duration_s=0 paths=2 nar_bytes_in=3125011worker2 # [ 106.786247] nix-daemon[842]: reaped child process 1310, status = succeeded5012worker2 # [ 106.789875] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:16Z level=info event=scheduler_connected addr=lb:500515013worker2 # [ 106.793819] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5014client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.22 seconds)5015worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache5016worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.03 seconds)5017worker1 # [ 107.000808] systemd[1]: Starting Nix worker-protocol over gRPC...5018worker2 # [ 106.996871] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:16Z level=info event=scheduler_connected addr=lb:500515019worker2 # [ 107.000954] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5020worker1 # [ 107.143135] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:16Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5021worker1 # [ 107.145597] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:16Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5022worker1 # [ 107.149517] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:16Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15023worker1 # [ 107.156075] nix-grpc-daemon[1222]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5024worker1 # [ 107.157564] nix-grpc-daemon[1222]: E0000 00:00:1790179276.774042 1222 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5025worker1 # [ 107.160680] nix-grpc-daemon[1222]: E0000 00:00:1790179276.780467 1222 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5026worker1 # [ 107.164061] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:16Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket5027worker1 # [ 107.165810] systemd[1]: Started Nix worker-protocol over gRPC.5028lb # [ 107.090354] niks3-server[953]: 2026/09/23 16:01:16 INFO lead: acquired remote=192.168.1.3:438305029lb: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux5030lb: must succeed: curl -sf localhost:9901/clusters5031worker1 # [ 107.196177] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:16Z level=info event=scheduler_take_over5032worker1 # [ 107.201650] nix-grpc-daemon[1222]: E0000 00:00:1790179276.793508 1262 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5033worker1 # [ 107.203958] nix-grpc-daemon[1222]: E0000 00:00:1790179276.810013 1262 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5034worker1 # [ 107.213733] nix-grpc-daemon[1222]: E0000 00:00:1790179276.810236 1262 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5035worker1 # [ 107.216666] nix-grpc-daemon[1222]: E0000 00:00:1790179276.832796 1262 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5036worker1 # [ 107.232292] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:16Z level=info event=scheduler_connected addr=lb:500515037worker1 # [ 107.234395] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5038lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5039worker2 # [ 107.197873] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:16Z level=info event=scheduler_connected addr=lb:500515040worker2 # [ 107.203023] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5041worker1 # [ 107.435606] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:17Z level=info event=scheduler_connected addr=lb:500515042worker1 # [ 107.445116] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:17Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:592365043worker1 # [ 107.451185] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:17Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05044worker2 # [ 107.403889] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:17Z level=info event=scheduler_connected addr=lb:500515045worker1 # [ 107.504167] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:17Z level=info event=oidc_jwks provider=mock keys=15046worker1 # [ 107.508604] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:17Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:592365047worker1 # [ 107.514118] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:17Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05048worker1 # [ 107.917702] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:17Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:592385049lb # [ 108.088711] envoy[897]: conn peer=[2001:db8:1::4]:47366 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=3074 tx=1089 ms=669685050lb # [ 108.093889] envoy[897]: conn peer=[2001:db8:1::1]:43578 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=295051lb # [ 108.099588] envoy[897]: conn peer=[2001:db8:1::3]:35908 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2003 tx=969 ms=700095052lb # [ 108.105716] envoy[897]: conn peer=[2001:db8:1::1]:43590 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1010 tx=282 ms=9745053lb: must succeed: curl -sf localhost:9901/clusters5054lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5055lb: (finished: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux, in 1.14 seconds)5056(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 10.62 seconds)5057subtest: a niks3 restart does not cost the leader its role5058lb: must succeed: curl -sf localhost:9901/clusters5059lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5060lb: must succeed: curl -sf localhost:9901/clusters5061lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5062lb # [ 108.381300] niks3-server[953]: 2026/09/23 16:01:17 INFO Shutdown signal received, draining in-flight requests timeout=10s5063worker2 # [ 108.377259] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:17Z level=info event=scheduler_lock_lost status=2005064worker1 # [ 108.467239] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:18Z level=info event=scheduler_lock_lost status=2005065lb # [ 108.389539] systemd[1]: Stopping niks3 server...5066lb # [ 108.390485] niks3-server[953]: 2026/09/23 16:01:17 INFO lead: released remote=192.168.1.3:438305067lb # [ 112.129991] systemd[1]: niks3.service: Deactivated successfully.5068lb # [ 112.134856] systemd[1]: Stopped niks3 server.5069lb # [ 112.138971] systemd[1]: niks3.service: Consumed 659ms CPU time over 1min 30.456s wall clock time, 27.8M memory peak, 100.7K incoming IP traffic, 144K outgoing IP traffic.5070lb # [ 112.162121] systemd[1]: Starting niks3 server...5071lb # [ 112.268431] niks3-server[1114]: 2026/09/23 16:01:21 goose: no migrations to run. current version: 202609231200005072lb # [ 112.275432] niks3-server[1114]: 2026/09/23 16:01:21 OK 1_commit_pending_closure.sql (7.23ms)5073lb # [ 112.279471] niks3-server[1114]: 2026/09/23 16:01:21 OK 2_object_stats_trigger.sql (3.98ms)5074lb # [ 112.283213] niks3-server[1114]: 2026/09/23 16:01:21 OK 3_commit_push.sql (3.34ms)5075lb # [ 112.285371] niks3-server[1114]: 2026/09/23 16:01:21 goose: up to current file version: 35076lb # [ 112.286575] niks3-server[1114]: 2026/09/23 16:01:21 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key5077lb # [ 112.298014] niks3-server[1114]: 2026/09/23 16:01:21 INFO Read proxy enabled — serving cache objects from S35078lb # [ 112.299510] niks3-server[1114]: 2026/09/23 16:01:21 INFO Using socket-activated listener address=0.0.0.0:57515079lb # [ 112.308469] niks3-server[1114]: 2026/09/23 16:01:21 INFO systemd watchdog enabled interval=15s5080lb # [ 112.312682] niks3-server[1114]: 2026/09/23 16:01:21 INFO Starting HTTP server address=0.0.0.0:57515081lb # [ 112.314069] systemd[1]: Started niks3 server.5082lb: waiting for unit niks3.service5083lb # [ 112.318846] niks3-server[1114]: 2026/09/23 16:01:21 INFO lead: acquired remote=192.168.1.3:438405084lb: (finished: waiting for unit niks3.service, in 0.03 seconds)5085lb: must succeed: curl -sf localhost:9901/clusters5086lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5087worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5088worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5089lb: must succeed: curl -sf localhost:9901/clusters5090lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5091lb: must succeed: curl -sf localhost:9901/clusters5092lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5093worker1: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5094worker1: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.06 seconds)5095client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&25096worker1 # [ 112.903191] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:432025097worker1 # [ 112.918427] nix-daemon[849]: accepted connection from pid 1222, user nix-grpc-daemon (trusted)5098client # don't know how to build these paths:5099client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5100worker1 # [ 112.940638] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:432025101worker1 # [ 112.956451] nix-grpc-daemon[1222]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 6 ms (attempt 1/5)5102worker1 # [ 112.971723] nix-grpc-daemon[1222]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 26 ms (attempt 2/5)5103worker1 # [ 113.006926] nix-grpc-daemon[1222]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 109 ms (attempt 3/5)5104worker1 # [ 113.117835] nix-grpc-daemon[1222]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 699 ms (attempt 4/5)5105worker1 # [ 113.820596] nix-grpc-daemon[1222]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5106worker1 # [ 113.825114] nix-grpc-daemon[1222]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5107worker1 # [ 113.873195] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:23Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=05108worker1 # [ 113.880673] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:23Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.3:50051 clients=15109worker1 # [ 113.890640] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:432025110worker1 # [ 113.903200] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:432025111client # copying 1 paths to 'lb:50051'...5112client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'...5113worker1 # [ 113.917408] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:432025114worker1 # [ 113.933101] nix-daemon[849]: accepted connection from pid 1222, user nix-grpc-daemon (trusted)5115worker1 # [ 113.965384] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:23Z level=info event=niks3_push_started pid=12945116worker1 # [ 114.101365] nix-grpc-daemon[1294]: time=2026-09-23T16:01:23.720Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""5117worker1 # [ 114.212114] nix-daemon[849]: accepted connection from pid 1307, user nix-grpc-daemon (trusted)5118worker1 # [ 114.233173] nix-daemon[849]: reaped child process 1310, status = succeeded5119lb # [ 114.153071] niks3-server[1114]: 2026/09/23 16:01:23 INFO Received push request method=POST path=/api/pushes5120worker1 # [ 114.249711] nix-grpc-daemon[1294]: time=2026-09-23T16:01:23.868Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5121worker1 # [ 114.251206] nix-grpc-daemon[1294]: time=2026-09-23T16:01:23.868Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)"5122lb # [ 114.213294] niks3-server[1114]: 2026/09/23 16:01:23 INFO Received sign narinfos request method=POST path=/api/pushes/35/sign5123lb # [ 114.225462] niks3-server[1114]: 2026/09/23 16:01:23 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls5124lb # [ 114.255173] niks3-server[1114]: 2026/09/23 16:01:23 INFO Signed narinfos id=35 count=15125worker1 # [ 114.340116] nix-grpc-daemon[1294]: time=2026-09-23T16:01:23.959Z level=INFO msg="Uploading 1 narinfos"5126lb # [ 114.266211] niks3-server[1114]: 2026/09/23 16:01:23 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst5127lb # [ 114.272799] niks3-server[1114]: 2026/09/23 16:01:23 INFO Received complete push request method=POST path=/api/pushes/35/complete5128lb # [ 114.289319] niks3-server[1114]: 2026/09/23 16:01:23 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.narinfo5129worker1 # [ 114.375842] nix-grpc-daemon[1294]: time=2026-09-23T16:01:23.995Z level=INFO msg="Upload complete. (270ms)"5130worker1 # [ 114.387346] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43202 duration_s=0 paths=1 nar_bytes_in=5445131worker1 # [ 114.404945] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:432025132worker1 # [ 114.416512] nix-daemon[849]: accepted connection from pid 1222, user nix-grpc-daemon (trusted)5133worker1 # [ 114.422513] nix-daemon[849]: reaped child process 1292, status = succeeded5134client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051...5135worker1 # [ 114.434662] nix-daemon[849]: accepted connection from pid 1222, user nix-grpc-daemon (trusted)5136client # farm-a> node-a: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5137client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5138client # farm-a> LOG-farm-a-after-niks3-restart5139worker1 # [ 115.701710] nix-daemon[849]: reaped child process 1314, status = succeeded5140worker1 # [ 115.826429] nix-daemon[849]: accepted connection from pid 1330, user nix-grpc-daemon (trusted)5141worker1 # [ 115.845512] nix-daemon[849]: reaped child process 1333, status = succeeded5142lb # [ 115.769708] niks3-server[1114]: 2026/09/23 16:01:25 INFO Received push request method=POST path=/api/pushes5143worker1 # [ 115.864453] nix-grpc-daemon[1294]: time=2026-09-23T16:01:25.482Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5144worker1 # [ 115.866238] nix-grpc-daemon[1294]: time=2026-09-23T16:01:25.482Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)"5145lb # [ 115.820214] niks3-server[1114]: 2026/09/23 16:01:25 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5146worker1 # [ 115.911674] nix-grpc-daemon[1294]: time=2026-09-23T16:01:25.530Z level=INFO msg="Uploading 1 narinfos"5147lb # [ 115.834493] niks3-server[1114]: 2026/09/23 16:01:25 INFO Received sign narinfos request method=POST path=/api/pushes/36/sign5148lb # [ 115.841302] niks3-server[1114]: 2026/09/23 16:01:25 INFO Signed narinfos id=36 count=15149lb # [ 115.843910] niks3-server[1114]: 2026/09/23 16:01:25 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls5150lb # [ 115.854884] niks3-server[1114]: 2026/09/23 16:01:25 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5151lb # [ 115.857923] niks3-server[1114]: 2026/09/23 16:01:25 INFO Received complete push request method=POST path=/api/pushes/36/complete5152lb # [ 115.861384] niks3-server[1114]: 2026/09/23 16:01:25 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.narinfo5153worker1 # [ 115.946228] nix-grpc-daemon[1294]: time=2026-09-23T16:01:25.565Z level=INFO msg="Upload complete. (297ms)"5154worker1 # [ 115.968959] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43202 duration_s=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=15155worker1 # [ 115.983877] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:25Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=05156worker1 # [ 115.996990] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:25Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.3:50051 clients=15157worker1 # [ 116.003964] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:432025158worker1 # [ 116.010266] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:432025159worker1 # [ 116.016168] nix-daemon[849]: reaped child process 1312, status = succeeded5160client # copying 1 paths to 'lb:50051'...5161worker1 # [ 116.018582] nix-grpc-daemon[1294]: time=2026-09-23T16:01:25.634Z level=INFO msg="All 1 paths already cached"5162client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'...5163worker1 # [ 116.022766] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:432025164worker1 # [ 116.040343] nix-daemon[849]: accepted connection from pid 1222, user nix-grpc-daemon (trusted)5165worker1 # [ 116.173745] nix-daemon[849]: accepted connection from pid 1339, user nix-grpc-daemon (trusted)5166worker1 # [ 116.194408] nix-daemon[849]: reaped child process 1342, status = succeeded5167lb # [ 116.110630] niks3-server[1114]: 2026/09/23 16:01:25 INFO Received push request method=POST path=/api/pushes5168worker1 # [ 116.201019] nix-grpc-daemon[1294]: time=2026-09-23T16:01:25.820Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5169worker1 # [ 116.202484] nix-grpc-daemon[1294]: time=2026-09-23T16:01:25.820Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)"5170worker1 # [ 116.232449] nix-grpc-daemon[1294]: time=2026-09-23T16:01:25.852Z level=INFO msg="Uploading 1 narinfos"5171lb # [ 116.140192] niks3-server[1114]: 2026/09/23 16:01:25 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst5172lb # [ 116.156498] niks3-server[1114]: 2026/09/23 16:01:25 INFO Received sign narinfos request method=POST path=/api/pushes/37/sign5173lb # [ 116.165742] niks3-server[1114]: 2026/09/23 16:01:25 INFO Signed narinfos id=37 count=15174lb # [ 116.177039] niks3-server[1114]: 2026/09/23 16:01:25 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls5175lb # [ 116.187387] niks3-server[1114]: 2026/09/23 16:01:25 INFO Received complete push request method=POST path=/api/pushes/37/complete5176lb # [ 116.192266] niks3-server[1114]: 2026/09/23 16:01:25 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo5177worker1 # [ 116.277550] nix-grpc-daemon[1294]: time=2026-09-23T16:01:25.897Z level=INFO msg="Upload complete. (231ms)"5178worker1 # [ 116.288288] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43202 duration_s=0 paths=1 nar_bytes_in=6565179worker1 # [ 116.304807] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:432025180worker1 # [ 116.315761] nix-daemon[849]: accepted connection from pid 1222, user nix-grpc-daemon (trusted)5181worker1 # [ 116.318723] nix-daemon[849]: reaped child process 1337, status = succeeded5182client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051...5183worker1 # [ 116.332316] nix-daemon[849]: accepted connection from pid 1222, user nix-grpc-daemon (trusted)5184client # farm-b> node-a: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5185client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5186client # farm-b> LOG-farm-b-after-niks3-restart5187worker1 # [ 117.593502] nix-daemon[849]: reaped child process 1346, status = succeeded5188worker1 # [ 117.727492] nix-daemon[849]: accepted connection from pid 1362, user nix-grpc-daemon (trusted)5189worker1 # [ 117.746133] nix-daemon[849]: reaped child process 1365, status = succeeded5190lb # [ 117.670900] niks3-server[1114]: 2026/09/23 16:01:27 INFO Received push request method=POST path=/api/pushes5191worker1 # [ 117.764504] nix-grpc-daemon[1294]: time=2026-09-23T16:01:27.383Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5192worker1 # [ 117.766395] nix-grpc-daemon[1294]: time=2026-09-23T16:01:27.383Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)"5193lb # [ 117.711081] niks3-server[1114]: 2026/09/23 16:01:27 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst5194lb # [ 117.728113] niks3-server[1114]: 2026/09/23 16:01:27 INFO Received sign narinfos request method=POST path=/api/pushes/38/sign5195worker1 # [ 117.815061] nix-grpc-daemon[1294]: time=2026-09-23T16:01:27.434Z level=INFO msg="Uploading 1 narinfos"5196lb # [ 117.736791] niks3-server[1114]: 2026/09/23 16:01:27 INFO Signed narinfos id=38 count=15197lb # [ 117.744342] niks3-server[1114]: 2026/09/23 16:01:27 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls5198lb # [ 117.751686] niks3-server[1114]: 2026/09/23 16:01:27 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5199lb # [ 117.761075] niks3-server[1114]: 2026/09/23 16:01:27 INFO Received complete push request method=POST path=/api/pushes/38/complete5200lb # [ 117.771482] niks3-server[1114]: 2026/09/23 16:01:27 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.narinfo5201worker1 # [ 117.862890] nix-grpc-daemon[1294]: time=2026-09-23T16:01:27.482Z level=INFO msg="Upload complete. (316ms)"5202worker1 # [ 117.882722] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43202 duration_s=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=15203worker1 # [ 117.900287] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:27Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=05204worker1 # [ 117.911678] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:27Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.3:50051 clients=15205worker1 # [ 117.918508] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:432025206worker1 # [ 117.924940] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:432025207worker1 # [ 117.931366] nix-daemon[849]: reaped child process 1344, status = succeeded5208client # copying 1 paths to 'lb:50051'...5209worker1 # [ 117.934011] nix-grpc-daemon[1294]: time=2026-09-23T16:01:27.549Z level=INFO msg="All 2 paths already cached"5210client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'...5211worker1 # [ 117.938488] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:432025212worker1 # [ 117.951814] nix-daemon[849]: accepted connection from pid 1222, user nix-grpc-daemon (trusted)5213worker1 # [ 118.090196] nix-daemon[849]: accepted connection from pid 1371, user nix-grpc-daemon (trusted)5214worker1 # [ 118.113309] nix-daemon[849]: reaped child process 1374, status = succeeded5215lb # [ 118.035260] niks3-server[1114]: 2026/09/23 16:01:27 INFO Received push request method=POST path=/api/pushes5216worker1 # [ 118.125527] nix-grpc-daemon[1294]: time=2026-09-23T16:01:27.745Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5217worker1 # [ 118.127040] nix-grpc-daemon[1294]: time=2026-09-23T16:01:27.745Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)"5218lb # [ 118.065212] niks3-server[1114]: 2026/09/23 16:01:27 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls5219lb # [ 118.076325] niks3-server[1114]: 2026/09/23 16:01:27 INFO Received sign narinfos request method=POST path=/api/pushes/39/sign5220worker1 # [ 118.164653] nix-grpc-daemon[1294]: time=2026-09-23T16:01:27.784Z level=INFO msg="Uploading 1 narinfos"5221lb # [ 118.088742] niks3-server[1114]: 2026/09/23 16:01:27 INFO Signed narinfos id=39 count=15222lb # [ 118.095059] niks3-server[1114]: 2026/09/23 16:01:27 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst5223lb # [ 118.108903] niks3-server[1114]: 2026/09/23 16:01:27 INFO Received complete push request method=POST path=/api/pushes/39/complete5224worker1 # [ 118.194262] nix-grpc-daemon[1294]: time=2026-09-23T16:01:27.814Z level=INFO msg="Upload complete. (230ms)"5225lb # [ 118.114603] niks3-server[1114]: 2026/09/23 16:01:27 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo5226worker1 # [ 118.204775] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:43202 duration_s=0 paths=1 nar_bytes_in=7845227worker1 # [ 118.220625] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:432025228worker1 # [ 118.231088] nix-daemon[849]: accepted connection from pid 1222, user nix-grpc-daemon (trusted)5229worker1 # [ 118.233349] nix-daemon[849]: reaped child process 1369, status = succeeded5230client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051...5231worker1 # [ 118.246884] nix-daemon[849]: accepted connection from pid 1222, user nix-grpc-daemon (trusted)5232client # farm-top> node-a: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5233client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5234client # farm-top> LOG-farm-top-after-niks3-restart5235worker1 # [ 119.502146] nix-daemon[849]: reaped child process 1378, status = succeeded5236worker1 # [ 119.620557] nix-daemon[849]: accepted connection from pid 1394, user nix-grpc-daemon (trusted)5237worker1 # [ 119.642780] nix-daemon[849]: reaped child process 1397, status = succeeded5238lb # [ 119.566974] niks3-server[1114]: 2026/09/23 16:01:29 INFO Received push request method=POST path=/api/pushes5239worker1 # [ 119.660442] nix-grpc-daemon[1294]: time=2026-09-23T16:01:29.278Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5240worker1 # [ 119.662070] nix-grpc-daemon[1294]: time=2026-09-23T16:01:29.278Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)"5241lb # [ 119.607590] niks3-server[1114]: 2026/09/23 16:01:29 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5242lb # [ 119.621544] niks3-server[1114]: 2026/09/23 16:01:29 INFO Received sign narinfos request method=POST path=/api/pushes/40/sign5243worker1 # [ 119.708745] nix-grpc-daemon[1294]: time=2026-09-23T16:01:29.328Z level=INFO msg="Uploading 1 narinfos"5244lb # [ 119.635125] niks3-server[1114]: 2026/09/23 16:01:29 INFO Signed narinfos id=40 count=15245lb # [ 119.640273] niks3-server[1114]: 2026/09/23 16:01:29 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls5246lb # [ 119.643063] niks3-server[1114]: 2026/09/23 16:01:29 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst5247lb # [ 119.656512] niks3-server[1114]: 2026/09/23 16:01:29 INFO Received complete push request method=POST path=/api/pushes/40/complete5248worker1 # [ 119.743448] nix-grpc-daemon[1294]: time=2026-09-23T16:01:29.363Z level=INFO msg="Upload complete. (286ms)"5249lb # [ 119.664251] niks3-server[1114]: 2026/09/23 16:01:29 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo5250worker1 # [ 119.757345] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:43202 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=15251worker1 # [ 119.778814] nix-daemon[849]: reaped child process 1376, status = succeeded5252client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2, in 7.12 seconds)5253client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath5254client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath, in 0.19 seconds)5255(finished: subtest: a niks3 restart does not cost the leader its role, in 11.65 seconds)5256subtest: scheduler down: the other node takes the lock and keeps it5257lb: must succeed: curl -sf localhost:9901/clusters5258lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5259lb: must succeed: curl -sf localhost:9901/clusters5260lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5261lb: must succeed: curl -sf localhost:9901/clusters5262lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5263lb: must succeed: curl -sf localhost:9901/clusters5264lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5265worker1 # [ 120.265245] systemd[1]: Stopping Nix worker-protocol over gRPC...5266worker1 # [ 121.178420] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:30Z level=info event=signal_exit5267lb # [ 121.316579] niks3-server[1114]: 2026/09/23 16:01:30 INFO lead: released remote=192.168.1.3:438405268lb # [ 122.303929] niks3-server[1114]: 2026/09/23 16:01:31 INFO lead: acquired remote=192.168.1.4:430725269worker2 # [ 122.297209] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:31Z level=info event=scheduler_take_over5270worker1 # [ 122.393871] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:32Z level=info event=scheduler_restarting5271lb # [ 122.311210] envoy[897]: [2026-09-23 16:01:31.925][897][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream}5272worker2 # [ 122.313269] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:31Z level=info event=scheduler_restarting addr=lb:500515273worker1 # [ 122.403304] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:32Z level=info event=scheduler_restarting addr=lb:500515274worker1 # [ 122.414679] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:32Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:592365275worker1 # [ 122.420601] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:32Z level=info event=worker_gone addr=192.168.1.4:500515276worker1 # [ 122.425299] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:32Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:592365277worker1 # [ 122.430377] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:32Z level=info event=worker_gone addr=192.168.1.3:500515278worker1 # [ 122.453908] nix-daemon[849]: reaped child process 1284, status = succeeded5279worker2 # [ 122.514376] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:32Z level=info event=scheduler_connected addr=lb:500515280worker1 # [ 122.616982] nix-grpc-daemon[1222]: ts=2026-09-23T16:01:32Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5281worker2 # [ 122.535654] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:32Z level=info event=oidc_jwks provider=mock keys=15282worker2 # [ 122.540620] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:32Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:403965283worker2 # [ 122.546085] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:32Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05284worker2 # [ 122.560865] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:32Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:404005285worker1 # [ 124.563452] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5286worker1 # [ 124.568863] systemd[1]: Stopped Nix worker-protocol over gRPC.5287worker1 # [ 124.577456] systemd[1]: nix-grpc-daemon.service: Consumed 1.271s CPU time over 17.566s wall clock time, 38.3M memory peak, 187.5K incoming IP traffic, 88.2K outgoing IP traffic.5288worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5289worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.08 seconds)5290worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5291worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)5292client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&25293client # don't know how to build these paths:5294client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5295worker2 # [ 124.851715] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:34Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:487485296worker2 # [ 124.859821] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:487485297worker2 # [ 124.886484] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:34Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=05298worker2 # [ 124.893651] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:34Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.4:50051 clients=15299worker2 # [ 124.903000] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:487485300worker2 # [ 124.914376] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:487485301client # copying 1 paths to 'lb:50051'...5302client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'...5303worker2 # [ 124.930098] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:487485304worker2 # [ 124.945190] nix-daemon[842]: accepted connection from pid 1252, user nix-grpc-daemon (trusted)5305worker2 # [ 125.095761] nix-daemon[842]: accepted connection from pid 1348, user nix-grpc-daemon (trusted)5306worker2 # [ 125.115625] nix-daemon[842]: reaped child process 1351, status = succeeded5307lb # [ 125.131370] niks3-server[1114]: 2026/09/23 16:01:34 INFO Received push request method=POST path=/api/pushes5308worker2 # [ 125.132464] nix-grpc-daemon[1312]: time=2026-09-23T16:01:34.755Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5309worker2 # [ 125.134617] nix-grpc-daemon[1312]: time=2026-09-23T16:01:34.755Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)"5310lb # [ 125.169873] niks3-server[1114]: 2026/09/23 16:01:34 INFO Received sign narinfos request method=POST path=/api/pushes/41/sign5311worker2 # [ 125.171050] nix-grpc-daemon[1312]: time=2026-09-23T16:01:34.794Z level=INFO msg="Uploading 1 narinfos"5312lb # [ 125.181006] niks3-server[1114]: 2026/09/23 16:01:34 INFO Signed narinfos id=41 count=15313lb # [ 125.190821] niks3-server[1114]: 2026/09/23 16:01:34 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls5314lb # [ 125.203333] niks3-server[1114]: 2026/09/23 16:01:34 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst5315lb # [ 125.216868] niks3-server[1114]: 2026/09/23 16:01:34 INFO Received complete push request method=POST path=/api/pushes/41/complete5316worker2 # [ 125.217205] nix-grpc-daemon[1312]: time=2026-09-23T16:01:34.840Z level=INFO msg="Upload complete. (256ms)"5317lb # [ 125.227203] niks3-server[1114]: 2026/09/23 16:01:34 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo5318worker2 # [ 125.230166] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48748 duration_s=0 paths=1 nar_bytes_in=5205319worker2 # [ 125.244333] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:487485320worker2 # [ 125.253970] nix-daemon[842]: accepted connection from pid 1252, user nix-grpc-daemon (trusted)5321worker2 # [ 125.260385] nix-daemon[842]: reaped child process 1346, status = succeeded5322client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051...5323client # farm-a> worker2: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5324worker2 # [ 125.273396] nix-daemon[842]: accepted connection from pid 1252, user nix-grpc-daemon (trusted)5325client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5326client # farm-a> LOG-farm-a-on-standby5327worker2 # [ 126.115689] nix-daemon[842]: reaped child process 1355, status = succeeded5328worker2 # [ 126.242031] nix-daemon[842]: accepted connection from pid 1371, user nix-grpc-daemon (trusted)5329worker2 # [ 126.259221] nix-daemon[842]: reaped child process 1374, status = succeeded5330lb # [ 126.278121] niks3-server[1114]: 2026/09/23 16:01:35 INFO Received push request method=POST path=/api/pushes5331worker2 # [ 126.278958] nix-grpc-daemon[1312]: time=2026-09-23T16:01:35.900Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5332worker2 # [ 126.280602] nix-grpc-daemon[1312]: time=2026-09-23T16:01:35.900Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)"5333worker2 # [ 126.319181] nix-grpc-daemon[1312]: time=2026-09-23T16:01:35.942Z level=INFO msg="Uploading 1 narinfos"5334lb # [ 126.318789] niks3-server[1114]: 2026/09/23 16:01:35 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5335lb # [ 126.331853] niks3-server[1114]: 2026/09/23 16:01:35 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls5336lb # [ 126.342376] niks3-server[1114]: 2026/09/23 16:01:35 INFO Received sign narinfos request method=POST path=/api/pushes/42/sign5337lb # [ 126.356151] niks3-server[1114]: 2026/09/23 16:01:35 INFO Signed narinfos id=42 count=15338worker2 # [ 126.353053] nix-grpc-daemon[1312]: time=2026-09-23T16:01:35.976Z level=INFO msg="Upload complete. (269ms)"5339lb # [ 126.361578] niks3-server[1114]: 2026/09/23 16:01:35 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5340lb # [ 126.366078] niks3-server[1114]: 2026/09/23 16:01:35 INFO Received complete push request method=POST path=/api/pushes/42/complete5341lb # [ 126.370406] niks3-server[1114]: 2026/09/23 16:01:35 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo5342worker2 # [ 126.377099] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48748 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=15343worker2 # [ 126.391433] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:35Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=05344worker2 # [ 126.396793] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:35Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.4:50051 clients=15345worker2 # [ 126.401161] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:487485346worker2 # [ 126.405180] nix-daemon[842]: reaped child process 1353, status = succeeded5347worker2 # [ 126.408731] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:487485348worker2 # [ 126.420296] nix-grpc-daemon[1312]: time=2026-09-23T16:01:36.044Z level=INFO msg="All 1 paths already cached"5349client # copying 1 paths to 'lb:50051'...5350client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'...5351worker2 # [ 126.429901] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:487485352worker2 # [ 126.440761] nix-daemon[842]: accepted connection from pid 1252, user nix-grpc-daemon (trusted)5353worker2 # [ 126.578653] nix-daemon[842]: accepted connection from pid 1380, user nix-grpc-daemon (trusted)5354worker2 # [ 126.596414] nix-daemon[842]: reaped child process 1383, status = succeeded5355lb # [ 126.610700] niks3-server[1114]: 2026/09/23 16:01:36 INFO Received push request method=POST path=/api/pushes5356worker2 # [ 126.610591] nix-grpc-daemon[1312]: time=2026-09-23T16:01:36.234Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5357worker2 # [ 126.612115] nix-grpc-daemon[1312]: time=2026-09-23T16:01:36.234Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)"5358lb # [ 126.641388] niks3-server[1114]: 2026/09/23 16:01:36 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls5359worker2 # [ 126.641885] nix-grpc-daemon[1312]: time=2026-09-23T16:01:36.265Z level=INFO msg="Uploading 1 narinfos"5360lb # [ 126.654571] niks3-server[1114]: 2026/09/23 16:01:36 INFO Received sign narinfos request method=POST path=/api/pushes/43/sign5361lb # [ 126.664715] niks3-server[1114]: 2026/09/23 16:01:36 INFO Signed narinfos id=43 count=15362lb # [ 126.675925] niks3-server[1114]: 2026/09/23 16:01:36 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst5363worker2 # [ 126.675806] nix-grpc-daemon[1312]: time=2026-09-23T16:01:36.299Z level=INFO msg="Upload complete. (221ms)"5364lb # [ 126.686384] niks3-server[1114]: 2026/09/23 16:01:36 INFO Received complete push request method=POST path=/api/pushes/43/complete5365lb # [ 126.692442] niks3-server[1114]: 2026/09/23 16:01:36 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo5366worker2 # [ 126.684293] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48748 duration_s=0 paths=1 nar_bytes_in=6405367worker2 # [ 126.699890] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:487485368worker2 # [ 126.710464] nix-daemon[842]: accepted connection from pid 1252, user nix-grpc-daemon (trusted)5369worker2 # [ 126.713818] nix-daemon[842]: reaped child process 1378, status = succeeded5370client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051...5371client # farm-b> worker2: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5372worker2 # [ 126.728198] nix-daemon[842]: accepted connection from pid 1252, user nix-grpc-daemon (trusted)5373client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5374client # farm-b> LOG-farm-b-on-standby5375lb # [ 128.086815] envoy[897]: conn peer=[2001:db8:1::1]:38230 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9779 tx=4465 ms=68735376lb # [ 128.091873] envoy[897]: conn peer=[2001:db8:1::3]:38820 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1217 tx=530 ms=153885377worker2 # [ 128.194372] nix-daemon[842]: reaped child process 1387, status = succeeded5378worker2 # [ 128.321893] nix-daemon[842]: accepted connection from pid 1403, user nix-grpc-daemon (trusted)5379worker2 # [ 128.341234] nix-daemon[842]: reaped child process 1406, status = succeeded5380lb # [ 128.357574] niks3-server[1114]: 2026/09/23 16:01:37 INFO Received push request method=POST path=/api/pushes5381worker2 # [ 128.359074] nix-grpc-daemon[1312]: time=2026-09-23T16:01:37.981Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5382worker2 # [ 128.360850] nix-grpc-daemon[1312]: time=2026-09-23T16:01:37.981Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)"5383lb # [ 128.398689] niks3-server[1114]: 2026/09/23 16:01:38 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5384worker2 # [ 128.402323] nix-grpc-daemon[1312]: time=2026-09-23T16:01:38.025Z level=INFO msg="Uploading 1 narinfos"5385lb # [ 128.413936] niks3-server[1114]: 2026/09/23 16:01:38 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst5386lb # [ 128.423584] niks3-server[1114]: 2026/09/23 16:01:38 INFO Received sign narinfos request method=POST path=/api/pushes/44/sign5387lb # [ 128.433715] niks3-server[1114]: 2026/09/23 16:01:38 INFO Signed narinfos id=44 count=15388lb # [ 128.435635] niks3-server[1114]: 2026/09/23 16:01:38 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls5389lb # [ 128.441676] niks3-server[1114]: 2026/09/23 16:01:38 INFO Received complete push request method=POST path=/api/pushes/44/complete5390worker2 # [ 128.432340] nix-grpc-daemon[1312]: time=2026-09-23T16:01:38.056Z level=INFO msg="Upload complete. (292ms)"5391lb # [ 128.444464] niks3-server[1114]: 2026/09/23 16:01:38 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo5392worker2 # [ 128.452347] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48748 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=15393worker2 # [ 128.466563] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:38Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=05394worker2 # [ 128.476208] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:38Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.4:50051 clients=15395worker2 # [ 128.484390] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:487485396client # copying 1 paths to 'lb:50051'...5397worker2 # [ 128.489960] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:487485398client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'...5399worker2 # [ 128.494283] nix-daemon[842]: reaped child process 1385, status = succeeded5400worker2 # [ 128.498454] nix-grpc-daemon[1312]: time=2026-09-23T16:01:38.112Z level=INFO msg="All 2 paths already cached"5401worker2 # [ 128.513954] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:487485402worker2 # [ 128.520239] nix-daemon[842]: accepted connection from pid 1252, user nix-grpc-daemon (trusted)5403worker2 # [ 128.641340] nix-daemon[842]: accepted connection from pid 1412, user nix-grpc-daemon (trusted)5404worker2 # [ 128.658981] nix-daemon[842]: reaped child process 1415, status = succeeded5405lb # [ 128.674008] niks3-server[1114]: 2026/09/23 16:01:38 INFO Received push request method=POST path=/api/pushes5406worker2 # [ 128.673982] nix-grpc-daemon[1312]: time=2026-09-23T16:01:38.297Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5407worker2 # [ 128.675500] nix-grpc-daemon[1312]: time=2026-09-23T16:01:38.297Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)"5408lb # [ 128.708285] niks3-server[1114]: 2026/09/23 16:01:38 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst5409worker2 # [ 128.706835] nix-grpc-daemon[1312]: time=2026-09-23T16:01:38.330Z level=INFO msg="Uploading 1 narinfos"5410lb # [ 128.723057] niks3-server[1114]: 2026/09/23 16:01:38 INFO Received sign narinfos request method=POST path=/api/pushes/45/sign5411lb # [ 128.733018] niks3-server[1114]: 2026/09/23 16:01:38 INFO Signed narinfos id=45 count=15412lb # [ 128.738878] niks3-server[1114]: 2026/09/23 16:01:38 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls5413worker2 # [ 128.740145] nix-grpc-daemon[1312]: time=2026-09-23T16:01:38.363Z level=INFO msg="Upload complete. (217ms)"5414lb # [ 128.749279] niks3-server[1114]: 2026/09/23 16:01:38 INFO Received complete push request method=POST path=/api/pushes/45/complete5415lb # [ 128.753381] niks3-server[1114]: 2026/09/23 16:01:38 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo5416worker2 # [ 128.751153] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48748 duration_s=0 paths=1 nar_bytes_in=7685417worker2 # [ 128.762611] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:487485418worker2 # [ 128.771970] nix-daemon[842]: accepted connection from pid 1252, user nix-grpc-daemon (trusted)5419worker2 # [ 128.774885] nix-daemon[842]: reaped child process 1410, status = succeeded5420client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051...5421worker2 # [ 128.786493] nix-daemon[842]: accepted connection from pid 1252, user nix-grpc-daemon (trusted)5422client # farm-top> worker2: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5423client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5424client # farm-top> LOG-farm-top-on-standby5425worker2 # [ 130.291517] nix-daemon[842]: reaped child process 1419, status = succeeded5426worker2 # [ 130.377384] nix-daemon[842]: accepted connection from pid 1435, user nix-grpc-daemon (trusted)5427worker2 # [ 130.396856] nix-daemon[842]: reaped child process 1438, status = succeeded5428lb # [ 130.410700] niks3-server[1114]: 2026/09/23 16:01:40 INFO Received push request method=POST path=/api/pushes5429worker2 # [ 130.413948] nix-grpc-daemon[1312]: time=2026-09-23T16:01:40.036Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5430worker2 # [ 130.415753] nix-grpc-daemon[1312]: time=2026-09-23T16:01:40.036Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)"5431lb # [ 130.451575] niks3-server[1114]: 2026/09/23 16:01:40 INFO Received sign narinfos request method=POST path=/api/pushes/46/sign5432worker2 # [ 130.452544] nix-grpc-daemon[1312]: time=2026-09-23T16:01:40.076Z level=INFO msg="Uploading 1 narinfos"5433lb # [ 130.469676] niks3-server[1114]: 2026/09/23 16:01:40 INFO Signed narinfos id=46 count=15434lb # [ 130.485311] niks3-server[1114]: 2026/09/23 16:01:40 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst5435lb # [ 130.499047] niks3-server[1114]: 2026/09/23 16:01:40 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5436worker2 # [ 130.506591] nix-grpc-daemon[1312]: time=2026-09-23T16:01:40.130Z level=INFO msg="Upload complete. (305ms)"5437lb # [ 130.514457] niks3-server[1114]: 2026/09/23 16:01:40 INFO Received complete push request method=POST path=/api/pushes/46/complete5438lb # [ 130.526099] niks3-server[1114]: 2026/09/23 16:01:40 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo5439worker2 # [ 130.521241] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:40Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48748 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=15440lb # [ 130.536738] niks3-server[1114]: 2026/09/23 16:01:40 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls5441worker2 # [ 130.542365] nix-daemon[842]: reaped child process 1417, status = succeeded5442client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2, in 5.93 seconds)5443client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath5444client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath, in 0.18 seconds)5445worker1 # [ 130.899608] systemd[1]: Starting Nix worker-protocol over gRPC...5446worker1 # [ 131.031200] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:40Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5447worker1 # [ 131.033522] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:40Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5448worker1 # [ 131.037380] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:40Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15449worker1 # [ 131.043822] nix-grpc-daemon[1415]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5450worker1 # [ 131.064138] nix-grpc-daemon[1415]: E0000 00:00:1790179300.661596 1415 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5451worker1 # [ 131.066528] nix-grpc-daemon[1415]: E0000 00:00:1790179300.663616 1415 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5452lb: must succeed: curl -sf localhost:9901/clusters5453worker1 # [ 131.092117] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:40Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket5454worker1 # [ 131.094335] nix-grpc-daemon[1415]: E0000 00:00:1790179300.672188 1455 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5455worker1 # [ 131.096984] nix-grpc-daemon[1415]: E0000 00:00:1790179300.694414 1455 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5456worker1 # [ 131.099286] nix-grpc-daemon[1415]: E0000 00:00:1790179300.694636 1455 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5457worker1 # [ 131.101747] nix-grpc-daemon[1415]: E0000 00:00:1790179300.711145 1455 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5458worker1 # [ 131.107230] systemd[1]: Started Nix worker-protocol over gRPC.5459worker1 # [ 131.110016] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:40Z level=info event=scheduler_connected addr=lb:500515460worker2 # [ 131.021706] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:40Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:403965461worker2 # [ 131.030119] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:40Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05462lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5463worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5464worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5465lb: must succeed: curl -sf localhost:9901/clusters5466lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5467lb: must succeed: curl -sf localhost:9901/clusters5468lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5469worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5470worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5471(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 11.43 seconds)5472subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5473client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs5474client # [ 131.402200] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs.5475client # Running as unit: rs.service; invocation ID: 6ce73a8ea224422cbe3a294aba45d94a5476client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs, in 0.09 seconds)5477worker1 # [ 131.716196] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:384405478worker1 # [ 131.732937] nix-daemon[849]: accepted connection from pid 1415, user nix-grpc-daemon (trusted)5479client # [ 131.664399] nix[1405]: don't know how to build these paths:5480client # [ 131.666051] nix[1405]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5481worker2 # [ 131.677903] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:41Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=05482worker2 # [ 131.685850] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:41Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.4:50051 clients=15483worker2 # [ 131.692845] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:487485484worker2 # [ 131.703822] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:487485485client # [ 131.720158] nix[1405]: copying 1 paths to 'lb:50051'...5486worker2 # [ 131.717919] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:487485487client # [ 131.725328] nix[1405]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'...5488worker2 # [ 131.741299] nix-daemon[842]: accepted connection from pid 1252, user nix-grpc-daemon (trusted)5489worker2 # [ 131.894358] nix-daemon[842]: accepted connection from pid 1451, user nix-grpc-daemon (trusted)5490worker2 # [ 131.912570] nix-daemon[842]: reaped child process 1454, status = succeeded5491lb # [ 131.929983] niks3-server[1114]: 2026/09/23 16:01:41 INFO Received push request method=POST path=/api/pushes5492worker2 # [ 131.934573] nix-grpc-daemon[1312]: time=2026-09-23T16:01:41.557Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5493worker2 # [ 131.936318] nix-grpc-daemon[1312]: time=2026-09-23T16:01:41.557Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)"5494lb # [ 131.965759] niks3-server[1114]: 2026/09/23 16:01:41 INFO Received sign narinfos request method=POST path=/api/pushes/47/sign5495worker2 # [ 131.965252] nix-grpc-daemon[1312]: time=2026-09-23T16:01:41.588Z level=INFO msg="Uploading 1 narinfos"5496lb # [ 131.980189] niks3-server[1114]: 2026/09/23 16:01:41 INFO Signed narinfos id=47 count=15497lb # [ 131.987635] niks3-server[1114]: 2026/09/23 16:01:41 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst5498lb # [ 132.001106] niks3-server[1114]: 2026/09/23 16:01:41 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls5499lb # [ 132.015375] niks3-server[1114]: 2026/09/23 16:01:41 INFO Received complete push request method=POST path=/api/pushes/47/complete5500lb # [ 132.019474] niks3-server[1114]: 2026/09/23 16:01:41 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo5501worker2 # [ 132.016208] nix-grpc-daemon[1312]: time=2026-09-23T16:01:41.639Z level=INFO msg="Upload complete. (267ms)"5502worker2 # [ 132.026354] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48748 duration_s=0 paths=1 nar_bytes_in=4165503worker2 # [ 132.040401] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:487485504worker2 # [ 132.059015] nix-daemon[842]: accepted connection from pid 1252, user nix-grpc-daemon (trusted)5505worker2 # [ 132.063409] nix-daemon[842]: reaped child process 1449, status = succeeded5506client # [ 132.080729] nix[1405]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051...5507worker2 # [ 132.079073] nix-daemon[842]: accepted connection from pid 1252, user nix-grpc-daemon (trusted)5508lb: must succeed: curl -sf localhost:9901/clusters5509lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5510lb: must succeed: curl -sf localhost:9901/clusters5511lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5512lb: must succeed: curl -sf localhost:9901/clusters5513lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5514lb: must succeed: curl -sf localhost:9901/clusters5515lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5516worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5517worker1 # [ 134.253303] systemd-journald[269]: Received client request to rotate journal, rotating.5518worker1 # [ 134.273490] systemd-journald[269]: Vacuuming done, freed 0B of archived journals from /var/log/journal/db72aa263333491a82503ede879c4344.5519worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.09 seconds)5520worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service5521worker2 # [ 134.234016] systemd[1]: Stopping Nix worker-protocol over gRPC...5522worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds)5523client: must succeed: sleep 5.05524worker2 # [ 134.237747] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:43Z level=info event=draining5525lb # [ 138.091393] envoy[897]: conn peer=[2001:db8:1::1]:39410 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9619 tx=4373 ms=56845526client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5527worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5528worker2 # [ 139.795839] nix-daemon[842]: reaped child process 1458, status = succeeded5529worker2 # [ 139.872289] nix-daemon[842]: accepted connection from pid 1499, user nix-grpc-daemon (trusted)5530worker2 # [ 139.890887] nix-daemon[842]: reaped child process 1502, status = succeeded5531lb # [ 139.904706] niks3-server[1114]: 2026/09/23 16:01:49 INFO Received push request method=POST path=/api/pushes5532worker2 # [ 139.907803] nix-grpc-daemon[1312]: time=2026-09-23T16:01:49.529Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5533worker2 # [ 139.909671] nix-grpc-daemon[1312]: time=2026-09-23T16:01:49.529Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)"5534lb # [ 139.945564] niks3-server[1114]: 2026/09/23 16:01:49 INFO Received sign narinfos request method=POST path=/api/pushes/48/sign5535worker2 # [ 139.946410] nix-grpc-daemon[1312]: time=2026-09-23T16:01:49.569Z level=INFO msg="Uploading 1 narinfos"5536lb # [ 139.962750] niks3-server[1114]: 2026/09/23 16:01:49 INFO Signed narinfos id=48 count=15537lb # [ 139.970865] niks3-server[1114]: 2026/09/23 16:01:49 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls5538lb # [ 139.977886] niks3-server[1114]: 2026/09/23 16:01:49 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5539lb # [ 139.988534] niks3-server[1114]: 2026/09/23 16:01:49 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5540lb # [ 139.991413] niks3-server[1114]: 2026/09/23 16:01:49 INFO Received complete push request method=POST path=/api/pushes/48/complete5541lb # [ 139.995701] niks3-server[1114]: 2026/09/23 16:01:49 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo5542worker2 # [ 139.987653] nix-grpc-daemon[1312]: time=2026-09-23T16:01:49.611Z level=INFO msg="Upload complete. (288ms)"5543worker2 # [ 140.001197] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48748 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=15544worker2 # [ 140.019835] nix-daemon[842]: reaped child process 1456, status = succeeded5545client # [ 140.036296] systemd[1]: rs.service: Deactivated successfully.5546client # [ 140.041830] systemd[1]: rs.service: Consumed 212ms CPU time over 8.635s wall clock time, 41.2M memory peak, 7.4K incoming IP traffic, 8.4K outgoing IP traffic.5547worker2 # [ 140.239145] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:49Z level=info event=signal_exit5548lb # [ 140.308197] niks3-server[1114]: 2026/09/23 16:01:49 INFO lead: released remote=192.168.1.4:430725549lb # [ 140.973749] niks3-server[1114]: 2026/09/23 16:01:50 INFO lead: acquired remote=192.168.1.3:516505550worker1 # [ 141.055518] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:50Z level=info event=scheduler_take_over5551worker2 # [ 141.303237] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:50Z level=info event=scheduler_restarting5552lb # [ 141.310541] envoy[897]: [2026-09-23 16:01:50.924][897][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream}5553worker1 # [ 141.402879] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:51Z level=info event=scheduler_restarting addr=lb:500515554worker2 # [ 141.313962] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:50Z level=info event=scheduler_restarting addr=lb:500515555worker2 # [ 141.323853] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:50Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:403965556worker2 # [ 141.329604] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:50Z level=info event=worker_gone addr=192.168.1.3:500515557worker2 # [ 141.334046] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:50Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:403965558worker2 # [ 141.339513] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:50Z level=info event=worker_gone addr=192.168.1.4:500515559worker2 # [ 141.366233] nix-daemon[842]: reaped child process 1304, status = succeeded5560worker1 # [ 141.603897] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:51Z level=info event=scheduler_connected addr=lb:500515561worker1 # [ 141.613399] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:51Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:456765562worker1 # [ 141.619605] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:51Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05563worker2 # [ 141.541711] nix-grpc-daemon[1252]: ts=2026-09-23T16:01:51Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5564worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.20 seconds)5565client: waiting for success: ! systemctl is-active rs5566worker1 # [ 141.872716] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:51Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:456785567client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds)5568client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5569client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds)5570client: must fail: journalctl -u rs | grep -q 'reconnecting'5571client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5572lb: must succeed: curl -sf localhost:9901/clusters5573lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5574worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5575worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5576lb: must succeed: curl -sf localhost:9901/clusters5577lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5578worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5579worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5580worker2 # [ 143.188227] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5581worker2 # [ 143.194577] systemd[1]: Stopped Nix worker-protocol over gRPC.5582worker2 # [ 143.200561] systemd[1]: nix-grpc-daemon.service: Consumed 1.786s CPU time over 41.114s wall clock time, 39.7M memory peak, 345.9K incoming IP traffic, 172K outgoing IP traffic.5583worker2 # [ 143.217693] systemd[1]: Starting scheduler-token.service...5584worker2 # [ 143.293290] systemd[1]: scheduler-token.service: Deactivated successfully.5585worker2 # [ 143.300518] systemd[1]: Finished scheduler-token.service.5586worker2 # [ 143.302307] systemd[1]: scheduler-token.service: Consumed 44ms CPU time over 78ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5587worker2 # [ 143.309869] systemd[1]: Starting Nix worker-protocol over gRPC...5588worker2 # [ 143.443904] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:53Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5589worker2 # [ 143.446390] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:53Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5590worker2 # [ 143.450194] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:53Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15591worker2 # [ 143.456119] nix-grpc-daemon[1513]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5592worker2 # [ 143.457590] nix-grpc-daemon[1513]: E0000 00:00:1790179313.078365 1513 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5593worker2 # [ 143.460978] nix-grpc-daemon[1513]: E0000 00:00:1790179313.084676 1513 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5594worker2 # [ 143.465353] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:53Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket5595worker2 # [ 143.467123] systemd[1]: Started Nix worker-protocol over gRPC.5596worker2 # [ 143.474479] nix-grpc-daemon[1513]: E0000 00:00:1790179313.096159 1554 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5597worker2 # [ 143.500210] nix-grpc-daemon[1513]: E0000 00:00:1790179313.113916 1554 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5598worker2 # [ 143.502451] nix-grpc-daemon[1513]: E0000 00:00:1790179313.114075 1554 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5599worker2 # [ 143.510363] nix-grpc-daemon[1513]: E0000 00:00:1790179313.133287 1554 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5600worker2 # [ 143.522124] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:53Z level=info event=scheduler_connected addr=lb:500515601worker1 # [ 143.623249] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:53Z level=info event=oidc_jwks provider=mock keys=15602lb: must succeed: curl -sf localhost:9901/clusters5603worker1 # [ 143.631298] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:53Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:456765604worker1 # [ 143.631495] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:53Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05605lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5606worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5607worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5608(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.95 seconds)5609subtest: scheduler restart mid-build: build finishes, no double build5610client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid5611client # [ 144.346401] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid.5612client # Running as unit: mid.service; invocation ID: 12483d6c775f454b8b3bd61fb8079eb85613client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid, in 0.09 seconds)5614worker2 # [ 144.551419] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:416385615worker2 # [ 144.565886] nix-daemon[842]: accepted connection from pid 1513, user nix-grpc-daemon (trusted)5616client # [ 144.586535] nix[1441]: don't know how to build these paths:5617client # [ 144.587894] nix[1441]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5618worker1 # [ 144.688564] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:54Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05619worker1 # [ 144.698214] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:54Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.3:50051 clients=15620worker1 # [ 144.709079] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384405621worker1 # [ 144.717407] nix-grpc-daemon[1415]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 34 ms (attempt 1/5)5622worker1 # [ 144.752965] nix-grpc-daemon[1415]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 16 ms (attempt 2/5)5623worker1 # [ 144.770911] nix-grpc-daemon[1415]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 352 ms (attempt 3/5)5624worker1 # [ 145.132538] nix-grpc-daemon[1415]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 50 ms (attempt 4/5)5625worker1 # [ 145.184811] nix-grpc-daemon[1415]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5626worker1 # [ 145.191470] nix-grpc-daemon[1415]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5627worker1 # [ 145.210042] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:384405628client # [ 145.137636] nix[1441]: copying 1 paths to 'lb:50051'...5629client # [ 145.143013] nix[1441]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'...5630worker1 # [ 145.229747] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:384405631worker1 # [ 145.245749] nix-daemon[849]: accepted connection from pid 1415, user nix-grpc-daemon (trusted)5632worker1 # [ 145.281587] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:54Z level=info event=niks3_push_started pid=15405633worker1 # [ 145.429486] nix-grpc-daemon[1540]: time=2026-09-23T16:01:55.048Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""5634worker1 # [ 145.538594] nix-daemon[849]: accepted connection from pid 1553, user nix-grpc-daemon (trusted)5635worker1 # [ 145.557050] nix-daemon[849]: reaped child process 1556, status = succeeded5636lb # [ 145.482388] niks3-server[1114]: 2026/09/23 16:01:55 INFO Received push request method=POST path=/api/pushes5637worker1 # [ 145.574442] nix-grpc-daemon[1540]: time=2026-09-23T16:01:55.193Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5638worker1 # [ 145.575994] nix-grpc-daemon[1540]: time=2026-09-23T16:01:55.193Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)"5639lb # [ 145.523075] niks3-server[1114]: 2026/09/23 16:01:55 INFO Received sign narinfos request method=POST path=/api/pushes/49/sign5640worker1 # [ 145.617884] nix-grpc-daemon[1540]: time=2026-09-23T16:01:55.235Z level=INFO msg="Uploading 1 narinfos"5641lb # [ 145.537188] niks3-server[1114]: 2026/09/23 16:01:55 INFO Signed narinfos id=49 count=15642lb # [ 145.545058] niks3-server[1114]: 2026/09/23 16:01:55 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls5643lb # [ 145.553190] niks3-server[1114]: 2026/09/23 16:01:55 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst5644lb # [ 145.568474] niks3-server[1114]: 2026/09/23 16:01:55 INFO Received complete push request method=POST path=/api/pushes/49/complete5645worker1 # [ 145.657099] nix-grpc-daemon[1540]: time=2026-09-23T16:01:55.276Z level=INFO msg="Upload complete. (223ms)"5646worker1 # [ 145.662071] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38440 duration_s=0 paths=1 nar_bytes_in=4245647lb # [ 145.580774] niks3-server[1114]: 2026/09/23 16:01:55 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo5648worker1 # [ 145.668330] nix-grpc-daemon[1415]: ts=2026-09-23T16:01:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:384405649worker1 # [ 145.679269] nix-daemon[849]: accepted connection from pid 1415, user nix-grpc-daemon (trusted)5650worker1 # [ 145.680883] nix-daemon[849]: reaped child process 1538, status = succeeded5651worker1 # [ 145.689042] nix-daemon[849]: accepted connection from pid 1415, user nix-grpc-daemon (trusted)5652client # [ 145.603711] nix[1441]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5653lb: must succeed: curl -sf localhost:9901/clusters5654lb # [ 148.090760] envoy[897]: conn peer=[2001:db8:1::1]:32890 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2966 tx=1542 ms=83895655lb # [ 148.095772] envoy[897]: conn peer=[2001:db8:1::4]:35754 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=15816 tx=1257 ms=392555656lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5657lb: must succeed: curl -sf localhost:9901/clusters5658lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5659worker1: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5660worker1 # [ 148.277663] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1415 (nix-grpc-daemon) on client request.5661worker1 # [ 148.279268] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5662lb # [ 148.205805] niks3-server[1114]: 2026/09/23 16:01:57 INFO lead: released remote=192.168.1.3:516505663worker1: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.06 seconds)5664client: must succeed: sleep 5.05665client # [ 148.209003] nix[1441]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5666worker1 # [ 148.300396] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5667worker2 # [ 148.205080] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:57Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"5668worker1 # [ 148.301986] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5669worker1 # [ 148.303205] systemd[1]: nix-grpc-daemon.service: Consumed 524ms CPU time over 17.400s wall clock time, 35M memory peak, 217.4K incoming IP traffic, 53.3K outgoing IP traffic.5670client # [ 148.214363] nix[1441]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again5671lb # [ 148.217649] envoy[897]: [2026-09-23 16:01:57.826][897][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 13, upstream_reset_after_response_started{connection_termination}5672worker1 # [ 148.333307] nix-daemon[849]: reaped child process 1562, status = succeeded5673worker1 # [ 148.335057] nix-daemon[849]: reaped child process 1474, status = succeeded5674worker1 # [ 148.344415] nix-daemon[1564]: unexpected Nix daemon error: error: interrupted by the user5675worker1 # [ 148.348968] nix-daemon[849]: reaped child process 1564, status = failed with exit code 15676worker1 # [ 148.478421] systemd[1]: nix-grpc-daemon.service: Scheduled restart job, restart counter is at 1.5677worker1 # [ 148.490789] systemd[1]: Starting Nix worker-protocol over gRPC...5678worker2 # [ 148.405309] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:58Z level=info event=scheduler_connected addr=lb:500515679worker2 # [ 148.442328] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:58Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure"5680lb # [ 148.482865] niks3-server[1114]: 2026/09/23 16:01:58 INFO lead: acquired remote=192.168.1.4:409045681worker2 # [ 148.478802] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:58Z level=info event=scheduler_take_over5682worker1 # [ 148.642350] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:58Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5683worker1 # [ 148.644631] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:58Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5684worker1 # [ 148.648345] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:58Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15685worker1 # [ 148.654906] nix-grpc-daemon[1601]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5686worker1 # [ 148.668170] nix-grpc-daemon[1601]: E0000 00:00:1790179318.272682 1601 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5687worker1 # [ 148.670517] nix-grpc-daemon[1601]: E0000 00:00:1790179318.274703 1601 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5688worker1 # [ 148.698144] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:58Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket5689worker1 # [ 148.700344] nix-grpc-daemon[1601]: E0000 00:00:1790179318.282549 1641 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5690worker1 # [ 148.702880] nix-grpc-daemon[1601]: E0000 00:00:1790179318.302664 1641 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5691worker1 # [ 148.708357] nix-grpc-daemon[1601]: E0000 00:00:1790179318.302810 1641 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5692worker1 # [ 148.712548] nix-grpc-daemon[1601]: E0000 00:00:1790179318.317192 1641 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5693worker1 # [ 148.715052] systemd[1]: Started Nix worker-protocol over gRPC.5694worker1 # [ 148.718447] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:58Z level=info event=scheduler_connected addr=lb:500515695worker1 # [ 148.725768] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:58Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5696worker2 # [ 148.643685] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:58Z level=info event=scheduler_connected addr=lb:500515697worker2 # [ 148.649598] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:58Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5698client # [ 148.712698] nix[1441]: scheduler at lb:50051: scheduler passive, another node serves, reconnecting5699worker1 # [ 148.926881] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:58Z level=info event=scheduler_connected addr=lb:500515700worker1 # [ 148.933385] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:58Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5701worker2 # [ 148.850644] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:58Z level=info event=scheduler_connected addr=lb:500515702worker2 # [ 148.856265] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:58Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5703worker1 # [ 149.134202] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:58Z level=info event=scheduler_connected addr=lb:500515704worker1 # [ 149.140981] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:58Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5705worker2 # [ 149.057338] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:58Z level=info event=scheduler_connected addr=lb:500515706worker2 # [ 149.062779] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:58Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5707worker1 # [ 149.341891] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:58Z level=info event=scheduler_connected addr=lb:500515708worker1 # [ 149.348691] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:58Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5709worker2 # [ 149.265494] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:58Z level=info event=scheduler_connected addr=lb:500515710worker2 # [ 149.270850] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:58Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5711worker1 # [ 149.549582] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:59Z level=info event=scheduler_connected addr=lb:500515712worker1 # [ 149.559091] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:59Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5713worker2 # [ 149.472365] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:59Z level=info event=scheduler_connected addr=lb:500515714worker2 # [ 149.499731] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:59Z level=info event=oidc_jwks provider=mock keys=15715worker2 # [ 149.505000] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:59Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:416425716worker2 # [ 149.511130] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:59Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05717worker1 # [ 149.762040] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:59Z level=info event=scheduler_connected addr=lb:500515718worker1 # [ 149.767367] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:59Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5719worker2 # [ 149.719602] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:59Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05720worker2 # [ 149.727349] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:59Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=15721worker2 # [ 149.740443] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416385722worker2 # [ 149.751049] nix-grpc-daemon[1513]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 32 ms (attempt 1/5)5723worker2 # [ 149.792194] nix-grpc-daemon[1513]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 63 ms (attempt 2/5)5724worker2 # [ 149.856830] nix-grpc-daemon[1513]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 233 ms (attempt 3/5)5725worker1 # [ 149.968088] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:59Z level=info event=scheduler_connected addr=lb:500515726worker1 # [ 149.974705] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:59Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5727worker1 # [ 150.175576] nix-grpc-daemon[1601]: ts=2026-09-23T16:01:59Z level=info event=scheduler_connected addr=lb:500515728worker2 # [ 150.087566] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:59Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:416425729worker2 # [ 150.095062] nix-grpc-daemon[1513]: ts=2026-09-23T16:01:59Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05730worker2 # [ 150.100732] nix-grpc-daemon[1513]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 21 ms (attempt 4/5)5731worker2 # [ 150.118750] nix-grpc-daemon[1513]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5732worker2 # [ 150.123968] nix-grpc-daemon[1513]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5733worker2 # [ 150.142475] nix-grpc-daemon[1513]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' from 'http://lb:5751'...5734worker2 # [ 150.171246] nix-daemon[842]: accepted connection from pid 1513, user nix-grpc-daemon (trusted)5735worker2 # [ 150.189231] nix-daemon[842]: accepted connection from pid 1513, user nix-grpc-daemon (trusted)5736client # [ 150.192743] nix[1441]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5737worker2 # [ 150.505892] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:00Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:416465738client: (finished: must succeed: sleep 5.0, in 5.06 seconds)5739client: waiting for success: ! systemctl is-active mid5740worker2 # [ 153.614277] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:03Z level=info event=niks3_push_started pid=16215741worker2 # [ 153.650203] nix-daemon[842]: reaped child process 1596, status = succeeded5742worker2 # [ 153.774302] nix-grpc-daemon[1621]: time=2026-09-23T16:02:03.396Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""5743worker2 # [ 153.884698] nix-daemon[842]: accepted connection from pid 1634, user nix-grpc-daemon (trusted)5744worker2 # [ 153.906067] nix-daemon[842]: reaped child process 1637, status = succeeded5745lb # [ 153.915273] niks3-server[1114]: 2026/09/23 16:02:03 INFO Received push request method=POST path=/api/pushes5746worker2 # [ 153.918575] nix-grpc-daemon[1621]: time=2026-09-23T16:02:03.540Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5747worker2 # [ 153.922332] nix-grpc-daemon[1621]: time=2026-09-23T16:02:03.540Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)"5748lb # [ 153.978499] niks3-server[1114]: 2026/09/23 16:02:03 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls5749lb # [ 153.997998] niks3-server[1114]: 2026/09/23 16:02:03 INFO Received sign narinfos request method=POST path=/api/pushes/50/sign5750worker2 # [ 153.995666] nix-grpc-daemon[1621]: time=2026-09-23T16:02:03.617Z level=INFO msg="Uploading 1 narinfos"5751lb # [ 154.007901] niks3-server[1114]: 2026/09/23 16:02:03 INFO Signed narinfos id=50 count=15752lb # [ 154.020654] niks3-server[1114]: 2026/09/23 16:02:03 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5753lb # [ 154.029182] niks3-server[1114]: 2026/09/23 16:02:03 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst5754lb # [ 154.038783] niks3-server[1114]: 2026/09/23 16:02:03 INFO Received complete push request method=POST path=/api/pushes/50/complete5755lb # [ 154.045865] niks3-server[1114]: 2026/09/23 16:02:03 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo5756worker2 # [ 154.043603] nix-grpc-daemon[1621]: time=2026-09-23T16:02:03.667Z level=INFO msg="Upload complete. (266ms)"5757worker2 # [ 154.060471] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41638 duration_s=4 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=15758worker2 # [ 154.078585] nix-daemon[842]: reaped child process 1594, status = succeeded5759client # [ 154.095282] systemd[1]: mid.service: Deactivated successfully.5760client # [ 154.099750] systemd[1]: mid.service: Consumed 212ms CPU time over 9.753s wall clock time, 39.5M memory peak, 8.9K incoming IP traffic, 10.9K outgoing IP traffic.5761client: (finished: waiting for success: ! systemctl is-active mid, in 1.12 seconds)5762client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }5763client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.05 seconds)5764(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.41 seconds)5765subtest: interrupting the client stops the build on the worker5766client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr5767client # [ 154.732422] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr.5768client # Running as unit: intr.service; invocation ID: c9b52d8587f14a4d9596b9cc5e6153fc5769client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr, in 0.05 seconds)5770worker1 # [ 155.001452] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:04Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:563325771worker1 # [ 155.017206] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)5772client # [ 154.950024] nix[1477]: don't know how to build these paths:5773client # [ 154.952155] nix[1477]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5774worker2 # [ 154.961246] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:04Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=05775worker2 # [ 154.967182] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:04Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.3:50051 clients=15776worker1 # [ 155.061348] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563325777worker1 # [ 155.076710] nix-grpc-daemon[1601]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 26 ms (attempt 1/5)5778worker1 # [ 155.112505] nix-grpc-daemon[1601]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 85 ms (attempt 2/5)5779worker1 # [ 155.206549] nix-grpc-daemon[1601]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 185 ms (attempt 3/5)5780worker1 # [ 155.401138] nix-grpc-daemon[1601]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 605 ms (attempt 4/5)5781worker1 # [ 156.014296] nix-grpc-daemon[1601]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5782worker1 # [ 156.026247] nix-grpc-daemon[1601]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5783worker1 # [ 156.043323] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563325784client # [ 155.969392] nix[1477]: copying 1 paths to 'lb:50051'...5785worker1 # [ 156.058499] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:05Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563325786client # [ 155.974481] nix[1477]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'...5787worker1 # [ 156.076377] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)5788worker1 # [ 156.106282] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:05Z level=info event=niks3_push_started pid=16775789worker1 # [ 156.227721] nix-grpc-daemon[1677]: time=2026-09-23T16:02:05.847Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""5790worker1 # [ 156.341267] nix-daemon[849]: accepted connection from pid 1690, user nix-grpc-daemon (trusted)5791worker1 # [ 156.358572] nix-daemon[849]: reaped child process 1693, status = succeeded5792lb # [ 156.286024] niks3-server[1114]: 2026/09/23 16:02:05 INFO Received push request method=POST path=/api/pushes5793worker1 # [ 156.380974] nix-grpc-daemon[1677]: time=2026-09-23T16:02:05.999Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5794worker1 # [ 156.382425] nix-grpc-daemon[1677]: time=2026-09-23T16:02:05.999Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)"5795lb # [ 156.329840] niks3-server[1114]: 2026/09/23 16:02:05 INFO Received sign narinfos request method=POST path=/api/pushes/51/sign5796worker1 # [ 156.423463] nix-grpc-daemon[1677]: time=2026-09-23T16:02:06.042Z level=INFO msg="Uploading 1 narinfos"5797lb # [ 156.345957] niks3-server[1114]: 2026/09/23 16:02:05 INFO Signed narinfos id=51 count=15798lb # [ 156.353938] niks3-server[1114]: 2026/09/23 16:02:05 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls5799lb # [ 156.370517] niks3-server[1114]: 2026/09/23 16:02:05 INFO Received complete push request method=POST path=/api/pushes/51/complete5800worker1 # [ 156.460676] nix-grpc-daemon[1677]: time=2026-09-23T16:02:06.080Z level=INFO msg="Upload complete. (229ms)"5801lb # [ 156.380367] niks3-server[1114]: 2026/09/23 16:02:05 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo5802lb # [ 156.388422] niks3-server[1114]: 2026/09/23 16:02:06 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst5803worker1 # [ 156.468272] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=4245804worker1 # [ 156.484947] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563325805worker1 # [ 156.491087] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)5806worker1 # [ 156.497377] nix-daemon[849]: reaped child process 1675, status = succeeded5807worker1 # [ 156.508538] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)5808client # [ 156.421328] nix[1477]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051...5809lb # [ 158.089487] envoy[897]: conn peer=[2001:db8:1::3]:49090 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=709 tx=328 ms=171975810lb # [ 158.094707] envoy[897]: conn peer=[2001:db8:1::1]:44274 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4127 tx=2025 ms=95245811client: must succeed: systemctl kill -s INT intr5812worker1 # [ 158.520311] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:08Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5813client # [ 158.432251] systemd[1]: intr.service: Sent signal SIGINT to main process 1477 (nix) on client request.5814client # [ 158.441270] nix[1477]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5815worker1 # [ 158.540919] nix-daemon[1697]: unexpected Nix daemon error: error: interrupted by the user5816client # [ 158.454782] nix[1477]: error: interrupted by the user5817client: (finished: must succeed: systemctl kill -s INT intr, in 0.08 seconds)5818worker1 # [ 158.550998] nix-daemon[849]: reaped child process 1695, status = succeeded5819worker1 # [ 158.554148] nix-daemon[849]: reaped child process 1697, status = failed with exit code 15820client # [ 158.479791] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE5821client # [ 158.485583] systemd[1]: intr.service: Failed with result 'exit-code'.5822client # [ 158.488648] systemd[1]: intr.service: Consumed 171ms CPU time over 3.750s wall clock time, 41.5M memory peak, 6.5K incoming IP traffic, 8.3K outgoing IP traffic.5823(finished: subtest: interrupting the client stops the build on the worker, in 3.87 seconds)5824subtest: a deploy mid-build drains: the build finishes, the new generation takes over5825client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw5826client # [ 158.627163] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw.5827client # Running as unit: sw.service; invocation ID: 6aa403cdcaa14f69b8c7324a9cd476355828client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw, in 0.08 seconds)5829client # [ 158.818308] nix[1498]: don't know how to build these paths:5830client # [ 158.819926] nix[1498]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv5831worker1 # [ 158.903060] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:08Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:563325832worker2 # [ 158.830301] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:08Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=05833worker2 # [ 158.838378] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:08Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.4:50051 clients=15834worker2 # [ 158.838728] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416385835worker2 # [ 158.849923] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:416385836client # [ 158.878925] nix[1498]: copying 1 paths to 'lb:50051'...5837client # [ 158.880870] nix[1498]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'...5838worker2 # [ 158.875510] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:416385839worker2 # [ 158.890684] nix-daemon[842]: accepted connection from pid 1513, user nix-grpc-daemon (trusted)5840worker2 # [ 159.046592] nix-daemon[842]: accepted connection from pid 1669, user nix-grpc-daemon (trusted)5841worker2 # [ 159.067945] nix-daemon[842]: reaped child process 1672, status = succeeded5842lb # [ 159.080187] niks3-server[1114]: 2026/09/23 16:02:08 INFO Received push request method=POST path=/api/pushes5843worker2 # [ 159.080572] nix-grpc-daemon[1621]: time=2026-09-23T16:02:08.703Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5844worker2 # [ 159.082560] nix-grpc-daemon[1621]: time=2026-09-23T16:02:08.703Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)"5845lb # [ 159.120186] niks3-server[1114]: 2026/09/23 16:02:08 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst5846worker2 # [ 159.120611] nix-grpc-daemon[1621]: time=2026-09-23T16:02:08.743Z level=INFO msg="Uploading 1 narinfos"5847lb # [ 159.131244] niks3-server[1114]: 2026/09/23 16:02:08 INFO Received sign narinfos request method=POST path=/api/pushes/52/sign5848lb # [ 159.141721] niks3-server[1114]: 2026/09/23 16:02:08 INFO Signed narinfos id=52 count=15849lb # [ 159.148343] niks3-server[1114]: 2026/09/23 16:02:08 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls5850lb # [ 159.161865] niks3-server[1114]: 2026/09/23 16:02:08 INFO Received complete push request method=POST path=/api/pushes/52/complete5851lb # [ 159.172573] niks3-server[1114]: 2026/09/23 16:02:08 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo5852worker2 # [ 159.168593] nix-grpc-daemon[1621]: time=2026-09-23T16:02:08.792Z level=INFO msg="Upload complete. (262ms)"5853worker2 # [ 159.178994] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41638 duration_s=0 paths=1 nar_bytes_in=4165854worker2 # [ 159.192382] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:416385855worker2 # [ 159.207333] nix-daemon[842]: accepted connection from pid 1513, user nix-grpc-daemon (trusted)5856worker2 # [ 159.213854] nix-daemon[842]: reaped child process 1667, status = succeeded5857worker2 # [ 159.226057] nix-daemon[842]: accepted connection from pid 1513, user nix-grpc-daemon (trusted)5858client # [ 159.228949] nix[1498]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051...5859worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&25860worker2 # Checking switch inhibitors... done5861worker2 # [ 161.605554] nixos[1708]: switching to system configuration /nix/store/2s5bc50462smm7jcgcka8hp4k9ri5jih-nixos-system-worker2-test5862worker2 # activating the configuration...5863worker2 # [ 161.612385] systemd[1]: Stopped target Remote File Systems.5864worker2 # [ 161.613243] systemd[1]: Stopped target Local File Systems.5865worker2 # setting up /etc...5866worker2 # [ 162.160929] systemd[1]: Reload requested from client PID 1708 ('.switch-to-conf') (unit backdoor.service)...5867worker2 # [ 162.162486] systemd[1]: Reloading...5868worker2 # [ 167.089426] systemd[1]: Reloading finished in 4913 ms.5869worker2 # restarting sysinit-reactivation.target5870worker2 # [ 167.220752] systemd[1]: Starting Re-setup tmpfiles on a system that is already running....5871worker2 # [ 167.339443] systemd[1]: Finished Re-setup tmpfiles on a system that is already running..5872worker2 # [ 167.344353] systemd[1]: Reached target Reactivate sysinit units.5873worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service5874worker2 # [ 167.348954] systemd[1]: Reloading D-Bus System Message Bus...5875worker2 # [ 167.350097] dbus-broker-launch[494]: Caught SIGHUP, trigger reload.5876worker2 # [ 167.361010] systemd[1]: Reloading Nix worker-protocol over gRPC...5877worker2 # [ 167.370100] dbus-broker-launch[494]: Looking up NSS user entry for 'systemd-timesync'...5878worker2 # [ 167.373786] dbus-broker-launch[494]: NSS returned no entry for 'systemd-timesync'5879worker2 # [ 167.385120] dbus-broker-launch[494]: Invalid user-name in /nix/store/ng2ydfkv03gvmdws4f4mjf9wj6qj3kq6-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"5880worker2 # [ 167.396489] systemd[1]: Reloaded Nix worker-protocol over gRPC.5881worker2 # [ 167.413726] systemd[1]: Reloaded D-Bus System Message Bus.5882worker2 # [ 167.420370] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os5883worker2 # [ 167.430070] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki5884worker2 # [ 167.431373] systemd[1]: Reached target Remote File Systems.5885worker2 # [ 167.440227] systemd[1]: Starting linger-users.service...5886worker2 # [ 167.441016] systemd[1]: Reached target Local File Systems.5887worker2 # [ 167.441755] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore5888worker2 # [ 167.455978] systemd[1]: Starting Create SUID/SGID Wrappers...5889worker2 # [ 167.459811] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met.5890worker2 # [ 167.465064] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc675891worker2 # [ 167.474745] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore5892worker2 # [ 167.485329] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:17Z level=info event=draining5893worker2 # [ 167.492379] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os5894worker2 # [ 167.493987] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os5895worker2 # [ 167.505851] systemd[1]: linger-users.service: Deactivated successfully.5896worker2 # [ 167.509390] systemd[1]: Finished linger-users.service.5897worker2 # [ 167.806913] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully.5898worker2 # [ 167.808070] systemd[1]: Finished Create SUID/SGID Wrappers.5899worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service5900worker2 # [ 168.064935] nixos[1708]: finished switching to system configuration /nix/store/2s5bc50462smm7jcgcka8hp4k9ri5jih-nixos-system-worker2-test5901worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 6.89 seconds)5902worker2: must succeed: pgrep -f 'read -t [9]0 x'5903lb # [ 168.092556] envoy[897]: conn peer=[2001:db8:1::1]:40470 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3013 tx=850 ms=35595904worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.06 seconds)5905client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&25906client # don't know how to build these paths:5907client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5908worker1 # [ 168.453111] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:18Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:563325909worker1 # [ 168.461714] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563325910worker2 # [ 168.393519] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:18Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=05911worker2 # [ 168.395672] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:18Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.3:50051 clients=15912worker1 # [ 168.489668] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563325913worker1 # [ 168.505017] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563325914client # copying 1 paths to 'lb:50051'...5915client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'...5916worker1 # [ 168.521740] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563325917worker1 # [ 168.541942] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)5918worker1 # [ 168.716057] nix-daemon[849]: accepted connection from pid 1744, user nix-grpc-daemon (trusted)5919worker1 # [ 168.733325] nix-daemon[849]: reaped child process 1747, status = succeeded5920lb # [ 168.660693] niks3-server[1114]: 2026/09/23 16:02:18 INFO Received push request method=POST path=/api/pushes5921worker1 # [ 168.752290] nix-grpc-daemon[1677]: time=2026-09-23T16:02:18.371Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5922worker1 # [ 168.753821] nix-grpc-daemon[1677]: time=2026-09-23T16:02:18.371Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)"5923lb # [ 168.702608] niks3-server[1114]: 2026/09/23 16:02:18 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls5924worker1 # [ 168.797090] nix-grpc-daemon[1677]: time=2026-09-23T16:02:18.416Z level=INFO msg="Uploading 1 narinfos"5925lb # [ 168.716672] niks3-server[1114]: 2026/09/23 16:02:18 INFO Received sign narinfos request method=POST path=/api/pushes/53/sign5926lb # [ 168.723703] niks3-server[1114]: 2026/09/23 16:02:18 INFO Signed narinfos id=53 count=15927lb # [ 168.731811] niks3-server[1114]: 2026/09/23 16:02:18 INFO Received complete push request method=POST path=/api/pushes/53/complete5928worker1 # [ 168.826247] nix-grpc-daemon[1677]: time=2026-09-23T16:02:18.446Z level=INFO msg="Upload complete. (267ms)"5929lb # [ 168.747347] niks3-server[1114]: 2026/09/23 16:02:18 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo5930lb # [ 168.755042] niks3-server[1114]: 2026/09/23 16:02:18 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst5931worker1 # [ 168.837456] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=5285932worker1 # [ 168.855261] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563325933worker1 # [ 168.861788] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)5934worker1 # [ 168.867051] nix-daemon[849]: reaped child process 1742, status = succeeded5935client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051...5936client # farm-a> node-a: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5937worker1 # [ 168.880819] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)5938client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5939client # farm-a> LOG-farm-a-during-drain5940worker1 # [ 169.747825] nix-daemon[849]: reaped child process 1751, status = succeeded5941worker1 # [ 169.863534] nix-daemon[849]: accepted connection from pid 1767, user nix-grpc-daemon (trusted)5942worker1 # [ 169.883931] nix-daemon[849]: reaped child process 1770, status = succeeded5943lb # [ 169.808815] niks3-server[1114]: 2026/09/23 16:02:19 INFO Received push request method=POST path=/api/pushes5944worker1 # [ 169.904500] nix-grpc-daemon[1677]: time=2026-09-23T16:02:19.523Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5945worker1 # [ 169.906392] nix-grpc-daemon[1677]: time=2026-09-23T16:02:19.523Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)"5946lb # [ 169.857467] niks3-server[1114]: 2026/09/23 16:02:19 INFO Received sign narinfos request method=POST path=/api/pushes/54/sign5947worker1 # [ 169.947397] nix-grpc-daemon[1677]: time=2026-09-23T16:02:19.566Z level=INFO msg="Uploading 1 narinfos"5948lb # [ 169.871026] niks3-server[1114]: 2026/09/23 16:02:19 INFO Signed narinfos id=54 count=15949lb # [ 169.879525] niks3-server[1114]: 2026/09/23 16:02:19 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5950lb # [ 169.888907] niks3-server[1114]: 2026/09/23 16:02:19 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst5951lb # [ 169.902006] niks3-server[1114]: 2026/09/23 16:02:19 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls5952lb # [ 169.908259] niks3-server[1114]: 2026/09/23 16:02:19 INFO Received complete push request method=POST path=/api/pushes/54/complete5953worker1 # [ 169.988276] nix-grpc-daemon[1677]: time=2026-09-23T16:02:19.608Z level=INFO msg="Upload complete. (297ms)"5954lb # [ 169.912157] niks3-server[1114]: 2026/09/23 16:02:19 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo5955worker2 # [ 169.913577] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:19Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=05956worker2 # [ 169.915451] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:19Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.3:50051 clients=15957worker1 # [ 170.001253] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=15958worker1 # [ 170.024365] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563325959worker1 # [ 170.030168] nix-daemon[849]: reaped child process 1749, status = succeeded5960worker1 # [ 170.040595] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563325961worker1 # [ 170.054091] nix-grpc-daemon[1677]: time=2026-09-23T16:02:19.673Z level=INFO msg="All 1 paths already cached"5962client # copying 1 paths to 'lb:50051'...5963worker1 # [ 170.065444] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563325964client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'...5965worker1 # [ 170.087837] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)5966worker1 # [ 170.245158] nix-daemon[849]: accepted connection from pid 1774, user nix-grpc-daemon (trusted)5967worker1 # [ 170.261875] nix-daemon[849]: reaped child process 1777, status = succeeded5968lb # [ 170.187485] niks3-server[1114]: 2026/09/23 16:02:19 INFO Received push request method=POST path=/api/pushes5969worker1 # [ 170.277644] nix-grpc-daemon[1677]: time=2026-09-23T16:02:19.897Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5970worker1 # [ 170.279126] nix-grpc-daemon[1677]: time=2026-09-23T16:02:19.897Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)"5971lb # [ 170.217240] niks3-server[1114]: 2026/09/23 16:02:19 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst5972worker1 # [ 170.310152] nix-grpc-daemon[1677]: time=2026-09-23T16:02:19.929Z level=INFO msg="Uploading 1 narinfos"5973lb # [ 170.231367] niks3-server[1114]: 2026/09/23 16:02:19 INFO Received sign narinfos request method=POST path=/api/pushes/55/sign5974lb # [ 170.241951] niks3-server[1114]: 2026/09/23 16:02:19 INFO Signed narinfos id=55 count=15975lb # [ 170.247520] niks3-server[1114]: 2026/09/23 16:02:19 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls5976lb # [ 170.256708] niks3-server[1114]: 2026/09/23 16:02:19 INFO Received complete push request method=POST path=/api/pushes/55/complete5977lb # [ 170.263328] niks3-server[1114]: 2026/09/23 16:02:19 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo5978worker1 # [ 170.350053] nix-grpc-daemon[1677]: time=2026-09-23T16:02:19.969Z level=INFO msg="Upload complete. (245ms)"5979worker1 # [ 170.360345] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=6405980worker1 # [ 170.376577] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563325981worker1 # [ 170.384793] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)5982worker1 # [ 170.390228] nix-daemon[849]: reaped child process 1772, status = succeeded5983worker1 # [ 170.401074] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)5984client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051...5985client # farm-b> node-a: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv5986client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5987client # farm-b> LOG-farm-b-during-drain5988worker1 # [ 172.069615] nix-daemon[849]: reaped child process 1781, status = succeeded5989worker1 # [ 172.201718] nix-daemon[849]: accepted connection from pid 1797, user nix-grpc-daemon (trusted)5990worker1 # [ 172.219425] nix-daemon[849]: reaped child process 1800, status = succeeded5991lb # [ 172.145621] niks3-server[1114]: 2026/09/23 16:02:21 INFO Received push request method=POST path=/api/pushes5992worker1 # [ 172.237497] nix-grpc-daemon[1677]: time=2026-09-23T16:02:21.856Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5993worker1 # [ 172.239240] nix-grpc-daemon[1677]: time=2026-09-23T16:02:21.856Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)"5994lb # [ 172.187243] niks3-server[1114]: 2026/09/23 16:02:21 INFO Received sign narinfos request method=POST path=/api/pushes/56/sign5995worker1 # [ 172.278349] nix-grpc-daemon[1677]: time=2026-09-23T16:02:21.897Z level=INFO msg="Uploading 1 narinfos"5996lb # [ 172.201412] niks3-server[1114]: 2026/09/23 16:02:21 INFO Signed narinfos id=56 count=15997lb # [ 172.208255] niks3-server[1114]: 2026/09/23 16:02:21 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv5998lb # [ 172.216452] niks3-server[1114]: 2026/09/23 16:02:21 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls5999worker1 # [ 172.310996] nix-grpc-daemon[1677]: time=2026-09-23T16:02:21.930Z level=INFO msg="Upload complete. (294ms)"6000lb # [ 172.231460] niks3-server[1114]: 2026/09/23 16:02:21 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst6001lb # [ 172.238030] niks3-server[1114]: 2026/09/23 16:02:21 INFO Received complete push request method=POST path=/api/pushes/56/complete6002lb # [ 172.243173] niks3-server[1114]: 2026/09/23 16:02:21 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo6003worker2 # [ 172.241562] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:21Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=06004worker1 # [ 172.325319] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=16005worker2 # [ 172.244213] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:21Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.3:50051 clients=16006worker1 # [ 172.348230] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326007worker1 # [ 172.355215] nix-daemon[849]: reaped child process 1779, status = succeeded6008worker1 # [ 172.358882] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563326009worker1 # [ 172.371973] nix-grpc-daemon[1677]: time=2026-09-23T16:02:21.991Z level=INFO msg="All 2 paths already cached"6010client # copying 1 paths to 'lb:50051'...6011worker1 # [ 172.383278] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563326012client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'...6013worker1 # [ 172.411483] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6014worker1 # [ 172.567580] nix-daemon[849]: accepted connection from pid 1804, user nix-grpc-daemon (trusted)6015worker1 # [ 172.588517] nix-daemon[849]: reaped child process 1807, status = succeeded6016lb # [ 172.512205] niks3-server[1114]: 2026/09/23 16:02:22 INFO Received push request method=POST path=/api/pushes6017worker1 # [ 172.602315] nix-grpc-daemon[1677]: time=2026-09-23T16:02:22.221Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6018worker1 # [ 172.604162] nix-grpc-daemon[1677]: time=2026-09-23T16:02:22.221Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)"6019lb # [ 172.541673] niks3-server[1114]: 2026/09/23 16:02:22 INFO Received sign narinfos request method=POST path=/api/pushes/57/sign6020worker1 # [ 172.629328] nix-grpc-daemon[1677]: time=2026-09-23T16:02:22.248Z level=INFO msg="Uploading 1 narinfos"6021lb # [ 172.554503] niks3-server[1114]: 2026/09/23 16:02:22 INFO Signed narinfos id=57 count=16022lb # [ 172.564895] niks3-server[1114]: 2026/09/23 16:02:22 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls6023lb # [ 172.580870] niks3-server[1114]: 2026/09/23 16:02:22 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst6024worker1 # [ 172.670008] nix-grpc-daemon[1677]: time=2026-09-23T16:02:22.289Z level=INFO msg="Upload complete. (240ms)"6025lb # [ 172.591230] niks3-server[1114]: 2026/09/23 16:02:22 INFO Received complete push request method=POST path=/api/pushes/57/complete6026worker1 # [ 172.678195] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=7686027lb # [ 172.599743] niks3-server[1114]: 2026/09/23 16:02:22 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo6028worker1 # [ 172.696581] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326029worker1 # [ 172.703080] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6030worker1 # [ 172.708879] nix-daemon[849]: reaped child process 1802, status = succeeded6031client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051...6032worker1 # [ 172.720167] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6033client # farm-top> node-a: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6034client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6035client # farm-top> LOG-farm-top-during-drain6036worker1 # [ 173.773388] nix-daemon[849]: reaped child process 1811, status = succeeded6037worker1 # [ 173.885158] nix-daemon[849]: accepted connection from pid 1827, user nix-grpc-daemon (trusted)6038worker1 # [ 173.906515] nix-daemon[849]: reaped child process 1830, status = succeeded6039lb # [ 173.829950] niks3-server[1114]: 2026/09/23 16:02:23 INFO Received push request method=POST path=/api/pushes6040worker1 # [ 173.922174] nix-grpc-daemon[1677]: time=2026-09-23T16:02:23.540Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6041worker1 # [ 173.923716] nix-grpc-daemon[1677]: time=2026-09-23T16:02:23.540Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)"6042lb # [ 173.873595] niks3-server[1114]: 2026/09/23 16:02:23 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls6043worker1 # [ 173.965506] nix-grpc-daemon[1677]: time=2026-09-23T16:02:23.585Z level=INFO msg="Uploading 1 narinfos"6044lb # [ 173.886050] niks3-server[1114]: 2026/09/23 16:02:23 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst6045lb # [ 173.897561] niks3-server[1114]: 2026/09/23 16:02:23 INFO Received sign narinfos request method=POST path=/api/pushes/58/sign6046lb # [ 173.908252] niks3-server[1114]: 2026/09/23 16:02:23 INFO Signed narinfos id=58 count=16047lb # [ 173.912768] niks3-server[1114]: 2026/09/23 16:02:23 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6048lb # [ 173.917326] niks3-server[1114]: 2026/09/23 16:02:23 INFO Received complete push request method=POST path=/api/pushes/58/complete6049worker1 # [ 173.998032] nix-grpc-daemon[1677]: time=2026-09-23T16:02:23.617Z level=INFO msg="Upload complete. (264ms)"6050lb # [ 173.920947] niks3-server[1114]: 2026/09/23 16:02:23 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo6051worker1 # [ 174.009109] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=16052worker1 # [ 174.028557] nix-daemon[849]: reaped child process 1809, status = succeeded6053client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2, in 5.83 seconds)6054client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath6055client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath, in 0.18 seconds)6056worker2: must succeed: pkill -f 'read -t [9]0 x'6057worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds)6058client: waiting for success: ! systemctl is-active sw6059worker2 # [ 174.289379] nix-daemon[842]: reaped child process 1676, status = succeeded6060worker2 # [ 174.397375] nix-daemon[842]: accepted connection from pid 1854, user nix-grpc-daemon (trusted)6061worker2 # [ 174.417429] nix-daemon[842]: reaped child process 1857, status = succeeded6062lb # [ 174.423884] niks3-server[1114]: 2026/09/23 16:02:24 INFO Received push request method=POST path=/api/pushes6063worker2 # [ 174.425793] nix-grpc-daemon[1621]: time=2026-09-23T16:02:24.048Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6064worker2 # [ 174.427281] nix-grpc-daemon[1621]: time=2026-09-23T16:02:24.048Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)"6065lb # [ 174.469170] niks3-server[1114]: 2026/09/23 16:02:24 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv6066worker2 # [ 174.471014] nix-grpc-daemon[1621]: time=2026-09-23T16:02:24.094Z level=INFO msg="Uploading 1 narinfos"6067lb # [ 174.481876] niks3-server[1114]: 2026/09/23 16:02:24 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls6068lb # [ 174.493843] niks3-server[1114]: 2026/09/23 16:02:24 INFO Received sign narinfos request method=POST path=/api/pushes/59/sign6069lb # [ 174.502972] niks3-server[1114]: 2026/09/23 16:02:24 INFO Signed narinfos id=59 count=16070lb # [ 174.514070] niks3-server[1114]: 2026/09/23 16:02:24 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6071lb # [ 174.523694] niks3-server[1114]: 2026/09/23 16:02:24 INFO Received complete push request method=POST path=/api/pushes/59/complete6072worker2 # [ 174.520113] nix-grpc-daemon[1621]: time=2026-09-23T16:02:24.143Z level=INFO msg="Upload complete. (254ms)"6073lb # [ 174.530586] niks3-server[1114]: 2026/09/23 16:02:24 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo6074worker2 # [ 174.533036] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41638 duration_s=15 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=16075worker2 # [ 174.552840] nix-daemon[842]: reaped child process 1674, status = succeeded6076client # [ 174.570117] systemd[1]: sw.service: Deactivated successfully.6077client # [ 174.574302] systemd[1]: sw.service: Consumed 190ms CPU time over 15.941s wall clock time, 41.2M memory peak, 7.5K incoming IP traffic, 8.7K outgoing IP traffic.6078client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds)6079client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6080client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.05 seconds)6081worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6082worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 0.06 seconds)6083lb: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux6084lb: must succeed: curl -sf localhost:9901/clusters6085lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6086worker2 # [ 175.480582] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:25Z level=info event=signal_exit6087lb: must succeed: curl -sf localhost:9901/clusters6088lb # [ 176.489312] niks3-server[1114]: 2026/09/23 16:02:26 INFO lead: released remote=192.168.1.4:409046089lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6090worker2 # [ 177.482078] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:27Z level=info event=scheduler_restarting6091worker1 # [ 177.573041] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:27Z level=info event=scheduler_restarting addr=lb:500516092lb # [ 177.488867] envoy[897]: [2026-09-23 16:02:27.102][897][warning][config] [./source/extensions/config_subscription/grpc/grpc_stream.h:189] StreamEndpoints gRPC config stream to sched closed: 1, Cancelled on the server side{via_upstream}6093worker2 # [ 177.486614] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:27Z level=info event=scheduler_restarting addr=lb:500516094worker2 # [ 177.492228] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:27Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:416426095worker2 # [ 177.505442] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:27Z level=info event=worker_gone addr=192.168.1.3:500516096worker2 # [ 177.509991] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:27Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:416426097worker2 # [ 177.514723] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:27Z level=info event=worker_gone addr=192.168.1.4:500516098worker2 # [ 177.538310] nix-daemon[842]: reaped child process 1569, status = succeeded6099lb: must succeed: curl -sf localhost:9901/clusters6100lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6101worker1 # [ 177.773356] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:27Z level=info event=scheduler_connected addr=lb:500516102worker1 # [ 177.779027] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6103worker2 # [ 177.702948] nix-grpc-daemon[1513]: ts=2026-09-23T16:02:27Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6104worker1 # [ 177.979849] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:27Z level=info event=scheduler_connected addr=lb:500516105worker1 # [ 177.986087] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6106lb # [ 178.091187] envoy[897]: conn peer=[2001:db8:1::1]:58634 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9677 tx=4441 ms=55746107lb # [ 178.097672] envoy[897]: conn peer=[2001:db8:1::1]:40472 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3008 tx=1536 ms=157316108lb # [ 178.102495] envoy[897]: conn peer=[2001:db8:1::4]:44484 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=6137 tx=811 ms=341716109worker1 # [ 178.186151] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:27Z level=info event=scheduler_connected addr=lb:500516110worker1 # [ 178.191341] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6111worker1 # [ 178.392307] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:28Z level=info event=scheduler_connected addr=lb:500516112worker1 # [ 178.397600] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6113worker1 # [ 178.600124] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:28Z level=info event=scheduler_connected addr=lb:500516114worker1 # [ 178.604231] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6115worker1 # [ 178.664955] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:28Z level=info event=scheduler_take_over6116lb # [ 178.584293] niks3-server[1114]: 2026/09/23 16:02:28 INFO lead: acquired remote=192.168.1.3:519186117lb: must succeed: curl -sf localhost:9901/clusters6118lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6119worker1 # [ 178.800778] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:28Z level=info event=scheduler_connected addr=lb:500516120worker1 # [ 178.805948] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6121worker1 # [ 179.006704] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:28Z level=info event=scheduler_connected addr=lb:500516122worker1 # [ 179.011861] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6123worker1 # [ 179.212660] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:28Z level=info event=scheduler_connected addr=lb:500516124worker1 # [ 179.217737] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6125worker1 # [ 179.418601] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:29Z level=info event=scheduler_connected addr=lb:500516126worker1 # [ 179.423771] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6127worker2 # [ 179.511684] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6128worker2 # [ 179.517085] systemd[1]: nix-grpc-daemon.service: Consumed 909ms CPU time over 36.206s wall clock time, 37.7M memory peak, 242.2K incoming IP traffic, 81.2K outgoing IP traffic.6129worker1 # [ 179.627108] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:29Z level=info event=scheduler_connected addr=lb:500516130worker1 # [ 179.631500] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:29Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:457906131worker1 # [ 179.636369] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:29Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=06132lb: must succeed: curl -sf localhost:9901/clusters6133lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6134worker2 # [ 179.785882] systemd[1]: nix-grpc-daemon.service: Scheduled restart job, restart counter is at 1.6135worker2 # [ 179.798510] systemd[1]: Starting scheduler-token.service...6136worker2 # [ 179.892076] systemd[1]: scheduler-token.service: Deactivated successfully.6137worker2 # [ 179.900665] systemd[1]: Finished scheduler-token.service.6138worker2 # [ 179.902870] systemd[1]: scheduler-token.service: Consumed 59ms CPU time over 98ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6139worker2 # [ 179.911683] systemd[1]: Starting Nix worker-protocol over gRPC...6140worker2 # [ 180.046516] nix-grpc-daemon[1873]: ts=2026-09-23T16:02:29Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6141worker2 # [ 180.048819] nix-grpc-daemon[1873]: ts=2026-09-23T16:02:29Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6142worker2 # [ 180.052573] nix-grpc-daemon[1873]: ts=2026-09-23T16:02:29Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16143worker2 # [ 180.057956] nix-grpc-daemon[1873]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6144worker2 # [ 180.059433] nix-grpc-daemon[1873]: E0000 00:00:1790179349.680894 1873 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6145worker2 # [ 180.063336] nix-grpc-daemon[1873]: E0000 00:00:1790179349.686911 1873 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6146worker2 # [ 180.100054] nix-grpc-daemon[1873]: ts=2026-09-23T16:02:29Z level=info event=startup listen=[::]:50051 proxy_socket=/nix/var/nix/daemon-socket/socket6147worker2 # [ 180.101806] nix-grpc-daemon[1873]: E0000 00:00:1790179349.695833 1914 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6148worker2 # [ 180.106582] nix-grpc-daemon[1873]: E0000 00:00:1790179349.710886 1914 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6149worker2 # [ 180.109450] nix-grpc-daemon[1873]: E0000 00:00:1790179349.711024 1914 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6150worker2 # [ 180.112155] systemd[1]: Started Nix worker-protocol over gRPC.6151worker2 # [ 180.117146] nix-grpc-daemon[1873]: E0000 00:00:1790179349.729536 1914 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6152worker2 # [ 180.121420] nix-grpc-daemon[1873]: ts=2026-09-23T16:02:29Z level=info event=scheduler_connected addr=lb:500516153worker1 # [ 180.221961] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:29Z level=info event=oidc_jwks provider=mock keys=16154worker1 # [ 180.227969] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:29Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:457906155worker1 # [ 180.233437] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:29Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=06156lb: must succeed: curl -sf localhost:9901/clusters6157lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6158worker1 # [ 181.573540] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:31Z level=info event=eds_stream_open cn=lb-1 peer=ipv4:192.168.1.2:582786159lb: must succeed: curl -sf localhost:9901/clusters6160lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6161lb: (finished: waiting until exactly ['worker1', 'worker2'] are healthy members of aarch64-linux, in 6.50 seconds)6162lb: must succeed: curl -sf localhost:9901/clusters6163lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6164worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6165worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6166(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 23.45 seconds)6167subtest: queue gauges show a burst while every build blocks6168lb: must succeed: curl -sf localhost:9901/clusters6169lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6170lb: must succeed: curl -sf localhost:9901/clusters6171lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6172worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6173worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6174client: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]'6175client # [ 182.257021] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto --impure --expr "map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ \"q0\" \"q1\" \"q2\" \"q3\" ]".6176client # Running as unit: burst.service; invocation ID: 459f0bb1a5d84d04ac706a3aa1955b8f6177client: (finished: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]', in 0.09 seconds)6178lb: must succeed: curl -sf localhost:9901/clusters6179lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6180lb: must succeed: curl -sf localhost:9901/clusters6181lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6182worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6183worker1 # [ 182.537630] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:32Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:563326184client # [ 182.459741] nix[1554]: don't know how to build these paths:6185client # [ 182.461071] nix[1554]: /nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv6186client # [ 182.462116] nix[1554]: /nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv6187client # [ 182.463080] nix[1554]: /nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv6188client # [ 182.464176] nix[1554]: /nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv6189worker1 # [ 182.554186] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:32Z level=debug event=want client=1 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv cached=06190worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds)6191worker1 # [ 182.561786] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:32Z level=debug event=assigned drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv worker=192.168.1.3:50051 clients=16192worker2 # [ 182.475296] nix-grpc-daemon[1873]: ts=2026-09-23T16:02:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:429646193worker1 # [ 182.567624] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:32Z level=debug event=want client=1 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv cached=06194worker1 # [ 182.573737] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:32Z level=debug event=assigned drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv worker=192.168.1.4:50051 clients=16195client # [ 182.496049] nix[1554]: copying 1 paths to 'lb:50051'...6196worker1 # [ 182.579165] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:32Z level=debug event=want client=1 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv cached=06197worker2 # [ 182.490961] nix-daemon[842]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)6198client # [ 182.497452] nix[1554]: copying path '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' to 'lb:50051'...6199worker1 # [ 182.587054] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:32Z level=debug event=want client=1 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv cached=06200worker1 # [ 182.595177] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326201worker1 # [ 182.604502] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563326202worker1 # [ 182.607377] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563326203worker2 # [ 182.517825] nix-grpc-daemon[1873]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 52 ms (attempt 1/5)6204worker1 # [ 182.613814] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6205worker2 # [ 182.578791] nix-grpc-daemon[1873]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 25 ms (attempt 2/5)6206worker2 # [ 182.605483] nix-grpc-daemon[1873]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 22 ms (attempt 3/5)6207worker2 # [ 182.629261] nix-grpc-daemon[1873]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 618 ms (attempt 4/5)6208worker1 # [ 182.739238] nix-daemon[849]: accepted connection from pid 1845, user nix-grpc-daemon (trusted)6209worker1 # [ 182.758075] nix-daemon[849]: reaped child process 1848, status = succeeded6210lb # [ 182.682724] niks3-server[1114]: 2026/09/23 16:02:32 INFO Received push request method=POST path=/api/pushes6211worker1 # [ 182.774025] nix-grpc-daemon[1677]: time=2026-09-23T16:02:32.393Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6212worker1 # [ 182.775673] nix-grpc-daemon[1677]: time=2026-09-23T16:02:32.393Z level=INFO msg="Uploading d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv (416B)"6213worker1 # [ 182.805005] nix-grpc-daemon[1677]: time=2026-09-23T16:02:32.424Z level=INFO msg="Uploading 1 narinfos"6214lb # [ 182.714548] niks3-server[1114]: 2026/09/23 16:02:32 INFO Received sign narinfos request method=POST path=/api/pushes/60/sign6215lb # [ 182.727277] niks3-server[1114]: 2026/09/23 16:02:32 INFO Signed narinfos id=60 count=16216lb # [ 182.738433] niks3-server[1114]: 2026/09/23 16:02:32 INFO Registered completed upload object_key=nar/06ismykzsix0w29371dm1hbzzf3ljf9jn7dyh8skp5wqfkkw0vlk.nar.zst6217lb # [ 182.750292] niks3-server[1114]: 2026/09/23 16:02:32 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.ls6218lb # [ 182.768783] niks3-server[1114]: 2026/09/23 16:02:32 INFO Received complete push request method=POST path=/api/pushes/60/complete6219worker1 # [ 182.856275] nix-grpc-daemon[1677]: time=2026-09-23T16:02:32.476Z level=INFO msg="Upload complete. (243ms)"6220lb # [ 182.778085] niks3-server[1114]: 2026/09/23 16:02:32 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.narinfo6221worker1 # [ 182.867775] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=4166222worker1 # [ 182.880584] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326223worker1 # [ 182.893340] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6224worker1 # [ 182.903219] nix-daemon[849]: reaped child process 1843, status = succeeded6225client # [ 182.821767] nix[1554]: building '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' on lb:50051...6226worker1 # [ 182.910819] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6227worker2 # [ 183.249775] nix-grpc-daemon[1873]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6228worker2 # [ 183.256727] nix-grpc-daemon[1873]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6229worker2 # [ 183.274488] nix-grpc-daemon[1873]: ts=2026-09-23T16:02:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:429646230client # [ 183.292131] nix[1554]: copying 1 paths to 'lb:50051'...6231worker2 # [ 183.289606] nix-grpc-daemon[1873]: ts=2026-09-23T16:02:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:429646232client # [ 183.296181] nix[1554]: copying path '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' to 'lb:50051'...6233worker2 # [ 183.304288] nix-daemon[842]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)6234worker2 # [ 183.337414] nix-grpc-daemon[1873]: ts=2026-09-23T16:02:32Z level=info event=niks3_push_started pid=19336235lb: must succeed: curl -sf localhost:9901/clusters6236worker2 # [ 183.489640] nix-grpc-daemon[1933]: time=2026-09-23T16:02:33.113Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""6237lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6238lb: must succeed: curl -sf localhost:9901/clusters6239worker2 # [ 183.603109] nix-daemon[842]: accepted connection from pid 1945, user nix-grpc-daemon (trusted)6240lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6241worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6242worker2 # [ 183.623972] nix-daemon[842]: reaped child process 1948, status = succeeded6243lb # [ 183.633251] niks3-server[1114]: 2026/09/23 16:02:33 INFO Received push request method=POST path=/api/pushes6244worker2 # [ 183.632198] nix-grpc-daemon[1933]: time=2026-09-23T16:02:33.255Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6245worker2 # [ 183.633708] nix-grpc-daemon[1933]: time=2026-09-23T16:02:33.255Z level=INFO msg="Uploading dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv (416B)"6246worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6247lb # [ 183.667007] niks3-server[1114]: 2026/09/23 16:02:33 INFO Received sign narinfos request method=POST path=/api/pushes/61/sign6248worker2 # [ 183.663838] nix-grpc-daemon[1933]: time=2026-09-23T16:02:33.286Z level=INFO msg="Uploading 1 narinfos"6249lb # [ 183.672598] niks3-server[1114]: 2026/09/23 16:02:33 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.ls6250lb # [ 183.677163] niks3-server[1114]: 2026/09/23 16:02:33 INFO Signed narinfos id=61 count=16251lb # [ 183.681159] niks3-server[1114]: 2026/09/23 16:02:33 INFO Registered completed upload object_key=nar/06mdwsnf16qmfi86y8r33asi4vcrp5z7kdxqki87psgfpgvvpdki.nar.zst6252lb # [ 183.690032] niks3-server[1114]: 2026/09/23 16:02:33 INFO Received complete push request method=POST path=/api/pushes/61/complete6253worker2 # [ 183.686894] nix-grpc-daemon[1933]: time=2026-09-23T16:02:33.310Z level=INFO msg="Upload complete. (193ms)"6254lb # [ 183.696738] niks3-server[1114]: 2026/09/23 16:02:33 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.narinfo6255worker2 # [ 183.694386] nix-grpc-daemon[1873]: ts=2026-09-23T16:02:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42964 duration_s=0 paths=1 nar_bytes_in=4166256worker2 # [ 183.705397] nix-grpc-daemon[1873]: ts=2026-09-23T16:02:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:429646257worker2 # [ 183.713075] nix-daemon[842]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)6258worker2 # [ 183.717948] nix-daemon[842]: reaped child process 1931, status = succeeded6259client # [ 183.735628] nix[1554]: building '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' on lb:50051...6260worker2 # [ 183.734435] nix-daemon[842]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)6261worker1 # [ 184.017404] nix-daemon[849]: reaped child process 1852, status = succeeded6262worker1 # [ 184.087096] nix-daemon[849]: accepted connection from pid 1876, user nix-grpc-daemon (trusted)6263worker1 # [ 184.104980] nix-daemon[849]: reaped child process 1879, status = succeeded6264lb # [ 184.029738] niks3-server[1114]: 2026/09/23 16:02:33 INFO Received push request method=POST path=/api/pushes6265worker1 # [ 184.120401] nix-grpc-daemon[1677]: time=2026-09-23T16:02:33.739Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6266worker1 # [ 184.121962] nix-grpc-daemon[1677]: time=2026-09-23T16:02:33.739Z level=INFO msg="Uploading k6zivhvzgmlyxfmy64fzd759bxlhf9b5-slow-q0 (120B)"6267lb # [ 184.071970] niks3-server[1114]: 2026/09/23 16:02:33 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.ls6268lb # [ 184.083568] niks3-server[1114]: 2026/09/23 16:02:33 INFO Registered completed upload object_key=nar/0xr809jj32p5n6mnc6i7lqsz48wrm5f9jp14m77k724041711iq7.nar.zst6269worker1 # [ 184.176869] nix-grpc-daemon[1677]: time=2026-09-23T16:02:33.796Z level=INFO msg="Uploading 1 narinfos"6270lb # [ 184.097782] niks3-server[1114]: 2026/09/23 16:02:33 INFO Received sign narinfos request method=POST path=/api/pushes/62/sign6271lb # [ 184.105634] niks3-server[1114]: 2026/09/23 16:02:33 INFO Signed narinfos id=62 count=16272lb # [ 184.111560] niks3-server[1114]: 2026/09/23 16:02:33 INFO Registered completed upload object_key=log/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv6273lb # [ 184.120429] niks3-server[1114]: 2026/09/23 16:02:33 INFO Received complete push request method=POST path=/api/pushes/62/complete6274lb # [ 184.130500] niks3-server[1114]: 2026/09/23 16:02:33 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.narinfo6275worker1 # [ 184.211544] nix-grpc-daemon[1677]: time=2026-09-23T16:02:33.831Z level=INFO msg="Upload complete. (290ms)"6276worker1 # [ 184.230786] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=1 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv assign_id=1 outputs=16277worker1 # [ 184.252310] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:33Z level=debug event=assigned drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv worker=192.168.1.3:50051 clients=16278worker1 # [ 184.256634] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326279worker1 # [ 184.261448] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563326280worker1 # [ 184.266199] nix-daemon[849]: reaped child process 1850, status = succeeded6281client # [ 184.184182] nix[1554]: copying 1 paths to 'lb:50051'...6282worker1 # [ 184.269795] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563326283client # [ 184.186538] nix[1554]: copying path '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' to 'lb:50051'...6284worker1 # [ 184.284633] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6285worker1 # [ 184.436320] nix-daemon[849]: accepted connection from pid 1883, user nix-grpc-daemon (trusted)6286worker1 # [ 184.455730] nix-daemon[849]: reaped child process 1886, status = succeeded6287lb # [ 184.377895] niks3-server[1114]: 2026/09/23 16:02:33 INFO Received push request method=POST path=/api/pushes6288worker1 # [ 184.467964] nix-grpc-daemon[1677]: time=2026-09-23T16:02:34.087Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6289worker1 # [ 184.469619] nix-grpc-daemon[1677]: time=2026-09-23T16:02:34.087Z level=INFO msg="Uploading jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv (416B)"6290lb # [ 184.408349] niks3-server[1114]: 2026/09/23 16:02:34 INFO Received sign narinfos request method=POST path=/api/pushes/63/sign6291worker1 # [ 184.494960] nix-grpc-daemon[1677]: time=2026-09-23T16:02:34.114Z level=INFO msg="Uploading 1 narinfos"6292lb # [ 184.421500] niks3-server[1114]: 2026/09/23 16:02:34 INFO Signed narinfos id=63 count=16293lb # [ 184.427398] niks3-server[1114]: 2026/09/23 16:02:34 INFO Registered completed upload object_key=nar/1fyfjzlx8vymsfmdv5cv8ymgjkr18fwm737qvys29plwwiwmyvwr.nar.zst6294lb # [ 184.446126] niks3-server[1114]: 2026/09/23 16:02:34 INFO Received complete push request method=POST path=/api/pushes/63/complete6295lb # [ 184.457205] niks3-server[1114]: 2026/09/23 16:02:34 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.ls6296lb # [ 184.466435] niks3-server[1114]: 2026/09/23 16:02:34 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.narinfo6297worker1 # [ 184.569054] nix-grpc-daemon[1677]: time=2026-09-23T16:02:34.188Z level=INFO msg="Upload complete. (271ms)"6298worker1 # [ 184.579749] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=4166299worker1 # [ 184.600296] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326300worker1 # [ 184.613108] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6301worker1 # [ 184.616617] nix-daemon[849]: reaped child process 1881, status = succeeded6302client # [ 184.542975] nix[1554]: building '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' on lb:50051...6303worker1 # [ 184.630940] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6304worker1 # [ 186.264467] nix-daemon[849]: reaped child process 1890, status = succeeded6305worker1 # [ 186.363505] nix-daemon[849]: accepted connection from pid 1915, user nix-grpc-daemon (trusted)6306worker1 # [ 186.385552] nix-daemon[849]: reaped child process 1918, status = succeeded6307lb # [ 186.303102] niks3-server[1114]: 2026/09/23 16:02:35 INFO Received push request method=POST path=/api/pushes6308worker1 # [ 186.400425] nix-grpc-daemon[1677]: time=2026-09-23T16:02:36.017Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6309worker1 # [ 186.401879] nix-grpc-daemon[1677]: time=2026-09-23T16:02:36.017Z level=INFO msg="Uploading gjyi8j90pk5flrsqqdhj6h132pcb7n33-slow-q2 (120B)"6310lb # [ 186.351412] niks3-server[1114]: 2026/09/23 16:02:35 INFO Received sign narinfos request method=POST path=/api/pushes/64/sign6311worker1 # [ 186.440857] nix-grpc-daemon[1677]: time=2026-09-23T16:02:36.060Z level=INFO msg="Uploading 1 narinfos"6312lb # [ 186.367045] niks3-server[1114]: 2026/09/23 16:02:35 INFO Signed narinfos id=64 count=16313lb # [ 186.376670] niks3-server[1114]: 2026/09/23 16:02:35 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.ls6314lb # [ 186.384440] niks3-server[1114]: 2026/09/23 16:02:35 INFO Registered completed upload object_key=log/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv6315lb # [ 186.387524] niks3-server[1114]: 2026/09/23 16:02:35 INFO Registered completed upload object_key=nar/15imdx7zyy6mzjjv6qvdzywh0d9w1n6ikc0rq4rq95aw9x6swfzj.nar.zst6316lb # [ 186.401847] niks3-server[1114]: 2026/09/23 16:02:36 INFO Received complete push request method=POST path=/api/pushes/64/complete6317lb # [ 186.404551] niks3-server[1114]: 2026/09/23 16:02:36 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.narinfo6318worker1 # [ 186.488894] nix-grpc-daemon[1677]: time=2026-09-23T16:02:36.108Z level=INFO msg="Upload complete. (242ms)"6319worker2 # [ 186.403086] nix-daemon[842]: reaped child process 1955, status = succeeded6320worker1 # [ 186.505193] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=1 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv assign_id=3 outputs=16321worker1 # [ 186.524629] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:36Z level=debug event=assigned drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker=192.168.1.3:50051 clients=16322worker1 # [ 186.528573] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326323worker1 # [ 186.531750] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563326324worker1 # [ 186.536350] nix-daemon[849]: reaped child process 1888, status = succeeded6325client # [ 186.457375] nix[1554]: copying 1 paths to 'lb:50051'...6326worker1 # [ 186.544437] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563326327client # [ 186.461736] nix[1554]: copying path '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' to 'lb:50051'...6328worker1 # [ 186.554125] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6329worker2 # [ 186.534288] nix-daemon[842]: accepted connection from pid 1981, user nix-grpc-daemon (trusted)6330worker2 # [ 186.553538] nix-daemon[842]: reaped child process 1984, status = succeeded6331lb # [ 186.568795] niks3-server[1114]: 2026/09/23 16:02:36 INFO Received push request method=POST path=/api/pushes6332worker2 # [ 186.570565] nix-grpc-daemon[1933]: time=2026-09-23T16:02:36.192Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6333worker2 # [ 186.573252] nix-grpc-daemon[1933]: time=2026-09-23T16:02:36.192Z level=INFO msg="Uploading y3qkgrgq2zzpgvi2rf5chq94i627wkdb-slow-q3 (120B)"6334lb # [ 186.604813] niks3-server[1114]: 2026/09/23 16:02:36 INFO Received sign narinfos request method=POST path=/api/pushes/65/sign6335worker1 # [ 186.693349] nix-daemon[849]: accepted connection from pid 1922, user nix-grpc-daemon (trusted)6336worker2 # [ 186.606888] nix-grpc-daemon[1933]: time=2026-09-23T16:02:36.230Z level=INFO msg="Uploading 1 narinfos"6337lb # [ 186.622579] niks3-server[1114]: 2026/09/23 16:02:36 INFO Signed narinfos id=65 count=16338worker1 # [ 186.713056] nix-daemon[849]: reaped child process 1925, status = succeeded6339lb # [ 186.632297] niks3-server[1114]: 2026/09/23 16:02:36 INFO Registered completed upload object_key=nar/1v1grzkvw3yj09x0dkdm2avihvqvnri156qzp768hk85zbw7lkn9.nar.zst6340worker1 # [ 186.731516] nix-grpc-daemon[1677]: time=2026-09-23T16:02:36.351Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6341worker1 # [ 186.733289] nix-grpc-daemon[1677]: time=2026-09-23T16:02:36.351Z level=INFO msg="Uploading pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv (416B)"6342lb # [ 186.651358] niks3-server[1114]: 2026/09/23 16:02:36 INFO Registered completed upload object_key=log/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv6343worker2 # [ 186.651095] nix-grpc-daemon[1933]: time=2026-09-23T16:02:36.274Z level=INFO msg="Upload complete. (301ms)"6344lb # [ 186.661137] niks3-server[1114]: 2026/09/23 16:02:36 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.ls6345lb # [ 186.669413] niks3-server[1114]: 2026/09/23 16:02:36 INFO Received push request method=POST path=/api/pushes6346lb # [ 186.674668] niks3-server[1114]: 2026/09/23 16:02:36 INFO Received complete push request method=POST path=/api/pushes/65/complete6347worker2 # [ 186.665189] nix-grpc-daemon[1873]: ts=2026-09-23T16:02:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42964 duration_s=2 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv assign_id=2 outputs=16348worker1 # [ 186.764910] nix-grpc-daemon[1677]: time=2026-09-23T16:02:36.384Z level=INFO msg="Uploading 1 narinfos"6349lb # [ 186.684252] niks3-server[1114]: 2026/09/23 16:02:36 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.narinfo6350lb # [ 186.687682] niks3-server[1114]: 2026/09/23 16:02:36 INFO Received sign narinfos request method=POST path=/api/pushes/66/sign6351worker2 # [ 186.683796] nix-daemon[842]: reaped child process 1952, status = succeeded6352lb # [ 186.699997] niks3-server[1114]: 2026/09/23 16:02:36 INFO Signed narinfos id=66 count=16353lb # [ 186.707950] niks3-server[1114]: 2026/09/23 16:02:36 INFO Registered completed upload object_key=nar/1k21jrz411xc9gkca2pnr9mrjq1s3bg3zhsam3nnv4j066dgzgm7.nar.zst6354lb # [ 186.712164] niks3-server[1114]: 2026/09/23 16:02:36 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.ls6355worker1 # [ 186.797505] nix-grpc-daemon[1677]: time=2026-09-23T16:02:36.417Z level=INFO msg="Upload complete. (231ms)"6356lb # [ 186.718615] niks3-server[1114]: 2026/09/23 16:02:36 INFO Received complete push request method=POST path=/api/pushes/66/complete6357lb # [ 186.721603] niks3-server[1114]: 2026/09/23 16:02:36 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.narinfo6358worker1 # [ 186.808330] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:36Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=4166359worker1 # [ 186.824688] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326360worker1 # [ 186.831607] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6361worker1 # [ 186.840611] nix-daemon[849]: reaped child process 1920, status = succeeded6362client # [ 186.763099] nix[1554]: building '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' on lb:50051...6363worker1 # [ 186.852619] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6364worker1 # [ 188.732557] nix-daemon[849]: reaped child process 1929, status = succeeded6365worker1 # [ 188.809913] nix-daemon[849]: accepted connection from pid 1954, user nix-grpc-daemon (trusted)6366worker1 # [ 188.828888] nix-daemon[849]: reaped child process 1957, status = succeeded6367lb # [ 188.754178] niks3-server[1114]: 2026/09/23 16:02:38 INFO Received push request method=POST path=/api/pushes6368worker1 # [ 188.848148] nix-grpc-daemon[1677]: time=2026-09-23T16:02:38.464Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6369worker1 # [ 188.849775] nix-grpc-daemon[1677]: time=2026-09-23T16:02:38.464Z level=INFO msg="Uploading l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx-slow-q1 (120B)"6370lb # [ 188.798460] niks3-server[1114]: 2026/09/23 16:02:38 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.ls6371worker1 # [ 188.890181] nix-grpc-daemon[1677]: time=2026-09-23T16:02:38.509Z level=INFO msg="Uploading 1 narinfos"6372lb # [ 188.811702] niks3-server[1114]: 2026/09/23 16:02:38 INFO Received sign narinfos request method=POST path=/api/pushes/67/sign6373lb # [ 188.821145] niks3-server[1114]: 2026/09/23 16:02:38 INFO Signed narinfos id=67 count=16374lb # [ 188.826430] niks3-server[1114]: 2026/09/23 16:02:38 INFO Registered completed upload object_key=nar/0c1c367r3msa4fz6imwp70zxi4fxvrkb86v3hv921wsrf13yd8v3.nar.zst6375lb # [ 188.838588] niks3-server[1114]: 2026/09/23 16:02:38 INFO Registered completed upload object_key=log/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv6376lb # [ 188.844460] niks3-server[1114]: 2026/09/23 16:02:38 INFO Received complete push request method=POST path=/api/pushes/67/complete6377lb # [ 188.849371] niks3-server[1114]: 2026/09/23 16:02:38 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.narinfo6378worker1 # [ 188.930986] nix-grpc-daemon[1677]: time=2026-09-23T16:02:38.549Z level=INFO msg="Upload complete. (264ms)"6379worker1 # [ 188.943267] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=2 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv assign_id=4 outputs=16380worker1 # [ 188.964676] nix-daemon[849]: reaped child process 1927, status = succeeded6381client # [ 188.917607] systemd[1]: burst.service: Deactivated successfully.6382client # [ 188.922129] systemd[1]: burst.service: Consumed 229ms CPU time over 6.664s wall clock time, 55.6M memory peak, 14.1K incoming IP traffic, 19.2K outgoing IP traffic.6383client: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }6384client: (finished: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }, in 0.07 seconds)6385lb: must succeed: curl -sf localhost:9901/clusters6386lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6387lb: must succeed: curl -sf localhost:9901/clusters6388lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6389worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6390worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6391(finished: subtest: queue gauges show a burst while every build blocks, in 8.08 seconds)6392subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6393client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16394worker1 # [ 190.415637] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:563326395worker2 # [ 190.332834] nix-grpc-daemon[1873]: ts=2026-09-23T16:02:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:429646396worker1 # [ 190.466316] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:40Z level=debug event=want client=2 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=06397worker1 # [ 190.473342] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:40Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=16398worker1 # [ 190.482657] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:40Z level=debug event=want client=2 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=06399worker1 # [ 190.490682] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:40Z level=debug event=want client=2 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=06400worker1 # [ 190.498278] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326401worker1 # [ 190.505150] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563326402worker1 # [ 190.511685] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:40Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563326403worker1 # [ 190.526566] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6404worker1 # [ 190.674385] nix-daemon[849]: accepted connection from pid 1970, user nix-grpc-daemon (trusted)6405worker1 # [ 190.698094] nix-daemon[849]: reaped child process 1973, status = succeeded6406lb # [ 190.615299] niks3-server[1114]: 2026/09/23 16:02:40 INFO Received push request method=POST path=/api/pushes6407worker1 # [ 190.707507] nix-grpc-daemon[1677]: time=2026-09-23T16:02:40.326Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6408worker1 # [ 190.709143] nix-grpc-daemon[1677]: time=2026-09-23T16:02:40.326Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)"6409lb # [ 190.643014] niks3-server[1114]: 2026/09/23 16:02:40 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls6410worker1 # [ 190.736557] nix-grpc-daemon[1677]: time=2026-09-23T16:02:40.356Z level=INFO msg="Uploading 1 narinfos"6411lb # [ 190.651645] niks3-server[1114]: 2026/09/23 16:02:40 INFO Received sign narinfos request method=POST path=/api/pushes/68/sign6412lb # [ 190.657334] niks3-server[1114]: 2026/09/23 16:02:40 INFO Signed narinfos id=68 count=16413lb # [ 190.661648] niks3-server[1114]: 2026/09/23 16:02:40 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst6414lb # [ 190.678312] niks3-server[1114]: 2026/09/23 16:02:40 INFO Received complete push request method=POST path=/api/pushes/68/complete6415worker1 # [ 190.764042] nix-grpc-daemon[1677]: time=2026-09-23T16:02:40.383Z level=INFO msg="Upload complete. (223ms)"6416lb # [ 190.685472] niks3-server[1114]: 2026/09/23 16:02:40 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo6417worker1 # [ 190.770965] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=5126418worker1 # [ 190.786437] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326419worker1 # [ 190.793352] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6420worker1 # [ 190.798762] nix-daemon[849]: reaped child process 1968, status = succeeded6421worker1 # [ 190.810049] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6422worker1 # [ 192.115660] nix-daemon[849]: reaped child process 1977, status = succeeded6423worker1 # [ 192.182342] nix-daemon[849]: accepted connection from pid 1993, user nix-grpc-daemon (trusted)6424worker1 # [ 192.202899] nix-daemon[849]: reaped child process 1996, status = succeeded6425lb # [ 192.126620] niks3-server[1114]: 2026/09/23 16:02:41 INFO Received push request method=POST path=/api/pushes6426worker1 # [ 192.219652] nix-grpc-daemon[1677]: time=2026-09-23T16:02:41.837Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6427worker1 # [ 192.222871] nix-grpc-daemon[1677]: time=2026-09-23T16:02:41.837Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)"6428lb # [ 192.172196] niks3-server[1114]: 2026/09/23 16:02:41 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6429worker1 # [ 192.264448] nix-grpc-daemon[1677]: time=2026-09-23T16:02:41.884Z level=INFO msg="Uploading 1 narinfos"6430lb # [ 192.186225] niks3-server[1114]: 2026/09/23 16:02:41 INFO Received sign narinfos request method=POST path=/api/pushes/69/sign6431lb # [ 192.195278] niks3-server[1114]: 2026/09/23 16:02:41 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv6432lb # [ 192.205681] niks3-server[1114]: 2026/09/23 16:02:41 INFO Signed narinfos id=69 count=16433lb # [ 192.211387] niks3-server[1114]: 2026/09/23 16:02:41 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls6434lb # [ 192.223133] niks3-server[1114]: 2026/09/23 16:02:41 INFO Received complete push request method=POST path=/api/pushes/69/complete6435worker1 # [ 192.310484] nix-grpc-daemon[1677]: time=2026-09-23T16:02:41.930Z level=INFO msg="Upload complete. (287ms)"6436lb # [ 192.230451] niks3-server[1114]: 2026/09/23 16:02:41 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo6437worker1 # [ 192.329209] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=5 outputs=16438worker1 # [ 192.346114] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:41Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=16439worker1 # [ 192.356819] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326440worker1 # [ 192.364423] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:41Z level=debug event=want client=2 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=06441worker1 # [ 192.378343] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563326442worker1 # [ 192.387823] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563326443worker1 # [ 192.394179] nix-daemon[849]: reaped child process 1975, status = succeeded6444worker1 # [ 192.398510] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6445worker1 # [ 192.520666] nix-daemon[849]: accepted connection from pid 2002, user nix-grpc-daemon (trusted)6446worker1 # [ 192.539523] nix-daemon[849]: reaped child process 2005, status = succeeded6447lb # [ 192.461900] niks3-server[1114]: 2026/09/23 16:02:42 INFO Received push request method=POST path=/api/pushes6448worker1 # [ 192.553348] nix-grpc-daemon[1677]: time=2026-09-23T16:02:42.172Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6449worker1 # [ 192.555063] nix-grpc-daemon[1677]: time=2026-09-23T16:02:42.172Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)"6450lb # [ 192.498052] niks3-server[1114]: 2026/09/23 16:02:42 INFO Received sign narinfos request method=POST path=/api/pushes/70/sign6451worker1 # [ 192.587313] nix-grpc-daemon[1677]: time=2026-09-23T16:02:42.207Z level=INFO msg="Uploading 1 narinfos"6452lb # [ 192.510698] niks3-server[1114]: 2026/09/23 16:02:42 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst6453lb # [ 192.522194] niks3-server[1114]: 2026/09/23 16:02:42 INFO Signed narinfos id=70 count=16454lb # [ 192.528305] niks3-server[1114]: 2026/09/23 16:02:42 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls6455lb # [ 192.540766] niks3-server[1114]: 2026/09/23 16:02:42 INFO Received complete push request method=POST path=/api/pushes/70/complete6456worker1 # [ 192.627958] nix-grpc-daemon[1677]: time=2026-09-23T16:02:42.247Z level=INFO msg="Upload complete. (234ms)"6457lb # [ 192.547827] niks3-server[1114]: 2026/09/23 16:02:42 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo6458worker1 # [ 192.638425] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=5126459worker1 # [ 192.655767] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326460worker1 # [ 192.662248] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6461worker1 # [ 192.667332] nix-daemon[849]: reaped child process 2000, status = succeeded6462worker1 # [ 192.680481] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6463worker1 # [ 193.315523] nix-daemon[849]: reaped child process 2009, status = succeeded6464worker1 # [ 193.411758] nix-daemon[849]: accepted connection from pid 2025, user nix-grpc-daemon (trusted)6465worker1 # [ 193.432753] nix-daemon[849]: reaped child process 2028, status = succeeded6466lb # [ 193.354657] niks3-server[1114]: 2026/09/23 16:02:42 INFO Received push request method=POST path=/api/pushes6467worker1 # [ 193.443670] nix-grpc-daemon[1677]: time=2026-09-23T16:02:43.062Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6468worker1 # [ 193.445261] nix-grpc-daemon[1677]: time=2026-09-23T16:02:43.062Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)"6469lb # [ 193.385131] niks3-server[1114]: 2026/09/23 16:02:43 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls6470worker1 # [ 193.474315] nix-grpc-daemon[1677]: time=2026-09-23T16:02:43.094Z level=INFO msg="Uploading 1 narinfos"6471lb # [ 193.393837] niks3-server[1114]: 2026/09/23 16:02:43 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6472lb # [ 193.403647] niks3-server[1114]: 2026/09/23 16:02:43 INFO Received sign narinfos request method=POST path=/api/pushes/71/sign6473lb # [ 193.412579] niks3-server[1114]: 2026/09/23 16:02:43 INFO Signed narinfos id=71 count=16474lb # [ 193.414571] niks3-server[1114]: 2026/09/23 16:02:43 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv6475lb # [ 193.421435] niks3-server[1114]: 2026/09/23 16:02:43 INFO Received complete push request method=POST path=/api/pushes/71/complete6476lb # [ 193.425302] niks3-server[1114]: 2026/09/23 16:02:43 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo6477worker1 # [ 193.506167] nix-grpc-daemon[1677]: time=2026-09-23T16:02:43.126Z level=INFO msg="Upload complete. (216ms)"6478worker1 # [ 193.519581] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=6 outputs=16479worker1 # [ 193.532310] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:43Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=16480worker1 # [ 193.535548] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326481worker1 # [ 193.544166] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:43Z level=debug event=want client=2 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=06482worker1 # [ 193.547577] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563326483worker1 # [ 193.550353] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563326484worker1 # [ 193.561717] nix-daemon[849]: reaped child process 2007, status = succeeded6485worker1 # [ 193.563107] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6486worker1 # [ 193.678232] nix-daemon[849]: accepted connection from pid 2034, user nix-grpc-daemon (trusted)6487worker1 # [ 193.699210] nix-daemon[849]: reaped child process 2037, status = succeeded6488lb # [ 193.621906] niks3-server[1114]: 2026/09/23 16:02:43 INFO Received push request method=POST path=/api/pushes6489worker1 # [ 193.711308] nix-grpc-daemon[1677]: time=2026-09-23T16:02:43.330Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6490worker1 # [ 193.712930] nix-grpc-daemon[1677]: time=2026-09-23T16:02:43.330Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)"6491worker1 # [ 193.738637] nix-grpc-daemon[1677]: time=2026-09-23T16:02:43.358Z level=INFO msg="Uploading 1 narinfos"6492lb # [ 193.651875] niks3-server[1114]: 2026/09/23 16:02:43 INFO Received sign narinfos request method=POST path=/api/pushes/72/sign6493lb # [ 193.666630] niks3-server[1114]: 2026/09/23 16:02:43 INFO Signed narinfos id=72 count=16494lb # [ 193.673851] niks3-server[1114]: 2026/09/23 16:02:43 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls6495lb # [ 193.686270] niks3-server[1114]: 2026/09/23 16:02:43 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst6496worker1 # [ 193.774481] nix-grpc-daemon[1677]: time=2026-09-23T16:02:43.394Z level=INFO msg="Upload complete. (206ms)"6497lb # [ 193.695010] niks3-server[1114]: 2026/09/23 16:02:43 INFO Received complete push request method=POST path=/api/pushes/72/complete6498lb # [ 193.700673] niks3-server[1114]: 2026/09/23 16:02:43 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo6499worker1 # [ 193.784289] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=5126500worker1 # [ 193.792805] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326501worker1 # [ 193.803259] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6502worker1 # [ 193.812374] nix-daemon[849]: reaped child process 2032, status = succeeded6503worker1 # [ 193.822212] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6504worker1 # [ 194.700526] nix-daemon[849]: reaped child process 2041, status = succeeded6505worker1 # [ 194.795769] nix-daemon[849]: accepted connection from pid 2057, user nix-grpc-daemon (trusted)6506worker1 # [ 194.814845] nix-daemon[849]: reaped child process 2060, status = succeeded6507lb # [ 194.739048] niks3-server[1114]: 2026/09/23 16:02:44 INFO Received push request method=POST path=/api/pushes6508worker1 # [ 194.832212] nix-grpc-daemon[1677]: time=2026-09-23T16:02:44.449Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6509worker1 # [ 194.833772] nix-grpc-daemon[1677]: time=2026-09-23T16:02:44.449Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)"6510lb # [ 194.778495] niks3-server[1114]: 2026/09/23 16:02:44 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv6511worker1 # [ 194.873963] nix-grpc-daemon[1677]: time=2026-09-23T16:02:44.493Z level=INFO msg="Uploading 1 narinfos"6512lb # [ 194.792672] niks3-server[1114]: 2026/09/23 16:02:44 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls6513lb # [ 194.803058] niks3-server[1114]: 2026/09/23 16:02:44 INFO Received sign narinfos request method=POST path=/api/pushes/73/sign6514lb # [ 194.815670] niks3-server[1114]: 2026/09/23 16:02:44 INFO Signed narinfos id=73 count=16515worker1 # [ 194.902665] nix-grpc-daemon[1677]: time=2026-09-23T16:02:44.522Z level=INFO msg="Upload complete. (270ms)"6516lb # [ 194.823688] niks3-server[1114]: 2026/09/23 16:02:44 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6517lb # [ 194.833012] niks3-server[1114]: 2026/09/23 16:02:44 INFO Received complete push request method=POST path=/api/pushes/73/complete6518lb # [ 194.838246] niks3-server[1114]: 2026/09/23 16:02:44 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo6519worker1 # [ 194.917813] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=7 outputs=16520worker1 # [ 194.940281] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:44Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=16521worker1 # [ 194.949787] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326522worker1 # [ 194.954918] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:44Z level=debug event=want client=2 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=06523worker1 # [ 194.960397] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563326524worker1 # [ 194.965696] nix-grpc-daemon[1677]: time=2026-09-23T16:02:44.584Z level=INFO msg="All 1 paths already cached"6525worker1 # [ 194.969746] nix-daemon[849]: reaped child process 2039, status = succeeded6526worker1 # [ 194.984215] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563326527worker1 # [ 194.989458] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6528worker1 # [ 195.117625] nix-daemon[849]: accepted connection from pid 2066, user nix-grpc-daemon (trusted)6529worker1 # [ 195.134440] nix-daemon[849]: reaped child process 2069, status = succeeded6530lb # [ 195.059897] niks3-server[1114]: 2026/09/23 16:02:44 INFO Received push request method=POST path=/api/pushes6531worker1 # [ 195.149740] nix-grpc-daemon[1677]: time=2026-09-23T16:02:44.769Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6532worker1 # [ 195.151453] nix-grpc-daemon[1677]: time=2026-09-23T16:02:44.769Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)"6533lb # [ 195.088591] niks3-server[1114]: 2026/09/23 16:02:44 INFO Received sign narinfos request method=POST path=/api/pushes/74/sign6534worker1 # [ 195.176489] nix-grpc-daemon[1677]: time=2026-09-23T16:02:44.796Z level=INFO msg="Uploading 1 narinfos"6535lb # [ 195.101913] niks3-server[1114]: 2026/09/23 16:02:44 INFO Signed narinfos id=74 count=16536lb # [ 195.111685] niks3-server[1114]: 2026/09/23 16:02:44 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst6537lb # [ 195.127934] niks3-server[1114]: 2026/09/23 16:02:44 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls6538worker1 # [ 195.218251] nix-grpc-daemon[1677]: time=2026-09-23T16:02:44.838Z level=INFO msg="Upload complete. (225ms)"6539lb # [ 195.138129] niks3-server[1114]: 2026/09/23 16:02:44 INFO Received complete push request method=POST path=/api/pushes/74/complete6540lb # [ 195.143683] niks3-server[1114]: 2026/09/23 16:02:44 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo6541worker1 # [ 195.228368] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=6246542worker1 # [ 195.245693] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326543worker1 # [ 195.252425] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6544worker1 # [ 195.257810] nix-daemon[849]: reaped child process 2064, status = succeeded6545worker1 # [ 195.271753] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6546worker1 # [ 196.939620] nix-daemon[849]: reaped child process 2073, status = succeeded6547worker1 # [ 197.049185] nix-daemon[849]: accepted connection from pid 2089, user nix-grpc-daemon (trusted)6548worker1 # [ 197.067691] nix-daemon[849]: reaped child process 2092, status = succeeded6549lb # [ 196.993887] niks3-server[1114]: 2026/09/23 16:02:46 INFO Received push request method=POST path=/api/pushes6550worker1 # [ 197.086334] nix-grpc-daemon[1677]: time=2026-09-23T16:02:46.704Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6551worker1 # [ 197.087952] nix-grpc-daemon[1677]: time=2026-09-23T16:02:46.704Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)"6552lb # [ 197.032573] niks3-server[1114]: 2026/09/23 16:02:46 INFO Received sign narinfos request method=POST path=/api/pushes/75/sign6553worker1 # [ 197.122070] nix-grpc-daemon[1677]: time=2026-09-23T16:02:46.741Z level=INFO msg="Uploading 1 narinfos"6554lb # [ 197.045922] niks3-server[1114]: 2026/09/23 16:02:46 INFO Signed narinfos id=75 count=16555lb # [ 197.054993] niks3-server[1114]: 2026/09/23 16:02:46 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst6556lb # [ 197.067700] niks3-server[1114]: 2026/09/23 16:02:46 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls6557lb # [ 197.073044] niks3-server[1114]: 2026/09/23 16:02:46 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv6558lb # [ 197.077987] niks3-server[1114]: 2026/09/23 16:02:46 INFO Received complete push request method=POST path=/api/pushes/75/complete6559worker1 # [ 197.163278] nix-grpc-daemon[1677]: time=2026-09-23T16:02:46.783Z level=INFO msg="Upload complete. (280ms)"6560lb # [ 197.084467] niks3-server[1114]: 2026/09/23 16:02:46 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo6561worker1 # [ 197.185135] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=8 outputs=16562worker1 # [ 197.200961] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:46Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=16563worker1 # [ 197.212199] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326564worker1 # [ 197.218384] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:46Z level=debug event=want client=2 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=06565worker1 # [ 197.224448] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563326566worker1 # [ 197.237925] nix-daemon[849]: reaped child process 2071, status = succeeded6567worker1 # [ 197.239987] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6568worker1 # [ 197.248566] nix-grpc-daemon[1677]: time=2026-09-23T16:02:46.843Z level=INFO msg="All 1 paths already cached"6569worker1 # [ 197.253791] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563326570worker1 # [ 197.379825] nix-daemon[849]: accepted connection from pid 2098, user nix-grpc-daemon (trusted)6571worker1 # [ 197.398275] nix-daemon[849]: reaped child process 2101, status = succeeded6572lb # [ 197.322151] niks3-server[1114]: 2026/09/23 16:02:46 INFO Received push request method=POST path=/api/pushes6573worker1 # [ 197.412745] nix-grpc-daemon[1677]: time=2026-09-23T16:02:47.032Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6574worker1 # [ 197.414239] nix-grpc-daemon[1677]: time=2026-09-23T16:02:47.032Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)"6575worker1 # [ 197.437606] nix-grpc-daemon[1677]: time=2026-09-23T16:02:47.057Z level=INFO msg="Uploading 1 narinfos"6576lb # [ 197.351141] niks3-server[1114]: 2026/09/23 16:02:46 INFO Received sign narinfos request method=POST path=/api/pushes/76/sign6577lb # [ 197.363695] niks3-server[1114]: 2026/09/23 16:02:46 INFO Signed narinfos id=76 count=16578lb # [ 197.370435] niks3-server[1114]: 2026/09/23 16:02:46 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst6579lb # [ 197.386283] niks3-server[1114]: 2026/09/23 16:02:46 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls6580worker1 # [ 197.473984] nix-grpc-daemon[1677]: time=2026-09-23T16:02:47.093Z level=INFO msg="Upload complete. (220ms)"6581lb # [ 197.395352] niks3-server[1114]: 2026/09/23 16:02:46 INFO Received complete push request method=POST path=/api/pushes/76/complete6582lb # [ 197.400643] niks3-server[1114]: 2026/09/23 16:02:47 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo6583worker1 # [ 197.484356] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=6246584worker1 # [ 197.498017] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326585worker1 # [ 197.504253] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6586worker1 # [ 197.510520] nix-daemon[849]: reaped child process 2096, status = succeeded6587worker1 # [ 197.522051] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6588lb # [ 198.094169] envoy[897]: conn peer=[2001:db8:1::1]:51962 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9744 tx=5256 ms=64366589worker1 # [ 198.571855] nix-daemon[849]: reaped child process 2105, status = succeeded6590worker1 # [ 198.667866] nix-daemon[849]: accepted connection from pid 2121, user nix-grpc-daemon (trusted)6591worker1 # [ 198.685336] nix-daemon[849]: reaped child process 2124, status = succeeded6592lb # [ 198.611021] niks3-server[1114]: 2026/09/23 16:02:48 INFO Received push request method=POST path=/api/pushes6593worker1 # [ 198.702628] nix-grpc-daemon[1677]: time=2026-09-23T16:02:48.321Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6594worker1 # [ 198.707754] nix-grpc-daemon[1677]: time=2026-09-23T16:02:48.321Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)"6595lb # [ 198.658657] niks3-server[1114]: 2026/09/23 16:02:48 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls6596worker1 # [ 198.749842] nix-grpc-daemon[1677]: time=2026-09-23T16:02:48.369Z level=INFO msg="Uploading 1 narinfos"6597lb # [ 198.670158] niks3-server[1114]: 2026/09/23 16:02:48 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst6598lb # [ 198.681234] niks3-server[1114]: 2026/09/23 16:02:48 INFO Received sign narinfos request method=POST path=/api/pushes/77/sign6599lb # [ 198.691996] niks3-server[1114]: 2026/09/23 16:02:48 INFO Signed narinfos id=77 count=16600worker1 # [ 198.779027] nix-grpc-daemon[1677]: time=2026-09-23T16:02:48.398Z level=INFO msg="Upload complete. (245ms)"6601lb # [ 198.698799] niks3-server[1114]: 2026/09/23 16:02:48 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv6602lb # [ 198.704548] niks3-server[1114]: 2026/09/23 16:02:48 INFO Received complete push request method=POST path=/api/pushes/77/complete6603lb # [ 198.710833] niks3-server[1114]: 2026/09/23 16:02:48 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo6604worker1 # [ 198.795275] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=9 outputs=16605worker1 # [ 198.809052] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:48Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=16606worker1 # [ 198.820989] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326607worker1 # [ 198.826372] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:48Z level=debug event=want client=2 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=06608worker1 # [ 198.831431] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563326609worker1 # [ 198.836379] nix-grpc-daemon[1677]: time=2026-09-23T16:02:48.455Z level=INFO msg="All 1 paths already cached"6610worker1 # [ 198.838938] nix-daemon[849]: reaped child process 2103, status = succeeded6611worker1 # [ 198.841660] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563326612worker1 # [ 198.855785] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6613worker1 # [ 198.980246] nix-daemon[849]: accepted connection from pid 2130, user nix-grpc-daemon (trusted)6614worker1 # [ 199.001605] nix-daemon[849]: reaped child process 2133, status = succeeded6615lb # [ 198.918576] niks3-server[1114]: 2026/09/23 16:02:48 INFO Received push request method=POST path=/api/pushes6616worker1 # [ 199.009846] nix-grpc-daemon[1677]: time=2026-09-23T16:02:48.629Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6617worker1 # [ 199.011285] nix-grpc-daemon[1677]: time=2026-09-23T16:02:48.629Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)"6618lb # [ 198.948975] niks3-server[1114]: 2026/09/23 16:02:48 INFO Received sign narinfos request method=POST path=/api/pushes/78/sign6619worker1 # [ 199.039070] nix-grpc-daemon[1677]: time=2026-09-23T16:02:48.658Z level=INFO msg="Uploading 1 narinfos"6620lb # [ 198.963609] niks3-server[1114]: 2026/09/23 16:02:48 INFO Signed narinfos id=78 count=16621lb # [ 198.971968] niks3-server[1114]: 2026/09/23 16:02:48 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst6622lb # [ 198.988124] niks3-server[1114]: 2026/09/23 16:02:48 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls6623lb # [ 198.991454] niks3-server[1114]: 2026/09/23 16:02:48 INFO Received complete push request method=POST path=/api/pushes/78/complete6624worker1 # [ 199.079502] nix-grpc-daemon[1677]: time=2026-09-23T16:02:48.699Z level=INFO msg="Upload complete. (218ms)"6625lb # [ 198.998174] niks3-server[1114]: 2026/09/23 16:02:48 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo6626worker1 # [ 199.087711] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=6246627worker1 # [ 199.102651] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326628worker1 # [ 199.109055] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6629worker1 # [ 199.113833] nix-daemon[849]: reaped child process 2128, status = succeeded6630worker1 # [ 199.127457] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6631worker1 # [ 199.767307] nix-daemon[849]: reaped child process 2137, status = succeeded6632worker1 # [ 199.857260] nix-daemon[849]: accepted connection from pid 2153, user nix-grpc-daemon (trusted)6633worker1 # [ 199.876277] nix-daemon[849]: reaped child process 2156, status = succeeded6634lb # [ 199.801209] niks3-server[1114]: 2026/09/23 16:02:49 INFO Received push request method=POST path=/api/pushes6635worker1 # [ 199.892413] nix-grpc-daemon[1677]: time=2026-09-23T16:02:49.511Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6636worker1 # [ 199.893922] nix-grpc-daemon[1677]: time=2026-09-23T16:02:49.511Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)"6637lb # [ 199.839609] niks3-server[1114]: 2026/09/23 16:02:49 INFO Received sign narinfos request method=POST path=/api/pushes/79/sign6638worker1 # [ 199.929867] nix-grpc-daemon[1677]: time=2026-09-23T16:02:49.549Z level=INFO msg="Uploading 1 narinfos"6639lb # [ 199.854183] niks3-server[1114]: 2026/09/23 16:02:49 INFO Signed narinfos id=79 count=16640lb # [ 199.864987] niks3-server[1114]: 2026/09/23 16:02:49 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv6641lb # [ 199.882204] niks3-server[1114]: 2026/09/23 16:02:49 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls6642lb # [ 199.893883] niks3-server[1114]: 2026/09/23 16:02:49 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst6643lb # [ 199.898610] niks3-server[1114]: 2026/09/23 16:02:49 INFO Received complete push request method=POST path=/api/pushes/79/complete6644worker1 # [ 199.980280] nix-grpc-daemon[1677]: time=2026-09-23T16:02:49.600Z level=INFO msg="Upload complete. (244ms)"6645lb # [ 199.901876] niks3-server[1114]: 2026/09/23 16:02:49 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo6646worker1 # [ 200.000570] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=10 outputs=16647worker1 # [ 200.016955] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:49Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=16648worker1 # [ 200.025087] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326649worker1 # [ 200.029813] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:49Z level=debug event=want client=2 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=06650worker1 # [ 200.035009] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563326651worker1 # [ 200.040116] nix-daemon[849]: reaped child process 2135, status = succeeded6652worker1 # [ 200.051989] nix-grpc-daemon[1677]: time=2026-09-23T16:02:49.653Z level=INFO msg="All 2 paths already cached"6653worker1 # [ 200.054523] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6654worker1 # [ 200.058072] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563326655worker1 # [ 200.182107] nix-daemon[849]: accepted connection from pid 2162, user nix-grpc-daemon (trusted)6656worker1 # [ 200.198002] nix-daemon[849]: reaped child process 2165, status = succeeded6657lb # [ 200.123659] niks3-server[1114]: 2026/09/23 16:02:49 INFO Received push request method=POST path=/api/pushes6658worker1 # [ 200.214082] nix-grpc-daemon[1677]: time=2026-09-23T16:02:49.833Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6659worker1 # [ 200.215617] nix-grpc-daemon[1677]: time=2026-09-23T16:02:49.833Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)"6660lb # [ 200.153304] niks3-server[1114]: 2026/09/23 16:02:49 INFO Received sign narinfos request method=POST path=/api/pushes/80/sign6661worker1 # [ 200.240675] nix-grpc-daemon[1677]: time=2026-09-23T16:02:49.860Z level=INFO msg="Uploading 1 narinfos"6662lb # [ 200.165810] niks3-server[1114]: 2026/09/23 16:02:49 INFO Signed narinfos id=80 count=16663lb # [ 200.172721] niks3-server[1114]: 2026/09/23 16:02:49 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst6664lb # [ 200.191821] niks3-server[1114]: 2026/09/23 16:02:49 INFO Received complete push request method=POST path=/api/pushes/80/complete6665worker1 # [ 200.279552] nix-grpc-daemon[1677]: time=2026-09-23T16:02:49.899Z level=INFO msg="Upload complete. (216ms)"6666lb # [ 200.200571] niks3-server[1114]: 2026/09/23 16:02:49 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls6667lb # [ 200.206194] niks3-server[1114]: 2026/09/23 16:02:49 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo6668worker1 # [ 200.288422] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=7526669worker1 # [ 200.303734] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326670worker1 # [ 200.310744] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6671worker1 # [ 200.316754] nix-daemon[849]: reaped child process 2160, status = succeeded6672worker1 # [ 200.328864] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6673worker1 # [ 201.567973] nix-daemon[849]: reaped child process 2169, status = succeeded6674worker1 # [ 201.674860] nix-daemon[849]: accepted connection from pid 2185, user nix-grpc-daemon (trusted)6675worker1 # [ 201.698494] nix-daemon[849]: reaped child process 2188, status = succeeded6676lb # [ 201.616826] niks3-server[1114]: 2026/09/23 16:02:51 INFO Received push request method=POST path=/api/pushes6677worker1 # [ 201.710006] nix-grpc-daemon[1677]: time=2026-09-23T16:02:51.327Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6678worker1 # [ 201.711663] nix-grpc-daemon[1677]: time=2026-09-23T16:02:51.328Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)"6679lb # [ 201.657900] niks3-server[1114]: 2026/09/23 16:02:51 INFO Received sign narinfos request method=POST path=/api/pushes/81/sign6680worker1 # [ 201.751668] nix-grpc-daemon[1677]: time=2026-09-23T16:02:51.371Z level=INFO msg="Uploading 1 narinfos"6681lb # [ 201.674783] niks3-server[1114]: 2026/09/23 16:02:51 INFO Signed narinfos id=81 count=16682lb # [ 201.681786] niks3-server[1114]: 2026/09/23 16:02:51 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst6683lb # [ 201.691556] niks3-server[1114]: 2026/09/23 16:02:51 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv6684lb # [ 201.700854] niks3-server[1114]: 2026/09/23 16:02:51 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls6685lb # [ 201.703638] niks3-server[1114]: 2026/09/23 16:02:51 INFO Received complete push request method=POST path=/api/pushes/81/complete6686worker1 # [ 201.790024] nix-grpc-daemon[1677]: time=2026-09-23T16:02:51.409Z level=INFO msg="Upload complete. (248ms)"6687lb # [ 201.710377] niks3-server[1114]: 2026/09/23 16:02:51 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo6688worker1 # [ 201.801832] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=11 outputs=16689worker1 # [ 201.816369] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:51Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=16690worker1 # [ 201.825522] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326691worker1 # [ 201.830564] nix-daemon[849]: reaped child process 2167, status = succeeded6692worker1 # [ 201.834043] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563326693worker1 # [ 201.845606] nix-grpc-daemon[1677]: time=2026-09-23T16:02:51.465Z level=INFO msg="All 2 paths already cached"6694worker1 # [ 201.854503] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563326695worker1 # [ 201.867129] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6696worker1 # [ 202.007669] nix-daemon[849]: accepted connection from pid 2192, user nix-grpc-daemon (trusted)6697worker1 # [ 202.032591] nix-daemon[849]: reaped child process 2195, status = succeeded6698lb # [ 201.955558] niks3-server[1114]: 2026/09/23 16:02:51 INFO Received push request method=POST path=/api/pushes6699worker1 # [ 202.046035] nix-grpc-daemon[1677]: time=2026-09-23T16:02:51.665Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6700worker1 # [ 202.047575] nix-grpc-daemon[1677]: time=2026-09-23T16:02:51.665Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)"6701lb # [ 201.989832] niks3-server[1114]: 2026/09/23 16:02:51 INFO Received sign narinfos request method=POST path=/api/pushes/82/sign6702worker1 # [ 202.078707] nix-grpc-daemon[1677]: time=2026-09-23T16:02:51.698Z level=INFO msg="Uploading 1 narinfos"6703lb # [ 202.003255] niks3-server[1114]: 2026/09/23 16:02:51 INFO Signed narinfos id=82 count=16704lb # [ 202.010788] niks3-server[1114]: 2026/09/23 16:02:51 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls6705lb # [ 202.026066] niks3-server[1114]: 2026/09/23 16:02:51 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst6706lb # [ 202.038813] niks3-server[1114]: 2026/09/23 16:02:51 INFO Received complete push request method=POST path=/api/pushes/82/complete6707worker1 # [ 202.117533] nix-grpc-daemon[1677]: time=2026-09-23T16:02:51.737Z level=INFO msg="Upload complete. (238ms)"6708lb # [ 202.044307] niks3-server[1114]: 2026/09/23 16:02:51 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo6709worker1 # [ 202.128666] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=7526710worker1 # [ 202.144770] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326711worker1 # [ 202.153059] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6712worker1 # [ 202.158923] nix-daemon[849]: reaped child process 2190, status = succeeded6713worker1 # [ 202.170117] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6714worker1 # [ 203.279293] nix-daemon[849]: reaped child process 2199, status = succeeded6715worker1 # [ 203.350714] nix-daemon[849]: accepted connection from pid 2215, user nix-grpc-daemon (trusted)6716worker1 # [ 203.369297] nix-daemon[849]: reaped child process 2218, status = succeeded6717lb # [ 203.293433] niks3-server[1114]: 2026/09/23 16:02:52 INFO Received push request method=POST path=/api/pushes6718worker1 # [ 203.387649] nix-grpc-daemon[1677]: time=2026-09-23T16:02:53.006Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6719worker1 # [ 203.389284] nix-grpc-daemon[1677]: time=2026-09-23T16:02:53.006Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)"6720lb # [ 203.338485] niks3-server[1114]: 2026/09/23 16:02:52 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst6721worker1 # [ 203.432916] nix-grpc-daemon[1677]: time=2026-09-23T16:02:53.052Z level=INFO msg="Uploading 1 narinfos"6722lb # [ 203.352849] niks3-server[1114]: 2026/09/23 16:02:52 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls6723lb # [ 203.361551] niks3-server[1114]: 2026/09/23 16:02:52 INFO Received sign narinfos request method=POST path=/api/pushes/83/sign6724lb # [ 203.371341] niks3-server[1114]: 2026/09/23 16:02:52 INFO Signed narinfos id=83 count=16725lb # [ 203.376716] niks3-server[1114]: 2026/09/23 16:02:52 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv6726lb # [ 203.381054] niks3-server[1114]: 2026/09/23 16:02:52 INFO Received complete push request method=POST path=/api/pushes/83/complete6727worker1 # [ 203.461701] nix-grpc-daemon[1677]: time=2026-09-23T16:02:53.081Z level=INFO msg="Upload complete. (279ms)"6728lb # [ 203.384393] niks3-server[1114]: 2026/09/23 16:02:52 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo6729worker1 # [ 203.477224] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=12 outputs=16730worker1 # [ 203.497052] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:53Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=16731worker1 # [ 203.501288] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326732worker1 # [ 203.506857] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563326733worker1 # [ 203.511602] nix-daemon[849]: reaped child process 2197, status = succeeded6734worker1 # [ 203.514777] nix-grpc-daemon[1677]: time=2026-09-23T16:02:53.134Z level=INFO msg="All 2 paths already cached"6735worker1 # [ 203.523042] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:563326736worker1 # [ 203.535902] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6737worker1 # [ 203.682563] nix-daemon[849]: accepted connection from pid 2222, user nix-grpc-daemon (trusted)6738worker1 # [ 203.700515] nix-daemon[849]: reaped child process 2225, status = succeeded6739lb # [ 203.624791] niks3-server[1114]: 2026/09/23 16:02:53 INFO Received push request method=POST path=/api/pushes6740worker1 # [ 203.714594] nix-grpc-daemon[1677]: time=2026-09-23T16:02:53.334Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6741worker1 # [ 203.716132] nix-grpc-daemon[1677]: time=2026-09-23T16:02:53.334Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)"6742lb # [ 203.653785] niks3-server[1114]: 2026/09/23 16:02:53 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst6743worker1 # [ 203.746783] nix-grpc-daemon[1677]: time=2026-09-23T16:02:53.366Z level=INFO msg="Uploading 1 narinfos"6744lb # [ 203.668320] niks3-server[1114]: 2026/09/23 16:02:53 INFO Received sign narinfos request method=POST path=/api/pushes/84/sign6745lb # [ 203.677912] niks3-server[1114]: 2026/09/23 16:02:53 INFO Signed narinfos id=84 count=16746lb # [ 203.692107] niks3-server[1114]: 2026/09/23 16:02:53 INFO Received complete push request method=POST path=/api/pushes/84/complete6747lb # [ 203.702621] niks3-server[1114]: 2026/09/23 16:02:53 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls6748lb # [ 203.706788] niks3-server[1114]: 2026/09/23 16:02:53 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo6749worker1 # [ 203.788259] nix-grpc-daemon[1677]: time=2026-09-23T16:02:53.407Z level=INFO msg="Upload complete. (239ms)"6750worker1 # [ 203.800333] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=0 paths=1 nar_bytes_in=7526751worker1 # [ 203.816754] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:563326752worker1 # [ 203.824465] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6753worker1 # [ 203.835075] nix-daemon[849]: reaped child process 2220, status = succeeded6754worker1 # [ 203.843680] nix-daemon[849]: accepted connection from pid 1601, user nix-grpc-daemon (trusted)6755worker1 # [ 205.353848] nix-daemon[849]: reaped child process 2229, status = succeeded6756worker1 # [ 205.424377] nix-daemon[849]: accepted connection from pid 2245, user nix-grpc-daemon (trusted)6757worker1 # [ 205.443281] nix-daemon[849]: reaped child process 2248, status = succeeded6758lb # [ 205.367153] niks3-server[1114]: 2026/09/23 16:02:54 INFO Received push request method=POST path=/api/pushes6759worker1 # [ 205.459731] nix-grpc-daemon[1677]: time=2026-09-23T16:02:55.077Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6760worker1 # [ 205.461431] nix-grpc-daemon[1677]: time=2026-09-23T16:02:55.077Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)"6761lb # [ 205.409912] niks3-server[1114]: 2026/09/23 16:02:55 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls6762worker1 # [ 205.503008] nix-grpc-daemon[1677]: time=2026-09-23T16:02:55.122Z level=INFO msg="Uploading 1 narinfos"6763lb # [ 205.424469] niks3-server[1114]: 2026/09/23 16:02:55 INFO Received sign narinfos request method=POST path=/api/pushes/85/sign6764lb # [ 205.433358] niks3-server[1114]: 2026/09/23 16:02:55 INFO Signed narinfos id=85 count=16765lb # [ 205.435865] niks3-server[1114]: 2026/09/23 16:02:55 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv6766lb # [ 205.452099] niks3-server[1114]: 2026/09/23 16:02:55 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst6767lb # [ 205.456523] niks3-server[1114]: 2026/09/23 16:02:55 INFO Received complete push request method=POST path=/api/pushes/85/complete6768lb # [ 205.462785] niks3-server[1114]: 2026/09/23 16:02:55 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo6769worker1 # [ 205.533102] nix-grpc-daemon[1677]: time=2026-09-23T16:02:55.152Z level=INFO msg="Upload complete. (278ms)"6770worker1 # [ 205.549433] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:56332 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=13 outputs=16771worker1 # [ 205.567030] nix-daemon[849]: reaped child process 2227, status = succeeded6772client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 15.43 seconds)6773worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6774worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6775client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]'6776client # [ 205.662481] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6777client # Running as unit: gpu.service; invocation ID: 3a69f8bb4bfe4cabb24287c0432ca03a6778client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds)6779lb: must succeed: curl -sf localhost:9901/clusters6780lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6781lb: must succeed: curl -sf localhost:9901/clusters6782lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6783worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6784client # [ 205.858377] nix[1628]: don't know how to build these paths:6785client # [ 205.859473] nix[1628]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv6786worker2 # [ 205.854009] nix-grpc-daemon[1873]: ts=2026-09-23T16:02:55Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:429646787worker1 # [ 205.950518] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:563326788worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6789worker1 # [ 205.972780] nix-grpc-daemon[1601]: ts=2026-09-23T16:02:55Z level=debug event=want client=3 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=06790client # [ 205.887994] nix[1628]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip}, waiting up to 30s for a worker6791lb: must succeed: curl -sf localhost:9901/clusters6792lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6793lb: must succeed: curl -sf localhost:9901/clusters6794lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6795worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6796worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6797client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6798lb # [ 208.090805] envoy[897]: conn peer=[2001:db8:1::1]:51974 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27127 tx=12356 ms=151536799client # [ 235.893871] nix[1628]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip}6800client # [ 235.906666] nix[1628]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed6801client # [ 235.946186] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6802client # [ 235.954426] systemd[1]: gpu.service: Failed with result 'exit-code'.6803client # [ 235.958578] systemd[1]: gpu.service: Consumed 191ms CPU time over 30.289s wall clock time, 37.4M memory peak, 5.6K incoming IP traffic, 5.2K outgoing IP traffic.6804client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.06 seconds)6805client: must succeed: journalctl -u gpu -o cat6806client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)6807lb: must succeed: curl -sf localhost:9901/clusters6808lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6809lb: must succeed: curl -sf localhost:9901/clusters6810lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6811worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6812worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6813(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 46.23 seconds)6814subtest: low disk drains a worker and builds go to the other6815worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6816worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.14 seconds)6817worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6818worker1 # [ 237.711752] nix-grpc-daemon[1601]: ts=2026-09-23T16:03:27Z level=info event=unhealthy reason=min_free6819lb # [ 238.097609] envoy[897]: conn peer=[2001:db8:1::1]:59716 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=931 tx=613 ms=300676820worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.10 seconds)6821client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&26822client # don't know how to build these paths:6823client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6824worker2 # [ 238.788355] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:28Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:429646825worker2 # [ 238.795811] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:429646826worker1 # [ 238.914635] nix-grpc-daemon[1601]: ts=2026-09-23T16:03:28Z level=debug event=want client=4 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=06827worker1 # [ 238.920997] nix-grpc-daemon[1601]: ts=2026-09-23T16:03:28Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=16828worker2 # [ 238.833739] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:429646829worker2 # [ 238.849280] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:429646830client # copying 1 paths to 'lb:50051'...6831client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'...6832worker2 # [ 238.866626] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:429646833worker2 # [ 238.882228] nix-daemon[842]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)6834worker2 # [ 239.038982] nix-daemon[842]: accepted connection from pid 2000, user nix-grpc-daemon (trusted)6835worker2 # [ 239.059299] nix-daemon[842]: reaped child process 2003, status = succeeded6836lb # [ 239.074317] niks3-server[1114]: 2026/09/23 16:03:28 INFO Received push request method=POST path=/api/pushes6837worker2 # [ 239.077155] nix-grpc-daemon[1933]: time=2026-09-23T16:03:28.700Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6838worker2 # [ 239.079738] nix-grpc-daemon[1933]: time=2026-09-23T16:03:28.702Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)"6839worker2 # [ 239.118913] nix-grpc-daemon[1933]: time=2026-09-23T16:03:28.742Z level=INFO msg="Uploading 1 narinfos"6840lb # [ 239.116948] niks3-server[1114]: 2026/09/23 16:03:28 INFO Received sign narinfos request method=POST path=/api/pushes/86/sign6841lb # [ 239.132064] niks3-server[1114]: 2026/09/23 16:03:28 INFO Signed narinfos id=86 count=16842lb # [ 239.143551] niks3-server[1114]: 2026/09/23 16:03:28 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst6843lb # [ 239.151908] niks3-server[1114]: 2026/09/23 16:03:28 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls6844lb # [ 239.164223] niks3-server[1114]: 2026/09/23 16:03:28 INFO Received complete push request method=POST path=/api/pushes/86/complete6845worker2 # [ 239.165528] nix-grpc-daemon[1933]: time=2026-09-23T16:03:28.789Z level=INFO msg="Upload complete. (266ms)"6846lb # [ 239.187995] niks3-server[1114]: 2026/09/23 16:03:28 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo6847worker2 # [ 239.175753] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42964 duration_s=0 paths=1 nar_bytes_in=5126848worker2 # [ 239.190643] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:429646849worker2 # [ 239.204663] nix-daemon[842]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)6850worker2 # [ 239.211768] nix-daemon[842]: reaped child process 1998, status = succeeded6851client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051...6852worker2 # [ 239.225902] nix-daemon[842]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)6853client # farm-a> worker2-next: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6854client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6855client # farm-a> LOG-farm-a-drain6856worker2 # [ 240.536864] nix-daemon[842]: reaped child process 2007, status = succeeded6857worker2 # [ 240.609447] nix-daemon[842]: accepted connection from pid 2023, user nix-grpc-daemon (trusted)6858worker2 # [ 240.629713] nix-daemon[842]: reaped child process 2026, status = succeeded6859lb # [ 240.637423] niks3-server[1114]: 2026/09/23 16:03:30 INFO Received push request method=POST path=/api/pushes6860worker2 # [ 240.640399] nix-grpc-daemon[1933]: time=2026-09-23T16:03:30.263Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6861worker2 # [ 240.641975] nix-grpc-daemon[1933]: time=2026-09-23T16:03:30.263Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)"6862lb # [ 240.679994] niks3-server[1114]: 2026/09/23 16:03:30 INFO Received sign narinfos request method=POST path=/api/pushes/87/sign6863worker2 # [ 240.681370] nix-grpc-daemon[1933]: time=2026-09-23T16:03:30.304Z level=INFO msg="Uploading 1 narinfos"6864lb # [ 240.693580] niks3-server[1114]: 2026/09/23 16:03:30 INFO Signed narinfos id=87 count=16865lb # [ 240.695904] niks3-server[1114]: 2026/09/23 16:03:30 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6866lb # [ 240.708355] niks3-server[1114]: 2026/09/23 16:03:30 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls6867worker2 # [ 240.710555] nix-grpc-daemon[1933]: time=2026-09-23T16:03:30.334Z level=INFO msg="Upload complete. (270ms)"6868lb # [ 240.718887] niks3-server[1114]: 2026/09/23 16:03:30 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6869lb # [ 240.726108] niks3-server[1114]: 2026/09/23 16:03:30 INFO Received complete push request method=POST path=/api/pushes/87/complete6870lb # [ 240.731326] niks3-server[1114]: 2026/09/23 16:03:30 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo6871worker2 # [ 240.725170] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42964 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=14 outputs=16872worker1 # [ 240.821753] nix-grpc-daemon[1601]: ts=2026-09-23T16:03:30Z level=debug event=want client=4 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=06873worker1 # [ 240.822121] nix-grpc-daemon[1601]: ts=2026-09-23T16:03:30Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=16874worker2 # [ 240.742144] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:429646875worker2 # [ 240.752683] nix-daemon[842]: reaped child process 2005, status = succeeded6876worker2 # [ 240.755934] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:429646877worker2 # [ 240.768894] nix-grpc-daemon[1933]: time=2026-09-23T16:03:30.392Z level=INFO msg="All 1 paths already cached"6878client # copying 1 paths to 'lb:50051'...6879client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'...6880worker2 # [ 240.778626] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:429646881worker2 # [ 240.793069] nix-daemon[842]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)6882worker2 # [ 240.936056] nix-daemon[842]: accepted connection from pid 2030, user nix-grpc-daemon (trusted)6883worker2 # [ 240.953462] nix-daemon[842]: reaped child process 2033, status = succeeded6884lb # [ 240.967321] niks3-server[1114]: 2026/09/23 16:03:30 INFO Received push request method=POST path=/api/pushes6885worker2 # [ 240.969384] nix-grpc-daemon[1933]: time=2026-09-23T16:03:30.592Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6886worker2 # [ 240.970993] nix-grpc-daemon[1933]: time=2026-09-23T16:03:30.592Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)"6887lb # [ 240.998381] niks3-server[1114]: 2026/09/23 16:03:30 INFO Received sign narinfos request method=POST path=/api/pushes/88/sign6888worker2 # [ 240.997386] nix-grpc-daemon[1933]: time=2026-09-23T16:03:30.620Z level=INFO msg="Uploading 1 narinfos"6889lb # [ 241.012118] niks3-server[1114]: 2026/09/23 16:03:30 INFO Signed narinfos id=88 count=16890lb # [ 241.014878] niks3-server[1114]: 2026/09/23 16:03:30 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls6891lb # [ 241.026735] niks3-server[1114]: 2026/09/23 16:03:30 INFO Received complete push request method=POST path=/api/pushes/88/complete6892lb # [ 241.031827] niks3-server[1114]: 2026/09/23 16:03:30 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst6893worker2 # [ 241.036616] nix-grpc-daemon[1933]: time=2026-09-23T16:03:30.660Z level=INFO msg="Upload complete. (232ms)"6894lb # [ 241.046778] niks3-server[1114]: 2026/09/23 16:03:30 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo6895worker2 # [ 241.046894] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42964 duration_s=0 paths=1 nar_bytes_in=6326896worker2 # [ 241.063452] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:429646897worker2 # [ 241.070032] nix-daemon[842]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)6898worker2 # [ 241.075702] nix-daemon[842]: reaped child process 2028, status = succeeded6899worker2 # [ 241.086713] nix-daemon[842]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)6900client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051...6901client # farm-b> worker2-next: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6902client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6903client # farm-b> LOG-farm-b-drain6904worker2 # [ 242.949619] nix-daemon[842]: reaped child process 2037, status = succeeded6905worker2 # [ 243.078158] nix-daemon[842]: accepted connection from pid 2053, user nix-grpc-daemon (trusted)6906worker2 # [ 243.096870] nix-daemon[842]: reaped child process 2056, status = succeeded6907lb # [ 243.113819] niks3-server[1114]: 2026/09/23 16:03:32 INFO Received push request method=POST path=/api/pushes6908worker2 # [ 243.118242] nix-grpc-daemon[1933]: time=2026-09-23T16:03:32.739Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6909worker2 # [ 243.120307] nix-grpc-daemon[1933]: time=2026-09-23T16:03:32.739Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)"6910lb # [ 243.159696] niks3-server[1114]: 2026/09/23 16:03:32 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls6911worker2 # [ 243.161001] nix-grpc-daemon[1933]: time=2026-09-23T16:03:32.784Z level=INFO msg="Uploading 1 narinfos"6912lb # [ 243.172645] niks3-server[1114]: 2026/09/23 16:03:32 INFO Received sign narinfos request method=POST path=/api/pushes/89/sign6913lb # [ 243.180380] niks3-server[1114]: 2026/09/23 16:03:32 INFO Signed narinfos id=89 count=16914lb # [ 243.182789] niks3-server[1114]: 2026/09/23 16:03:32 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6915lb # [ 243.197906] niks3-server[1114]: 2026/09/23 16:03:32 INFO Received complete push request method=POST path=/api/pushes/89/complete6916lb # [ 243.203583] niks3-server[1114]: 2026/09/23 16:03:32 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst6917lb # [ 243.207789] niks3-server[1114]: 2026/09/23 16:03:32 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo6918worker2 # [ 243.193678] nix-grpc-daemon[1933]: time=2026-09-23T16:03:32.817Z level=INFO msg="Upload complete. (293ms)"6919worker2 # [ 243.205436] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42964 duration_s=2 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=15 outputs=16920worker1 # [ 243.303270] nix-grpc-daemon[1601]: ts=2026-09-23T16:03:32Z level=debug event=want client=4 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=06921worker1 # [ 243.309658] nix-grpc-daemon[1601]: ts=2026-09-23T16:03:32Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=16922worker2 # [ 243.225593] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:429646923worker2 # [ 243.235739] nix-daemon[842]: reaped child process 2035, status = succeeded6924worker2 # [ 243.244625] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:429646925worker2 # [ 243.259823] nix-grpc-daemon[1933]: time=2026-09-23T16:03:32.883Z level=INFO msg="All 2 paths already cached"6926client # copying 1 paths to 'lb:50051'...6927client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'...6928worker2 # [ 243.271370] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:429646929worker2 # [ 243.285617] nix-daemon[842]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)6930worker2 # [ 243.433173] nix-daemon[842]: accepted connection from pid 2060, user nix-grpc-daemon (trusted)6931worker2 # [ 243.454547] nix-daemon[842]: reaped child process 2063, status = succeeded6932lb # [ 243.466048] niks3-server[1114]: 2026/09/23 16:03:33 INFO Received push request method=POST path=/api/pushes6933worker2 # [ 243.466210] nix-grpc-daemon[1933]: time=2026-09-23T16:03:33.089Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6934worker2 # [ 243.467699] nix-grpc-daemon[1933]: time=2026-09-23T16:03:33.089Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)"6935lb # [ 243.495531] niks3-server[1114]: 2026/09/23 16:03:33 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst6936lb # [ 243.509145] niks3-server[1114]: 2026/09/23 16:03:33 INFO Received sign narinfos request method=POST path=/api/pushes/90/sign6937worker2 # [ 243.506168] nix-grpc-daemon[1933]: time=2026-09-23T16:03:33.129Z level=INFO msg="Uploading 1 narinfos"6938lb # [ 243.521117] niks3-server[1114]: 2026/09/23 16:03:33 INFO Signed narinfos id=90 count=16939lb # [ 243.528769] niks3-server[1114]: 2026/09/23 16:03:33 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls6940lb # [ 243.542849] niks3-server[1114]: 2026/09/23 16:03:33 INFO Received complete push request method=POST path=/api/pushes/90/complete6941worker2 # [ 243.542717] nix-grpc-daemon[1933]: time=2026-09-23T16:03:33.166Z level=INFO msg="Upload complete. (242ms)"6942lb # [ 243.552723] niks3-server[1114]: 2026/09/23 16:03:33 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo6943worker2 # [ 243.553221] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:42964 duration_s=0 paths=1 nar_bytes_in=7606944worker2 # [ 243.570694] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:429646945worker2 # [ 243.577870] nix-daemon[842]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)6946worker2 # [ 243.587501] nix-daemon[842]: reaped child process 2058, status = succeeded6947client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051...6948client # farm-top> worker2-next: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6949worker2 # [ 243.600179] nix-daemon[842]: accepted connection from pid 1873, user nix-grpc-daemon (trusted)6950client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6951client # farm-top> LOG-farm-top-drain6952worker2 # [ 244.438763] nix-daemon[842]: reaped child process 2068, status = succeeded6953worker2 # [ 244.554472] nix-daemon[842]: accepted connection from pid 2084, user nix-grpc-daemon (trusted)6954worker2 # [ 244.575235] nix-daemon[842]: reaped child process 2087, status = succeeded6955lb # [ 244.591751] niks3-server[1114]: 2026/09/23 16:03:34 INFO Received push request method=POST path=/api/pushes6956worker2 # [ 244.594546] nix-grpc-daemon[1933]: time=2026-09-23T16:03:34.215Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6957worker2 # [ 244.596074] nix-grpc-daemon[1933]: time=2026-09-23T16:03:34.215Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)"6958lb # [ 244.637795] niks3-server[1114]: 2026/09/23 16:03:34 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls6959worker2 # [ 244.639516] nix-grpc-daemon[1933]: time=2026-09-23T16:03:34.263Z level=INFO msg="Uploading 1 narinfos"6960lb # [ 244.649378] niks3-server[1114]: 2026/09/23 16:03:34 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst6961lb # [ 244.658466] niks3-server[1114]: 2026/09/23 16:03:34 INFO Received sign narinfos request method=POST path=/api/pushes/91/sign6962lb # [ 244.670706] niks3-server[1114]: 2026/09/23 16:03:34 INFO Signed narinfos id=91 count=16963worker2 # [ 244.668952] nix-grpc-daemon[1933]: time=2026-09-23T16:03:34.292Z level=INFO msg="Upload complete. (257ms)"6964lb # [ 244.677933] niks3-server[1114]: 2026/09/23 16:03:34 INFO Received complete push request method=POST path=/api/pushes/91/complete6965lb # [ 244.681523] niks3-server[1114]: 2026/09/23 16:03:34 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6966lb # [ 244.686978] niks3-server[1114]: 2026/09/23 16:03:34 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo6967worker2 # [ 244.678964] nix-grpc-daemon[1873]: ts=2026-09-23T16:03:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:42964 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=16 outputs=16968worker2 # [ 244.697823] nix-daemon[842]: reaped child process 2066, status = succeeded6969client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2, in 6.17 seconds)6970client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath6971client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath, in 0.17 seconds)6972worker1: must succeed: rm /nix/.rw-store/fill6973worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.16 seconds)6974worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'6975worker1 # [ 245.718108] nix-grpc-daemon[1601]: ts=2026-09-23T16:03:35Z level=info event=healthy reason=min_free6976worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.10 seconds)6977(finished: subtest: low disk drains a worker and builds go to the other, in 9.92 seconds)6978(finished: run the VM test script, in 247.17 seconds)6979test script finished in 247.39s6980cleanup6981kill QemuMachine (pid 49)6982client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6983client # [2026-09-23T16:03:35Z INFO virtiofsd] Client disconnected, shutting down6984client # [2026-09-23T16:03:35Z INFO virtiofsd] Client disconnected, shutting down6985client # [2026-09-23T16:03:35Z INFO virtiofsd] Client disconnected, shutting down6986kill QemuMachine (pid 50)6987lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6988lb # [2026-09-23T16:03:35Z INFO virtiofsd] Client disconnected, shutting down6989lb # [2026-09-23T16:03:35Z INFO virtiofsd] Client disconnected, shutting down6990lb # [2026-09-23T16:03:35Z INFO virtiofsd] Client disconnected, shutting down6991kill QemuMachine (pid 51)6992worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6993worker1 # [2026-09-23T16:03:36Z INFO virtiofsd] Client disconnected, shutting down6994worker1 # [2026-09-23T16:03:36Z INFO virtiofsd] Client disconnected, shutting down6995worker1 # [2026-09-23T16:03:36Z INFO virtiofsd] Client disconnected, shutting down6996kill QemuMachine (pid 52)6997worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6998worker2 # [2026-09-23T16:03:36Z INFO virtiofsd] Client disconnected, shutting down6999worker2 # [2026-09-23T16:03:36Z INFO virtiofsd] Client disconnected, shutting down7000worker2 # [2026-09-23T16:03:36Z INFO virtiofsd] Client disconnected, shutting down7001(finished: cleanup, in 1.03 seconds)7002additionally exposed symbols:7003 client, lb, worker1, worker2,7004 vlan1,7005 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh