vm-test-run-nix-grpc-farm
checks.aarch64-linux.farm
· build #476
· raw
1tribuchet: building on eliza2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker1 # Disk image does not exist, creating the virtualisation disk image...13client # Disk image does not exist, creating the virtualisation disk image...14worker1 # Formatting '/build/vm-state-worker1/tmp.2AzWFgMvdZ', fmt=raw size=107374182415client: QEMU running (pid 50)16worker1: QEMU running (pid 51)17client # Formatting '/build/vm-state-client/tmp.PGr8mw5bAa', fmt=raw size=107374182418worker1 # mke2fs 1.47.4 (6-Mar-2025)19lb: QEMU running (pid 49)20lb # Disk image does not exist, creating the virtualisation disk image...21client # mke2fs 1.47.4 (6-Mar-2025)22lb # Formatting '/build/vm-state-lb/tmp.aokvPdJzLa', fmt=raw size=107374182423client # Discarding device blocks: 0/262144 done24lb # mke2fs 1.47.4 (6-Mar-2025)25client # Creating filesystem with 262144 4k blocks and 65536 inodes26lb # Discarding device blocks: 0/262144 done27lb # Creating filesystem with 262144 4k blocks and 65536 inodes28client # Filesystem UUID: 660d2273-7218-43a0-9a33-e96d4e27a8a829worker1 # Discarding device blocks: 0/262144 done30worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes31lb # Filesystem UUID: 3c3aa78d-c8ea-47b8-ab27-a5bfbb66822132worker1 # Filesystem UUID: 9d48dbc4-4996-454b-9856-73a0894ebe2333lb # Superblock backups stored on blocks:34worker1 # Superblock backups stored on blocks:35lb # 32768, 98304, 163840, 22937636worker1 # 32768, 98304, 163840, 22937637client # Superblock backups stored on blocks:38lb # 39lb # Allocating group tables: 0/8 done40client # 32768, 98304, 163840, 22937641worker2: QEMU running (pid 52)42worker1 # 43worker2 # Disk image does not exist, creating the virtualisation disk image...44lb # Writing inode tables: 0/8 done45client # 46(finished: start all VMs, in 0.48 seconds)47worker2 # Formatting '/build/vm-state-worker2/tmp.mP1jRioH5K', fmt=raw size=107374182448worker1 # Allocating group tables: 0/8 done49client: waiting for unit network-addresses-eth1.service50worker2 # mke2fs 1.47.4 (6-Mar-2025)51worker1 # Writing inode tables: 0/8 done52client: waiting for the VM to finish booting53worker2 # Discarding device blocks: 0/262144 done54worker1 # Creating journal (8192 blocks): done55client # Allocating group tables: 0/8 done56lb # Creating journal (8192 blocks): done57worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes58worker1 # Writing superblocks and filesystem accounting information: 0/8 done59client # Writing inode tables: 0/8 done60lb # Writing superblocks and filesystem accounting information: 0/8 done61worker2 # Filesystem UUID: ec466e96-c9d5-46e4-9e97-0b84620c9eb762worker1 # 63client # Creating journal (8192 blocks): done64lb # 65worker2 # Superblock backups stored on blocks:66worker1 # Virtualisation disk image created.67client # Writing superblocks and filesystem accounting information: 0/8 done68lb # Virtualisation disk image created.69worker2 # 32768, 98304, 163840, 22937670worker1 # Starting virtiofs daemons...71client # 72lb # Starting virtiofs daemons...73worker2 # 74worker1 # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)75client # Virtualisation disk image created.76lb # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)77worker2 # Allocating group tables: 0/8 done78worker1 # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether79worker1 # [2026-09-21T13:21:41Z INFO virtiofsd] Waiting for vhost-user socket connection...80lb # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether81client # Starting virtiofs daemons...82worker1 # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)83lb # [2026-09-21T13:21:41Z INFO virtiofsd] Waiting for vhost-user socket connection...84worker2 # Writing inode tables: 0/8 done85lb # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)86worker2 # Creating journal (8192 blocks): done87client # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)88worker1 # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether89lb # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether90worker2 # Writing superblocks and filesystem accounting information: 0/8 done91client # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether92worker1 # [2026-09-21T13:21:41Z INFO virtiofsd] Waiting for vhost-user socket connection...93lb # [2026-09-21T13:21:41Z INFO virtiofsd] Waiting for vhost-user socket connection...94worker2 # 95client # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)96worker1 # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)97lb # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)98worker2 # Virtualisation disk image created.99client # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether100worker1 # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether101lb # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether102worker2 # Starting virtiofs daemons...103client # [2026-09-21T13:21:41Z INFO virtiofsd] Waiting for vhost-user socket connection...104worker1 # [2026-09-21T13:21:41Z INFO virtiofsd] Waiting for vhost-user socket connection...105lb # [2026-09-21T13:21:41Z INFO virtiofsd] Waiting for vhost-user socket connection...106worker2 # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)107client # [2026-09-21T13:21:41Z INFO virtiofsd] Waiting for vhost-user socket connection...108worker1 # [2026-09-21T13:21:41Z INFO virtiofsd] Client connected, servicing requests109lb # [2026-09-21T13:21:41Z INFO virtiofsd] Client connected, servicing requests110worker2 # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether111client # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)112worker1 # [2026-09-21T13:21:41Z INFO virtiofsd] Client connected, servicing requests113lb # [2026-09-21T13:21:41Z INFO virtiofsd] Client connected, servicing requests114worker2 # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)115client # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether116worker1 # [2026-09-21T13:21:41Z INFO virtiofsd] Client connected, servicing requests117lb # [2026-09-21T13:21:41Z INFO virtiofsd] Client connected, servicing requests118worker2 # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether119client # [2026-09-21T13:21:41Z INFO virtiofsd] Waiting for vhost-user socket connection...120worker2 # [2026-09-21T13:21:41Z INFO virtiofsd] Waiting for vhost-user socket connection...121client # [2026-09-21T13:21:41Z INFO virtiofsd] Client connected, servicing requests122worker2 # [2026-09-21T13:21:41Z INFO virtiofsd] Waiting for vhost-user socket connection...123client # [2026-09-21T13:21:41Z INFO virtiofsd] Client connected, servicing requests124worker2 # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)125client # [2026-09-21T13:21:41Z INFO virtiofsd] Client connected, servicing requests126worker2 # [2026-09-21T13:21:41Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether127worker2 # [2026-09-21T13:21:41Z INFO virtiofsd] Waiting for vhost-user socket connection...128worker2 # [2026-09-21T13:21:41Z INFO virtiofsd] Client connected, servicing requests129worker2 # [2026-09-21T13:21:41Z INFO virtiofsd] Client connected, servicing requests130worker2 # [2026-09-21T13:21:41Z INFO virtiofsd] Client connected, servicing requests131worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]132worker1 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026133worker1 # [ 0.000000] KASLR enabled134worker1 # [ 0.000000] random: crng init done135worker1 # [ 0.000000] Machine model: linux,dummy-virt136worker1 # [ 0.000000] efi: UEFI not found.137worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT138worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]139worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]140worker1 # [ 0.000000] Zone ranges:141worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]142worker1 # [ 0.000000] DMA32 empty143worker1 # [ 0.000000] Normal empty144worker1 # [ 0.000000] Device empty145worker1 # [ 0.000000] Movable zone start for each node146worker1 # [ 0.000000] Early memory node ranges147worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]148worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]149worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000150worker1 # [ 0.000000] psci: probing for conduit method from DT.151worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware.152worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs153worker1 # [ 0.000000] psci: Trusted OS migration not required154worker1 # [ 0.000000] psci: SMC Calling Convention v1.1155worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)156worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296157worker1 # [ 0.000000] Detected PIPT I-cache on CPU0158worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)159worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface160worker1 # [ 0.000000] CPU features: detected: Spectre-v4161worker1 # [ 0.000000] CPU features: detected: Spectre-BHB162worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38163worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23164worker1 # [ 0.000000] alternatives: applying boot alternatives165worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/r5b98ymy3fa8ic2qg3yky4fhnwpld5a2-nixos-system-worker1-test/init regInfo=/nix/store/blmvi9vsa7y5hmfj1zny5y7q1wcgrnb7-closure-info/registration console=ttyAMA0,115200n8 console=tty0166worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/blmvi9vsa7y5hmfj1zny5y7q1wcgrnb7-closure-info/registration", will be passed to user space.167worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes168worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)169worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)170worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB171worker1 # [ 0.000000] software IO TLB: area num 1.172worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB173worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)174worker1 # [ 0.000000] Fallback order for Node 0: 0175worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216176worker1 # [ 0.000000] Policy zone: DMA177worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off178worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1179worker1 # [ 0.000000] allocated 3145728 bytes of page_ext180worker1 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages181worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups182worker1 # [ 0.000000] rcu: Hierarchical RCU implementation.183worker1 # [ 0.000000] rcu: RCU event tracing is enabled.184worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.185worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled.186worker1 # [ 0.000000] Rude variant of Tasks RCU enabled.187worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled.188worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.189worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1190worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.191worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.192worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.193worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0194worker1 # [ 0.000000] GICv3: 256 SPIs implemented195worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented196worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq197worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI198worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0199worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000200worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]201worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)202worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)203worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000204worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000205worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.206worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns207worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).208worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns209worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns210worker1 # [ 0.000033] arm-pv: using stolen time PV211worker1 # [ 0.000664] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)212worker1 # [ 0.000813] Console: colour dummy device 80x25213worker1 # [ 0.000820] printk: legacy console [tty0] enabled214worker1 # [ 0.001010] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)215worker1 # [ 0.001016] pid_max: default: 32768 minimum: 301216worker1 # [ 0.001099] LSM: initializing lsm=capability,landlock,yama,bpf,ima217worker1 # [ 0.001258] landlock: Up and running.218worker1 # [ 0.001260] Yama: becoming mindful.219worker1 # [ 0.001909] LSM support for eBPF active220lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]221worker1 # [ 0.002063] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)222lb # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026223worker1 # [ 0.002104] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)224client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]225lb # [ 0.000000] KASLR enabled226worker1 # [ 0.004031] rcu: Hierarchical SRCU implementation.227lb # [ 0.000000] random: crng init done228lb # [ 0.000000] Machine model: linux,dummy-virt229worker1 # [ 0.004036] rcu: Max phase no-delay instances is 1000.230client # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026231lb # [ 0.000000] efi: UEFI not found.232worker1 # [ 0.005308] fsl-mc MSI: its@8080000 domain created233client # [ 0.000000] KASLR enabled234client # [ 0.000000] random: crng init done235worker1 # [ 0.005398] EFI services will not be available.236lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT237client # [ 0.000000] Machine model: linux,dummy-virt238worker1 # [ 0.005484] smp: Bringing up secondary CPUs ...239client # [ 0.000000] efi: UEFI not found.240lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]241worker1 # [ 0.005492] smp: Brought up 1 node, 1 CPU242worker1 # [ 0.005495] SMP: Total of 1 processors activated.243lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]244client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT245lb # [ 0.000000] Zone ranges:246worker1 # [ 0.005501] CPU: All CPU(s) started at EL1247client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]248lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]249worker1 # [ 0.005513] CPU features: detected: Branch Target Identification250lb # [ 0.000000] DMA32 empty251client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]252lb # [ 0.000000] Normal empty253worker1 # [ 0.005518] CPU features: detected: ARMv8.4 Translation Table Level254client # [ 0.000000] Zone ranges:255lb # [ 0.000000] Device empty256lb # [ 0.000000] Movable zone start for each node257client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]258worker1 # [ 0.005521] CPU features: detected: Instruction cache invalidation not required for I/D coherence259lb # [ 0.000000] Early memory node ranges260client # [ 0.000000] DMA32 empty261client # [ 0.000000] Normal empty262lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]263client # [ 0.000000] Device empty264worker1 # [ 0.005524] CPU features: detected: Data cache clean to the PoU not required for I/D coherence265client # [ 0.000000] Movable zone start for each node266lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]267client # [ 0.000000] Early memory node ranges268worker1 # [ 0.005528] CPU features: detected: Common not Private translations269lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000270worker1 # [ 0.005532] CPU features: detected: CRC32 instructions271client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]272lb # [ 0.000000] psci: probing for conduit method from DT.273worker1 # [ 0.005534] CPU features: detected: Data cache clean to Point of Deep Persistence274client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]275lb # [ 0.000000] psci: PSCIv1.3 detected in firmware.276client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000277lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs278worker1 # [ 0.005538] CPU features: detected: Data cache clean to Point of Persistence279lb # [ 0.000000] psci: Trusted OS migration not required280client # [ 0.000000] psci: probing for conduit method from DT.281worker1 # [ 0.005541] CPU features: detected: Data independent timing control (DIT)282lb # [ 0.000000] psci: SMC Calling Convention v1.1283client # [ 0.000000] psci: PSCIv1.3 detected in firmware.284worker1 # [ 0.005544] CPU features: detected: E0PD285client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs286lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)287worker1 # [ 0.005546] CPU features: detected: Enhanced Counter Virtualization288client # [ 0.000000] psci: Trusted OS migration not required289client # [ 0.000000] psci: SMC Calling Convention v1.1290lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296291worker1 # [ 0.005550] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)292lb # [ 0.000000] Detected PIPT I-cache on CPU0293worker1 # [ 0.005553] CPU features: detected: Enhanced Virtualization Traps294client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)295lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)296worker1 # [ 0.005556] CPU features: detected: Fine Grained Traps297client # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296298lb # [ 0.000000] CPU features: detected: GICv3 CPU interface299worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]300client # [ 0.000000] Detected PIPT I-cache on CPU0301worker1 # [ 0.005560] CPU features: detected: Generic authentication (architected QARMA5 algorithm)302lb # [ 0.000000] CPU features: detected: Spectre-v4303worker1 # [ 0.005566] CPU features: detected: RCpc load-acquire (LDAPR)304lb # [ 0.000000] CPU features: detected: Spectre-BHB305client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)306worker2 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026307worker1 # [ 0.005568] CPU features: detected: LSE atomic instructions308client # [ 0.000000] CPU features: detected: GICv3 CPU interface309lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38310worker2 # [ 0.000000] KASLR enabled311worker2 # [ 0.000000] random: crng init done312client # [ 0.000000] CPU features: detected: Spectre-v4313worker1 # [ 0.005571] CPU features: detected: Privileged Access Never314lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23315worker2 # [ 0.000000] Machine model: linux,dummy-virt316worker1 # [ 0.005574] CPU features: detected: PMUv3317client # [ 0.000000] CPU features: detected: Spectre-BHB318worker2 # [ 0.000000] efi: UEFI not found.319lb # [ 0.000000] alternatives: applying boot alternatives320worker1 # [ 0.005577] CPU features: detected: RAS Extension Support321client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38322worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT323worker1 # [ 0.005580] CPU features: detected: RASv1p1 Extension Support324client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23325worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]326worker1 # [ 0.005582] CPU features: detected: Random Number Generator327client # [ 0.000000] alternatives: applying boot alternatives328worker1 # [ 0.005585] CPU features: detected: Speculation barrier (SB)329worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]330worker2 # [ 0.000000] Zone ranges:331worker1 # [ 0.005588] CPU features: detected: Stage-2 Force Write-Back332worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]333lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/vc02c2vazd61rmklvp1rjw8zynvx4pcy-nixos-system-lb-test/init regInfo=/nix/store/h3l2jk1vn5bwms7512h3yf5fbh3fmard-closure-info/registration console=ttyAMA0,115200n8 console=tty0334worker2 # [ 0.000000] DMA32 empty335worker1 # [ 0.005591] CPU features: detected: TLB range maintenance instructions336worker2 # [ 0.000000] Normal empty337worker2 # [ 0.000000] Device empty338worker1 # [ 0.005596] CPU features: detected: Speculative Store Bypassing Safe (SSBS)339worker2 # [ 0.000000] Movable zone start for each node340worker1 # [ 0.005636] alternatives: applying system-wide alternatives341worker2 # [ 0.000000] Early memory node ranges342client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/s3qhwgxsdcwiy6iwmdamf1bqgk3qgrb9-nixos-system-client-test/init regInfo=/nix/store/z7i9ys3n1axh8vj4xancyfigl20vpi6q-closure-info/registration console=ttyAMA0,115200n8 console=tty0343lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/h3l2jk1vn5bwms7512h3yf5fbh3fmard-closure-info/registration", will be passed to user space.344worker1 # [ 0.008672] CPU features: detected: BBM Level 2 without TLB conflict abort345worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]346lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes347worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]348worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000349client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/z7i9ys3n1axh8vj4xancyfigl20vpi6q-closure-info/registration", will be passed to user space.350lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)351worker1 # [ 0.008893] Memory: 1405664K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124664K reserved, 32768K cma-reserved)352worker2 # [ 0.000000] psci: probing for conduit method from DT.353worker1 # [ 0.009342] devtmpfs: initialized354client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes355lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)356worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware.357worker1 # [ 0.011185] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)358worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs359lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB360client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)361lb # [ 0.000000] software IO TLB: area num 1.362worker2 # [ 0.000000] psci: Trusted OS migration not required363worker1 # [ 0.011209] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).364worker2 # [ 0.000000] psci: SMC Calling Convention v1.1365client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)366lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB367worker1 # [ 0.011385] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL368client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB369worker1 # [ 0.011390] 0 pages in range for non-PLT usage370worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)371lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)372client # [ 0.000000] software IO TLB: area num 1.373worker1 # [ 0.011391] 508272 pages in range for PLT usage374lb # [ 0.000000] Fallback order for Node 0: 0375worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296376worker1 # [ 0.011512] pinctrl core: initialized pinctrl subsystem377client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB378worker2 # [ 0.000000] Detected PIPT I-cache on CPU0379lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216380worker1 # [ 0.012287] DMI not present or invalid.381lb # [ 0.000000] Policy zone: DMA382client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)383worker1 # [ 0.016069] NET: Registered PF_NETLINK/PF_ROUTE protocol family384worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)385client # [ 0.000000] Fallback order for Node 0: 0386lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off387worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface388worker1 # [ 0.018611] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations389client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216390lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1391worker2 # [ 0.000000] CPU features: detected: Spectre-v4392client # [ 0.000000] Policy zone: DMA393lb # [ 0.000000] allocated 3145728 bytes of page_ext394worker2 # [ 0.000000] CPU features: detected: Spectre-BHB395worker1 # [ 0.018809] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations396client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off397lb # [ 0.000000] ftrace: allocating 74950 entries in 294 pages398worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38399worker1 # [ 0.019008] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations400client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1401lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups402worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23403client # [ 0.000000] allocated 3145728 bytes of page_ext404worker1 # [ 0.019031] audit: initializing netlink subsys (disabled)405lb # [ 0.000000] rcu: Hierarchical RCU implementation.406worker2 # [ 0.000000] alternatives: applying boot alternatives407lb # [ 0.000000] rcu: RCU event tracing is enabled.408worker1 # [ 0.019633] thermal_sys: Registered thermal governor 'fair_share'409client # [ 0.000000] ftrace: allocating 74950 entries in 294 pages410client # [ 0.000000] ftrace: allocated 294 pages with 4 groups411lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.412worker1 # [ 0.019635] thermal_sys: Registered thermal governor 'bang_bang'413client # [ 0.000000] rcu: Hierarchical RCU implementation.414lb # [ 0.000000] Trampoline variant of Tasks RCU enabled.415worker1 # [ 0.019639] thermal_sys: Registered thermal governor 'step_wise'416client # [ 0.000000] rcu: RCU event tracing is enabled.417lb # [ 0.000000] Rude variant of Tasks RCU enabled.418worker1 # [ 0.019642] thermal_sys: Registered thermal governor 'user_space'419lb # [ 0.000000] Tracing variant of Tasks RCU enabled.420client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.421worker1 # [ 0.019647] thermal_sys: Registered thermal governor 'power_allocator'422client # [ 0.000000] Trampoline variant of Tasks RCU enabled.423worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/zd415i8p1sflcjhd4jcz8axkgzv8xbyr-nixos-system-worker2-test/init regInfo=/nix/store/z8cam0grqnkhf6g44z7bhh8ad3jx1sq9-closure-info/registration console=ttyAMA0,115200n8 console=tty0424lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.425client # [ 0.000000] Rude variant of Tasks RCU enabled.426worker1 # [ 0.019681] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1427lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1428client # [ 0.000000] Tracing variant of Tasks RCU enabled.429worker1 # [ 0.019690] cpuidle: using governor ladder430worker1 # [ 0.019696] cpuidle: using governor menu431client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.432worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/z8cam0grqnkhf6g44z7bhh8ad3jx1sq9-closure-info/registration", will be passed to user space.433lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.434worker1 # [ 0.019888] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.435client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1436lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.437worker1 # [ 0.019904] ASID allocator initialised with 65536 entries438worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes439worker1 # [ 0.021170] Serial: AMBA PL011 UART driver440client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.441lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.442worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)443worker1 # [ 0.026601] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1444client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.445lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0446worker1 # [ 0.026731] printk: console [ttyAMA0] enabled447worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)448lb # [ 0.000000] GICv3: 256 SPIs implemented449client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.450lb # [ 0.000000] GICv3: 0 Extended SPIs implemented451worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB452lb # [ 0.000000] Root IRQ handler: gic_handle_irq453client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0454worker2 # [ 0.000000] software IO TLB: area num 1.455client # [ 0.000000] GICv3: 256 SPIs implemented456lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI457worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB458client # [ 0.000000] GICv3: 0 Extended SPIs implemented459lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0460client # [ 0.000000] Root IRQ handler: gic_handle_irq461worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)462lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000463client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI464worker2 # [ 0.000000] Fallback order for Node 0: 0465lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]466client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0467worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216468worker2 # [ 0.000000] Policy zone: DMA469client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000470lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1)471client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]472worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off473lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1)474worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1475client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)476lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a00000477worker2 # [ 0.000000] allocated 3145728 bytes of page_ext478worker2 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages479client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)480lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a10000481worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups482client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000483lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.484worker2 # [ 0.000000] rcu: Hierarchical RCU implementation.485client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000486worker2 # [ 0.000000] rcu: RCU event tracing is enabled.487lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns488worker1 # [ 0.149488] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages489client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.490worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.491lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).492worker1 # [ 0.149505] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page493worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled.494worker2 # [ 0.000000] Rude variant of Tasks RCU enabled.495worker1 # [ 0.149510] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages496client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns497lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns498worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled.499client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).500worker1 # [ 0.149514] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page501lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns502worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.503worker1 # [ 0.149519] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages504lb # [ 0.000030] arm-pv: using stolen time PV505client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns506worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1507worker1 # [ 0.149523] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page508client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns509lb # [ 0.000458] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)510worker1 # [ 0.149527] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages511worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.512client # [ 0.000031] arm-pv: using stolen time PV513lb # [ 0.000605] Console: colour dummy device 80x25514worker1 # [ 0.149531] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page515lb # [ 0.000613] printk: legacy console [tty0] enabled516worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.517client # [ 0.000454] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)518client # [ 0.000630] Console: colour dummy device 80x25519worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.520lb # [ 0.000832] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)521client # [ 0.000637] printk: legacy console [tty0] enabled522lb # [ 0.000838] pid_max: default: 32768 minimum: 301523worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0524worker1 # [ 0.156814] fbcon: Taking over console525worker2 # [ 0.000000] GICv3: 256 SPIs implemented526lb # [ 0.000912] LSM: initializing lsm=capability,landlock,yama,bpf,ima527worker1 # [ 0.156828] ACPI: Interpreter disabled.528client # [ 0.000824] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)529lb # [ 0.001050] landlock: Up and running.530worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented531client # [ 0.000831] pid_max: default: 32768 minimum: 301532worker1 # [ 0.158678] iommu: Default domain type: Translated533lb # [ 0.001053] Yama: becoming mindful.534worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq535lb # [ 0.001537] LSM support for eBPF active536client # [ 0.000903] LSM: initializing lsm=capability,landlock,yama,bpf,ima537worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI538worker1 # [ 0.158688] iommu: DMA domain TLB invalidation policy: strict mode539client # [ 0.001089] landlock: Up and running.540worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0541lb # [ 0.001670] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)542client # [ 0.001092] Yama: becoming mindful.543client # [ 0.001563] LSM support for eBPF active544worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000545lb # [ 0.001726] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)546worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]547client # [ 0.001708] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)548lb # [ 0.003546] rcu: Hierarchical SRCU implementation.549lb # [ 0.003551] rcu: Max phase no-delay instances is 1000.550worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1)551client # [ 0.001740] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)552lb # [ 0.004789] fsl-mc MSI: its@8080000 domain created553client # [ 0.003588] rcu: Hierarchical SRCU implementation.554lb # [ 0.004886] EFI services will not be available.555worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1)556client # [ 0.003593] rcu: Max phase no-delay instances is 1000.557lb # [ 0.004955] smp: Bringing up secondary CPUs ...558client # [ 0.004894] fsl-mc MSI: its@8080000 domain created559worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a00000560lb # [ 0.004963] smp: Brought up 1 node, 1 CPU561client # [ 0.004984] EFI services will not be available.562lb # [ 0.004966] SMP: Total of 1 processors activated.563worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a10000564client # [ 0.005062] smp: Bringing up secondary CPUs ...565lb # [ 0.004971] CPU: All CPU(s) started at EL1566client # [ 0.005071] smp: Brought up 1 node, 1 CPU567worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.568worker1 # [ 0.165926] SCSI subsystem initialized569lb # [ 0.004982] CPU features: detected: Branch Target Identification570client # [ 0.005074] SMP: Total of 1 processors activated.571client # [ 0.005079] CPU: All CPU(s) started at EL1572lb # [ 0.004987] CPU features: detected: ARMv8.4 Translation Table Level573worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns574worker1 # [ 0.166575] usbcore: registered new interface driver usbfs575client # [ 0.005090] CPU features: detected: Branch Target Identification576worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).577lb # [ 0.004990] CPU features: detected: Instruction cache invalidation not required for I/D coherence578worker1 # [ 0.166606] usbcore: registered new interface driver hub579client # [ 0.005095] CPU features: detected: ARMv8.4 Translation Table Level580worker1 # [ 0.166622] usbcore: registered new device driver usb581lb # [ 0.004993] CPU features: detected: Data cache clean to the PoU not required for I/D coherence582worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns583worker1 # [ 0.166890] pps_core: LinuxPPS API ver. 1 registered584client # [ 0.005098] CPU features: detected: Instruction cache invalidation not required for I/D coherence585lb # [ 0.004997] CPU features: detected: Common not Private translations586worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns587worker1 # [ 0.166896] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>588lb # [ 0.005000] CPU features: detected: CRC32 instructions589worker2 # [ 0.000031] arm-pv: using stolen time PV590client # [ 0.005101] CPU features: detected: Data cache clean to the PoU not required for I/D coherence591worker1 # [ 0.166907] PTP clock support registered592worker1 # [ 0.166961] EDAC MC: Ver: 3.0.0593client # [ 0.005105] CPU features: detected: Common not Private translations594lb # [ 0.005003] CPU features: detected: Data cache clean to Point of Deep Persistence595worker2 # [ 0.000384] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)596client # [ 0.005108] CPU features: detected: CRC32 instructions597lb # [ 0.005006] CPU features: detected: Data cache clean to Point of Persistence598worker1 # [ 0.171529] scmi_core: SCMI protocol bus registered599worker2 # [ 0.000566] Console: colour dummy device 80x25600client # [ 0.005111] CPU features: detected: Data cache clean to Point of Deep Persistence601worker2 # [ 0.000573] printk: legacy console [tty0] enabled602lb # [ 0.005010] CPU features: detected: Data independent timing control (DIT)603worker1 # [ 0.172467] FPGA manager framework604lb # [ 0.005013] CPU features: detected: E0PD605client # [ 0.005115] CPU features: detected: Data cache clean to Point of Persistence606worker1 # [ 0.173433] vgaarb: loaded607worker2 # [ 0.000763] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)608lb # [ 0.005015] CPU features: detected: Enhanced Counter Virtualization609client # [ 0.005118] CPU features: detected: Data independent timing control (DIT)610worker2 # [ 0.000770] pid_max: default: 32768 minimum: 301611worker1 # [ 0.174047] clocksource: Switched to clocksource arch_sys_counter612lb # [ 0.005018] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)613worker2 # [ 0.000850] LSM: initializing lsm=capability,landlock,yama,bpf,ima614worker2 # [ 0.000990] landlock: Up and running.615lb # [ 0.005022] CPU features: detected: Enhanced Virtualization Traps616worker2 # [ 0.000993] Yama: becoming mindful.617lb # [ 0.005025] CPU features: detected: Fine Grained Traps618worker2 # [ 0.001404] LSM support for eBPF active619lb # [ 0.005029] CPU features: detected: Generic authentication (architected QARMA5 algorithm)620worker2 # [ 0.001553] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)621lb # [ 0.005033] CPU features: detected: RCpc load-acquire (LDAPR)622worker1 # [ 0.177827] VFS: Disk quotas dquot_6.6.0623worker2 # [ 0.001591] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)624lb # [ 0.005036] CPU features: detected: LSE atomic instructions625worker2 # [ 0.003408] rcu: Hierarchical SRCU implementation.626worker1 # [ 0.177864] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)627lb # [ 0.005039] CPU features: detected: Privileged Access Never628worker2 # [ 0.003414] rcu: Max phase no-delay instances is 1000.629lb # [ 0.005042] CPU features: detected: PMUv3630worker2 # [ 0.004667] fsl-mc MSI: its@8080000 domain created631lb # [ 0.005045] CPU features: detected: RAS Extension Support632worker2 # [ 0.004757] EFI services will not be available.633worker2 # [ 0.004824] smp: Bringing up secondary CPUs ...634lb # [ 0.005047] CPU features: detected: RASv1p1 Extension Support635worker2 # [ 0.004832] smp: Brought up 1 node, 1 CPU636worker1 # [ 0.181587] netfs: FS-Cache loaded637lb # [ 0.005050] CPU features: detected: Random Number Generator638worker2 # [ 0.004835] SMP: Total of 1 processors activated.639worker1 # [ 0.181708] pnp: PnP ACPI: disabled640lb # [ 0.005053] CPU features: detected: Speculation barrier (SB)641worker2 # [ 0.004840] CPU: All CPU(s) started at EL1642client # [ 0.005121] CPU features: detected: E0PD643lb # [ 0.005055] CPU features: detected: Stage-2 Force Write-Back644worker2 # [ 0.004850] CPU features: detected: Branch Target Identification645client # [ 0.005123] CPU features: detected: Enhanced Counter Virtualization646lb # [ 0.005058] CPU features: detected: TLB range maintenance instructions647worker2 # [ 0.004855] CPU features: detected: ARMv8.4 Translation Table Level648client # [ 0.005127] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)649lb # [ 0.005063] CPU features: detected: Speculative Store Bypassing Safe (SSBS)650client # [ 0.005130] CPU features: detected: Enhanced Virtualization Traps651worker2 # [ 0.004858] CPU features: detected: Instruction cache invalidation not required for I/D coherence652lb # [ 0.005101] alternatives: applying system-wide alternatives653client # [ 0.005133] CPU features: detected: Fine Grained Traps654worker2 # [ 0.004862] CPU features: detected: Data cache clean to the PoU not required for I/D coherence655worker1 # [ 0.185604] NET: Registered PF_INET protocol family656lb # [ 0.008023] CPU features: detected: BBM Level 2 without TLB conflict abort657client # [ 0.005136] CPU features: detected: Generic authentication (architected QARMA5 algorithm)658worker2 # [ 0.004866] CPU features: detected: Common not Private translations659worker1 # [ 0.185873] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)660client # [ 0.005141] CPU features: detected: RCpc load-acquire (LDAPR)661worker2 # [ 0.004869] CPU features: detected: CRC32 instructions662client # [ 0.005144] CPU features: detected: LSE atomic instructions663lb # [ 0.008303] Memory: 1405740K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124636K reserved, 32768K cma-reserved)664worker2 # [ 0.004872] CPU features: detected: Data cache clean to Point of Deep Persistence665lb # [ 0.008677] devtmpfs: initialized666client # [ 0.005147] CPU features: detected: Privileged Access Never667client # [ 0.005149] CPU features: detected: PMUv3668worker2 # [ 0.004876] CPU features: detected: Data cache clean to Point of Persistence669lb # [ 0.010599] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)670client # [ 0.005152] CPU features: detected: RAS Extension Support671worker2 # [ 0.004879] CPU features: detected: Data independent timing control (DIT)672lb # [ 0.010620] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).673client # [ 0.005155] CPU features: detected: RASv1p1 Extension Support674worker2 # [ 0.004882] CPU features: detected: E0PD675lb # [ 0.010800] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL676client # [ 0.005157] CPU features: detected: Random Number Generator677worker2 # [ 0.004885] CPU features: detected: Enhanced Counter Virtualization678client # [ 0.005160] CPU features: detected: Speculation barrier (SB)679worker2 # [ 0.004888] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)680client # [ 0.005163] CPU features: detected: Stage-2 Force Write-Back681worker2 # [ 0.004891] CPU features: detected: Enhanced Virtualization Traps682client # [ 0.005165] CPU features: detected: TLB range maintenance instructions683worker2 # [ 0.004894] CPU features: detected: Fine Grained Traps684client # [ 0.005170] CPU features: detected: Speculative Store Bypassing Safe (SSBS)685worker2 # [ 0.004898] CPU features: detected: Generic authentication (architected QARMA5 algorithm)686client # [ 0.005209] alternatives: applying system-wide alternatives687worker2 # [ 0.004903] CPU features: detected: RCpc load-acquire (LDAPR)688client # [ 0.008140] CPU features: detected: BBM Level 2 without TLB conflict abort689worker2 # [ 0.004906] CPU features: detected: LSE atomic instructions690worker2 # [ 0.004908] CPU features: detected: Privileged Access Never691worker2 # [ 0.004911] CPU features: detected: PMUv3692client # [ 0.008307] Memory: 1405616K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124660K reserved, 32768K cma-reserved)693worker2 # [ 0.004913] CPU features: detected: RAS Extension Support694client # [ 0.008677] devtmpfs: initialized695worker2 # [ 0.004916] CPU features: detected: RASv1p1 Extension Support696client # [ 0.010440] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)697worker2 # [ 0.004919] CPU features: detected: Random Number Generator698lb # [ 0.010805] 0 pages in range for non-PLT usage699client # [ 0.010461] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).700worker2 # [ 0.004921] CPU features: detected: Speculation barrier (SB)701lb # [ 0.010806] 508272 pages in range for PLT usage702client # [ 0.010635] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL703worker2 # [ 0.004924] CPU features: detected: Stage-2 Force Write-Back704lb # [ 0.010923] pinctrl core: initialized pinctrl subsystem705client # [ 0.010639] 0 pages in range for non-PLT usage706lb # [ 0.011749] DMI not present or invalid.707worker2 # [ 0.004927] CPU features: detected: TLB range maintenance instructions708client # [ 0.010640] 508272 pages in range for PLT usage709lb # [ 0.014774] NET: Registered PF_NETLINK/PF_ROUTE protocol family710client # [ 0.010757] pinctrl core: initialized pinctrl subsystem711worker2 # [ 0.004931] CPU features: detected: Speculative Store Bypassing Safe (SSBS)712client # [ 0.011489] DMI not present or invalid.713lb # [ 0.017185] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations714worker2 # [ 0.004968] alternatives: applying system-wide alternatives715client # [ 0.014577] NET: Registered PF_NETLINK/PF_ROUTE protocol family716lb # [ 0.017378] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations717worker2 # [ 0.007935] CPU features: detected: BBM Level 2 without TLB conflict abort718client # [ 0.017027] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations719lb # [ 0.017573] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations720client # [ 0.017227] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations721lb # [ 0.017597] audit: initializing netlink subsys (disabled)722worker2 # [ 0.008167] Memory: 1405680K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124632K reserved, 32768K cma-reserved)723worker2 # [ 0.008542] devtmpfs: initialized724lb # [ 0.018148] thermal_sys: Registered thermal governor 'fair_share'725client # [ 0.017420] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations726worker2 # [ 0.010369] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)727lb # [ 0.018150] thermal_sys: Registered thermal governor 'bang_bang'728client # [ 0.017442] audit: initializing netlink subsys (disabled)729client # [ 0.018029] thermal_sys: Registered thermal governor 'fair_share'730lb # [ 0.018154] thermal_sys: Registered thermal governor 'step_wise'731worker2 # [ 0.010391] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).732client # [ 0.018031] thermal_sys: Registered thermal governor 'bang_bang'733lb # [ 0.018156] thermal_sys: Registered thermal governor 'user_space'734worker2 # [ 0.010562] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL735client # [ 0.018034] thermal_sys: Registered thermal governor 'step_wise'736lb # [ 0.018161] thermal_sys: Registered thermal governor 'power_allocator'737client # [ 0.018037] thermal_sys: Registered thermal governor 'user_space'738lb # [ 0.018200] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1739lb # [ 0.018208] cpuidle: using governor ladder740client # [ 0.018042] thermal_sys: Registered thermal governor 'power_allocator'741lb # [ 0.018214] cpuidle: using governor menu742client # [ 0.018078] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1743lb # [ 0.018409] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.744client # [ 0.018086] cpuidle: using governor ladder745client # [ 0.018092] cpuidle: using governor menu746lb # [ 0.018424] ASID allocator initialised with 65536 entries747lb # [ 0.019581] Serial: AMBA PL011 UART driver748client # [ 0.018278] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.749client # [ 0.018292] ASID allocator initialised with 65536 entries750lb # [ 0.024883] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1751client # [ 0.019492] Serial: AMBA PL011 UART driver752lb # [ 0.025005] printk: console [ttyAMA0] enabled753client # [ 0.024665] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1754client # [ 0.024817] printk: console [ttyAMA0] enabled755lb # [ 0.145544] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages756lb # [ 0.145561] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page757worker2 # [ 0.010566] 0 pages in range for non-PLT usage758client # [ 0.145128] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages759lb # [ 0.145566] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages760worker2 # [ 0.010567] 508272 pages in range for PLT usage761client # [ 0.145146] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page762worker2 # [ 0.010676] pinctrl core: initialized pinctrl subsystem763lb # [ 0.145570] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page764worker2 # [ 0.011435] DMI not present or invalid.765client # [ 0.145151] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages766lb # [ 0.145575] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages767worker2 # [ 0.014413] NET: Registered PF_NETLINK/PF_ROUTE protocol family768client # [ 0.145156] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page769lb # [ 0.145579] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page770worker2 # [ 0.016864] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations771client # [ 0.145160] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages772lb # [ 0.145583] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages773worker2 # [ 0.017057] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations774client # [ 0.145164] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page775lb # [ 0.145587] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page776worker2 # [ 0.017254] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations777client # [ 0.145169] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages778worker2 # [ 0.017276] audit: initializing netlink subsys (disabled)779client # [ 0.145173] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page780worker1 # [ 0.215231] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)781worker2 # [ 0.017855] thermal_sys: Registered thermal governor 'fair_share'782lb # [ 0.152947] fbcon: Taking over console783worker1 # [ 0.215273] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)784worker2 # [ 0.017857] thermal_sys: Registered thermal governor 'bang_bang'785lb # [ 0.152961] ACPI: Interpreter disabled.786worker2 # [ 0.017861] thermal_sys: Registered thermal governor 'step_wise'787lb # [ 0.154763] iommu: Default domain type: Translated788worker1 # [ 0.215303] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)789client # [ 0.152607] fbcon: Taking over console790lb # [ 0.154773] iommu: DMA domain TLB invalidation policy: strict mode791worker2 # [ 0.017863] thermal_sys: Registered thermal governor 'user_space'792client # [ 0.152621] ACPI: Interpreter disabled.793worker1 # [ 0.215373] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)794worker2 # [ 0.017868] thermal_sys: Registered thermal governor 'power_allocator'795lb # [ 0.156412] SCSI subsystem initialized796worker1 # [ 0.215515] TCP: Hash tables configured (established 16384 bind 16384)797worker2 # [ 0.017894] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1798worker1 # [ 0.215592] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)799worker2 # [ 0.017902] cpuidle: using governor ladder800worker2 # [ 0.017908] cpuidle: using governor menu801worker1 # [ 0.215625] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)802worker2 # [ 0.018094] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.803worker1 # [ 0.215660] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)804worker2 # [ 0.018108] ASID allocator initialised with 65536 entries805worker1 # [ 0.215765] NET: Registered PF_UNIX/PF_LOCAL protocol family806worker2 # [ 0.019265] Serial: AMBA PL011 UART driver807worker1 # [ 0.215815] NET: Registered PF_XDP protocol family808client # [ 0.158890] iommu: Default domain type: Translated809worker2 # [ 0.024435] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1810worker1 # [ 0.215832] PCI: CLS 0 bytes, default 64811client # [ 0.158899] iommu: DMA domain TLB invalidation policy: strict mode812worker2 # [ 0.024580] printk: console [ttyAMA0] enabled813worker1 # [ 0.216070] Trying to unpack rootfs image as initramfs...814client # [ 0.160572] SCSI subsystem initialized815worker2 # [ 0.144782] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages816worker2 # [ 0.144801] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page817client # [ 0.161262] usbcore: registered new interface driver usbfs818client # [ 0.161294] usbcore: registered new interface driver hub819worker2 # [ 0.144806] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages820lb # [ 0.164031] usbcore: registered new interface driver usbfs821worker1 # [ 0.231774] kvm [1]: HYP mode not available822client # [ 0.161310] usbcore: registered new device driver usb823worker2 # [ 0.144811] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page824lb # [ 0.164063] usbcore: registered new interface driver hub825client # [ 0.161594] pps_core: LinuxPPS API ver. 1 registered826lb # [ 0.164078] usbcore: registered new device driver usb827worker2 # [ 0.144815] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages828lb # [ 0.164354] pps_core: LinuxPPS API ver. 1 registered829client # [ 0.161600] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>830worker2 # [ 0.144819] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page831client # [ 0.161611] PTP clock support registered832lb # [ 0.164360] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>833client # [ 0.161661] EDAC MC: Ver: 3.0.0834worker2 # [ 0.144823] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages835lb # [ 0.164370] PTP clock support registered836lb # [ 0.164408] EDAC MC: Ver: 3.0.0837worker2 # [ 0.144828] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page838client # [ 0.166215] scmi_core: SCMI protocol bus registered839lb # [ 0.168922] scmi_core: SCMI protocol bus registered840client # [ 0.167188] FPGA manager framework841client # [ 0.168139] vgaarb: loaded842lb # [ 0.169885] FPGA manager framework843worker2 # [ 0.152107] fbcon: Taking over console844lb # [ 0.170872] vgaarb: loaded845client # [ 0.168779] clocksource: Switched to clocksource arch_sys_counter846worker2 # [ 0.152121] ACPI: Interpreter disabled.847worker2 # [ 0.153958] iommu: Default domain type: Translated848lb # [ 0.171489] clocksource: Switched to clocksource arch_sys_counter849worker2 # [ 0.153969] iommu: DMA domain TLB invalidation policy: strict mode850lb # [ 0.172071] VFS: Disk quotas dquot_6.6.0851worker2 # [ 0.155643] SCSI subsystem initialized852lb # [ 0.172097] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)853lb # [ 0.174439] netfs: FS-Cache loaded854lb # [ 0.174549] pnp: PnP ACPI: disabled855client # [ 0.174886] VFS: Disk quotas dquot_6.6.0856client # [ 0.174926] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)857client # [ 0.176533] netfs: FS-Cache loaded858client # [ 0.176647] pnp: PnP ACPI: disabled859worker2 # [ 0.163293] usbcore: registered new interface driver usbfs860worker2 # [ 0.163336] usbcore: registered new interface driver hub861client # [ 0.180523] NET: Registered PF_INET protocol family862worker2 # [ 0.163353] usbcore: registered new device driver usb863worker2 # [ 0.163676] pps_core: LinuxPPS API ver. 1 registered864client # [ 0.180779] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)865lb # [ 0.180311] NET: Registered PF_INET protocol family866lb # [ 0.180582] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)867worker2 # [ 0.163682] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>868worker2 # [ 0.163692] PTP clock support registered869worker2 # [ 0.163733] EDAC MC: Ver: 3.0.0870worker2 # [ 0.168313] scmi_core: SCMI protocol bus registered871worker2 # [ 0.169265] FPGA manager framework872worker2 # [ 0.170205] vgaarb: loaded873worker2 # [ 0.170827] clocksource: Switched to clocksource arch_sys_counter874worker2 # [ 0.172216] VFS: Disk quotas dquot_6.6.0875worker2 # [ 0.172249] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)876worker2 # [ 0.173769] netfs: FS-Cache loaded877worker2 # [ 0.173881] pnp: PnP ACPI: disabled878worker2 # [ 0.179720] NET: Registered PF_INET protocol family879worker2 # [ 0.180009] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)880client # [ 0.209742] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)881lb # [ 0.210320] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)882client # [ 0.209786] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)883lb # [ 0.210366] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)884client # [ 0.209812] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)885client # [ 0.209881] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)886lb # [ 0.210394] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)887client # [ 0.210022] TCP: Hash tables configured (established 16384 bind 16384)888lb # [ 0.210471] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)889client # [ 0.210101] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)890lb # [ 0.210616] TCP: Hash tables configured (established 16384 bind 16384)891client # [ 0.210136] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)892lb # [ 0.210705] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)893client # [ 0.210173] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)894lb # [ 0.210742] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)895client # [ 0.210271] NET: Registered PF_UNIX/PF_LOCAL protocol family896lb # [ 0.210777] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)897client # [ 0.210290] NET: Registered PF_XDP protocol family898lb # [ 0.210878] NET: Registered PF_UNIX/PF_LOCAL protocol family899client # [ 0.210309] PCI: CLS 0 bytes, default 64900lb # [ 0.210899] NET: Registered PF_XDP protocol family901client # [ 0.210548] Trying to unpack rootfs image as initramfs...902lb # [ 0.210917] PCI: CLS 0 bytes, default 64903lb # [ 0.211163] Trying to unpack rootfs image as initramfs...904client # [ 0.226482] kvm [1]: HYP mode not available905lb # [ 0.229289] kvm [1]: HYP mode not available906worker2 # [ 0.209585] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)907worker2 # [ 0.209634] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)908worker2 # [ 0.209664] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)909worker2 # [ 0.209734] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)910worker2 # [ 0.209876] TCP: Hash tables configured (established 16384 bind 16384)911worker2 # [ 0.209957] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)912worker2 # [ 0.209992] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)913worker2 # [ 0.210028] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)914worker2 # [ 0.210128] NET: Registered PF_UNIX/PF_LOCAL protocol family915worker2 # [ 0.210171] NET: Registered PF_XDP protocol family916worker2 # [ 0.210190] PCI: CLS 0 bytes, default 64917worker2 # [ 0.210438] Trying to unpack rootfs image as initramfs...918worker2 # [ 0.228462] kvm [1]: HYP mode not available919worker1 # [ 0.322601] Initialise system trusted keyrings920worker1 # [ 0.323348] workingset: timestamp_bits=42 max_order=19 bucket_order=0921worker1 # [ 0.324662] squashfs: version 4.0 (2009/01/31) Phillip Lougher922worker1 # [ 0.325431] 9p: Installing v9fs 9p2000 file system support923worker1 # [ 0.354162] Key type asymmetric registered924worker1 # [ 0.354189] Asymmetric key parser 'x509' registered925worker1 # [ 0.354253] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)926worker1 # [ 0.356402] io scheduler mq-deadline registered927worker1 # [ 0.356413] io scheduler kyber registered928worker1 # [ 0.366225] pl061_gpio 9030000.pl061: PL061 GPIO chip registered929worker1 # [ 0.367689] ledtrig-cpu: registered to indicate activity on CPUs930worker1 # [ 0.368081] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:931worker1 # [ 0.368099] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000932worker1 # [ 0.368124] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000933worker1 # [ 0.368133] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000934worker1 # [ 0.368154] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits935worker1 # [ 0.368181] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]936worker1 # [ 0.368268] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00937worker1 # [ 0.368278] pci_bus 0000:00: root bus resource [bus 00-ff]938worker1 # [ 0.368283] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]939worker1 # [ 0.368288] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]940worker1 # [ 0.368293] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]941worker1 # [ 0.368347] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint942worker1 # [ 0.368777] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint943worker1 # [ 0.368959] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]944worker1 # [ 0.368976] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]945lb # [ 0.314748] Initialise system trusted keyrings946worker1 # [ 0.369005] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]947lb # [ 0.315535] workingset: timestamp_bits=42 max_order=19 bucket_order=0948worker1 # [ 0.369021] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]949worker1 # [ 0.369468] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint950worker1 # [ 0.369647] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]951worker1 # [ 0.369662] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]952client # [ 0.317366] Initialise system trusted keyrings953worker1 # [ 0.369691] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]954client # [ 0.318110] workingset: timestamp_bits=42 max_order=19 bucket_order=0955worker1 # [ 0.389422] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint956client # [ 0.319362] squashfs: version 4.0 (2009/01/31) Phillip Lougher957worker1 # [ 0.389608] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]958worker1 # [ 0.389623] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]959client # [ 0.320147] 9p: Installing v9fs 9p2000 file system support960worker1 # [ 0.389653] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]961lb # [ 0.323923] squashfs: version 4.0 (2009/01/31) Phillip Lougher962lb # [ 0.323990] 9p: Installing v9fs 9p2000 file system support963worker1 # [ 0.398201] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint964worker1 # [ 0.398403] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]965worker1 # [ 0.398419] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]966worker1 # [ 0.398449] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]967worker1 # [ 0.398468] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]968worker2 # [ 0.323526] Initialise system trusted keyrings969worker1 # [ 0.398952] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint970worker1 # [ 0.399145] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]971worker2 # [ 0.324290] workingset: timestamp_bits=42 max_order=19 bucket_order=0972worker1 # [ 0.399175] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]973worker2 # [ 0.325608] squashfs: version 4.0 (2009/01/31) Phillip Lougher974worker1 # [ 0.399654] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint975lb # [ 0.343981] Key type asymmetric registered976worker2 # [ 0.326379] 9p: Installing v9fs 9p2000 file system support977lb # [ 0.344003] Asymmetric key parser 'x509' registered978worker1 # [ 0.399836] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]979lb # [ 0.344085] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)980worker1 # [ 0.399866] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]981lb # [ 0.346270] io scheduler mq-deadline registered982worker1 # [ 0.400254] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint983lb # [ 0.346281] io scheduler kyber registered984worker1 # [ 0.400433] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]985worker1 # [ 0.400676] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint986worker1 # [ 0.400859] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]987worker1 # [ 0.400888] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]988client # [ 0.348847] Key type asymmetric registered989worker1 # [ 0.401332] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint990client # [ 0.348874] Asymmetric key parser 'x509' registered991worker1 # [ 0.401516] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]992client # [ 0.348939] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)993worker1 # [ 0.401546] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]994client # [ 0.351066] io scheduler mq-deadline registered995client # [ 0.351076] io scheduler kyber registered996worker1 # [ 0.401991] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint997worker1 # [ 0.402189] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]998worker1 # [ 0.402219] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]999worker1 # [ 0.402667] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1000worker1 # [ 0.402953] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1001worker1 # [ 0.402970] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1002worker1 # [ 0.402999] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1003worker1 # [ 0.403472] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1004worker1 # [ 0.403651] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1005worker1 # [ 0.403666] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1006worker1 # [ 0.403696] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1007worker1 # [ 0.404254] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1008worker1 # [ 0.404265] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1009client # [ 0.360922] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1010worker1 # [ 0.404270] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1011worker1 # [ 0.404315] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1012worker1 # [ 0.404361] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1013client # [ 0.362292] ledtrig-cpu: registered to indicate activity on CPUs1014worker1 # [ 0.404408] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1015client # [ 0.362684] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1016lb # [ 0.359626] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1017worker1 # [ 0.404454] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1018lb # [ 0.360238] ledtrig-cpu: registered to indicate activity on CPUs1019client # [ 0.362700] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001020worker1 # [ 0.404500] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1021lb # [ 0.360609] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1022client # [ 0.362712] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001023worker1 # [ 0.404545] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1024lb # [ 0.360626] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001025client # [ 0.362720] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001026worker1 # [ 0.404590] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1027lb # [ 0.360646] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001028client # [ 0.362742] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1029worker1 # [ 0.404636] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1030lb # [ 0.360655] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001031client # [ 0.362767] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1032worker1 # [ 0.404681] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1033worker2 # [ 0.355098] Key type asymmetric registered1034lb # [ 0.360676] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1035worker2 # [ 0.355126] Asymmetric key parser 'x509' registered1036client # [ 0.362840] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001037worker1 # [ 0.404750] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1038client # [ 0.362849] pci_bus 0000:00: root bus resource [bus 00-ff]1039worker2 # [ 0.355192] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)1040lb # [ 0.360701] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1041worker1 # [ 0.404795] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1042client # [ 0.362869] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1043worker2 # [ 0.357237] io scheduler mq-deadline registered1044lb # [ 0.360778] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001045worker2 # [ 0.357248] io scheduler kyber registered1046worker1 # [ 0.404817] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1047client # [ 0.362875] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1048lb # [ 0.360788] pci_bus 0000:00: root bus resource [bus 00-ff]1049worker1 # [ 0.404837] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1050lb # [ 0.360794] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1051client # [ 0.362880] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1052worker1 # [ 0.404858] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1053lb # [ 0.360799] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1054client # [ 0.362932] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1055worker1 # [ 0.404879] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1056lb # [ 0.360804] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1057client # [ 0.363364] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1058worker1 # [ 0.404900] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1059lb # [ 0.360877] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1060client # [ 0.363548] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1061worker1 # [ 0.404922] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1062client # [ 0.363564] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1063lb # [ 0.361318] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1064worker1 # [ 0.404945] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1065client # [ 0.363593] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1066lb # [ 0.361504] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1067worker1 # [ 0.404967] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1068client # [ 0.363609] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1069lb # [ 0.361520] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1070worker1 # [ 0.404988] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1071lb # [ 0.361550] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1072client # [ 0.364076] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1073worker1 # [ 0.405010] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1074lb # [ 0.361566] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1075client # [ 0.364259] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1076worker1 # [ 0.405032] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1077client # [ 0.364274] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1078lb # [ 0.362033] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1079worker1 # [ 0.405053] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1080client # [ 0.364304] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1081lb # [ 0.362216] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1082worker1 # [ 0.405073] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1083lb # [ 0.362232] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1084worker2 # [ 0.366982] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1085client # [ 0.364753] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1086worker1 # [ 0.405094] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1087lb # [ 0.362261] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1088worker1 # [ 0.405114] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1089client # [ 0.384392] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1090lb # [ 0.362710] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1091worker1 # [ 0.405135] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1092client # [ 0.384411] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1093lb # [ 0.362891] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1094worker1 # [ 0.405155] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1095client # [ 0.384441] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1096lb # [ 0.362907] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1097worker1 # [ 0.405180] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1098worker2 # [ 0.368329] ledtrig-cpu: registered to indicate activity on CPUs1099lb # [ 0.362937] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1100worker1 # [ 0.405190] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1101worker2 # [ 0.368700] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1102lb # [ 0.363384] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1103worker1 # [ 0.405194] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1104worker2 # [ 0.368717] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001105lb # [ 0.363594] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1106worker1 # [ 0.405982] pci 0000:00:07.0: enabling device (0000 -> 0002)1107client # [ 0.386984] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1108lb # [ 0.363609] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1109worker2 # [ 0.368730] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001110client # [ 0.387171] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1111lb # [ 0.363638] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1112client # [ 0.387187] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1113worker2 # [ 0.368738] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001114lb # [ 0.363657] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1115client # [ 0.387218] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1116worker2 # [ 0.368760] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1117client # [ 0.387236] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1118lb # [ 0.364112] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1119lb # [ 0.364298] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1120worker2 # [ 0.368784] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1121client # [ 0.387699] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1122lb # [ 0.364327] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1123client # [ 0.387887] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1124worker2 # [ 0.368867] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001125worker2 # [ 0.368876] pci_bus 0000:00: root bus resource [bus 00-ff]1126client # [ 0.387917] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1127lb # [ 0.364789] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1128worker2 # [ 0.368882] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1129lb # [ 0.364975] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1130client # [ 0.388365] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1131worker2 # [ 0.368888] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1132lb # [ 0.365005] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1133client # [ 0.388551] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1134worker2 # [ 0.368893] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1135worker1 # [ 0.466180] pci 0000:00:07.0: quirk_usb_early_handoff+0x0/0xa60 took 58792 usecs1136client # [ 0.388580] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1137lb # [ 0.365397] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1138worker2 # [ 0.368982] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1139lb # [ 0.365577] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1140worker2 # [ 0.369427] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1141lb # [ 0.365841] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1142worker2 # [ 0.369611] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1143lb # [ 0.366028] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1144worker2 # [ 0.369627] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1145lb # [ 0.366057] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1146worker2 # [ 0.369656] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1147lb # [ 0.366513] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1148worker2 # [ 0.369672] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1149lb # [ 0.366698] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1150worker2 # [ 0.370126] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1151lb # [ 0.366728] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1152worker2 # [ 0.370308] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1153lb # [ 0.367177] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1154worker2 # [ 0.370323] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1155lb # [ 0.367363] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1156worker2 # [ 0.370352] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1157client # [ 0.401082] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1158lb # [ 0.367393] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1159worker2 # [ 0.370797] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1160client # [ 0.401281] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1161worker2 # [ 0.390629] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1162client # [ 0.401531] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1163worker2 # [ 0.390650] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1164client # [ 0.401717] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1165worker2 # [ 0.390679] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1166client # [ 0.401746] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1167client # [ 0.402193] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1168client # [ 0.402378] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1169client # [ 0.402408] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1170worker2 # [ 0.393309] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1171worker2 # [ 0.393494] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1172client # [ 0.402857] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1173worker2 # [ 0.393510] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1174client # [ 0.403045] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1175worker2 # [ 0.393538] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1176client # [ 0.403075] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1177worker2 # [ 0.393556] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1178client # [ 0.403535] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1179worker2 # [ 0.394015] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1180client # [ 0.403809] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1181worker2 # [ 0.394200] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1182client # [ 0.403837] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1183worker2 # [ 0.394229] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1184client # [ 0.403867] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1185worker2 # [ 0.394680] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1186client # [ 0.404322] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1187client # [ 0.404505] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1188worker1 # [ 0.486834] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1189client # [ 0.404520] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1190worker1 # [ 0.488990] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1191client # [ 0.404550] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1192lb # [ 0.415973] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1193lb # [ 0.416289] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1194lb # [ 0.416307] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1195lb # [ 0.416337] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1196lb # [ 0.416805] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1197lb # [ 0.416989] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1198lb # [ 0.417005] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1199lb # [ 0.417035] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1200lb # [ 0.417605] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1201worker2 # [ 0.406937] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1202lb # [ 0.417616] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1203worker2 # [ 0.406981] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1204lb # [ 0.417621] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1205worker2 # [ 0.407416] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1206client # [ 0.425214] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1207lb # [ 0.417667] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1208worker2 # [ 0.407600] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1209client # [ 0.425235] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1210lb # [ 0.417730] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1211worker2 # [ 0.407874] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1212client # [ 0.425241] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1213lb # [ 0.417778] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1214worker1 # [ 0.500059] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1215worker2 # [ 0.408077] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1216client # [ 0.425290] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1217worker1 # [ 0.502043] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1218worker2 # [ 0.408106] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1219lb # [ 0.417827] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1220client # [ 0.425338] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1221worker2 # [ 0.408553] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1222lb # [ 0.417874] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1223worker2 # [ 0.408739] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1224client # [ 0.425386] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1225lb # [ 0.417922] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1226worker2 # [ 0.408768] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1227client # [ 0.425434] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1228lb # [ 0.417969] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1229worker2 # [ 0.409215] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1230client # [ 0.425482] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1231lb # [ 0.418016] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1232worker2 # [ 0.409399] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1233client # [ 0.425529] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1234lb # [ 0.418063] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1235worker2 # [ 0.409428] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1236client # [ 0.425577] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1237lb # [ 0.418140] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1238worker2 # [ 0.409879] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1239worker2 # [ 0.410166] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1240client # [ 0.425623] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1241lb # [ 0.418192] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1242worker2 # [ 0.410183] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1243lb # [ 0.418214] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1244client # [ 0.425670] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1245worker2 # [ 0.410213] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1246lb # [ 0.418235] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1247client # [ 0.425785] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1248worker2 # [ 0.410664] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1249lb # [ 0.418256] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1250client # [ 0.425837] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1251lb # [ 0.418277] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1252client # [ 0.425859] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1253lb # [ 0.418298] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1254client # [ 0.425881] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1255worker1 # [ 0.513564] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1256lb # [ 0.418319] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1257client # [ 0.425902] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1258lb # [ 0.418345] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1259client # [ 0.425923] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1260lb # [ 0.418366] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1261client # [ 0.425944] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1262lb # [ 0.418388] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1263client # [ 0.425965] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1264lb # [ 0.418410] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1265client # [ 0.425991] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1266client # [ 0.426013] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1267lb # [ 0.418432] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1268lb # [ 0.418453] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1269client # [ 0.426034] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1270lb # [ 0.418474] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1271client # [ 0.426056] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1272lb # [ 0.418494] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1273client # [ 0.426078] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1274lb # [ 0.418515] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1275client # [ 0.426100] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1276worker1 # [ 0.519569] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1277worker2 # [ 0.430917] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1278lb # [ 0.418536] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1279client # [ 0.426120] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1280worker1 # [ 0.521254] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1281worker2 # [ 0.430944] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1282lb # [ 0.418557] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1283client # [ 0.426141] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1284lb # [ 0.418582] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1285worker2 # [ 0.430974] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1286client # [ 0.426162] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1287lb # [ 0.418591] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1288worker2 # [ 0.431589] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1289client # [ 0.426183] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1290lb # [ 0.418596] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1291client # [ 0.426203] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1292worker2 # [ 0.431601] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1293lb # [ 0.419417] pci 0000:00:07.0: enabling device (0000 -> 0002)1294client # [ 0.426229] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1295worker2 # [ 0.431606] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1296client # [ 0.426239] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1297worker2 # [ 0.431652] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1298client # [ 0.426243] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1299worker2 # [ 0.431699] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1300client # [ 0.427069] pci 0000:00:07.0: enabling device (0000 -> 0002)1301worker2 # [ 0.431747] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1302worker2 # [ 0.431795] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1303worker2 # [ 0.431843] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1304worker2 # [ 0.431890] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1305worker2 # [ 0.431946] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1306worker2 # [ 0.431995] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1307worker1 # [ 0.531312] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1308worker1 # [ 0.533266] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1309worker2 # [ 0.432046] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1310worker2 # [ 0.432161] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1311worker2 # [ 0.432210] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1312worker2 # [ 0.432232] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1313worker1 # [ 0.536844] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1314worker2 # [ 0.432254] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1315worker2 # [ 0.432275] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1316worker2 # [ 0.432297] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1317worker2 # [ 0.432318] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1318worker2 # [ 0.432340] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1319worker2 # [ 0.432367] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1320worker2 # [ 0.432389] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1321worker2 # [ 0.432411] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1322worker2 # [ 0.432433] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1323worker2 # [ 0.432455] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1324worker2 # [ 0.432476] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1325worker1 # [ 0.543776] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1326worker2 # [ 0.432497] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1327worker2 # [ 0.432518] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1328worker2 # [ 0.432539] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1329worker2 # [ 0.432560] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1330worker2 # [ 0.432581] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1331worker2 # [ 0.432607] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1332worker2 # [ 0.432617] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1333worker2 # [ 0.432621] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1334worker2 # [ 0.433442] pci 0000:00:07.0: enabling device (0000 -> 0002)1335lb # [ 0.481597] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1336client # [ 0.483453] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1337lb # [ 0.484491] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1338worker1 # [ 0.556827] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1339lb # [ 0.486491] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1340worker1 # [ 0.559263] msm_serial: driver initialized1341worker1 # [ 0.559423] SuperH (H)SCI(F) driver initialized1342worker1 # [ 0.559476] STM32 USART driver initialized1343lb # [ 0.496920] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1344lb # [ 0.498954] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1345worker2 # [ 0.480713] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1346worker2 # [ 0.482818] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1347client # [ 0.493418] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1348client # [ 0.496761] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1349lb # [ 0.502498] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1350client # [ 0.498980] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1351lb # [ 0.508705] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1352lb # [ 0.510855] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1353client # [ 0.508965] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1354client # [ 0.511006] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1355client # [ 0.512732] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1356worker2 # [ 0.500899] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1357worker1 # [ 0.590820] loop: module loaded1358worker1 # [ 0.590999] virtio_blk virtio2: 1/0/0 default/read/poll queues1359lb # [ 0.522059] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1360worker2 # [ 0.504000] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1361worker1 # [ 0.592051] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1362worker2 # [ 0.506193] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1363client # [ 0.522309] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1364client # [ 0.524281] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1365lb # [ 0.528020] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1366lb # [ 0.530211] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1367worker1 # [ 0.602712] megasas: 07.734.00.00-rc11368worker2 # [ 0.516529] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1369worker2 # [ 0.518395] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1370worker1 # [ 0.603506] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1371client # [ 0.535435] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1372worker1 # [ 0.605612] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001373worker1 # [ 0.605636] Intel/Sharp Extended Query Table at 0x00311374worker2 # [ 0.521843] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1375client # [ 0.539018] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1376lb # [ 0.542983] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1377lb # [ 0.545407] msm_serial: driver initialized1378lb # [ 0.545538] SuperH (H)SCI(F) driver initialized1379lb # [ 0.545593] STM32 USART driver initialized1380worker1 # [ 0.615096] Using buffer write method1381worker2 # [ 0.529236] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1382worker1 # [ 0.615165] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1383worker1 # [ 0.619785] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001384worker1 # [ 0.619803] Intel/Sharp Extended Query Table at 0x00311385worker1 # [ 0.621178] Using buffer write method1386worker1 # [ 0.621202] Concatenating MTD devices:1387worker1 # [ 0.621206] (0): "0.flash"1388client # [ 0.551950] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1389worker1 # [ 0.621217] (1): "0.flash"1390worker1 # [ 0.621221] into device "0.flash"1391worker2 # [ 0.536079] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1392client # [ 0.554557] msm_serial: driver initialized1393worker2 # [ 0.538365] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1394client # [ 0.554705] SuperH (H)SCI(F) driver initialized1395client # [ 0.554764] STM32 USART driver initialized1396worker2 # [ 0.556026] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1397worker2 # [ 0.558608] msm_serial: driver initialized1398worker2 # [ 0.558746] SuperH (H)SCI(F) driver initialized1399worker2 # [ 0.558798] STM32 USART driver initialized1400lb # [ 0.578309] loop: module loaded1401lb # [ 0.578491] virtio_blk virtio2: 1/0/0 default/read/poll queues1402lb # [ 0.579240] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1403client # [ 0.585611] loop: module loaded1404client # [ 0.585774] virtio_blk virtio2: 1/0/0 default/read/poll queues1405client # [ 0.586557] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1406lb # [ 0.592131] megasas: 07.734.00.00-rc11407lb # [ 0.592842] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1408lb # [ 0.594854] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001409lb # [ 0.594877] Intel/Sharp Extended Query Table at 0x00311410lb # [ 0.596461] Using buffer write method1411lb # [ 0.596515] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1412lb # [ 0.598290] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001413lb # [ 0.598311] Intel/Sharp Extended Query Table at 0x00311414client # [ 0.597406] megasas: 07.734.00.00-rc11415lb # [ 0.607421] Using buffer write method1416client # [ 0.598144] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1417lb # [ 0.607448] Concatenating MTD devices:1418lb # [ 0.607452] (0): "0.flash"1419lb # [ 0.607455] (1): "0.flash"1420worker2 # [ 0.589600] loop: module loaded1421client # [ 0.600332] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001422lb # [ 0.607465] into device "0.flash"1423client # [ 0.600362] Intel/Sharp Extended Query Table at 0x00311424worker2 # [ 0.589793] virtio_blk virtio2: 1/0/0 default/read/poll queues1425client # [ 0.601877] Using buffer write method1426worker2 # [ 0.590574] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1427client # [ 0.601949] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1428client # [ 0.603695] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001429client # [ 0.603722] Intel/Sharp Extended Query Table at 0x00311430client # [ 0.622438] Using buffer write method1431client # [ 0.622483] Concatenating MTD devices:1432client # [ 0.622488] (0): "0.flash"1433client # [ 0.622491] (1): "0.flash"1434client # [ 0.622501] into device "0.flash"1435worker2 # [ 0.603536] megasas: 07.734.00.00-rc11436worker2 # [ 0.604325] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1437worker2 # [ 0.606593] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001438worker2 # [ 0.606621] Intel/Sharp Extended Query Table at 0x00311439worker2 # [ 0.608446] Using buffer write method1440worker2 # [ 0.608514] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1441worker2 # [ 0.610150] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001442worker2 # [ 0.610174] Intel/Sharp Extended Query Table at 0x00311443worker2 # [ 0.623769] Using buffer write method1444worker2 # [ 0.623803] Concatenating MTD devices:1445worker2 # [ 0.623807] (0): "0.flash"1446worker2 # [ 0.623811] (1): "0.flash"1447worker2 # [ 0.623820] into device "0.flash"1448worker1 # [ 0.868554] Freeing initrd memory: 26976K1449worker1 # [ 0.874568] tun: Universal TUN/TAP device driver, 1.61450worker1 # [ 0.878340] thunder_xcv, ver 1.01451worker1 # [ 0.878378] thunder_bgx, ver 1.01452worker1 # [ 0.878399] nicpf, ver 1.01453worker1 # [ 0.878947] e1000: Intel(R) PRO/1000 Network Driver1454worker1 # [ 0.878954] e1000: Copyright (c) 1999-2006 Intel Corporation.1455worker1 # [ 0.878993] e1000e: Intel(R) PRO/1000 Network Driver1456worker1 # [ 0.879000] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1457worker1 # [ 0.879028] igb: Intel(R) Gigabit Ethernet Network Driver1458worker1 # [ 0.879034] igb: Copyright (c) 2007-2014 Intel Corporation.1459worker1 # [ 0.879055] igbvf: Intel(R) Gigabit Virtual Function Network Driver1460worker1 # [ 0.879077] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1461worker1 # [ 0.879216] sky2: driver version 1.301462worker1 # [ 0.880769] usbcore: registered new interface driver usb-storage1463worker1 # [ 0.880881] usbcore: registered new interface driver usbserial_generic1464worker1 # [ 0.880894] usbserial: USB Serial support registered for generic1465worker1 # [ 0.881474] hv_vmbus: registering driver hyperv_keyboard1466worker1 # [ 0.882801] ehci-pci 0000:00:07.0: EHCI Host Controller1467worker1 # [ 0.882841] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11468worker1 # [ 0.883121] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001469worker1 # [ 0.895534] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001470worker1 # [ 0.895888] hub 1-0:1.0: USB hub found1471worker1 # [ 0.895908] hub 1-0:1.0: 6 ports detected1472worker1 # [ 0.898270] rtc-pl031 9010000.pl031: registered as rtc01473worker1 # [ 0.898298] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T13:21:42 UTC (1789996902)1474worker1 # [ 0.898668] i2c_dev: i2c /dev entries driver1475worker1 # [ 0.904150] sdhci: Secure Digital Host Controller Interface driver1476worker1 # [ 0.904161] sdhci: Copyright(c) Pierre Ossman1477worker1 # [ 0.904420] Synopsys Designware Multimedia Card Interface Driver1478worker1 # [ 0.904785] sdhci-pltfm: SDHCI platform and OF driver helper1479worker1 # [ 0.909009] hid: raw HID events driver (C) Jiri Kosina1480worker1 # [ 0.909263] usbcore: registered new interface driver usbhid1481worker1 # [ 0.909270] usbhid: USB HID core driver1482worker1 # [ 0.912184] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1483worker1 # [ 0.913692] drop_monitor: Initializing network drop monitor service1484worker1 # [ 0.913848] NET: Registered PF_INET6 protocol family1485worker1 # [ 0.916953] Segment Routing with IPv61486worker1 # [ 0.916971] In-situ OAM (IOAM) with IPv61487worker1 # [ 0.916998] NET: Registered PF_PACKET protocol family1488worker1 # [ 0.918873] 9pnet: Installing 9P2000 support1489worker1 # [ 0.918916] Key type dns_resolver registered1490worker1 # [ 0.925405] registered taskstats version 11491worker1 # [ 0.925558] Loading compiled-in X.509 certificates1492lb # [ 0.858536] Freeing initrd memory: 26948K1493lb # [ 0.864591] tun: Universal TUN/TAP device driver, 1.61494client # [ 0.861547] Freeing initrd memory: 26972K1495worker1 # [ 0.934200] Demotion targets for Node 0: null1496worker1 # [ 0.934302] Key type .fscrypt registered1497worker1 # [ 0.934308] Key type fscrypt-provisioning registered1498worker1 # [ 0.934400] ima: No TPM chip found, activating TPM-bypass!1499worker1 # [ 0.934419] ima: Allocated hash algorithm: sha11500worker1 # [ 0.934439] ima: No architecture policies found1501client # [ 0.867525] tun: Universal TUN/TAP device driver, 1.61502worker1 # [ 0.938616] input: gpio-keys as /devices/platform/gpio-keys/input/input01503lb # [ 0.868309] thunder_xcv, ver 1.01504lb # [ 0.868349] thunder_bgx, ver 1.01505lb # [ 0.868370] nicpf, ver 1.01506lb # [ 0.868923] e1000: Intel(R) PRO/1000 Network Driver1507lb # [ 0.868930] e1000: Copyright (c) 1999-2006 Intel Corporation.1508client # [ 0.871166] thunder_xcv, ver 1.01509client # [ 0.871214] thunder_bgx, ver 1.01510lb # [ 0.868953] e1000e: Intel(R) PRO/1000 Network Driver1511client # [ 0.871237] nicpf, ver 1.01512lb # [ 0.868960] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1513client # [ 0.871797] e1000: Intel(R) PRO/1000 Network Driver1514lb # [ 0.868990] igb: Intel(R) Gigabit Ethernet Network Driver1515client # [ 0.871804] e1000: Copyright (c) 1999-2006 Intel Corporation.1516client # [ 0.871853] e1000e: Intel(R) PRO/1000 Network Driver1517lb # [ 0.868995] igb: Copyright (c) 2007-2014 Intel Corporation.1518client # [ 0.871862] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1519lb # [ 0.869018] igbvf: Intel(R) Gigabit Virtual Function Network Driver1520client # [ 0.871892] igb: Intel(R) Gigabit Ethernet Network Driver1521lb # [ 0.869023] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1522lb # [ 0.869154] sky2: driver version 1.301523client # [ 0.871898] igb: Copyright (c) 2007-2014 Intel Corporation.1524lb # [ 0.870689] usbcore: registered new interface driver usb-storage1525client # [ 0.871921] igbvf: Intel(R) Gigabit Virtual Function Network Driver1526worker2 # [ 0.862597] Freeing initrd memory: 26944K1527lb # [ 0.870771] usbcore: registered new interface driver usbserial_generic1528client # [ 0.871927] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1529client # [ 0.872060] sky2: driver version 1.301530lb # [ 0.870785] usbserial: USB Serial support registered for generic1531lb # [ 0.871357] hv_vmbus: registering driver hyperv_keyboard1532lb # [ 0.872247] ehci-pci 0000:00:07.0: EHCI Host Controller1533lb # [ 0.872277] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11534client # [ 0.881511] ehci-pci 0000:00:07.0: EHCI Host Controller1535lb # [ 0.872548] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001536client # [ 0.881546] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11537lb # [ 0.883958] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001538client # [ 0.881786] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001539lb # [ 0.885730] hub 1-0:1.0: USB hub found1540lb # [ 0.885758] hub 1-0:1.0: 6 ports detected1541worker2 # [ 0.868634] tun: Universal TUN/TAP device driver, 1.61542client # [ 0.884199] usbcore: registered new interface driver usb-storage1543client # [ 0.884319] usbcore: registered new interface driver usbserial_generic1544client # [ 0.884334] usbserial: USB Serial support registered for generic1545worker1 # [ 0.956630] clk: Disabling unused clocks1546lb # [ 0.887859] rtc-pl031 9010000.pl031: registered as rtc01547worker1 # [ 0.956659] PM: genpd: Disabling unused power domains1548client # [ 0.887257] hv_vmbus: registering driver hyperv_keyboard1549lb # [ 0.887887] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T13:21:42 UTC (1789996902)1550lb # [ 0.888187] i2c_dev: i2c /dev entries driver1551client # [ 0.888837] rtc-pl031 9010000.pl031: registered as rtc01552worker1 # [ 0.960888] Freeing unused kernel memory: 4736K1553worker1 # [ 0.961083] Run /init as init process1554client # [ 0.888865] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T13:21:42 UTC (1789996902)1555client # [ 0.889165] i2c_dev: i2c /dev entries driver1556worker2 # [ 0.872336] thunder_xcv, ver 1.01557worker2 # [ 0.872378] thunder_bgx, ver 1.01558worker2 # [ 0.872401] nicpf, ver 1.01559worker2 # [ 0.872939] e1000: Intel(R) PRO/1000 Network Driver1560worker2 # [ 0.872946] e1000: Copyright (c) 1999-2006 Intel Corporation.1561lb # [ 0.893412] sdhci: Secure Digital Host Controller Interface driver1562worker2 # [ 0.872970] e1000e: Intel(R) PRO/1000 Network Driver1563lb # [ 0.893422] sdhci: Copyright(c) Pierre Ossman1564worker2 # [ 0.872979] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1565lb # [ 0.893722] Synopsys Designware Multimedia Card Interface Driver1566worker2 # [ 0.873008] igb: Intel(R) Gigabit Ethernet Network Driver1567lb # [ 0.894110] sdhci-pltfm: SDHCI platform and OF driver helper1568worker2 # [ 0.873014] igb: Copyright (c) 2007-2014 Intel Corporation.1569client # [ 0.894492] sdhci: Secure Digital Host Controller Interface driver1570worker2 # [ 0.873065] igbvf: Intel(R) Gigabit Virtual Function Network Driver1571client # [ 0.894503] sdhci: Copyright(c) Pierre Ossman1572lb # [ 0.898382] hid: raw HID events driver (C) Jiri Kosina1573worker2 # [ 0.873071] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1574client # [ 0.894770] Synopsys Designware Multimedia Card Interface Driver1575worker2 # [ 0.873204] sky2: driver version 1.301576lb # [ 0.898621] usbcore: registered new interface driver usbhid1577lb # [ 0.898627] usbhid: USB HID core driver1578client # [ 0.895134] sdhci-pltfm: SDHCI platform and OF driver helper1579worker2 # [ 0.874740] usbcore: registered new interface driver usb-storage1580client # [ 0.896655] hid: raw HID events driver (C) Jiri Kosina1581worker2 # [ 0.874824] usbcore: registered new interface driver usbserial_generic1582client # [ 0.900041] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001583worker2 # [ 0.875576] ehci-pci 0000:00:07.0: EHCI Host Controller1584worker2 # [ 0.875600] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11585worker2 # [ 0.875849] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001586lb # [ 0.901439] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1587client # [ 0.900990] usbcore: registered new interface driver usbhid1588client # [ 0.900998] usbhid: USB HID core driver1589lb # [ 0.902957] drop_monitor: Initializing network drop monitor service1590lb # [ 0.903109] NET: Registered PF_INET6 protocol family1591client # [ 0.903449] hub 1-0:1.0: USB hub found1592worker2 # [ 0.886661] usbserial: USB Serial support registered for generic1593client # [ 0.903936] hub 1-0:1.0: 6 ports detected1594worker2 # [ 0.886882] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001595lb # [ 0.906180] Segment Routing with IPv61596worker2 # [ 0.887172] hub 1-0:1.0: USB hub found1597lb # [ 0.906195] In-situ OAM (IOAM) with IPv61598worker1 # [ 0.976230] systemd[1]: Successfully made /usr/ read-only.1599worker2 # [ 0.887192] hub 1-0:1.0: 6 ports detected1600lb # [ 0.906240] NET: Registered PF_PACKET protocol family1601worker2 # [ 0.890379] hv_vmbus: registering driver hyperv_keyboard1602client # [ 0.904959] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1603client # [ 0.906464] drop_monitor: Initializing network drop monitor service1604client # [ 0.906628] NET: Registered PF_INET6 protocol family1605worker2 # [ 0.891918] rtc-pl031 9010000.pl031: registered as rtc01606lb # [ 0.908039] 9pnet: Installing 9P2000 support1607worker2 # [ 0.891951] rtc-pl031 9010000.pl031: setting system clock to 2026-09-21T13:21:42 UTC (1789996902)1608lb # [ 0.908084] Key type dns_resolver registered1609client # [ 0.909693] Segment Routing with IPv61610client # [ 0.909711] In-situ OAM (IOAM) with IPv61611worker2 # [ 0.892255] i2c_dev: i2c /dev entries driver1612client # [ 0.909740] NET: Registered PF_PACKET protocol family1613client # [ 0.911307] 9pnet: Installing 9P2000 support1614client # [ 0.911358] Key type dns_resolver registered1615lb # [ 0.914484] registered taskstats version 11616lb # [ 0.914646] Loading compiled-in X.509 certificates1617worker2 # [ 0.897497] sdhci: Secure Digital Host Controller Interface driver1618worker2 # [ 0.897507] sdhci: Copyright(c) Pierre Ossman1619worker2 # [ 0.897772] Synopsys Designware Multimedia Card Interface Driver1620worker2 # [ 0.898142] sdhci-pltfm: SDHCI platform and OF driver helper1621worker2 # [ 0.902332] hid: raw HID events driver (C) Jiri Kosina1622worker2 # [ 0.902588] usbcore: registered new interface driver usbhid1623worker2 # [ 0.902595] usbhid: USB HID core driver1624client # [ 0.918078] registered taskstats version 11625client # [ 0.918226] Loading compiled-in X.509 certificates1626lb # [ 0.923219] Demotion targets for Node 0: null1627lb # [ 0.923323] Key type .fscrypt registered1628lb # [ 0.923329] Key type fscrypt-provisioning registered1629worker2 # [ 0.905449] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1630lb # [ 0.923461] ima: No TPM chip found, activating TPM-bypass!1631lb # [ 0.923480] ima: Allocated hash algorithm: sha11632worker2 # [ 0.908072] drop_monitor: Initializing network drop monitor service1633lb # [ 0.926457] ima: No architecture policies found1634worker2 # [ 0.908242] NET: Registered PF_INET6 protocol family1635lb # [ 0.927625] input: gpio-keys as /devices/platform/gpio-keys/input/input01636worker2 # [ 0.910194] Segment Routing with IPv61637worker2 # [ 0.910225] In-situ OAM (IOAM) with IPv61638worker2 # [ 0.910263] NET: Registered PF_PACKET protocol family1639client # [ 0.927006] Demotion targets for Node 0: null1640client # [ 0.927114] Key type .fscrypt registered1641client # [ 0.927120] Key type fscrypt-provisioning registered1642client # [ 0.927213] ima: No TPM chip found, activating TPM-bypass!1643client # [ 0.927231] ima: Allocated hash algorithm: sha11644client # [ 0.927253] ima: No architecture policies found1645worker2 # [ 0.912087] 9pnet: Installing 9P2000 support1646worker2 # [ 0.912134] Key type dns_resolver registered1647client # [ 0.931263] input: gpio-keys as /devices/platform/gpio-keys/input/input01648worker2 # [ 0.918850] registered taskstats version 11649worker2 # [ 0.919003] Loading compiled-in X.509 certificates1650worker2 # [ 0.927663] Demotion targets for Node 0: null1651worker2 # [ 0.927776] Key type .fscrypt registered1652worker2 # [ 0.927782] Key type fscrypt-provisioning registered1653lb # [ 0.945062] clk: Disabling unused clocks1654worker2 # [ 0.927878] ima: No TPM chip found, activating TPM-bypass!1655lb # [ 0.945092] PM: genpd: Disabling unused power domains1656worker2 # [ 0.927896] ima: Allocated hash algorithm: sha11657worker2 # [ 0.927925] ima: No architecture policies found1658lb # [ 0.949257] Freeing unused kernel memory: 4736K1659lb # [ 0.949479] Run /init as init process1660worker2 # [ 0.932166] input: gpio-keys as /devices/platform/gpio-keys/input/input01661client # [ 0.948999] clk: Disabling unused clocks1662client # [ 0.949029] PM: genpd: Disabling unused power domains1663client # [ 0.953247] Freeing unused kernel memory: 4736K1664client # [ 0.953427] Run /init as init process1665lb # [ 0.965775] systemd[1]: Successfully made /usr/ read-only.1666worker2 # [ 0.950789] clk: Disabling unused clocks1667worker2 # [ 0.950818] PM: genpd: Disabling unused power domains1668client # [ 0.971034] systemd[1]: Successfully made /usr/ read-only.1669worker2 # [ 0.954939] Freeing unused kernel memory: 4736K1670worker2 # [ 0.955144] Run /init as init process1671worker2 # [ 0.969057] systemd[1]: Successfully made /usr/ read-only.1672worker1 # [ 1.142134] usb 1-1: new high-speed USB device number 2 using ehci-pci1673lb # [ 1.131545] usb 1-1: new high-speed USB device number 2 using ehci-pci1674client # [ 1.148831] usb 1-1: new high-speed USB device number 2 using ehci-pci1675worker2 # [ 1.134911] usb 1-1: new high-speed USB device number 2 using ehci-pci1676worker1 # [ 1.294379] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11677worker1 # [ 1.311122] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1678worker1 # [ 1.323281] systemd[1]: Detected virtualization qemu.1679worker1 # [ 1.325263] systemd[1]: Detected architecture arm64.1680worker1 # [ 1.327221] systemd[1]: Running in initrd.1681worker1 # [ 1.329807] systemd[1]: Initializing machine ID from random generator.1682worker1 # [ 1.332924] systemd[1]: Hostname set to <worker1>.1683lb # [ 1.284097] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11684client # [ 1.302976] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11685worker2 # [ 1.287438] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11686lb # [ 1.300581] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1687worker1 # [ 1.378360] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01688lb # [ 1.312826] systemd[1]: Detected virtualization qemu.1689lb # [ 1.314831] systemd[1]: Detected architecture arm64.1690lb # [ 1.316810] systemd[1]: Running in initrd.1691lb # [ 1.319563] systemd[1]: Initializing machine ID from random generator.1692client # [ 1.306700] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1693lb # [ 1.322392] systemd[1]: Hostname set to <lb>.1694client # [ 1.306770] systemd[1]: Detected virtualization qemu.1695client # [ 1.306857] systemd[1]: Detected architecture arm64.1696client # [ 1.306882] systemd[1]: Running in initrd.1697client # [ 1.307908] systemd[1]: Initializing machine ID from random generator.1698client # [ 1.308193] systemd[1]: Hostname set to <client>.1699worker2 # [ 1.303735] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1700worker2 # [ 1.315832] systemd[1]: Detected virtualization qemu.1701worker2 # [ 1.317815] systemd[1]: Detected architecture arm64.1702worker2 # [ 1.319752] systemd[1]: Running in initrd.1703worker2 # [ 1.322356] systemd[1]: Initializing machine ID from random generator.1704worker2 # [ 1.325244] systemd[1]: Hostname set to <worker2>.1705lb # [ 1.367791] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01706client # [ 1.389047] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01707worker2 # [ 1.375109] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01708worker1 # [ 1.502103] usb 1-2: new high-speed USB device number 3 using ehci-pci1709lb # [ 1.487541] usb 1-2: new high-speed USB device number 3 using ehci-pci1710worker2 # [ 1.494878] usb 1-2: new high-speed USB device number 3 using ehci-pci1711client # [ 1.512854] usb 1-2: new high-speed USB device number 3 using ehci-pci1712worker1 # [ 1.637330] systemd[1]: bpf-restrict-fs: LSM BPF program attached1713worker1 # [ 1.666207] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21714worker1 # [ 1.671890] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01715lb # [ 1.608985] systemd[1]: bpf-restrict-fs: LSM BPF program attached1716lb # [ 1.637486] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21717lb # [ 1.641315] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01718client # [ 1.643453] systemd[1]: bpf-restrict-fs: LSM BPF program attached1719worker2 # [ 1.630987] systemd[1]: bpf-restrict-fs: LSM BPF program attached1720client # [ 1.665251] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21721client # [ 1.670325] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01722worker2 # [ 1.659736] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21723lb # [ 1.681867] systemd[1]: Queued start job for default target Initrd Default Target.1724worker1 # [ 1.752209] systemd[1]: Queued start job for default target Initrd Default Target.1725worker2 # [ 1.665430] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01726lb # [ 1.689485] systemd[1]: Created slice Slice /system/modprobe.1727lb # [ 1.690487] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1728worker1 # [ 1.760456] systemd[1]: Created slice Slice /system/modprobe.1729lb # [ 1.691660] systemd[1]: Expecting device /dev/disk/by-label/nixos...1730lb # [ 1.692549] systemd[1]: Reached target Path Units.1731worker1 # [ 1.761657] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1732lb # [ 1.693242] systemd[1]: Reached target Slice Units.1733lb # [ 1.693949] systemd[1]: Reached target Swaps.1734worker1 # [ 1.763125] systemd[1]: Expecting device /dev/disk/by-label/nixos...1735lb # [ 1.694586] systemd[1]: Reached target Timer Units.1736worker1 # [ 1.764217] systemd[1]: Reached target Path Units.1737lb # [ 1.695428] systemd[1]: Listening on D-Bus System Message Bus Socket.1738worker1 # [ 1.765050] systemd[1]: Reached target Slice Units.1739worker1 # [ 1.765893] systemd[1]: Reached target Swaps.1740lb # [ 1.696590] systemd[1]: Listening on Journal Socket (/dev/log).1741worker1 # [ 1.766723] systemd[1]: Reached target Timer Units.1742lb # [ 1.697541] systemd[1]: Listening on Journal Sockets.1743lb # [ 1.698383] systemd[1]: Listening on udev Control Socket.1744worker1 # [ 1.767762] systemd[1]: Listening on D-Bus System Message Bus Socket.1745lb # [ 1.699306] systemd[1]: Listening on udev Kernel Socket.1746worker1 # [ 1.769001] systemd[1]: Listening on Journal Socket (/dev/log).1747lb # [ 1.700120] systemd[1]: Reached target Socket Units.1748worker1 # [ 1.770175] systemd[1]: Listening on Journal Sockets.1749worker1 # [ 1.771267] systemd[1]: Listening on udev Control Socket.1750lb # [ 1.702219] systemd[1]: Starting Create List of Static Device Nodes...1751worker1 # [ 1.772311] systemd[1]: Listening on udev Kernel Socket.1752worker1 # [ 1.773234] systemd[1]: Reached target Socket Units.1753lb # [ 1.703203] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1754worker1 # [ 1.775728] systemd[1]: Starting Create List of Static Device Nodes...1755worker1 # [ 1.775811] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1756lb # [ 1.711361] systemd[1]: Mounting Kernel Configuration File System...1757worker1 # [ 1.786218] systemd[1]: Mounting Kernel Configuration File System...1758lb # [ 1.723842] systemd[1]: Starting Journal Service...1759worker1 # [ 1.794331] systemd[1]: Starting Journal Service...1760lb # [ 1.744355] systemd[1]: Starting Load Kernel Modules...1761lb # [ 1.744450] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1762lb # [ 1.747886] systemd[1]: Starting Coldplug All udev Devices...1763worker1 # [ 1.818016] systemd[1]: Starting Load Kernel Modules...1764worker1 # [ 1.819141] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1765client # [ 1.756014] systemd[1]: Queued start job for default target Initrd Default Target.1766worker2 # [ 1.747814] systemd[1]: Queued start job for default target Initrd Default Target.1767lb # [ 1.768940] systemd[1]: Finished Create List of Static Device Nodes.1768client # [ 1.768444] systemd[1]: Created slice Slice /system/modprobe.1769client # [ 1.769686] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1770client # [ 1.770981] systemd[1]: Expecting device /dev/disk/by-label/nixos...1771client # [ 1.772047] systemd[1]: Reached target Path Units.1772client # [ 1.772885] systemd[1]: Reached target Slice Units.1773worker2 # [ 1.757296] systemd[1]: Created slice Slice /system/modprobe.1774client # [ 1.773711] systemd[1]: Reached target Swaps.1775client # [ 1.774452] systemd[1]: Reached target Timer Units.1776worker2 # [ 1.758513] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1777client # [ 1.775565] systemd[1]: Listening on D-Bus System Message Bus Socket.1778worker2 # [ 1.759934] systemd[1]: Expecting device /dev/disk/by-label/nixos...1779worker1 # [ 1.846423] systemd[1]: Starting Coldplug All udev Devices...1780worker2 # [ 1.761004] systemd[1]: Reached target Path Units.1781client # [ 1.776851] systemd[1]: Listening on Journal Socket (/dev/log).1782worker2 # [ 1.761819] systemd[1]: Reached target Slice Units.1783client # [ 1.777982] systemd[1]: Listening on Journal Sockets.1784worker2 # [ 1.762642] systemd[1]: Reached target Swaps.1785worker1 # [ 1.849860] systemd-journald[72]: Collecting audit messages is disabled.1786client # [ 1.778974] systemd[1]: Listening on udev Control Socket.1787worker2 # [ 1.763450] systemd[1]: Reached target Timer Units.1788client # [ 1.780009] systemd[1]: Listening on udev Kernel Socket.1789worker2 # [ 1.764472] systemd[1]: Listening on D-Bus System Message Bus Socket.1790worker2 # [ 1.765695] systemd[1]: Listening on Journal Socket (/dev/log).1791client # [ 1.780934] systemd[1]: Reached target Socket Units.1792worker2 # [ 1.767011] systemd[1]: Listening on Journal Sockets.1793client # [ 1.783389] systemd[1]: Starting Create List of Static Device Nodes...1794worker2 # [ 1.768030] systemd[1]: Listening on udev Control Socket.1795worker2 # [ 1.769071] systemd[1]: Listening on udev Kernel Socket.1796client # [ 1.784553] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1797worker2 # [ 1.769985] systemd[1]: Reached target Socket Units.1798lb # [ 1.787844] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1799lb # [ 1.788182] systemd[1]: Mounted Kernel Configuration File System.1800worker2 # [ 1.772477] systemd[1]: Starting Create List of Static Device Nodes...1801lb # [ 1.790465] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1802worker2 # [ 1.773645] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1803client # [ 1.792982] systemd[1]: Mounting Kernel Configuration File System...1804lb # [ 1.792350] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1805lb # [ 1.797374] systemd-journald[72]: Collecting audit messages is disabled.1806worker1 # [ 1.866165] systemd[1]: Finished Create List of Static Device Nodes.1807worker2 # [ 1.783028] systemd[1]: Mounting Kernel Configuration File System...1808worker1 # [ 1.871311] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1809worker1 # [ 1.871563] systemd[1]: Mounted Kernel Configuration File System.1810client # [ 1.804973] systemd[1]: Starting Journal Service...1811worker2 # [ 1.791165] systemd[1]: Starting Journal Service...1812lb # [ 1.808806] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01813lb # [ 1.809048] [drm] features: -virgl +edid -resource_blob -host_visible1814lb # [ 1.809059] [drm] features: -context_init1815lb # [ 1.809792] [drm] number of scanouts: 11816lb # [ 1.809810] [drm] number of cap sets: 01817client # [ 1.821038] systemd[1]: Starting Load Kernel Modules...1818client # [ 1.821952] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1819lb # [ 1.823794] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1820lb # [ 1.823812] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01821worker2 # [ 1.816339] systemd[1]: Starting Load Kernel Modules...1822worker2 # [ 1.817322] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1823worker1 # [ 1.904189] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1824worker1 # [ 1.918658] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1825worker1 # [ 1.921166] systemd[1]: Starting Create Static Device Nodes in /dev...1826worker1 # [ 1.923012] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1827lb # [ 1.856173] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1828worker2 # [ 1.843744] systemd-journald[72]: Collecting audit messages is disabled.1829worker1 # [ 1.931425] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01830worker1 # [ 1.931661] [drm] features: -virgl +edid -resource_blob -host_visible1831worker1 # [ 1.931670] [drm] features: -context_init1832worker1 # [ 1.932357] [drm] number of scanouts: 11833worker1 # [ 1.932374] [drm] number of cap sets: 01834worker2 # [ 1.847272] systemd[1]: Starting Coldplug All udev Devices...1835client # [ 1.861048] systemd[1]: Starting Coldplug All udev Devices...1836lb # [ 1.859781] Console: switching to colour frame buffer device 160x501837worker2 # [ 1.851042] systemd[1]: Finished Create List of Static Device Nodes.1838client # [ 1.865726] systemd[1]: Finished Create List of Static Device Nodes.1839worker2 # [ 1.852541] systemd[1]: Mounted Kernel Configuration File System.1840lb # [ 1.868220] systemd[1]: Starting Create Static Device Nodes in /dev...1841client # [ 1.870044] systemd-journald[72]: Collecting audit messages is disabled.1842client # [ 1.873288] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1843client # [ 1.874540] systemd[1]: Mounted Kernel Configuration File System.1844lb # [ 1.884170] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1845worker2 # [ 1.866985] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1846client # [ 1.882719] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1847worker1 # [ 1.958504] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1848worker1 # [ 1.958525] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01849lb # [ 1.900796] systemd[1]: Finished Load Kernel Modules.1850client # [ 1.900972] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1851lb # [ 1.904073] systemd[1]: Starting Apply Kernel Variables...1852client # [ 1.910062] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01853client # [ 1.910298] [drm] features: -virgl +edid -resource_blob -host_visible1854client # [ 1.910308] [drm] features: -context_init1855client # [ 1.911019] [drm] number of scanouts: 11856client # [ 1.911037] [drm] number of cap sets: 01857worker1 # [ 1.982368] Console: switching to colour frame buffer device 160x501858worker2 # [ 1.903474] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1859lb # [ 1.922706] systemd[1]: Finished Create Static Device Nodes in /dev.1860worker2 # [ 1.907219] systemd[1]: Starting Create Static Device Nodes in /dev...1861lb # [ 1.925097] systemd[1]: Reached target Preparation for Local File Systems.1862worker1 # [ 2.002838] systemd[1]: Finished Create Static Device Nodes in /dev.1863worker1 # [ 2.003069] systemd[1]: Reached target Preparation for Local File Systems.1864worker1 # [ 2.003095] systemd[1]: Reached target Local File Systems.1865worker1 # [ 2.007119] systemd[1]: Starting Rule-based Manager for Device Events and Files...1866client # [ 1.937125] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1867client # [ 1.937150] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01868lb # [ 1.939674] systemd[1]: Reached target Local File Systems.1869worker2 # [ 1.926072] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1870lb # [ 1.941405] systemd[1]: Starting Rule-based Manager for Device Events and Files...1871client # [ 1.943242] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1872worker1 # [ 2.014825] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1873worker1 # [ 2.022129] systemd[1]: Started Journal Service.1874client # [ 1.949187] systemd[1]: Starting Create Static Device Nodes in /dev...1875worker1 # [ 2.016440] systemd-modules-load[74]: Inserted module 'dm_mod'1876worker1 # [ 2.017720] systemd-modules-load[74]: Module 'virtio_balloon' is built in1877lb # [ 1.967808] systemd[1]: Finished Apply Kernel Variables.1878worker1 # [ 2.020252] systemd-modules-load[74]: Module 'virtio_console' is built in1879worker1 # [ 2.024391] systemd-modules-load[74]: Inserted module 'virtio_gpu'1880lb # [ 1.973188] systemd[1]: Started Journal Service.1881worker2 # [ 1.951103] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1882worker2 # [ 1.957452] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01883worker2 # [ 1.957693] [drm] features: -virgl +edid -resource_blob -host_visible1884client # [ 1.969118] Console: switching to colour frame buffer device 160x501885worker2 # [ 1.957704] [drm] features: -context_init1886worker2 # [ 1.958384] [drm] number of scanouts: 11887worker2 # [ 1.958402] [drm] number of cap sets: 01888client # [ 1.975821] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1889worker1 # [ 2.032410] systemd-modules-load[74]: Module 'virtio_rng' is built in1890worker1 # [ 2.036325] systemd[1]: Finished Load Kernel Modules.1891lb # [ 1.972286] systemd-modules-load[73]: Inserted module 'dm_mod'1892worker1 # [ 2.040335] systemd[1]: Starting Apply Kernel Variables...1893lb # [ 1.973367] systemd-modules-load[73]: Module 'virtio_balloon' is built in1894lb # [ 1.974572] systemd-modules-load[73]: Module 'virtio_console' is built in1895worker2 # [ 1.970978] systemd[1]: Finished Create Static Device Nodes in /dev.1896worker2 # [ 1.971363] systemd[1]: Reached target Preparation for Local File Systems.1897lb # [ 1.975624] systemd-modules-load[73]: Inserted module 'virtio_gpu'1898worker1 # [ 2.042676] systemd[1]: Starting Create System Files and Directories...1899worker2 # [ 1.971393] systemd[1]: Reached target Local File Systems.1900lb # [ 1.980594] systemd-modules-load[73]: Module 'virtio_rng' is built in1901worker2 # [ 1.975260] systemd[1]: Starting Rule-based Manager for Device Events and Files...1902lb # [ 1.985351] systemd[1]: Starting Create System Files and Directories...1903client # [ 1.998155] systemd[1]: Finished Load Kernel Modules.1904client # [ 2.001323] systemd[1]: Starting Apply Kernel Variables...1905worker2 # [ 1.987163] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1906worker2 # [ 1.987180] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01907client # [ 2.017893] systemd[1]: Finished Create Static Device Nodes in /dev.1908client # [ 2.018293] systemd[1]: Reached target Preparation for Local File Systems.1909client # [ 2.018331] systemd[1]: Reached target Local File Systems.1910client # [ 2.023436] systemd[1]: Starting Rule-based Manager for Device Events and Files...1911lb # [ 2.014993] systemd[1]: Finished Create System Files and Directories.1912worker2 # [ 2.007098] Console: switching to colour frame buffer device 160x501913lb # [ 2.024657] systemd-udevd[80]: Using default interface naming scheme 'v261'.1914worker1 # [ 2.093554] systemd[1]: Finished Apply Kernel Variables.1915worker1 # [ 2.097085] systemd[1]: Finished Create System Files and Directories.1916worker2 # [ 2.016515] systemd-modules-load[74]: Inserted module 'dm_mod'1917worker2 # [ 2.032497] systemd[1]: Started Journal Service.1918client # [ 2.049071] systemd[1]: Started Journal Service.1919worker2 # [ 2.019705] systemd-modules-load[74]: Module 'virtio_balloon' is built in1920worker1 # [ 2.104517] systemd-udevd[78]: Using default interface naming scheme 'v261'.1921worker2 # [ 2.024503] systemd-modules-load[74]: Module 'virtio_console' is built in1922worker2 # [ 2.025682] systemd[1]: Starting Create System Files and Directories...1923lb # [ 2.046711] systemd[1]: Started Rule-based Manager for Device Events and Files.1924worker2 # [ 2.047605] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1925client # [ 2.054712] systemd-modules-load[73]: Inserted module 'dm_mod'1926client # [ 2.057524] systemd-modules-load[73]: Module 'virtio_balloon' is built in1927worker2 # [ 2.044857] systemd-modules-load[74]: Inserted module 'virtio_gpu'1928worker2 # [ 2.045865] systemd-modules-load[74]: Module 'virtio_rng' is built in1929worker1 # [ 2.130094] systemd[1]: Started Rule-based Manager for Device Events and Files.1930client # [ 2.065237] systemd-modules-load[73]: Module 'virtio_console' is built in1931client # [ 2.066369] systemd-modules-load[73]: Inserted module 'virtio_gpu'1932client # [ 2.067367] systemd-modules-load[73]: Module 'virtio_rng' is built in1933client # [ 2.074765] systemd[1]: Starting Create System Files and Directories...1934worker2 # [ 2.061337] systemd[1]: Finished Load Kernel Modules.1935client # [ 2.084364] systemd[1]: Finished Apply Kernel Variables.1936worker2 # [ 2.072318] systemd[1]: Starting Apply Kernel Variables...1937client # [ 2.093532] systemd[1]: Finished Create System Files and Directories.1938worker2 # [ 2.077397] systemd-udevd[78]: Using default interface naming scheme 'v261'.1939worker2 # [ 2.089770] systemd[1]: Finished Create System Files and Directories.1940lb # [ 2.108114] systemd[1]: Starting Virtual Console Setup...1941client # [ 2.113281] systemd-udevd[80]: Using default interface naming scheme 'v261'.1942worker1 # [ 2.192123] systemd[1]: Starting Virtual Console Setup...1943worker2 # [ 2.111896] systemd[1]: Finished Apply Kernel Variables.1944client # [ 2.141728] systemd[1]: Started Rule-based Manager for Device Events and Files.1945worker2 # [ 2.124633] systemd[1]: Started Rule-based Manager for Device Events and Files.1946lb # [ 2.152440] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1947lb # [ 2.155835] systemd[1]: Finished Virtual Console Setup.1948worker1 # [ 2.240555] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1949worker1 # [ 2.244071] systemd[1]: Finished Virtual Console Setup.1950worker2 # [ 2.172526] systemd[1]: Starting Virtual Console Setup...1951client # [ 2.196169] systemd[1]: Starting Virtual Console Setup...1952worker2 # [ 2.220445] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1953worker2 # [ 2.223135] systemd[1]: Finished Virtual Console Setup.1954client # [ 2.245243] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1955client # [ 2.247939] systemd[1]: Finished Virtual Console Setup.1956worker1 # [ 2.825574] systemd[1]: Finished Coldplug All udev Devices.1957worker1 # [ 2.826538] systemd[1]: Reached target System Initialization.1958worker1 # [ 2.828095] systemd[1]: Reached target Basic System.1959lb # [ 2.778223] systemd[1]: Finished Coldplug All udev Devices.1960lb # [ 2.779449] systemd[1]: Reached target System Initialization.1961lb # [ 2.780492] systemd[1]: Reached target Basic System.1962worker2 # [ 2.825262] systemd[1]: Finished Coldplug All udev Devices.1963worker2 # [ 2.826206] systemd[1]: Reached target System Initialization.1964worker2 # [ 2.827012] systemd[1]: Reached target Basic System.1965client # [ 2.867636] systemd[1]: Finished Coldplug All udev Devices.1966client # [ 2.868623] systemd[1]: Reached target System Initialization.1967client # [ 2.869412] systemd[1]: Reached target Basic System.1968lb # [ 2.923997] (udev-worker)[93]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1969worker1 # [ 2.998764] (udev-worker)[99]: Network interface NamePolicy= disabled on kernel command line.1970worker1 # [ 3.010806] (udev-worker)[100]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1971worker1 # [ 3.015350] (udev-worker)[100]: Network interface NamePolicy= disabled on kernel command line.1972lb # [ 2.957012] (udev-worker)[104]: Network interface NamePolicy= disabled on kernel command line.1973lb # [ 2.967069] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line.1974worker2 # [ 3.011022] (udev-worker)[94]: Network interface NamePolicy= disabled on kernel command line.1975lb # [ 3.029994] systemd[1]: Found device /dev/disk/by-label/nixos.1976client # [ 3.031476] (udev-worker)[104]: Network interface NamePolicy= disabled on kernel command line.1977lb # [ 3.035924] systemd[1]: Reached target Initrd Root Device.1978worker1 # [ 3.106861] systemd[1]: Found device /dev/disk/by-label/nixos.1979lb # [ 3.040655] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1980worker1 # [ 3.107945] systemd[1]: Reached target Initrd Root Device.1981worker1 # [ 3.112094] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1982worker2 # [ 3.028444] (udev-worker)[93]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1983client # [ 3.048325] (udev-worker)[93]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1984worker2 # [ 3.032565] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line.1985client # [ 3.052462] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line.1986worker1 # [ 3.160590] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks1987lb # [ 3.095528] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks1988worker1 # [ 3.167908] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1989lb # [ 3.102475] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1990worker1 # [ 3.172906] systemd[1]: Mounting /sysroot...1991lb # [ 3.108330] systemd[1]: Mounting /sysroot...1992client # [ 3.128186] systemd[1]: Found device /dev/disk/by-label/nixos.1993client # [ 3.132560] systemd[1]: Reached target Initrd Root Device.1994worker2 # [ 3.117435] systemd[1]: Found device /dev/disk/by-label/nixos.1995client # [ 3.137258] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1996worker2 # [ 3.120750] systemd[1]: Reached target Initrd Root Device.1997worker2 # [ 3.124856] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1998worker1 # [ 3.226361] EXT4-fs (vda): mounted filesystem 9d48dbc4-4996-454b-9856-73a0894ebe23 r/w with ordered data mode. Quota mode: none.1999lb # [ 3.157530] EXT4-fs (vda): mounted filesystem 3c3aa78d-c8ea-47b8-ab27-a5bfbb668221 r/w with ordered data mode. Quota mode: none.2000lb # [ 3.147191] systemd[1]: Mounted /sysroot.
loading 1,646 hidden lines…
3647worker1 # [ 20.150878] nix-grpc-daemon[770]: ts=2026-09-21T13:22:01Z level=info event=scheduler_connected addr=lb:500513648worker2 # [ 20.070128] nix-grpc-daemon[772]: ts=2026-09-21T13:22:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3649worker1 # [ 20.155285] nix-grpc-daemon[770]: ts=2026-09-21T13:22:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3650worker1 # [ 20.351569] nix-grpc-daemon[770]: ts=2026-09-21T13:22:01Z level=info event=scheduler_connected addr=lb:500513651worker2 # [ 20.271860] nix-grpc-daemon[772]: ts=2026-09-21T13:22:01Z level=info event=scheduler_connected addr=lb:500513652worker1 # [ 20.357449] nix-grpc-daemon[770]: ts=2026-09-21T13:22:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3653worker2 # [ 20.278177] nix-grpc-daemon[772]: ts=2026-09-21T13:22:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3654worker1 # [ 20.558952] nix-grpc-daemon[770]: ts=2026-09-21T13:22:02Z level=info event=scheduler_connected addr=lb:500513655worker2 # [ 20.479310] nix-grpc-daemon[772]: ts=2026-09-21T13:22:02Z level=info event=scheduler_connected addr=lb:500513656worker1 # [ 20.565416] nix-grpc-daemon[770]: ts=2026-09-21T13:22:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3657worker2 # [ 20.484951] nix-grpc-daemon[772]: ts=2026-09-21T13:22:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3658worker1 # [ 20.766612] nix-grpc-daemon[770]: ts=2026-09-21T13:22:02Z level=info event=scheduler_connected addr=lb:500513659worker2 # [ 20.687424] nix-grpc-daemon[772]: ts=2026-09-21T13:22:02Z level=info event=scheduler_connected addr=lb:500513660worker1 # [ 20.776425] nix-grpc-daemon[770]: ts=2026-09-21T13:22:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3661worker2 # [ 20.693669] nix-grpc-daemon[772]: ts=2026-09-21T13:22:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3662worker1 # [ 20.975233] nix-grpc-daemon[770]: ts=2026-09-21T13:22:02Z level=info event=scheduler_connected addr=lb:500513663worker2 # [ 20.895898] nix-grpc-daemon[772]: ts=2026-09-21T13:22:02Z level=info event=scheduler_connected addr=lb:500513664worker1 # [ 20.981719] nix-grpc-daemon[770]: ts=2026-09-21T13:22:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3665worker2 # [ 20.901759] nix-grpc-daemon[772]: ts=2026-09-21T13:22:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3666worker1 # [ 21.185079] nix-grpc-daemon[770]: ts=2026-09-21T13:22:02Z level=info event=scheduler_connected addr=lb:500513667worker2 # [ 21.101525] nix-grpc-daemon[772]: ts=2026-09-21T13:22:02Z level=info event=scheduler_connected addr=lb:500513668worker2 # [ 21.106479] nix-grpc-daemon[772]: ts=2026-09-21T13:22:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3669worker1 # [ 21.190555] nix-grpc-daemon[770]: ts=2026-09-21T13:22:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3670worker2 # [ 21.307906] nix-grpc-daemon[772]: ts=2026-09-21T13:22:02Z level=info event=scheduler_connected addr=lb:500513671worker1 # [ 21.392409] nix-grpc-daemon[770]: ts=2026-09-21T13:22:03Z level=info event=scheduler_connected addr=lb:500513672worker2 # [ 21.313546] nix-grpc-daemon[772]: ts=2026-09-21T13:22:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3673worker1 # [ 21.399082] nix-grpc-daemon[770]: ts=2026-09-21T13:22:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3674worker2 # [ 21.516156] nix-grpc-daemon[772]: ts=2026-09-21T13:22:03Z level=info event=scheduler_connected addr=lb:500513675worker1 # [ 21.599829] nix-grpc-daemon[770]: ts=2026-09-21T13:22:03Z level=info event=scheduler_connected addr=lb:500513676worker1 # [ 21.605538] nix-grpc-daemon[770]: ts=2026-09-21T13:22:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3677worker2 # [ 21.521905] nix-grpc-daemon[772]: ts=2026-09-21T13:22:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3678worker1 # [ 21.806801] nix-grpc-daemon[770]: ts=2026-09-21T13:22:03Z level=info event=scheduler_connected addr=lb:500513679worker2 # [ 21.724629] nix-grpc-daemon[772]: ts=2026-09-21T13:22:03Z level=info event=scheduler_connected addr=lb:500513680worker1 # [ 21.812639] nix-grpc-daemon[770]: ts=2026-09-21T13:22:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3681worker2 # [ 21.730906] nix-grpc-daemon[772]: ts=2026-09-21T13:22:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3682worker1 # [ 22.013520] nix-grpc-daemon[770]: ts=2026-09-21T13:22:03Z level=info event=scheduler_connected addr=lb:500513683worker2 # [ 21.935301] nix-grpc-daemon[772]: ts=2026-09-21T13:22:03Z level=info event=scheduler_connected addr=lb:500513684worker1 # [ 22.019856] nix-grpc-daemon[770]: ts=2026-09-21T13:22:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3685worker2 # [ 21.940612] nix-grpc-daemon[772]: ts=2026-09-21T13:22:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3686worker1 # [ 22.222985] nix-grpc-daemon[770]: ts=2026-09-21T13:22:03Z level=info event=scheduler_connected addr=lb:500513687worker1 # [ 22.227528] nix-grpc-daemon[770]: ts=2026-09-21T13:22:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3688worker2 # [ 22.146244] nix-grpc-daemon[772]: ts=2026-09-21T13:22:03Z level=info event=scheduler_connected addr=lb:500513689worker2 # [ 22.150821] nix-grpc-daemon[772]: ts=2026-09-21T13:22:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3690worker1 # [ 22.424572] nix-grpc-daemon[770]: ts=2026-09-21T13:22:04Z level=info event=scheduler_connected addr=lb:500513691worker1 # [ 22.430261] nix-grpc-daemon[770]: ts=2026-09-21T13:22:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3692worker2 # [ 22.350007] nix-grpc-daemon[772]: ts=2026-09-21T13:22:03Z level=info event=scheduler_connected addr=lb:500513693worker2 # [ 22.355794] nix-grpc-daemon[772]: ts=2026-09-21T13:22:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3694worker1 # [ 22.631429] nix-grpc-daemon[770]: ts=2026-09-21T13:22:04Z level=info event=scheduler_connected addr=lb:500513695worker1 # [ 22.637539] nix-grpc-daemon[770]: ts=2026-09-21T13:22:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3696worker2 # [ 22.556760] nix-grpc-daemon[772]: ts=2026-09-21T13:22:04Z level=info event=scheduler_connected addr=lb:500513697worker2 # [ 22.562307] nix-grpc-daemon[772]: ts=2026-09-21T13:22:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3698worker1 # [ 22.838539] nix-grpc-daemon[770]: ts=2026-09-21T13:22:04Z level=info event=scheduler_connected addr=lb:500513699worker1 # [ 22.844391] nix-grpc-daemon[770]: ts=2026-09-21T13:22:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3700worker2 # [ 22.764413] nix-grpc-daemon[772]: ts=2026-09-21T13:22:04Z level=info event=scheduler_connected addr=lb:500513701worker2 # [ 22.768560] nix-grpc-daemon[772]: ts=2026-09-21T13:22:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3702worker1 # [ 23.045616] nix-grpc-daemon[770]: ts=2026-09-21T13:22:04Z level=info event=scheduler_connected addr=lb:500513703worker2 # [ 22.965863] nix-grpc-daemon[772]: ts=2026-09-21T13:22:04Z level=info event=scheduler_connected addr=lb:500513704worker1 # [ 23.051344] nix-grpc-daemon[770]: ts=2026-09-21T13:22:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3705worker2 # [ 22.971230] nix-grpc-daemon[772]: ts=2026-09-21T13:22:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3706worker1 # [ 23.255745] nix-grpc-daemon[770]: ts=2026-09-21T13:22:04Z level=info event=scheduler_connected addr=lb:500513707worker2 # [ 23.175316] nix-grpc-daemon[772]: ts=2026-09-21T13:22:04Z level=info event=scheduler_connected addr=lb:500513708worker1 # [ 23.260320] nix-grpc-daemon[770]: ts=2026-09-21T13:22:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3709worker2 # [ 23.179731] nix-grpc-daemon[772]: ts=2026-09-21T13:22:04Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3710worker1 # [ 23.455887] nix-grpc-daemon[770]: ts=2026-09-21T13:22:05Z level=info event=scheduler_connected addr=lb:500513711worker2 # [ 23.375392] nix-grpc-daemon[772]: ts=2026-09-21T13:22:04Z level=info event=scheduler_connected addr=lb:500513712worker1 # [ 23.461759] nix-grpc-daemon[770]: ts=2026-09-21T13:22:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3713worker2 # [ 23.380594] nix-grpc-daemon[772]: ts=2026-09-21T13:22:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3714worker1 # [ 23.662365] nix-grpc-daemon[770]: ts=2026-09-21T13:22:05Z level=info event=scheduler_connected addr=lb:500513715worker2 # [ 23.581867] nix-grpc-daemon[772]: ts=2026-09-21T13:22:05Z level=info event=scheduler_connected addr=lb:500513716worker1 # [ 23.667588] nix-grpc-daemon[770]: ts=2026-09-21T13:22:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3717worker2 # [ 23.587077] nix-grpc-daemon[772]: ts=2026-09-21T13:22:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3718worker1 # [ 23.868505] nix-grpc-daemon[770]: ts=2026-09-21T13:22:05Z level=info event=scheduler_connected addr=lb:500513719worker2 # [ 23.788109] nix-grpc-daemon[772]: ts=2026-09-21T13:22:05Z level=info event=scheduler_connected addr=lb:500513720worker1 # [ 23.873815] nix-grpc-daemon[770]: ts=2026-09-21T13:22:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3721worker2 # [ 23.793167] nix-grpc-daemon[772]: ts=2026-09-21T13:22:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3722worker1 # [ 24.074516] nix-grpc-daemon[770]: ts=2026-09-21T13:22:05Z level=info event=scheduler_connected addr=lb:500513723worker2 # [ 23.994364] nix-grpc-daemon[772]: ts=2026-09-21T13:22:05Z level=info event=scheduler_connected addr=lb:500513724worker1 # [ 24.084696] nix-grpc-daemon[770]: ts=2026-09-21T13:22:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3725worker2 # [ 23.999524] nix-grpc-daemon[772]: ts=2026-09-21T13:22:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3726worker1 # [ 24.283231] nix-grpc-daemon[770]: ts=2026-09-21T13:22:05Z level=info event=scheduler_connected addr=lb:500513727worker2 # [ 24.202466] nix-grpc-daemon[772]: ts=2026-09-21T13:22:05Z level=info event=scheduler_connected addr=lb:500513728worker1 # [ 24.287604] nix-grpc-daemon[770]: ts=2026-09-21T13:22:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3729worker2 # [ 24.206625] nix-grpc-daemon[772]: ts=2026-09-21T13:22:05Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3730worker1 # [ 24.483885] nix-grpc-daemon[770]: ts=2026-09-21T13:22:06Z level=info event=scheduler_connected addr=lb:500513731worker2 # [ 24.403303] nix-grpc-daemon[772]: ts=2026-09-21T13:22:06Z level=info event=scheduler_connected addr=lb:500513732worker1 # [ 24.489271] nix-grpc-daemon[770]: ts=2026-09-21T13:22:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3733worker2 # [ 24.408644] nix-grpc-daemon[772]: ts=2026-09-21T13:22:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3734worker1 # [ 24.689973] nix-grpc-daemon[770]: ts=2026-09-21T13:22:06Z level=info event=scheduler_connected addr=lb:500513735worker2 # [ 24.609685] nix-grpc-daemon[772]: ts=2026-09-21T13:22:06Z level=info event=scheduler_connected addr=lb:500513736worker1 # [ 24.695171] nix-grpc-daemon[770]: ts=2026-09-21T13:22:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3737worker2 # [ 24.614839] nix-grpc-daemon[772]: ts=2026-09-21T13:22:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3738worker1 # [ 24.896046] nix-grpc-daemon[770]: ts=2026-09-21T13:22:06Z level=info event=scheduler_connected addr=lb:500513739worker2 # [ 24.816134] nix-grpc-daemon[772]: ts=2026-09-21T13:22:06Z level=info event=scheduler_connected addr=lb:500513740worker1 # [ 24.901598] nix-grpc-daemon[770]: ts=2026-09-21T13:22:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3741worker2 # [ 24.821181] nix-grpc-daemon[772]: ts=2026-09-21T13:22:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3742worker1 # [ 25.102385] nix-grpc-daemon[770]: ts=2026-09-21T13:22:06Z level=info event=scheduler_connected addr=lb:500513743worker2 # [ 25.022219] nix-grpc-daemon[772]: ts=2026-09-21T13:22:06Z level=info event=scheduler_connected addr=lb:500513744worker1 # [ 25.107742] nix-grpc-daemon[770]: ts=2026-09-21T13:22:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3745worker2 # [ 25.027912] nix-grpc-daemon[772]: ts=2026-09-21T13:22:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3746worker1 # [ 25.310509] nix-grpc-daemon[770]: ts=2026-09-21T13:22:06Z level=info event=scheduler_connected addr=lb:500513747worker2 # [ 25.230977] nix-grpc-daemon[772]: ts=2026-09-21T13:22:06Z level=info event=scheduler_connected addr=lb:500513748worker1 # [ 25.314877] nix-grpc-daemon[770]: ts=2026-09-21T13:22:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3749worker2 # [ 25.235168] nix-grpc-daemon[772]: ts=2026-09-21T13:22:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3750worker1 # [ 25.511315] nix-grpc-daemon[770]: ts=2026-09-21T13:22:07Z level=info event=scheduler_connected addr=lb:500513751worker2 # [ 25.431608] nix-grpc-daemon[772]: ts=2026-09-21T13:22:07Z level=info event=scheduler_connected addr=lb:500513752worker1 # [ 25.516606] nix-grpc-daemon[770]: ts=2026-09-21T13:22:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3753worker2 # [ 25.436893] nix-grpc-daemon[772]: ts=2026-09-21T13:22:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3754worker1 # [ 25.717424] nix-grpc-daemon[770]: ts=2026-09-21T13:22:07Z level=info event=scheduler_connected addr=lb:500513755worker2 # [ 25.638091] nix-grpc-daemon[772]: ts=2026-09-21T13:22:07Z level=info event=scheduler_connected addr=lb:500513756worker1 # [ 25.722521] nix-grpc-daemon[770]: ts=2026-09-21T13:22:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3757worker2 # [ 25.643210] nix-grpc-daemon[772]: ts=2026-09-21T13:22:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3758worker1 # [ 25.923543] nix-grpc-daemon[770]: ts=2026-09-21T13:22:07Z level=info event=scheduler_connected addr=lb:500513759worker2 # [ 25.844383] nix-grpc-daemon[772]: ts=2026-09-21T13:22:07Z level=info event=scheduler_connected addr=lb:500513760worker1 # [ 25.928990] nix-grpc-daemon[770]: ts=2026-09-21T13:22:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3761worker2 # [ 25.849481] nix-grpc-daemon[772]: ts=2026-09-21T13:22:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3762worker1 # [ 26.129618] nix-grpc-daemon[770]: ts=2026-09-21T13:22:07Z level=info event=scheduler_connected addr=lb:500513763worker2 # [ 26.050665] nix-grpc-daemon[772]: ts=2026-09-21T13:22:07Z level=info event=scheduler_connected addr=lb:500513764worker1 # [ 26.134755] nix-grpc-daemon[770]: ts=2026-09-21T13:22:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3765worker2 # [ 26.056176] nix-grpc-daemon[772]: ts=2026-09-21T13:22:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3766worker1 # [ 26.337473] nix-grpc-daemon[770]: ts=2026-09-21T13:22:07Z level=info event=scheduler_connected addr=lb:500513767worker1 # [ 26.342181] nix-grpc-daemon[770]: ts=2026-09-21T13:22:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3768worker2 # [ 26.259315] nix-grpc-daemon[772]: ts=2026-09-21T13:22:07Z level=info event=scheduler_connected addr=lb:500513769worker2 # [ 26.263666] nix-grpc-daemon[772]: ts=2026-09-21T13:22:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3770worker1 # [ 26.543045] nix-grpc-daemon[770]: ts=2026-09-21T13:22:08Z level=info event=scheduler_connected addr=lb:500513771worker2 # [ 26.460361] nix-grpc-daemon[772]: ts=2026-09-21T13:22:08Z level=info event=scheduler_connected addr=lb:500513772worker1 # [ 26.548575] nix-grpc-daemon[770]: ts=2026-09-21T13:22:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3773worker2 # [ 26.465350] nix-grpc-daemon[772]: ts=2026-09-21T13:22:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3774worker2 # [ 26.662673] nix-grpc-daemon[772]: ts=2026-09-21T13:22:08Z level=info event=scheduler_connected addr=lb:500513775worker1 # [ 26.749480] nix-grpc-daemon[770]: ts=2026-09-21T13:22:08Z level=info event=scheduler_connected addr=lb:500513776worker2 # [ 26.668306] nix-grpc-daemon[772]: ts=2026-09-21T13:22:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3777worker1 # [ 26.754753] nix-grpc-daemon[770]: ts=2026-09-21T13:22:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3778worker2 # [ 26.869530] nix-grpc-daemon[772]: ts=2026-09-21T13:22:08Z level=info event=scheduler_connected addr=lb:500513779worker1 # [ 26.955537] nix-grpc-daemon[770]: ts=2026-09-21T13:22:08Z level=info event=scheduler_connected addr=lb:500513780worker2 # [ 26.875001] nix-grpc-daemon[772]: ts=2026-09-21T13:22:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3781worker1 # [ 26.960857] nix-grpc-daemon[770]: ts=2026-09-21T13:22:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3782worker2 # [ 27.076090] nix-grpc-daemon[772]: ts=2026-09-21T13:22:08Z level=info event=scheduler_connected addr=lb:500513783worker1 # [ 27.161713] nix-grpc-daemon[770]: ts=2026-09-21T13:22:08Z level=info event=scheduler_connected addr=lb:500513784worker2 # [ 27.081436] nix-grpc-daemon[772]: ts=2026-09-21T13:22:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3785worker1 # [ 27.166762] nix-grpc-daemon[770]: ts=2026-09-21T13:22:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3786worker1 # [ 27.370421] nix-grpc-daemon[770]: ts=2026-09-21T13:22:08Z level=info event=scheduler_connected addr=lb:500513787worker2 # [ 27.285148] nix-grpc-daemon[772]: ts=2026-09-21T13:22:08Z level=info event=scheduler_connected addr=lb:500513788worker2 # [ 27.290918] nix-grpc-daemon[772]: ts=2026-09-21T13:22:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3789worker1 # [ 27.379258] nix-grpc-daemon[770]: ts=2026-09-21T13:22:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3790worker2 # [ 27.486058] nix-grpc-daemon[772]: ts=2026-09-21T13:22:09Z level=info event=scheduler_connected addr=lb:500513791worker1 # [ 27.570884] nix-grpc-daemon[770]: ts=2026-09-21T13:22:09Z level=info event=scheduler_connected addr=lb:500513792worker2 # [ 27.491443] nix-grpc-daemon[772]: ts=2026-09-21T13:22:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3793worker1 # [ 27.576318] nix-grpc-daemon[770]: ts=2026-09-21T13:22:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3794worker2 # [ 27.692326] nix-grpc-daemon[772]: ts=2026-09-21T13:22:09Z level=info event=scheduler_connected addr=lb:500513795worker1 # [ 27.777514] nix-grpc-daemon[770]: ts=2026-09-21T13:22:09Z level=info event=scheduler_connected addr=lb:500513796worker2 # [ 27.695695] nix-grpc-daemon[772]: ts=2026-09-21T13:22:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3797worker1 # [ 27.782375] nix-grpc-daemon[770]: ts=2026-09-21T13:22:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3798worker2 # [ 27.897009] nix-grpc-daemon[772]: ts=2026-09-21T13:22:09Z level=info event=scheduler_connected addr=lb:500513799worker1 # [ 27.983356] nix-grpc-daemon[770]: ts=2026-09-21T13:22:09Z level=info event=scheduler_connected addr=lb:500513800worker2 # [ 27.902238] nix-grpc-daemon[772]: ts=2026-09-21T13:22:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3801worker1 # [ 27.988555] nix-grpc-daemon[770]: ts=2026-09-21T13:22:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3802worker2 # [ 28.103291] nix-grpc-daemon[772]: ts=2026-09-21T13:22:09Z level=info event=scheduler_connected addr=lb:500513803worker1 # [ 28.189271] nix-grpc-daemon[770]: ts=2026-09-21T13:22:09Z level=info event=scheduler_connected addr=lb:500513804worker2 # [ 28.109002] nix-grpc-daemon[772]: ts=2026-09-21T13:22:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3805worker1 # [ 28.194478] nix-grpc-daemon[770]: ts=2026-09-21T13:22:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3806worker2 # [ 28.311804] nix-grpc-daemon[772]: ts=2026-09-21T13:22:09Z level=info event=scheduler_connected addr=lb:500513807worker1 # [ 28.397206] nix-grpc-daemon[770]: ts=2026-09-21T13:22:10Z level=info event=scheduler_connected addr=lb:500513808worker2 # [ 28.316304] nix-grpc-daemon[772]: ts=2026-09-21T13:22:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3809worker1 # [ 28.401330] nix-grpc-daemon[770]: ts=2026-09-21T13:22:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3810worker2 # [ 28.513206] nix-grpc-daemon[772]: ts=2026-09-21T13:22:10Z level=info event=scheduler_connected addr=lb:500513811worker1 # [ 28.597482] nix-grpc-daemon[770]: ts=2026-09-21T13:22:10Z level=info event=scheduler_connected addr=lb:500513812worker1 # [ 28.602745] nix-grpc-daemon[770]: ts=2026-09-21T13:22:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3813worker2 # [ 28.519407] nix-grpc-daemon[772]: ts=2026-09-21T13:22:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3814worker1 # [ 28.803726] nix-grpc-daemon[770]: ts=2026-09-21T13:22:10Z level=info event=scheduler_connected addr=lb:500513815worker2 # [ 28.720335] nix-grpc-daemon[772]: ts=2026-09-21T13:22:10Z level=info event=scheduler_connected addr=lb:500513816worker1 # [ 28.809293] nix-grpc-daemon[770]: ts=2026-09-21T13:22:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3817worker2 # [ 28.727238] nix-grpc-daemon[772]: ts=2026-09-21T13:22:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3818worker1 # [ 29.009855] nix-grpc-daemon[770]: ts=2026-09-21T13:22:10Z level=info event=scheduler_connected addr=lb:500513819worker2 # [ 28.928823] nix-grpc-daemon[772]: ts=2026-09-21T13:22:10Z level=info event=scheduler_connected addr=lb:500513820worker1 # [ 29.015121] nix-grpc-daemon[770]: ts=2026-09-21T13:22:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3821worker2 # [ 28.934283] nix-grpc-daemon[772]: ts=2026-09-21T13:22:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3822worker1 # [ 29.215992] nix-grpc-daemon[770]: ts=2026-09-21T13:22:10Z level=info event=scheduler_connected addr=lb:500513823worker2 # [ 29.135327] nix-grpc-daemon[772]: ts=2026-09-21T13:22:10Z level=info event=scheduler_connected addr=lb:500513824worker1 # [ 29.221567] nix-grpc-daemon[770]: ts=2026-09-21T13:22:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3825worker2 # [ 29.140935] nix-grpc-daemon[772]: ts=2026-09-21T13:22:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3826worker1 # [ 29.423565] nix-grpc-daemon[770]: ts=2026-09-21T13:22:11Z level=info event=scheduler_connected addr=lb:500513827worker2 # [ 29.343874] nix-grpc-daemon[772]: ts=2026-09-21T13:22:10Z level=info event=scheduler_connected addr=lb:500513828worker1 # [ 29.428821] nix-grpc-daemon[770]: ts=2026-09-21T13:22:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3829worker2 # [ 29.348228] nix-grpc-daemon[772]: ts=2026-09-21T13:22:10Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3830worker2 # [ 29.544382] nix-grpc-daemon[772]: ts=2026-09-21T13:22:11Z level=info event=scheduler_connected addr=lb:500513831worker1 # [ 29.629778] nix-grpc-daemon[770]: ts=2026-09-21T13:22:11Z level=info event=scheduler_connected addr=lb:500513832worker2 # [ 29.549778] nix-grpc-daemon[772]: ts=2026-09-21T13:22:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3833worker1 # [ 29.635009] nix-grpc-daemon[770]: ts=2026-09-21T13:22:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3834worker2 # [ 29.751030] nix-grpc-daemon[772]: ts=2026-09-21T13:22:11Z level=info event=scheduler_connected addr=lb:500513835worker1 # [ 29.835753] nix-grpc-daemon[770]: ts=2026-09-21T13:22:11Z level=info event=scheduler_connected addr=lb:500513836worker2 # [ 29.756194] nix-grpc-daemon[772]: ts=2026-09-21T13:22:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3837worker1 # [ 29.840758] nix-grpc-daemon[770]: ts=2026-09-21T13:22:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3838worker2 # [ 29.957349] nix-grpc-daemon[772]: ts=2026-09-21T13:22:11Z level=info event=scheduler_connected addr=lb:500513839worker1 # [ 30.041727] nix-grpc-daemon[770]: ts=2026-09-21T13:22:11Z level=info event=scheduler_connected addr=lb:500513840worker2 # [ 29.962854] nix-grpc-daemon[772]: ts=2026-09-21T13:22:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3841worker1 # [ 30.047057] nix-grpc-daemon[770]: ts=2026-09-21T13:22:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3842worker1 # [ 30.247692] nix-grpc-daemon[770]: ts=2026-09-21T13:22:11Z level=info event=scheduler_connected addr=lb:500513843worker2 # [ 30.163884] nix-grpc-daemon[772]: ts=2026-09-21T13:22:11Z level=info event=scheduler_connected addr=lb:500513844worker2 # [ 30.169184] nix-grpc-daemon[772]: ts=2026-09-21T13:22:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3845worker1 # [ 30.253259] nix-grpc-daemon[770]: ts=2026-09-21T13:22:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3846worker2 # [ 30.371849] nix-grpc-daemon[772]: ts=2026-09-21T13:22:11Z level=info event=scheduler_connected addr=lb:500513847worker1 # [ 30.455854] nix-grpc-daemon[770]: ts=2026-09-21T13:22:12Z level=info event=scheduler_connected addr=lb:500513848worker2 # [ 30.376200] nix-grpc-daemon[772]: ts=2026-09-21T13:22:11Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3849worker1 # [ 30.460153] nix-grpc-daemon[770]: ts=2026-09-21T13:22:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3850worker1 # [ 30.657082] nix-grpc-daemon[770]: ts=2026-09-21T13:22:12Z level=info event=scheduler_connected addr=lb:500513851worker2 # [ 30.573239] nix-grpc-daemon[772]: ts=2026-09-21T13:22:12Z level=info event=scheduler_connected addr=lb:500513852worker1 # [ 30.662296] nix-grpc-daemon[770]: ts=2026-09-21T13:22:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3853worker2 # [ 30.578521] nix-grpc-daemon[772]: ts=2026-09-21T13:22:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3854worker1 # [ 30.863305] nix-grpc-daemon[770]: ts=2026-09-21T13:22:12Z level=info event=scheduler_connected addr=lb:500513855worker2 # [ 30.779562] nix-grpc-daemon[772]: ts=2026-09-21T13:22:12Z level=info event=scheduler_connected addr=lb:500513856worker1 # [ 30.868511] nix-grpc-daemon[770]: ts=2026-09-21T13:22:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3857worker2 # [ 30.784606] nix-grpc-daemon[772]: ts=2026-09-21T13:22:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3858worker1 # [ 31.069289] nix-grpc-daemon[770]: ts=2026-09-21T13:22:12Z level=info event=scheduler_connected addr=lb:500513859worker2 # [ 30.986517] nix-grpc-daemon[772]: ts=2026-09-21T13:22:12Z level=info event=scheduler_connected addr=lb:500513860worker1 # [ 31.074545] nix-grpc-daemon[770]: ts=2026-09-21T13:22:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3861worker2 # [ 30.995877] nix-grpc-daemon[772]: ts=2026-09-21T13:22:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3862lb # [ 31.070266] rustfs-bucket-start[1025]: mb s3://farm3863lb # [ 31.083608] systemd[1]: Finished rustfs-bucket.service.3864lb # [ 31.095629] systemd[1]: Starting niks3 server...3865worker1 # [ 31.275607] nix-grpc-daemon[770]: ts=2026-09-21T13:22:12Z level=info event=scheduler_connected addr=lb:500513866worker2 # [ 31.195174] nix-grpc-daemon[772]: ts=2026-09-21T13:22:12Z level=info event=scheduler_connected addr=lb:500513867worker1 # [ 31.281367] nix-grpc-daemon[770]: ts=2026-09-21T13:22:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3868worker2 # [ 31.200267] nix-grpc-daemon[772]: ts=2026-09-21T13:22:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3869lb # [ 31.287987] postgres[1040]: [1040] ERROR: relation "goose_db_version" does not exist at character 363870lb # [ 31.292200] postgres[1040]: [1040] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC3871lb # [ 31.322218] niks3-server[1034]: 2026/09/21 13:22:12 OK 20241026095416_initial_model.sql (18.77ms)3872lb # [ 31.325957] niks3-server[1034]: 2026/09/21 13:22:12 OK 20251210153512_drop_unused_gin_index.sql (4.06ms)3873lb # [ 31.331515] niks3-server[1034]: 2026/09/21 13:22:12 OK 20251218171726_add_pins.sql (5.54ms)3874lb # [ 31.336192] niks3-server[1034]: 2026/09/21 13:22:12 OK 20260628120000_add_object_size_and_stats.sql (4.65ms)3875lb # [ 31.341075] niks3-server[1034]: 2026/09/21 13:22:12 OK 20260905000000_add_claims.sql (4.84ms)3876lb # [ 31.345006] niks3-server[1034]: 2026/09/21 13:22:12 OK 20260920000000_drop_claims.sql (3.84ms)3877lb # [ 31.347013] niks3-server[1034]: 2026/09/21 13:22:12 goose: successfully migrated database to version: 202609200000003878lb # [ 31.350723] niks3-server[1034]: 2026/09/21 13:22:12 OK 1_commit_pending_closure.sql (5.69ms)3879lb # [ 31.353262] niks3-server[1034]: 2026/09/21 13:22:12 OK 2_object_stats_trigger.sql (2.42ms)3880lb # [ 31.355151] niks3-server[1034]: 2026/09/21 13:22:12 goose: up to current file version: 23881lb # [ 31.357311] niks3-server[1034]: 2026/09/21 13:22:12 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key3882lb # [ 31.379550] niks3-server[1034]: 2026/09/21 13:22:13 INFO Read proxy enabled — serving cache objects from S33883lb # [ 31.383654] niks3-server[1034]: 2026/09/21 13:22:13 INFO Using socket-activated listener address=0.0.0.0:57513884lb # [ 31.393621] niks3-server[1034]: 2026/09/21 13:22:13 INFO systemd watchdog enabled interval=15s3885lb # [ 31.394783] niks3-server[1034]: 2026/09/21 13:22:13 INFO Starting HTTP server address=0.0.0.0:57513886lb # [ 31.399206] systemd[1]: Started niks3 server.3887lb # [ 31.399996] niks3-server[1034]: 2026/09/21 13:22:13 INFO lead: acquired remote=192.168.1.4:560283888worker2 # [ 31.381157] nix-grpc-daemon[772]: ts=2026-09-21T13:22:13Z level=info event=scheduler_take_over3889lb # [ 31.403183] systemd[1]: Reached target Multi-User System.3890lb # [ 31.404399] systemd[1]: Startup finished in 938ms (kernel) + 4.883s (initrd) + 25.569s (userspace) = 31.391s.3891worker2 # [ 31.397496] nix-grpc-daemon[772]: ts=2026-09-21T13:22:13Z level=info event=scheduler_connected addr=lb:500513892worker1 # [ 31.483847] nix-grpc-daemon[770]: ts=2026-09-21T13:22:13Z level=info event=scheduler_connected addr=lb:500513893worker2 # [ 31.402691] nix-grpc-daemon[772]: ts=2026-09-21T13:22:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3894worker1 # [ 31.488264] nix-grpc-daemon[770]: ts=2026-09-21T13:22:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3895lb: (finished: waiting for unit niks3.service, in 19.05 seconds)3896lb: waiting for TCP port 5751 on localhost3897lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded!3898worker1 # [ 31.684125] nix-grpc-daemon[770]: ts=2026-09-21T13:22:13Z level=info event=scheduler_connected addr=lb:500513899lb: (finished: waiting for TCP port 5751 on localhost, in 0.05 seconds)3900lb: waiting for unit envoy.service3901worker2 # [ 31.603777] nix-grpc-daemon[772]: ts=2026-09-21T13:22:13Z level=info event=scheduler_connected addr=lb:500513902worker1 # [ 31.689426] nix-grpc-daemon[770]: ts=2026-09-21T13:22:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3903worker2 # [ 31.609117] nix-grpc-daemon[772]: ts=2026-09-21T13:22:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3904lb: (finished: waiting for unit envoy.service, in 0.04 seconds)3905lb: waiting for TCP port 50051 on localhost3906lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded!3907lb # [ 31.685793] envoy[897]: conn peer=[::1]:44950 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=33908lb: (finished: waiting for TCP port 50051 on localhost, in 0.03 seconds)3909worker1 # [ 31.890284] nix-grpc-daemon[770]: ts=2026-09-21T13:22:13Z level=info event=scheduler_connected addr=lb:500513910worker2 # [ 31.810305] nix-grpc-daemon[772]: ts=2026-09-21T13:22:13Z level=info event=scheduler_connected addr=lb:500513911worker1 # [ 31.895667] nix-grpc-daemon[770]: ts=2026-09-21T13:22:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3912worker2 # [ 31.820406] nix-grpc-daemon[772]: ts=2026-09-21T13:22:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3913worker2 # [ 31.853482] systemd[1]: Starting scheduler-token.service...3914worker2 # [ 31.923773] systemd[1]: scheduler-token.service: Deactivated successfully.3915worker2 # [ 31.928249] systemd[1]: Finished scheduler-token.service.3916worker2 # [ 31.929699] systemd[1]: scheduler-token.service: Consumed 40ms CPU time over 73ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.3917lb: waiting until exactly no endpoints of aarch64-linux are unhealthy3918lb: must succeed: curl -sf localhost:9901/clusters3919worker1 # [ 32.097435] nix-grpc-daemon[770]: ts=2026-09-21T13:22:13Z level=info event=scheduler_connected addr=lb:500513920lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)3921lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.08 seconds)3922subtest: one node schedules and both builders hold a WorkerSession on it3923lb: must succeed: curl -sf localhost:9901/clusters3924worker2 # [ 32.018017] nix-grpc-daemon[772]: ts=2026-09-21T13:22:13Z level=info event=scheduler_connected addr=lb:500513925worker1 # [ 32.101683] nix-grpc-daemon[770]: ts=2026-09-21T13:22:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3926worker2 # [ 32.023108] nix-grpc-daemon[772]: ts=2026-09-21T13:22:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"3927lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)3928worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3929worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)3930worker1 # [ 32.298545] nix-grpc-daemon[770]: ts=2026-09-21T13:22:13Z level=info event=scheduler_connected addr=lb:500513931worker2 # [ 32.220934] nix-grpc-daemon[772]: ts=2026-09-21T13:22:13Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:341863932worker2 # [ 32.227872] nix-grpc-daemon[772]: ts=2026-09-21T13:22:13Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=03933worker2 # [ 32.235239] nix-grpc-daemon[772]: ts=2026-09-21T13:22:13Z level=info event=scheduler_connected addr=lb:500513934worker2 # [ 32.246651] nix-grpc-daemon[772]: ts=2026-09-21T13:22:13Z level=info event=oidc_jwks provider=mock keys=13935worker2 # [ 32.251035] nix-grpc-daemon[772]: ts=2026-09-21T13:22:13Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:341863936worker2 # [ 32.256454] nix-grpc-daemon[772]: ts=2026-09-21T13:22:13Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=03937lb: must succeed: curl -sf localhost:9901/clusters3938lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)3939worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3940worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)3941lb: must succeed: curl -sf localhost:9901/clusters3942lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)3943lb: must succeed: curl -sf localhost:9901/clusters3944lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)3945lb: must succeed: curl -sf localhost:9901/clusters3946lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)3947lb: must succeed: curl -sf localhost:9901/clusters3948lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)3949worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open'3950worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.06 seconds)3951worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3952worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)3953lb: must succeed: curl -sf localhost:9901/clusters3954lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)3955lb: must succeed: curl -sf localhost:9901/clusters3956lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)3957worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics3958worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)3959worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3960worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)3961worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3962worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds)3963worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics3964worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)3965(finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 1.84 seconds)3966subtest: balancer auth: client cert, bearer token, nothing, unknown CN3967worker1 # [ 34.239277] nix-grpc-daemon[770]: ts=2026-09-21T13:22:15Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:365623968worker1 # [ 34.257018] systemd[1]: Started Nix Daemon.3969worker1 # [ 34.459769] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)3970worker1 # [ 34.478752] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 48 ms (attempt 1/5)3971worker1 # [ 34.530679] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 104 ms (attempt 2/5)3972worker1 # [ 34.636903] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 192 ms (attempt 3/5)3973worker1 # [ 34.831005] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 732 ms (attempt 4/5)3974worker1 # [ 35.567410] nix-grpc-daemon[770]: disabling binary cache 'https://cache.nixos.org' for 60 seconds3975worker1 # [ 35.567776] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org3976worker2 # [ 35.523205] nix-grpc-daemon[772]: ts=2026-09-21T13:22:17Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:341943977worker2 # [ 35.539062] systemd[1]: Started Nix Daemon.3978worker2 # [ 35.720987] nix-daemon[860]: accepted connection from pid 772, user nix-grpc-daemon (trusted)3979worker2 # [ 35.744598] nix-grpc-daemon[772]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 81 ms (attempt 1/5)3980worker2 # [ 35.830169] nix-grpc-daemon[772]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 63 ms (attempt 2/5)3981worker2 # [ 35.894881] nix-grpc-daemon[772]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 81 ms (attempt 3/5)3982worker2 # [ 35.977609] nix-grpc-daemon[772]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 676 ms (attempt 4/5)3983worker2 # [ 36.657305] nix-grpc-daemon[772]: disabling binary cache 'https://cache.nixos.org' for 60 seconds3984worker2 # [ 36.662708] nix-grpc-daemon[772]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org3985worker2 # [ 36.677734] nix-grpc-daemon[772]: ts=2026-09-21T13:22:18Z level=debug event=path_info_miss path=s3qhwgxsdcwiy6iwmdamf1bqgk3qgrb9-nixos-system-client-test error="error: path '/nix/store/s3qhwgxsdcwiy6iwmdamf1bqgk3qgrb9-nixos-system-client-test' is required, but there is no substituter that can build it"3986worker2 # [ 36.691570] nix-grpc-daemon[772]: ts=2026-09-21T13:22:18Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:341943987worker2 # [ 36.702576] nix-grpc-daemon[772]: ts=2026-09-21T13:22:18Z level=debug event=path_info_miss path=s3qhwgxsdcwiy6iwmdamf1bqgk3qgrb9-nixos-system-client-test error="error: path '/nix/store/s3qhwgxsdcwiy6iwmdamf1bqgk3qgrb9-nixos-system-client-test' is required, but there is no substituter that can build it"3988client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt3989client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.08 seconds)3990client: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key3991client: (finished: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds)3992worker1 # [ 37.121753] nix-grpc-daemon[770]: ts=2026-09-21T13:22:18Z level=info event=oidc_jwks provider=mock keys=13993worker1 # [ 37.122195] nix-grpc-daemon[770]: ts=2026-09-21T13:22:18Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:365623994worker2 # [ 37.056953] nix-grpc-daemon[772]: ts=2026-09-21T13:22:18Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:341943995worker1 # [ 37.156513] nix-grpc-daemon[770]: ts=2026-09-21T13:22:18Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:365623996worker2 # [ 37.068221] nix-grpc-daemon[772]: ts=2026-09-21T13:22:18Z level=debug event=path_info_miss path=s3qhwgxsdcwiy6iwmdamf1bqgk3qgrb9-nixos-system-client-test error="error: path '/nix/store/s3qhwgxsdcwiy6iwmdamf1bqgk3qgrb9-nixos-system-client-test' is required, but there is no substituter that can build it"3997worker1 # [ 37.170486] nix-grpc-daemon[770]: ts=2026-09-21T13:22:18Z level=debug event=path_info_miss path=s3qhwgxsdcwiy6iwmdamf1bqgk3qgrb9-nixos-system-client-test error="error: path '/nix/store/s3qhwgxsdcwiy6iwmdamf1bqgk3qgrb9-nixos-system-client-test' is required, but there is no substituter that can build it"3998client: must succeed: rm -r /var/lib/nix-grpc-store3999client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.03 seconds)4000worker2 # [ 37.326293] nix-grpc-daemon[772]: ts=2026-09-21T13:22:18Z level=info event=denied method=QueryMissing cn=- role=none4001worker2 # [ 37.491146] nix-grpc-daemon[772]: ts=2026-09-21T13:22:19Z level=info event=denied method=QueryMissing cn=stranger role=none4002(finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 3.65 seconds)4003subtest: DAG build is scheduled across workers and lands in the cache4004client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&24005client # don't know how to build these paths:4006client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4007worker1 # [ 37.743307] nix-grpc-daemon[770]: ts=2026-09-21T13:22:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:365624008worker1 # [ 37.752110] nix-grpc-daemon[770]: ts=2026-09-21T13:22:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:365624009worker2 # [ 37.692955] nix-grpc-daemon[772]: ts=2026-09-21T13:22:19Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=04010worker2 # [ 37.695436] nix-grpc-daemon[772]: ts=2026-09-21T13:22:19Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.3:50051 clients=14011worker1 # [ 37.826539] nix-grpc-daemon[770]: ts=2026-09-21T13:22:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624012worker1 # [ 37.843769] nix-grpc-daemon[770]: ts=2026-09-21T13:22:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:365624013client # copying 1 paths to 'lb:50051'...4014worker1 # [ 37.859597] nix-grpc-daemon[770]: ts=2026-09-21T13:22:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:365624015client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'...4016worker1 # [ 37.886756] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4017worker1 # [ 37.921654] nix-grpc-daemon[770]: ts=2026-09-21T13:22:19Z level=info event=niks3_push_started pid=8484018worker1 # [ 38.136164] nix-grpc-daemon[848]: time=2026-09-21T13:22:19.754Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""4019worker1 # [ 38.242393] nix-daemon[835]: accepted connection from pid 860, user nix-grpc-daemon (trusted)4020worker1 # [ 38.263845] nix-daemon[835]: reaped child process 863, status = succeeded4021worker1 # [ 38.369527] nix-daemon[835]: accepted connection from pid 865, user nix-grpc-daemon (trusted)4022worker1 # [ 38.387750] nix-daemon[835]: reaped child process 868, status = succeeded4023lb # [ 38.328687] niks3-server[1034]: 2026/09/21 13:22:19 INFO Received uploads request method=POST path=/api/pending_closures4024worker1 # [ 38.414390] nix-grpc-daemon[848]: time=2026-09-21T13:22:20.032Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4025worker1 # [ 38.415935] nix-grpc-daemon[848]: time=2026-09-21T13:22:20.032Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)"4026lb # [ 38.390134] niks3-server[1034]: 2026/09/21 13:22:20 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst4027lb # [ 38.403492] niks3-server[1034]: 2026/09/21 13:22:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign4028lb # [ 38.413034] niks3-server[1034]: 2026/09/21 13:22:20 INFO Signed narinfos id=1 count=14029worker1 # [ 38.483795] nix-grpc-daemon[848]: time=2026-09-21T13:22:20.102Z level=INFO msg="Uploading 1 narinfos"4030lb # [ 38.427527] niks3-server[1034]: 2026/09/21 13:22:20 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls4031lb # [ 38.438396] niks3-server[1034]: 2026/09/21 13:22:20 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete4032lb # [ 38.451322] niks3-server[1034]: 2026/09/21 13:22:20 INFO Completed upload id=14033worker1 # [ 38.518283] nix-grpc-daemon[848]: time=2026-09-21T13:22:20.137Z level=INFO msg="Upload complete. (379ms)"4034worker1 # [ 38.530323] nix-grpc-daemon[770]: ts=2026-09-21T13:22:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=0 paths=1 nar_bytes_in=5044035worker1 # [ 38.548536] nix-grpc-daemon[770]: ts=2026-09-21T13:22:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624036worker1 # [ 38.557517] nix-grpc-daemon[848]: time=2026-09-21T13:22:20.160Z level=WARN msg="Failed to register uploaded object" key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo error="server returned 404: object is not pending upload\n"4037worker1 # [ 38.568212] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4038worker1 # [ 38.570371] nix-daemon[835]: reaped child process 846, status = succeeded4039client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051...4040client # farm-a> node-a: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4041worker1 # [ 38.579148] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4042lb # [ 38.662172] envoy[897]: conn peer=[2001:db8:1::1]:42254 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=732 tx=430 ms=25884043lb # [ 38.667204] envoy[897]: conn peer=[2001:db8:1::1]:42266 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=774044lb # [ 38.671635] envoy[897]: conn peer=[2001:db8:1::1]:42274 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=104045lb # [ 38.676005] envoy[897]: conn peer=[2001:db8:1::1]:42282 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=124046client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds4047client # farm-a> LOG-farm-a-t14048worker1 # [ 39.853099] nix-daemon[835]: reaped child process 872, status = succeeded4049worker1 # [ 39.965326] nix-daemon[835]: accepted connection from pid 888, user nix-grpc-daemon (trusted)4050worker1 # [ 39.983686] nix-daemon[835]: reaped child process 891, status = succeeded4051lb # [ 39.924870] niks3-server[1034]: 2026/09/21 13:22:21 INFO Received uploads request method=POST path=/api/pending_closures4052worker1 # [ 40.015084] nix-grpc-daemon[848]: time=2026-09-21T13:22:21.629Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4053worker1 # [ 40.018128] nix-grpc-daemon[848]: time=2026-09-21T13:22:21.629Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)"4054lb # [ 39.998745] niks3-server[1034]: 2026/09/21 13:22:21 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst4055worker1 # [ 40.074822] nix-grpc-daemon[848]: time=2026-09-21T13:22:21.692Z level=INFO msg="Uploading 1 narinfos"4056lb # [ 40.009670] niks3-server[1034]: 2026/09/21 13:22:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign4057lb # [ 40.018473] niks3-server[1034]: 2026/09/21 13:22:21 INFO Signed narinfos id=2 count=14058lb # [ 40.023563] niks3-server[1034]: 2026/09/21 13:22:21 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv4059lb # [ 40.035025] niks3-server[1034]: 2026/09/21 13:22:21 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls4060lb # [ 40.038452] niks3-server[1034]: 2026/09/21 13:22:21 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete4061lb # [ 40.044851] niks3-server[1034]: 2026/09/21 13:22:21 INFO Completed upload id=24062lb # [ 40.049540] niks3-server[1034]: 2026/09/21 13:22:21 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo4063worker1 # [ 40.110670] nix-grpc-daemon[848]: time=2026-09-21T13:22:21.730Z level=INFO msg="Upload complete. (320ms)"4064worker1 # [ 40.125320] nix-grpc-daemon[770]: ts=2026-09-21T13:22:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=14065worker2 # [ 40.051292] nix-grpc-daemon[772]: ts=2026-09-21T13:22:21Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=04066worker1 # [ 40.143427] nix-daemon[835]: reaped child process 870, status = succeeded4067worker2 # [ 40.057521] nix-grpc-daemon[772]: ts=2026-09-21T13:22:21Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.3:50051 clients=14068worker1 # [ 40.184889] nix-grpc-daemon[770]: ts=2026-09-21T13:22:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624069worker1 # [ 40.199721] nix-grpc-daemon[770]: ts=2026-09-21T13:22:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:365624070worker1 # [ 40.214640] nix-grpc-daemon[848]: time=2026-09-21T13:22:21.834Z level=INFO msg="All 1 paths already cached"4071client # copying 1 paths to 'lb:50051'...4072worker1 # [ 40.224326] nix-grpc-daemon[770]: ts=2026-09-21T13:22:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:365624073client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'...4074worker1 # [ 40.244957] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4075worker1 # [ 40.401831] nix-daemon[835]: accepted connection from pid 895, user nix-grpc-daemon (trusted)4076worker1 # [ 40.427273] nix-daemon[835]: reaped child process 898, status = succeeded4077worker1 # [ 40.525562] nix-daemon[835]: accepted connection from pid 900, user nix-grpc-daemon (trusted)4078worker1 # [ 40.543573] nix-daemon[835]: reaped child process 903, status = succeeded4079lb # [ 40.480129] niks3-server[1034]: 2026/09/21 13:22:22 INFO Received uploads request method=POST path=/api/pending_closures4080worker1 # [ 40.556975] nix-grpc-daemon[848]: time=2026-09-21T13:22:22.176Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4081worker1 # [ 40.558506] nix-grpc-daemon[848]: time=2026-09-21T13:22:22.176Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)"4082lb # [ 40.513213] niks3-server[1034]: 2026/09/21 13:22:22 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst4083worker1 # [ 40.592358] nix-grpc-daemon[848]: time=2026-09-21T13:22:22.211Z level=INFO msg="Uploading 1 narinfos"4084lb # [ 40.529479] niks3-server[1034]: 2026/09/21 13:22:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign4085lb # [ 40.539474] niks3-server[1034]: 2026/09/21 13:22:22 INFO Signed narinfos id=3 count=14086lb # [ 40.547376] niks3-server[1034]: 2026/09/21 13:22:22 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls4087worker1 # [ 40.618851] nix-grpc-daemon[848]: time=2026-09-21T13:22:22.238Z level=INFO msg="Upload complete. (356ms)"4088lb # [ 40.554741] niks3-server[1034]: 2026/09/21 13:22:22 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete4089lb # [ 40.558973] niks3-server[1034]: 2026/09/21 13:22:22 INFO Completed upload id=34090lb # [ 40.562442] niks3-server[1034]: 2026/09/21 13:22:22 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo4091worker1 # [ 40.628214] nix-grpc-daemon[770]: ts=2026-09-21T13:22:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=0 paths=1 nar_bytes_in=6244092worker1 # [ 40.638338] nix-grpc-daemon[770]: ts=2026-09-21T13:22:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624093worker1 # [ 40.647262] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4094worker1 # [ 40.650071] nix-daemon[835]: reaped child process 893, status = succeeded4095client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051...4096client # farm-b> node-a: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4097worker1 # [ 40.663491] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4098client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4099client # farm-b> LOG-farm-b-t14100worker1 # [ 42.161948] nix-daemon[835]: reaped child process 907, status = succeeded4101worker1 # [ 42.230341] nix-daemon[835]: accepted connection from pid 925, user nix-grpc-daemon (trusted)4102worker1 # [ 42.246582] nix-daemon[835]: reaped child process 928, status = succeeded4103lb # [ 42.186925] niks3-server[1034]: 2026/09/21 13:22:23 INFO Received uploads request method=POST path=/api/pending_closures4104worker1 # [ 42.263988] nix-grpc-daemon[848]: time=2026-09-21T13:22:23.882Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4105worker1 # [ 42.265811] nix-grpc-daemon[848]: time=2026-09-21T13:22:23.882Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)"4106lb # [ 42.221730] niks3-server[1034]: 2026/09/21 13:22:23 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv4107lb # [ 42.235141] niks3-server[1034]: 2026/09/21 13:22:23 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst4108worker1 # [ 42.316595] nix-grpc-daemon[848]: time=2026-09-21T13:22:23.935Z level=INFO msg="Uploading 1 narinfos"4109lb # [ 42.254452] niks3-server[1034]: 2026/09/21 13:22:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign4110lb # [ 42.265492] niks3-server[1034]: 2026/09/21 13:22:23 INFO Signed narinfos id=4 count=14111lb # [ 42.272277] niks3-server[1034]: 2026/09/21 13:22:23 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls4112lb # [ 42.284392] niks3-server[1034]: 2026/09/21 13:22:23 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete4113lb # [ 42.295194] niks3-server[1034]: 2026/09/21 13:22:23 INFO Completed upload id=44114worker1 # [ 42.362371] nix-grpc-daemon[848]: time=2026-09-21T13:22:23.981Z level=INFO msg="Upload complete. (288ms)"4115lb # [ 42.298796] niks3-server[1034]: 2026/09/21 13:22:23 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo4116worker2 # [ 42.296981] nix-grpc-daemon[772]: ts=2026-09-21T13:22:23Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=04117worker2 # [ 42.302918] nix-grpc-daemon[772]: ts=2026-09-21T13:22:23Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.3:50051 clients=14118worker1 # [ 42.373177] nix-grpc-daemon[770]: ts=2026-09-21T13:22:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=14119worker1 # [ 42.394482] nix-daemon[835]: reaped child process 905, status = succeeded4120worker1 # [ 42.427580] nix-grpc-daemon[770]: ts=2026-09-21T13:22:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624121worker1 # [ 42.448649] nix-grpc-daemon[770]: ts=2026-09-21T13:22:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:365624122worker1 # [ 42.462714] nix-grpc-daemon[848]: time=2026-09-21T13:22:24.082Z level=INFO msg="All 2 paths already cached"4123client # copying 1 paths to 'lb:50051'...4124worker1 # [ 42.472939] nix-grpc-daemon[770]: ts=2026-09-21T13:22:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:365624125client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'...4126worker1 # [ 42.495514] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4127worker1 # [ 42.661471] nix-daemon[835]: accepted connection from pid 932, user nix-grpc-daemon (trusted)4128worker1 # [ 42.679816] nix-daemon[835]: reaped child process 935, status = succeeded4129worker1 # [ 42.781046] nix-daemon[835]: accepted connection from pid 937, user nix-grpc-daemon (trusted)4130worker1 # [ 42.797566] nix-daemon[835]: reaped child process 940, status = succeeded4131lb # [ 42.735503] niks3-server[1034]: 2026/09/21 13:22:24 INFO Received uploads request method=POST path=/api/pending_closures4132worker1 # [ 42.812091] nix-grpc-daemon[848]: time=2026-09-21T13:22:24.431Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4133worker1 # [ 42.813567] nix-grpc-daemon[848]: time=2026-09-21T13:22:24.431Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)"4134lb # [ 42.763076] niks3-server[1034]: 2026/09/21 13:22:24 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst4135worker1 # [ 42.842997] nix-grpc-daemon[848]: time=2026-09-21T13:22:24.462Z level=INFO msg="Uploading 1 narinfos"4136lb # [ 42.778651] niks3-server[1034]: 2026/09/21 13:22:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign4137lb # [ 42.790712] niks3-server[1034]: 2026/09/21 13:22:24 INFO Signed narinfos id=5 count=14138lb # [ 42.804454] niks3-server[1034]: 2026/09/21 13:22:24 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls4139worker1 # [ 42.876831] nix-grpc-daemon[848]: time=2026-09-21T13:22:24.496Z level=INFO msg="Upload complete. (363ms)"4140lb # [ 42.812491] niks3-server[1034]: 2026/09/21 13:22:24 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete4141worker1 # [ 42.882458] nix-grpc-daemon[770]: ts=2026-09-21T13:22:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=0 paths=1 nar_bytes_in=7524142lb # [ 42.821759] niks3-server[1034]: 2026/09/21 13:22:24 INFO Completed upload id=54143lb # [ 42.824998] niks3-server[1034]: 2026/09/21 13:22:24 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo4144worker1 # [ 42.895728] nix-grpc-daemon[770]: ts=2026-09-21T13:22:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624145worker1 # [ 42.903805] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4146worker1 # [ 42.906913] nix-daemon[835]: reaped child process 930, status = succeeded4147client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051...4148worker1 # [ 42.920929] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4149client # farm-top> node-a: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4150client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4151client # farm-top> LOG-farm-top-t14152worker1 # [ 44.036451] nix-daemon[835]: reaped child process 944, status = succeeded4153worker1 # [ 44.113870] nix-daemon[835]: accepted connection from pid 960, user nix-grpc-daemon (trusted)4154worker1 # [ 44.148170] nix-daemon[835]: reaped child process 963, status = succeeded4155lb # [ 44.083011] niks3-server[1034]: 2026/09/21 13:22:25 INFO Received uploads request method=POST path=/api/pending_closures4156worker1 # [ 44.162873] nix-grpc-daemon[848]: time=2026-09-21T13:22:25.780Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4157worker1 # [ 44.164591] nix-grpc-daemon[848]: time=2026-09-21T13:22:25.780Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)"4158lb # [ 44.127737] niks3-server[1034]: 2026/09/21 13:22:25 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst4159lb # [ 44.140392] niks3-server[1034]: 2026/09/21 13:22:25 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv4160worker1 # [ 44.217756] nix-grpc-daemon[848]: time=2026-09-21T13:22:25.837Z level=INFO msg="Uploading 1 narinfos"4161lb # [ 44.153011] niks3-server[1034]: 2026/09/21 13:22:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign4162lb # [ 44.161076] niks3-server[1034]: 2026/09/21 13:22:25 INFO Signed narinfos id=6 count=14163lb # [ 44.163626] niks3-server[1034]: 2026/09/21 13:22:25 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls4164worker1 # [ 44.242638] nix-grpc-daemon[848]: time=2026-09-21T13:22:25.862Z level=INFO msg="Upload complete. (312ms)"4165lb # [ 44.177576] niks3-server[1034]: 2026/09/21 13:22:25 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete4166lb # [ 44.184943] niks3-server[1034]: 2026/09/21 13:22:25 INFO Completed upload id=64167worker1 # [ 44.252938] nix-grpc-daemon[770]: ts=2026-09-21T13:22:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=14168lb # [ 44.189355] niks3-server[1034]: 2026/09/21 13:22:25 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo4169worker1 # [ 44.268288] nix-daemon[835]: reaped child process 942, status = succeeded4170client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 6.71 seconds)4171client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath4172client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.18 seconds)4173client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4174client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.02 seconds)4175client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4176client # copying 3 paths...4177client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'...4178client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'...4179client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'...4180client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.23 seconds)4181(finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.23 seconds)4182subtest: repeat is answered Cached by the scheduler without building4183worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4184worker1 # finding garbage collector roots...4185worker1 # deleting specified paths...4186worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4187worker1 # deleting unused links...4188worker1 # note: hard linking is currently saving 0.0 KiB4189worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.22 seconds)4190worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4191worker2 # finding garbage collector roots...4192worker2 # deleting specified paths...4193worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top'4194worker2 # deleting unused links...4195worker2 # note: hard linking is currently saving 0.0 KiB4196worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.24 seconds)4197client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&24198worker1 # [ 45.501928] nix-grpc-daemon[770]: ts=2026-09-21T13:22:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:365624199client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked):4200client # /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top4201worker2 # [ 45.439116] nix-grpc-daemon[772]: ts=2026-09-21T13:22:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:341944202worker2 # [ 45.468730] nix-grpc-daemon[772]: ts=2026-09-21T13:22:27Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=14203client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 0.31 seconds)4204client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath4205client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.15 seconds)4206lb: must succeed: curl -sf localhost:9901/clusters4207lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)4208lb: must succeed: curl -sf localhost:9901/clusters4209lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4210worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4211worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4212(finished: subtest: repeat is answered Cached by the scheduler without building, in 1.18 seconds)4213subtest: inputs already in the cache are not sent to the scheduler4214lb: must succeed: curl -sf localhost:9901/clusters4215lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4216lb: must succeed: curl -sf localhost:9901/clusters4217lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)4218worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4219worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4220client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&24221client # don't know how to build these paths:4222client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4223worker2 # [ 46.297689] nix-grpc-daemon[772]: ts=2026-09-21T13:22:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:341944224worker1 # [ 46.385614] nix-grpc-daemon[770]: ts=2026-09-21T13:22:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:365624225worker1 # [ 46.396328] nix-grpc-daemon[848]: time=2026-09-21T13:22:28.015Z level=INFO msg="All 2 paths already cached"4226worker2 # [ 46.327446] nix-grpc-daemon[772]: ts=2026-09-21T13:22:27Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=04227worker2 # [ 46.333259] nix-grpc-daemon[772]: ts=2026-09-21T13:22:27Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.3:50051 clients=14228worker1 # [ 46.459607] nix-grpc-daemon[770]: ts=2026-09-21T13:22:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624229worker1 # [ 46.474933] nix-grpc-daemon[770]: ts=2026-09-21T13:22:28Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:365624230worker1 # [ 46.489536] nix-grpc-daemon[848]: time=2026-09-21T13:22:28.109Z level=INFO msg="All 2 paths already cached"4231client # copying 1 paths to 'lb:50051'...4232client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'...4233worker1 # [ 46.500941] nix-grpc-daemon[770]: ts=2026-09-21T13:22:28Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:365624234worker1 # [ 46.515586] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4235worker1 # [ 46.671096] nix-daemon[835]: accepted connection from pid 981, user nix-grpc-daemon (trusted)4236worker1 # [ 46.696796] nix-daemon[835]: reaped child process 984, status = succeeded4237worker1 # [ 46.789871] nix-daemon[835]: accepted connection from pid 986, user nix-grpc-daemon (trusted)4238worker1 # [ 46.807296] nix-daemon[835]: reaped child process 989, status = succeeded4239lb # [ 46.747408] niks3-server[1034]: 2026/09/21 13:22:28 INFO Received uploads request method=POST path=/api/pending_closures4240worker1 # [ 46.823810] nix-grpc-daemon[848]: time=2026-09-21T13:22:28.443Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4241worker1 # [ 46.825426] nix-grpc-daemon[848]: time=2026-09-21T13:22:28.443Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)"4242lb # [ 46.775570] niks3-server[1034]: 2026/09/21 13:22:28 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst4243lb # [ 46.789312] niks3-server[1034]: 2026/09/21 13:22:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign4244worker1 # [ 46.862321] nix-grpc-daemon[848]: time=2026-09-21T13:22:28.481Z level=INFO msg="Uploading 1 narinfos"4245lb # [ 46.801133] niks3-server[1034]: 2026/09/21 13:22:28 INFO Signed narinfos id=7 count=14246lb # [ 46.803659] niks3-server[1034]: 2026/09/21 13:22:28 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls4247worker1 # [ 46.883525] nix-grpc-daemon[848]: time=2026-09-21T13:22:28.503Z level=INFO msg="Upload complete. (350ms)"4248lb # [ 46.818262] niks3-server[1034]: 2026/09/21 13:22:28 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete4249lb # [ 46.824763] niks3-server[1034]: 2026/09/21 13:22:28 INFO Completed upload id=74250worker1 # [ 46.892437] nix-grpc-daemon[770]: ts=2026-09-21T13:22:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=0 paths=1 nar_bytes_in=7524251lb # [ 46.826960] niks3-server[1034]: 2026/09/21 13:22:28 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo4252worker1 # [ 46.902891] nix-grpc-daemon[770]: ts=2026-09-21T13:22:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624253worker1 # [ 46.908675] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4254worker1 # [ 46.913193] nix-daemon[835]: reaped child process 979, status = succeeded4255client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051...4256client # farm-top> node-a: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4257worker1 # [ 46.925985] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4258client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4259client # farm-top> LOG-farm-top-t1b4260worker1 # [ 47.769771] nix-daemon[835]: reaped child process 993, status = succeeded4261worker1 # [ 47.879709] nix-daemon[835]: accepted connection from pid 1009, user nix-grpc-daemon (trusted)4262worker1 # [ 47.897487] nix-daemon[835]: reaped child process 1012, status = succeeded4263lb # [ 47.838248] niks3-server[1034]: 2026/09/21 13:22:29 INFO Received uploads request method=POST path=/api/pending_closures4264worker1 # [ 47.916928] nix-grpc-daemon[848]: time=2026-09-21T13:22:29.535Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4265worker1 # [ 47.921437] nix-grpc-daemon[848]: time=2026-09-21T13:22:29.536Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)"4266lb # [ 47.877086] niks3-server[1034]: 2026/09/21 13:22:29 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv4267lb # [ 47.889253] niks3-server[1034]: 2026/09/21 13:22:29 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst4268worker1 # [ 47.969716] nix-grpc-daemon[848]: time=2026-09-21T13:22:29.589Z level=INFO msg="Uploading 1 narinfos"4269lb # [ 47.906431] niks3-server[1034]: 2026/09/21 13:22:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign4270lb # [ 47.915409] niks3-server[1034]: 2026/09/21 13:22:29 INFO Signed narinfos id=8 count=14271lb # [ 47.924250] niks3-server[1034]: 2026/09/21 13:22:29 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls4272lb # [ 47.927864] niks3-server[1034]: 2026/09/21 13:22:29 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete4273lb # [ 47.935761] niks3-server[1034]: 2026/09/21 13:22:29 INFO Completed upload id=84274lb # [ 47.938677] niks3-server[1034]: 2026/09/21 13:22:29 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo4275worker1 # [ 48.002038] nix-grpc-daemon[848]: time=2026-09-21T13:22:29.621Z level=INFO msg="Upload complete. (266ms)"4276worker1 # [ 48.013410] nix-grpc-daemon[770]: ts=2026-09-21T13:22:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=14277worker1 # [ 48.031604] nix-daemon[835]: reaped child process 991, status = succeeded4278client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2, in 1.88 seconds)4279client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath4280client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.17 seconds)4281lb: must succeed: curl -sf localhost:9901/clusters4282lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4283lb: must succeed: curl -sf localhost:9901/clusters4284lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)4285worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true4286worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds)4287(finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.39 seconds)4288subtest: two clients wanting the same drv build it once4289client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup4290client # [ 48.399166] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup.4291client # Running as unit: dedup1.service; invocation ID: cc6acce72b1c45678dfbfb673f4e34524292client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 0.09 seconds)4293client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup4294client # [ 48.596548] nix[931]: don't know how to build these paths:4295client # [ 48.597824] nix[931]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4296worker2 # [ 48.577981] nix-grpc-daemon[772]: ts=2026-09-21T13:22:30Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:341944297worker2 # [ 48.590259] nix-grpc-daemon[772]: ts=2026-09-21T13:22:30Z level=debug event=want client=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04298worker2 # [ 48.598295] nix-grpc-daemon[772]: ts=2026-09-21T13:22:30Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.4:50051 clients=14299worker2 # [ 48.606464] nix-grpc-daemon[772]: ts=2026-09-21T13:22:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:341944300worker2 # [ 48.617301] nix-grpc-daemon[772]: ts=2026-09-21T13:22:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:341944301client # [ 48.646561] nix[931]: copying 1 paths to 'lb:50051'...4302client # [ 48.650282] nix[931]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'...4303worker2 # [ 48.632824] nix-grpc-daemon[772]: ts=2026-09-21T13:22:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:341944304lb # [ 48.662823] envoy[897]: conn peer=[2001:db8:1::1]:42290 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9523 tx=4070 ms=65384305worker2 # [ 48.647830] nix-daemon[860]: accepted connection from pid 772, user nix-grpc-daemon (trusted)4306worker2 # [ 48.674168] nix-grpc-daemon[772]: ts=2026-09-21T13:22:30Z level=info event=niks3_push_started pid=9124307worker2 # [ 48.836620] nix-grpc-daemon[912]: time=2026-09-21T13:22:30.458Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""4308worker2 # [ 48.949039] nix-daemon[860]: accepted connection from pid 924, user nix-grpc-daemon (trusted)4309worker2 # [ 48.970677] nix-daemon[860]: reaped child process 927, status = succeeded4310worker2 # [ 49.078773] nix-daemon[860]: accepted connection from pid 929, user nix-grpc-daemon (trusted)4311worker2 # [ 49.096033] nix-daemon[860]: reaped child process 932, status = succeeded4312lb # [ 48.667993] envoy[897]: conn peer=[2001:db8:1::1]:54054 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=480 ms=734313lb # [ 48.668173] envoy[897]: conn peer=[2001:db8:1::1]:54060 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4198 tx=1807 ms=16464314lb # [ 49.119439] niks3-server[1034]: 2026/09/21 13:22:30 INFO Received uploads request method=POST path=/api/pending_closures4315worker2 # [ 49.121308] nix-grpc-daemon[912]: time=2026-09-21T13:22:30.743Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4316worker2 # [ 49.122783] nix-grpc-daemon[912]: time=2026-09-21T13:22:30.743Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)"4317worker2 # [ 49.156480] nix-grpc-daemon[912]: time=2026-09-21T13:22:30.778Z level=INFO msg="Uploading 1 narinfos"4318lb # [ 49.166203] niks3-server[1034]: 2026/09/21 13:22:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign4319lb # [ 49.180931] niks3-server[1034]: 2026/09/21 13:22:30 INFO Signed narinfos id=9 count=14320lb # [ 49.190724] niks3-server[1034]: 2026/09/21 13:22:30 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst4321worker2 # [ 49.189620] nix-grpc-daemon[912]: time=2026-09-21T13:22:30.812Z level=INFO msg="Upload complete. (348ms)"4322lb # [ 49.206790] niks3-server[1034]: 2026/09/21 13:22:30 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls4323lb # [ 49.214816] niks3-server[1034]: 2026/09/21 13:22:30 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete4324lb # [ 49.221216] niks3-server[1034]: 2026/09/21 13:22:30 INFO Completed upload id=94325lb # [ 49.223844] niks3-server[1034]: 2026/09/21 13:22:30 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo4326worker2 # [ 49.196832] nix-grpc-daemon[772]: ts=2026-09-21T13:22:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34194 duration_s=0 paths=1 nar_bytes_in=4324327worker2 # [ 49.211575] nix-grpc-daemon[772]: ts=2026-09-21T13:22:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:341944328worker2 # [ 49.218813] nix-daemon[860]: accepted connection from pid 772, user nix-grpc-daemon (trusted)4329worker2 # [ 49.223500] nix-daemon[860]: reaped child process 910, status = succeeded4330client # [ 49.251566] nix[931]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4331worker2 # [ 49.236834] nix-daemon[860]: accepted connection from pid 772, user nix-grpc-daemon (trusted)4332client # [ 49.503273] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup.4333client # Running as unit: dedup2.service; invocation ID: d9dd347ec8d04e148941e6b3310bc9f34334client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 1.11 seconds)4335client # [ 49.739412] nix[947]: don't know how to build these paths:4336client # [ 49.741335] nix[947]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4337worker1 # [ 49.805235] nix-grpc-daemon[770]: ts=2026-09-21T13:22:31Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:365624338worker2 # [ 49.733773] nix-grpc-daemon[772]: ts=2026-09-21T13:22:31Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04339client # [ 49.760104] nix[947]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4340worker2 # [ 49.741487] nix-grpc-daemon[772]: ts=2026-09-21T13:22:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:341944341client: must succeed: sleep 24342client: (finished: must succeed: sleep 2, in 2.05 seconds)4343client: waiting for success: ! systemctl is-active dedup1 dedup24344worker2 # [ 53.065421] nix-daemon[860]: reaped child process 936, status = succeeded4345worker2 # [ 53.168547] nix-daemon[860]: accepted connection from pid 968, user nix-grpc-daemon (trusted)4346worker2 # [ 53.188708] nix-daemon[860]: reaped child process 971, status = succeeded4347lb # [ 53.211941] niks3-server[1034]: 2026/09/21 13:22:34 INFO Received uploads request method=POST path=/api/pending_closures4348worker2 # [ 53.206950] nix-grpc-daemon[912]: time=2026-09-21T13:22:34.827Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4349worker2 # [ 53.209631] nix-grpc-daemon[912]: time=2026-09-21T13:22:34.827Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)"4350lb # [ 53.256321] niks3-server[1034]: 2026/09/21 13:22:34 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4351lb # [ 53.269238] niks3-server[1034]: 2026/09/21 13:22:34 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4352lb # [ 53.283490] niks3-server[1034]: 2026/09/21 13:22:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign4353worker2 # [ 53.272110] nix-grpc-daemon[912]: time=2026-09-21T13:22:34.893Z level=INFO msg="Uploading 1 narinfos"4354lb # [ 53.290853] niks3-server[1034]: 2026/09/21 13:22:34 INFO Signed narinfos id=10 count=14355lb # [ 53.294916] niks3-server[1034]: 2026/09/21 13:22:34 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls4356lb # [ 53.304464] niks3-server[1034]: 2026/09/21 13:22:34 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4357lb # [ 53.310731] niks3-server[1034]: 2026/09/21 13:22:34 INFO Completed upload id=104358worker2 # [ 53.292857] nix-grpc-daemon[912]: time=2026-09-21T13:22:34.915Z level=INFO msg="Upload complete. (243ms)"4359lb # [ 53.313154] niks3-server[1034]: 2026/09/21 13:22:34 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo4360worker2 # [ 53.305075] nix-grpc-daemon[772]: ts=2026-09-21T13:22:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34194 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14361worker2 # [ 53.323607] nix-grpc-daemon[772]: ts=2026-09-21T13:22:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34194 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=5 outputs=14362worker2 # [ 53.330290] nix-daemon[860]: reaped child process 934, status = succeeded4363client # [ 53.382351] systemd[1]: dedup1.service: Deactivated successfully.4364client # [ 53.385636] systemd[1]: dedup1.service: Consumed 172ms CPU time over 4.986s wall clock time, 39.4M memory peak, 7K incoming IP traffic, 8.4K outgoing IP traffic.4365client # [ 53.393089] systemd[1]: dedup2.service: Deactivated successfully.4366client # [ 53.396716] systemd[1]: dedup2.service: Consumed 215ms CPU time over 3.890s wall clock time, 35.2M memory peak, 5.9K incoming IP traffic, 5.7K outgoing IP traffic.4367client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.12 seconds)4368client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4369client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.07 seconds)4370worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4371worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4372worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4373worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4374worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4375worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4376worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4377worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4378(finished: subtest: two clients wanting the same drv build it once, in 6.13 seconds)4379subtest: build hook: nix-daemon with builders = grpc://lb4380client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log4381client # [ 54.651873] systemd[1]: Started Nix Daemon.4382client # [ 54.768920] nix-daemon[989]: accepted connection from pid 986, user root (trusted)4383worker1 # [ 54.982983] nix-grpc-daemon[770]: ts=2026-09-21T13:22:36Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:365624384worker1 # [ 54.989986] nix-grpc-daemon[770]: ts=2026-09-21T13:22:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:365624385worker2 # [ 54.919515] nix-grpc-daemon[772]: ts=2026-09-21T13:22:36Z level=debug event=want client=6 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=04386worker2 # [ 54.925520] nix-grpc-daemon[772]: ts=2026-09-21T13:22:36Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.3:50051 clients=14387worker1 # [ 55.047086] nix-grpc-daemon[770]: ts=2026-09-21T13:22:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624388worker1 # [ 55.062444] nix-grpc-daemon[770]: ts=2026-09-21T13:22:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:365624389worker1 # [ 55.077128] nix-grpc-daemon[770]: ts=2026-09-21T13:22:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:365624390worker1 # [ 55.091642] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4391worker1 # [ 55.260176] nix-daemon[835]: accepted connection from pid 1037, user nix-grpc-daemon (trusted)4392worker1 # [ 55.280311] nix-daemon[835]: reaped child process 1040, status = succeeded4393worker1 # [ 55.381065] nix-daemon[835]: accepted connection from pid 1042, user nix-grpc-daemon (trusted)4394worker1 # [ 55.399216] nix-daemon[835]: reaped child process 1045, status = succeeded4395lb # [ 55.337887] niks3-server[1034]: 2026/09/21 13:22:36 INFO Received uploads request method=POST path=/api/pending_closures4396worker1 # [ 55.414131] nix-grpc-daemon[848]: time=2026-09-21T13:22:37.033Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4397worker1 # [ 55.415663] nix-grpc-daemon[848]: time=2026-09-21T13:22:37.033Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)"4398lb # [ 55.367916] niks3-server[1034]: 2026/09/21 13:22:36 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst4399worker1 # [ 55.447152] nix-grpc-daemon[848]: time=2026-09-21T13:22:37.066Z level=INFO msg="Uploading 1 narinfos"4400lb # [ 55.384119] niks3-server[1034]: 2026/09/21 13:22:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4401lb # [ 55.393955] niks3-server[1034]: 2026/09/21 13:22:37 INFO Signed narinfos id=11 count=14402lb # [ 55.399414] niks3-server[1034]: 2026/09/21 13:22:37 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls4403worker1 # [ 55.476667] nix-grpc-daemon[848]: time=2026-09-21T13:22:37.096Z level=INFO msg="Upload complete. (367ms)"4404lb # [ 55.409765] niks3-server[1034]: 2026/09/21 13:22:37 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4405lb # [ 55.413138] niks3-server[1034]: 2026/09/21 13:22:37 INFO Completed upload id=114406lb # [ 55.416497] niks3-server[1034]: 2026/09/21 13:22:37 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo4407worker1 # [ 55.484117] nix-grpc-daemon[770]: ts=2026-09-21T13:22:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=0 paths=1 nar_bytes_in=5124408worker1 # [ 55.498999] nix-grpc-daemon[770]: ts=2026-09-21T13:22:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624409worker1 # [ 55.506107] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4410worker1 # [ 55.511122] nix-daemon[835]: reaped child process 1035, status = succeeded4411worker1 # [ 55.522605] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4412worker1 # [ 56.772263] nix-daemon[835]: reaped child process 1049, status = succeeded4413worker1 # [ 56.851750] nix-daemon[835]: accepted connection from pid 1065, user nix-grpc-daemon (trusted)4414worker1 # [ 56.870529] nix-daemon[835]: reaped child process 1068, status = succeeded4415lb # [ 56.808965] niks3-server[1034]: 2026/09/21 13:22:38 INFO Received uploads request method=POST path=/api/pending_closures4416worker1 # [ 56.888275] nix-grpc-daemon[848]: time=2026-09-21T13:22:38.504Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4417worker1 # [ 56.889754] nix-grpc-daemon[848]: time=2026-09-21T13:22:38.504Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)"4418lb # [ 56.850461] niks3-server[1034]: 2026/09/21 13:22:38 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4419lb # [ 56.863894] niks3-server[1034]: 2026/09/21 13:22:38 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv4420worker1 # [ 56.940824] nix-grpc-daemon[848]: time=2026-09-21T13:22:38.560Z level=INFO msg="Uploading 1 narinfos"4421lb # [ 56.876335] niks3-server[1034]: 2026/09/21 13:22:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4422lb # [ 56.879663] niks3-server[1034]: 2026/09/21 13:22:38 INFO Signed narinfos id=12 count=14423lb # [ 56.891412] niks3-server[1034]: 2026/09/21 13:22:38 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls4424lb # [ 56.896914] niks3-server[1034]: 2026/09/21 13:22:38 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4425lb # [ 56.899789] niks3-server[1034]: 2026/09/21 13:22:38 INFO Completed upload id=124426worker1 # [ 56.963583] nix-grpc-daemon[848]: time=2026-09-21T13:22:38.583Z level=INFO msg="Upload complete. (228ms)"4427lb # [ 56.903799] niks3-server[1034]: 2026/09/21 13:22:38 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo4428worker1 # [ 56.973498] nix-grpc-daemon[770]: ts=2026-09-21T13:22:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=6 outputs=14429worker1 # [ 56.992568] nix-grpc-daemon[770]: ts=2026-09-21T13:22:38Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:365624430worker1 # [ 56.997219] nix-daemon[835]: reaped child process 1047, status = succeeded4431worker2 # [ 56.933733] nix-grpc-daemon[772]: ts=2026-09-21T13:22:38Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:341944432worker2 # [ 56.951375] nix-grpc-daemon[772]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'...4433worker2 # [ 56.971610] nix-grpc-daemon[772]: ts=2026-09-21T13:22:38Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34194 duration_s=0 paths=1 nar_bytes_out=1284434worker1 # [ 57.360854] nix-grpc-daemon[770]: ts=2026-09-21T13:22:38Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:365624435worker2 # [ 57.283297] nix-grpc-daemon[772]: ts=2026-09-21T13:22:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:341944436worker2 # [ 57.294948] nix-grpc-daemon[912]: time=2026-09-21T13:22:38.916Z level=INFO msg="All 1 paths already cached"4437worker2 # [ 57.319271] nix-grpc-daemon[772]: ts=2026-09-21T13:22:38Z level=debug event=want client=7 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=04438worker2 # [ 57.326394] nix-grpc-daemon[772]: ts=2026-09-21T13:22:38Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.3:50051 clients=14439worker1 # [ 57.417292] nix-grpc-daemon[770]: ts=2026-09-21T13:22:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624440worker1 # [ 57.436190] nix-grpc-daemon[770]: ts=2026-09-21T13:22:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:365624441worker1 # [ 57.461407] nix-grpc-daemon[848]: time=2026-09-21T13:22:39.080Z level=INFO msg="All 1 paths already cached"4442worker1 # [ 57.477259] nix-grpc-daemon[770]: ts=2026-09-21T13:22:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:365624443worker1 # [ 57.492387] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4444worker1 # [ 57.676081] nix-daemon[835]: accepted connection from pid 1072, user nix-grpc-daemon (trusted)4445worker1 # [ 57.698692] nix-daemon[835]: reaped child process 1075, status = succeeded4446worker1 # [ 57.806310] nix-daemon[835]: accepted connection from pid 1077, user nix-grpc-daemon (trusted)4447worker1 # [ 57.827010] nix-daemon[835]: reaped child process 1080, status = succeeded4448lb # [ 57.765156] niks3-server[1034]: 2026/09/21 13:22:39 INFO Received uploads request method=POST path=/api/pending_closures4449worker1 # [ 57.842606] nix-grpc-daemon[848]: time=2026-09-21T13:22:39.461Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4450worker1 # [ 57.844265] nix-grpc-daemon[848]: time=2026-09-21T13:22:39.461Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)"4451lb # [ 57.795138] niks3-server[1034]: 2026/09/21 13:22:39 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst4452worker1 # [ 57.877581] nix-grpc-daemon[848]: time=2026-09-21T13:22:39.497Z level=INFO msg="Uploading 1 narinfos"4453lb # [ 57.811544] niks3-server[1034]: 2026/09/21 13:22:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4454lb # [ 57.821476] niks3-server[1034]: 2026/09/21 13:22:39 INFO Signed narinfos id=13 count=14455lb # [ 57.828835] niks3-server[1034]: 2026/09/21 13:22:39 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4456lb # [ 57.834521] niks3-server[1034]: 2026/09/21 13:22:39 INFO Completed upload id=134457worker1 # [ 57.902534] nix-grpc-daemon[848]: time=2026-09-21T13:22:39.521Z level=INFO msg="Upload complete. (390ms)"4458lb # [ 57.839253] niks3-server[1034]: 2026/09/21 13:22:39 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls4459lb # [ 57.848119] niks3-server[1034]: 2026/09/21 13:22:39 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo4460worker1 # [ 57.912312] nix-grpc-daemon[770]: ts=2026-09-21T13:22:39Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=0 paths=1 nar_bytes_in=6244461worker1 # [ 57.924379] nix-grpc-daemon[770]: ts=2026-09-21T13:22:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624462worker1 # [ 57.933235] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4463worker1 # [ 57.935934] nix-daemon[835]: reaped child process 1070, status = succeeded4464worker1 # [ 57.947560] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4465lb # [ 58.664329] envoy[897]: conn peer=[2001:db8:1::1]:54066 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3018 tx=1453 ms=47444466lb # [ 58.664705] envoy[897]: conn peer=[2001:db8:1::1]:54080 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1430 tx=1085 ms=36134467lb # [ 58.664947] envoy[897]: conn peer=[2001:db8:1::1]:52396 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=334 ms=1134468lb # [ 58.665056] envoy[897]: conn peer=[2001:db8:1::1]:52394 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3414 tx=1717 ms=21374469worker1 # [ 59.203494] nix-daemon[835]: reaped child process 1084, status = succeeded4470worker1 # [ 59.312873] nix-daemon[835]: accepted connection from pid 1100, user nix-grpc-daemon (trusted)4471worker1 # [ 59.334866] nix-daemon[835]: reaped child process 1103, status = succeeded4472lb # [ 59.270363] niks3-server[1034]: 2026/09/21 13:22:40 INFO Received uploads request method=POST path=/api/pending_closures4473worker1 # [ 59.346265] nix-grpc-daemon[848]: time=2026-09-21T13:22:40.964Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4474worker1 # [ 59.347929] nix-grpc-daemon[848]: time=2026-09-21T13:22:40.964Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)"4475lb # [ 59.299899] niks3-server[1034]: 2026/09/21 13:22:40 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv4476lb # [ 59.316373] niks3-server[1034]: 2026/09/21 13:22:40 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst4477worker1 # [ 59.397636] nix-grpc-daemon[848]: time=2026-09-21T13:22:41.016Z level=INFO msg="Uploading 1 narinfos"4478lb # [ 59.330753] niks3-server[1034]: 2026/09/21 13:22:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4479lb # [ 59.337305] niks3-server[1034]: 2026/09/21 13:22:40 INFO Signed narinfos id=14 count=14480lb # [ 59.346090] niks3-server[1034]: 2026/09/21 13:22:40 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls4481lb # [ 59.356423] niks3-server[1034]: 2026/09/21 13:22:40 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4482lb # [ 59.364332] niks3-server[1034]: 2026/09/21 13:22:40 INFO Completed upload id=144483worker1 # [ 59.429648] nix-grpc-daemon[848]: time=2026-09-21T13:22:41.049Z level=INFO msg="Upload complete. (269ms)"4484lb # [ 59.369237] niks3-server[1034]: 2026/09/21 13:22:40 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo4485worker1 # [ 59.441207] nix-grpc-daemon[770]: ts=2026-09-21T13:22:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=7 outputs=14486worker2 # [ 59.366162] nix-grpc-daemon[772]: ts=2026-09-21T13:22:40Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:341944487worker2 # [ 59.374109] nix-grpc-daemon[772]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'...4488worker1 # [ 59.460685] nix-daemon[835]: reaped child process 1082, status = succeeded4489worker1 # [ 59.500579] nix-grpc-daemon[770]: ts=2026-09-21T13:22:41Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:365624490worker1 # [ 59.507892] nix-grpc-daemon[770]: ts=2026-09-21T13:22:41Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=0 paths=1 nar_bytes_out=1764491worker2 # [ 59.670636] nix-grpc-daemon[772]: ts=2026-09-21T13:22:41Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:341944492worker2 # [ 59.679138] nix-grpc-daemon[772]: ts=2026-09-21T13:22:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:341944493worker2 # [ 59.686998] nix-grpc-daemon[912]: time=2026-09-21T13:22:41.307Z level=INFO msg="All 2 paths already cached"4494worker2 # [ 59.699959] nix-grpc-daemon[772]: ts=2026-09-21T13:22:41Z level=debug event=want client=8 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=04495worker2 # [ 59.705868] nix-grpc-daemon[772]: ts=2026-09-21T13:22:41Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.3:50051 clients=14496worker1 # [ 59.832700] nix-grpc-daemon[770]: ts=2026-09-21T13:22:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624497worker1 # [ 59.849260] nix-grpc-daemon[770]: ts=2026-09-21T13:22:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:365624498worker1 # [ 59.864195] nix-grpc-daemon[848]: time=2026-09-21T13:22:41.483Z level=INFO msg="All 2 paths already cached"4499worker1 # [ 59.874086] nix-grpc-daemon[770]: ts=2026-09-21T13:22:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:365624500worker1 # [ 59.888197] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4501worker1 # [ 60.047695] nix-daemon[835]: accepted connection from pid 1107, user nix-grpc-daemon (trusted)4502worker1 # [ 60.075237] nix-daemon[835]: reaped child process 1110, status = succeeded4503worker1 # [ 60.168502] nix-daemon[835]: accepted connection from pid 1112, user nix-grpc-daemon (trusted)4504worker1 # [ 60.184689] nix-daemon[835]: reaped child process 1115, status = succeeded4505lb # [ 60.124871] niks3-server[1034]: 2026/09/21 13:22:41 INFO Received uploads request method=POST path=/api/pending_closures4506worker1 # [ 60.200966] nix-grpc-daemon[848]: time=2026-09-21T13:22:41.820Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4507worker1 # [ 60.202479] nix-grpc-daemon[848]: time=2026-09-21T13:22:41.820Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)"4508lb # [ 60.157000] niks3-server[1034]: 2026/09/21 13:22:41 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst4509worker1 # [ 60.233144] nix-grpc-daemon[848]: time=2026-09-21T13:22:41.852Z level=INFO msg="Uploading 1 narinfos"4510lb # [ 60.173374] niks3-server[1034]: 2026/09/21 13:22:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4511lb # [ 60.182998] niks3-server[1034]: 2026/09/21 13:22:41 INFO Signed narinfos id=15 count=14512lb # [ 60.190045] niks3-server[1034]: 2026/09/21 13:22:41 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls4513worker1 # [ 60.261333] nix-grpc-daemon[848]: time=2026-09-21T13:22:41.880Z level=INFO msg="Upload complete. (354ms)"4514lb # [ 60.196901] niks3-server[1034]: 2026/09/21 13:22:41 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4515lb # [ 60.202024] niks3-server[1034]: 2026/09/21 13:22:41 INFO Completed upload id=154516lb # [ 60.204950] niks3-server[1034]: 2026/09/21 13:22:41 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo4517worker1 # [ 60.268133] nix-grpc-daemon[770]: ts=2026-09-21T13:22:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=0 paths=1 nar_bytes_in=7524518worker1 # [ 60.281994] nix-grpc-daemon[770]: ts=2026-09-21T13:22:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624519worker1 # [ 60.291222] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4520worker1 # [ 60.294504] nix-daemon[835]: reaped child process 1105, status = succeeded4521worker1 # [ 60.304738] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4522worker1 # [ 61.343657] nix-daemon[835]: reaped child process 1119, status = succeeded4523worker1 # [ 61.435187] nix-daemon[835]: accepted connection from pid 1135, user nix-grpc-daemon (trusted)4524worker1 # [ 61.452951] nix-daemon[835]: reaped child process 1138, status = succeeded4525lb # [ 61.394415] niks3-server[1034]: 2026/09/21 13:22:43 INFO Received uploads request method=POST path=/api/pending_closures4526worker1 # [ 61.471445] nix-grpc-daemon[848]: time=2026-09-21T13:22:43.088Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4527worker1 # [ 61.473052] nix-grpc-daemon[848]: time=2026-09-21T13:22:43.088Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)"4528lb # [ 61.428920] niks3-server[1034]: 2026/09/21 13:22:43 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst4529lb # [ 61.445568] niks3-server[1034]: 2026/09/21 13:22:43 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv4530worker1 # [ 61.522217] nix-grpc-daemon[848]: time=2026-09-21T13:22:43.141Z level=INFO msg="Uploading 1 narinfos"4531lb # [ 61.458344] niks3-server[1034]: 2026/09/21 13:22:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4532lb # [ 61.468316] niks3-server[1034]: 2026/09/21 13:22:43 INFO Signed narinfos id=16 count=14533lb # [ 61.470649] niks3-server[1034]: 2026/09/21 13:22:43 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls4534lb # [ 61.483161] niks3-server[1034]: 2026/09/21 13:22:43 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4535lb # [ 61.486532] niks3-server[1034]: 2026/09/21 13:22:43 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo4536lb # [ 61.490365] niks3-server[1034]: 2026/09/21 13:22:43 INFO Completed upload id=164537worker1 # [ 61.556786] nix-grpc-daemon[848]: time=2026-09-21T13:22:43.176Z level=INFO msg="Upload complete. (240ms)"4538worker2 # [ 61.486386] nix-grpc-daemon[772]: ts=2026-09-21T13:22:43Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:341944539worker1 # [ 61.569127] nix-grpc-daemon[770]: ts=2026-09-21T13:22:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=8 outputs=14540worker1 # [ 61.584414] nix-daemon[835]: reaped child process 1117, status = succeeded4541worker2 # [ 61.498618] nix-grpc-daemon[772]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'...4542worker2 # [ 61.535423] nix-grpc-daemon[772]: ts=2026-09-21T13:22:43Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:341944543worker2 # [ 61.540548] nix-grpc-daemon[772]: ts=2026-09-21T13:22:43Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34194 duration_s=0 paths=1 nar_bytes_out=2324544client # [ 61.689076] nix-daemon[989]: reaped child process 991, status = succeeded4545client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.25 seconds)4546client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top4547client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.04 seconds)4548client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4549client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds)4550client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4551client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)4552(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.37 seconds)4553subtest: an input only one worker has still reaches the builder4554client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only4555client # this derivation will be built:4556client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv4557client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'...4558client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only, in 0.30 seconds)4559client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure4560client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.12 seconds)4561worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4562worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.19 seconds)4563worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only4564worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.03 seconds)4565client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&24566client # [ 62.658354] nix-daemon[989]: accepted connection from pid 1071, user root (trusted)4567client # this derivation will be built:4568client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4569client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4570worker2 # [ 62.791275] nix-grpc-daemon[772]: ts=2026-09-21T13:22:44Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:341944571worker1 # [ 62.881661] nix-grpc-daemon[770]: ts=2026-09-21T13:22:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:365624572worker1 # [ 63.049199] nix-daemon[835]: accepted connection from pid 1145, user nix-grpc-daemon (trusted)4573worker1 # [ 63.069970] nix-daemon[835]: reaped child process 1148, status = succeeded4574lb # [ 63.001528] niks3-server[1034]: 2026/09/21 13:22:44 INFO Received uploads request method=POST path=/api/pending_closures4575worker1 # [ 63.078061] nix-grpc-daemon[848]: time=2026-09-21T13:22:44.697Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4576worker1 # [ 63.079741] nix-grpc-daemon[848]: time=2026-09-21T13:22:44.697Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)"4577lb # [ 63.030733] niks3-server[1034]: 2026/09/21 13:22:44 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4578worker1 # [ 63.118731] nix-grpc-daemon[848]: time=2026-09-21T13:22:44.738Z level=INFO msg="Uploading 1 narinfos"4579lb # [ 63.045845] niks3-server[1034]: 2026/09/21 13:22:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4580lb # [ 63.055040] niks3-server[1034]: 2026/09/21 13:22:44 INFO Signed narinfos id=17 count=14581lb # [ 63.063383] niks3-server[1034]: 2026/09/21 13:22:44 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls4582lb # [ 63.077220] niks3-server[1034]: 2026/09/21 13:22:44 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4583worker1 # [ 63.151192] nix-grpc-daemon[848]: time=2026-09-21T13:22:44.770Z level=INFO msg="Upload complete. (270ms)"4584lb # [ 63.086676] niks3-server[1034]: 2026/09/21 13:22:44 INFO Completed upload id=174585lb # [ 63.092888] niks3-server[1034]: 2026/09/21 13:22:44 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo4586worker2 # [ 63.089642] nix-grpc-daemon[772]: ts=2026-09-21T13:22:44Z level=debug event=want client=9 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=04587worker2 # [ 63.097487] nix-grpc-daemon[772]: ts=2026-09-21T13:22:44Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=14588worker2 # [ 63.135480] nix-grpc-daemon[772]: ts=2026-09-21T13:22:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:341944589worker2 # [ 63.150866] nix-grpc-daemon[772]: ts=2026-09-21T13:22:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:341944590client # copying 2 paths to 'lb:50051'...4591client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'...4592worker2 # [ 63.169777] nix-grpc-daemon[772]: ts=2026-09-21T13:22:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:341944593client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'...4594worker2 # [ 63.186409] nix-daemon[860]: accepted connection from pid 772, user nix-grpc-daemon (trusted)4595worker2 # [ 63.358305] nix-daemon[860]: accepted connection from pid 998, user nix-grpc-daemon (trusted)4596worker2 # [ 63.381134] nix-daemon[860]: reaped child process 1001, status = succeeded4597worker2 # [ 63.485222] nix-daemon[860]: accepted connection from pid 1003, user nix-grpc-daemon (trusted)4598worker2 # [ 63.502317] nix-daemon[860]: reaped child process 1006, status = succeeded4599lb # [ 63.526251] niks3-server[1034]: 2026/09/21 13:22:45 INFO Received uploads request method=POST path=/api/pending_closures4600lb # [ 63.535667] niks3-server[1034]: 2026/09/21 13:22:45 INFO Received uploads request method=POST path=/api/pending_closures4601worker2 # [ 63.528957] nix-grpc-daemon[912]: time=2026-09-21T13:22:45.150Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4602worker2 # [ 63.530456] nix-grpc-daemon[912]: time=2026-09-21T13:22:45.150Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)"4603worker2 # [ 63.532194] nix-grpc-daemon[912]: time=2026-09-21T13:22:45.150Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)"4604lb # [ 63.572120] niks3-server[1034]: 2026/09/21 13:22:45 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst4605lb # [ 63.585529] niks3-server[1034]: 2026/09/21 13:22:45 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst4606lb # [ 63.599051] niks3-server[1034]: 2026/09/21 13:22:45 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls4607lb # [ 63.606929] niks3-server[1034]: 2026/09/21 13:22:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign4608worker2 # [ 63.592136] nix-grpc-daemon[912]: time=2026-09-21T13:22:45.214Z level=INFO msg="Uploading 3 narinfos"4609lb # [ 63.616486] niks3-server[1034]: 2026/09/21 13:22:45 INFO Signed narinfos id=18 count=14610lb # [ 63.619012] niks3-server[1034]: 2026/09/21 13:22:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign4611lb # [ 63.628588] niks3-server[1034]: 2026/09/21 13:22:45 INFO Signed narinfos id=19 count=24612lb # [ 63.633618] niks3-server[1034]: 2026/09/21 13:22:45 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls4613lb # [ 63.647915] niks3-server[1034]: 2026/09/21 13:22:45 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo4614lb # [ 63.660244] niks3-server[1034]: 2026/09/21 13:22:45 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete4615lb # [ 63.662685] niks3-server[1034]: 2026/09/21 13:22:45 INFO Completed upload id=184616lb # [ 63.670433] niks3-server[1034]: 2026/09/21 13:22:45 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete4617lb # [ 63.676351] niks3-server[1034]: 2026/09/21 13:22:45 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4618lb # [ 63.717167] niks3-server[1034]: 2026/09/21 13:22:45 INFO Completed upload id=194619worker2 # [ 63.700680] nix-grpc-daemon[912]: time=2026-09-21T13:22:45.323Z level=INFO msg="Upload complete. (494ms)"4620worker2 # [ 63.711595] nix-grpc-daemon[772]: ts=2026-09-21T13:22:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34194 duration_s=0 paths=2 nar_bytes_in=9844621lb # [ 63.731416] niks3-server[1034]: 2026/09/21 13:22:45 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo4622worker2 # [ 63.731365] nix-grpc-daemon[772]: ts=2026-09-21T13:22:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:341944623worker2 # [ 63.743913] nix-daemon[860]: accepted connection from pid 772, user nix-grpc-daemon (trusted)4624worker2 # [ 63.749516] nix-daemon[860]: reaped child process 996, status = succeeded4625worker2 # [ 63.768223] nix-grpc-daemon[772]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'...4626client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051...4627client # uses-input-w1onlya> worker2: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4628worker2 # [ 63.790387] nix-daemon[860]: accepted connection from pid 772, user nix-grpc-daemon (trusted)4629client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds4630worker2 # [ 65.032577] nix-daemon[860]: reaped child process 1011, status = succeeded4631worker2 # [ 65.158547] nix-daemon[860]: accepted connection from pid 1028, user nix-grpc-daemon (trusted)4632worker2 # [ 65.178541] nix-daemon[860]: reaped child process 1031, status = succeeded4633lb # [ 65.204879] niks3-server[1034]: 2026/09/21 13:22:46 INFO Received uploads request method=POST path=/api/pending_closures4634worker2 # [ 65.210849] nix-grpc-daemon[912]: time=2026-09-21T13:22:46.832Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4635lb # [ 65.249233] niks3-server[1034]: 2026/09/21 13:22:46 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls4636lb # [ 65.259850] niks3-server[1034]: 2026/09/21 13:22:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign4637worker2 # [ 65.247484] nix-grpc-daemon[912]: time=2026-09-21T13:22:46.869Z level=INFO msg="Uploading 1 narinfos"4638lb # [ 65.267839] niks3-server[1034]: 2026/09/21 13:22:46 INFO Signed narinfos id=20 count=14639lb # [ 65.274759] niks3-server[1034]: 2026/09/21 13:22:46 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4640lb # [ 65.287009] niks3-server[1034]: 2026/09/21 13:22:46 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete4641worker2 # [ 65.276329] nix-grpc-daemon[912]: time=2026-09-21T13:22:46.898Z level=INFO msg="Upload complete. (271ms)"4642lb # [ 65.295987] niks3-server[1034]: 2026/09/21 13:22:46 INFO Completed upload id=204643lb # [ 65.299951] niks3-server[1034]: 2026/09/21 13:22:46 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo4644client # copying 1 paths...4645worker2 # [ 65.289172] nix-grpc-daemon[772]: ts=2026-09-21T13:22:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34194 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=9 outputs=14646worker1 # [ 65.380525] nix-grpc-daemon[770]: ts=2026-09-21T13:22:46Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:365624647worker2 # [ 65.309704] nix-daemon[860]: reaped child process 1008, status = succeeded4648worker1 # [ 65.394440] nix-grpc-daemon[770]: copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'http://lb:5751'...4649client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'...4650worker2 # [ 65.355094] nix-grpc-daemon[772]: ts=2026-09-21T13:22:46Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:341944651worker2 # [ 65.360641] nix-grpc-daemon[772]: ts=2026-09-21T13:22:46Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:34194 duration_s=0 paths=1 nar_bytes_out=1204652client # [ 65.454828] nix-daemon[989]: reaped child process 1074, status = succeeded4653client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.99 seconds)4654client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&24655client # [ 65.617132] nix-daemon[989]: accepted connection from pid 1093, user root (trusted)4656client # this derivation will be built:4657client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4658worker2 # [ 65.749475] nix-grpc-daemon[772]: ts=2026-09-21T13:22:47Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:341944659client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4660worker1 # [ 65.838077] nix-grpc-daemon[770]: ts=2026-09-21T13:22:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:365624661worker1 # [ 65.853401] nix-grpc-daemon[848]: time=2026-09-21T13:22:47.470Z level=INFO msg="All 1 paths already cached"4662worker2 # [ 65.781782] nix-grpc-daemon[772]: ts=2026-09-21T13:22:47Z level=debug event=want client=10 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=04663worker2 # [ 65.787901] nix-grpc-daemon[772]: ts=2026-09-21T13:22:47Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=14664worker1 # [ 65.911541] nix-grpc-daemon[770]: ts=2026-09-21T13:22:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624665worker1 # [ 65.926917] nix-grpc-daemon[770]: ts=2026-09-21T13:22:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:365624666client # copying 1 paths to 'lb:50051'...4667client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'...4668worker1 # [ 65.948840] nix-grpc-daemon[770]: ts=2026-09-21T13:22:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:365624669worker1 # [ 65.963626] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4670worker1 # [ 65.985333] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 81 ms (attempt 1/5)4671worker1 # [ 66.074523] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 92 ms (attempt 2/5)4672worker1 # [ 66.168633] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 247 ms (attempt 3/5)4673worker1 # [ 66.418662] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 717 ms (attempt 4/5)4674worker1 # [ 67.137430] nix-grpc-daemon[770]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4675worker1 # [ 67.141511] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4676worker1 # [ 67.151748] nix-grpc-daemon[770]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'...4677worker1 # [ 67.333007] nix-daemon[835]: accepted connection from pid 1158, user nix-grpc-daemon (trusted)4678worker1 # [ 67.356284] nix-daemon[835]: reaped child process 1161, status = succeeded4679worker1 # [ 67.463574] nix-daemon[835]: accepted connection from pid 1163, user nix-grpc-daemon (trusted)4680worker1 # [ 67.482938] nix-daemon[835]: reaped child process 1166, status = succeeded4681lb # [ 67.421666] niks3-server[1034]: 2026/09/21 13:22:49 INFO Received uploads request method=POST path=/api/pending_closures4682worker1 # [ 67.499544] nix-grpc-daemon[848]: time=2026-09-21T13:22:49.118Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4683worker1 # [ 67.501202] nix-grpc-daemon[848]: time=2026-09-21T13:22:49.118Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)"4684lb # [ 67.457508] niks3-server[1034]: 2026/09/21 13:22:49 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst4685worker1 # [ 67.537319] nix-grpc-daemon[848]: time=2026-09-21T13:22:49.156Z level=INFO msg="Uploading 1 narinfos"4686lb # [ 67.473739] niks3-server[1034]: 2026/09/21 13:22:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign4687lb # [ 67.482312] niks3-server[1034]: 2026/09/21 13:22:49 INFO Signed narinfos id=21 count=14688lb # [ 67.492781] niks3-server[1034]: 2026/09/21 13:22:49 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls4689worker1 # [ 67.569999] nix-grpc-daemon[848]: time=2026-09-21T13:22:49.189Z level=INFO msg="Upload complete. (396ms)"4690lb # [ 67.503048] niks3-server[1034]: 2026/09/21 13:22:49 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete4691lb # [ 67.508934] niks3-server[1034]: 2026/09/21 13:22:49 INFO Completed upload id=214692lb # [ 67.512680] niks3-server[1034]: 2026/09/21 13:22:49 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo4693worker1 # [ 67.580893] nix-grpc-daemon[770]: ts=2026-09-21T13:22:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=1 paths=1 nar_bytes_in=5684694worker1 # [ 67.588807] nix-grpc-daemon[770]: ts=2026-09-21T13:22:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:365624695worker1 # [ 67.603314] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4696worker1 # [ 67.605647] nix-daemon[835]: reaped child process 1151, status = succeeded4697client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051...4698worker1 # [ 67.619529] nix-daemon[835]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4699client # uses-input-w1onlyb> node-a: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4700client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds4701lb # [ 68.662366] envoy[897]: conn peer=[2001:db8:1::1]:52406 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=375 ms=324702worker1 # [ 69.128900] nix-daemon[835]: reaped child process 1170, status = succeeded4703worker1 # [ 69.200140] nix-daemon[835]: accepted connection from pid 1186, user nix-grpc-daemon (trusted)4704worker1 # [ 69.221996] nix-daemon[835]: reaped child process 1189, status = succeeded4705lb # [ 68.667429] envoy[897]: conn peer=[2001:db8:1::1]:52404 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3887 tx=2068 ms=21734706lb # [ 68.667556] envoy[897]: conn peer=[2001:db8:1::1]:52420 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4339 tx=2302 ms=19274707lb # [ 68.667718] envoy[897]: conn peer=[2001:db8:1::1]:35740 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=438 tx=406 ms=654708lb # [ 68.667845] envoy[897]: conn peer=[2001:db8:1::1]:35752 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3892 tx=1973 ms=25814709lb # [ 68.667943] envoy[897]: conn peer=[2001:db8:1::1]:35766 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=328 ms=194710lb # [ 69.154466] niks3-server[1034]: 2026/09/21 13:22:50 INFO Received uploads request method=POST path=/api/pending_closures4711worker1 # [ 69.256601] nix-grpc-daemon[848]: time=2026-09-21T13:22:50.874Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4712lb # [ 69.214527] niks3-server[1034]: 2026/09/21 13:22:50 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls4713worker1 # [ 69.291427] nix-grpc-daemon[848]: time=2026-09-21T13:22:50.910Z level=INFO msg="Uploading 1 narinfos"4714lb # [ 69.228936] niks3-server[1034]: 2026/09/21 13:22:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign4715lb # [ 69.240211] niks3-server[1034]: 2026/09/21 13:22:50 INFO Signed narinfos id=22 count=14716lb # [ 69.242557] niks3-server[1034]: 2026/09/21 13:22:50 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv4717lb # [ 69.254633] niks3-server[1034]: 2026/09/21 13:22:50 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete4718lb # [ 69.259416] niks3-server[1034]: 2026/09/21 13:22:50 INFO Completed upload id=224719worker1 # [ 69.325216] nix-grpc-daemon[848]: time=2026-09-21T13:22:50.944Z level=INFO msg="Upload complete. (294ms)"4720lb # [ 69.261756] niks3-server[1034]: 2026/09/21 13:22:50 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo4721worker1 # [ 69.337235] nix-grpc-daemon[770]: ts=2026-09-21T13:22:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=10 outputs=14722worker1 # [ 69.356313] nix-daemon[835]: reaped child process 1168, status = succeeded4723client # copying 1 paths...4724worker1 # [ 69.385407] nix-grpc-daemon[770]: ts=2026-09-21T13:22:51Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:365624725client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'...4726worker1 # [ 69.413784] nix-grpc-daemon[770]: ts=2026-09-21T13:22:51Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:365624727worker1 # [ 69.419761] nix-grpc-daemon[770]: ts=2026-09-21T13:22:51Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=0 paths=1 nar_bytes_out=1204728client # [ 69.485744] nix-daemon[989]: reaped child process 1096, status = succeeded4729client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2, in 4.03 seconds)4730client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&24731client # [ 69.664183] nix-daemon[989]: accepted connection from pid 1115, user root (trusted)4732client # this derivation will be built:4733client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4734client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4735worker2 # [ 69.793063] nix-grpc-daemon[772]: ts=2026-09-21T13:22:51Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:341944736worker2 # [ 69.800724] nix-grpc-daemon[772]: ts=2026-09-21T13:22:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:341944737worker2 # [ 69.811260] nix-grpc-daemon[912]: time=2026-09-21T13:22:51.433Z level=INFO msg="All 1 paths already cached"4738worker2 # [ 69.830109] nix-grpc-daemon[772]: ts=2026-09-21T13:22:51Z level=debug event=want client=11 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=04739worker2 # [ 69.839921] nix-grpc-daemon[772]: ts=2026-09-21T13:22:51Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=14740worker2 # [ 69.849052] nix-grpc-daemon[772]: ts=2026-09-21T13:22:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:341944741worker2 # [ 69.856765] nix-grpc-daemon[772]: ts=2026-09-21T13:22:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:341944742worker2 # [ 69.868355] nix-grpc-daemon[912]: time=2026-09-21T13:22:51.490Z level=INFO msg="All 1 paths already cached"4743client # copying 1 paths to 'lb:50051'...4744client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'...4745worker2 # [ 69.879739] nix-grpc-daemon[772]: ts=2026-09-21T13:22:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:341944746worker2 # [ 69.894715] nix-daemon[860]: accepted connection from pid 772, user nix-grpc-daemon (trusted)4747worker2 # [ 70.049315] nix-daemon[860]: accepted connection from pid 1039, user nix-grpc-daemon (trusted)4748worker2 # [ 70.070631] nix-daemon[860]: reaped child process 1042, status = succeeded4749worker2 # [ 70.175327] nix-daemon[860]: accepted connection from pid 1044, user nix-grpc-daemon (trusted)4750worker2 # [ 70.193287] nix-daemon[860]: reaped child process 1047, status = succeeded4751lb # [ 70.215098] niks3-server[1034]: 2026/09/21 13:22:51 INFO Received uploads request method=POST path=/api/pending_closures4752worker2 # [ 70.207450] nix-grpc-daemon[912]: time=2026-09-21T13:22:51.829Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4753worker2 # [ 70.209102] nix-grpc-daemon[912]: time=2026-09-21T13:22:51.830Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)"4754lb # [ 70.243451] niks3-server[1034]: 2026/09/21 13:22:51 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst4755lb # [ 70.261598] niks3-server[1034]: 2026/09/21 13:22:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign4756worker2 # [ 70.251238] nix-grpc-daemon[912]: time=2026-09-21T13:22:51.873Z level=INFO msg="Uploading 1 narinfos"4757lb # [ 70.273202] niks3-server[1034]: 2026/09/21 13:22:51 INFO Signed narinfos id=23 count=14758lb # [ 70.275727] niks3-server[1034]: 2026/09/21 13:22:51 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls4759worker2 # [ 70.272997] nix-grpc-daemon[912]: time=2026-09-21T13:22:51.895Z level=INFO msg="Upload complete. (361ms)"4760lb # [ 70.290711] niks3-server[1034]: 2026/09/21 13:22:51 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete4761lb # [ 70.295768] niks3-server[1034]: 2026/09/21 13:22:51 INFO Completed upload id=234762worker2 # [ 70.280112] nix-grpc-daemon[772]: ts=2026-09-21T13:22:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34194 duration_s=0 paths=1 nar_bytes_in=5684763lb # [ 70.299379] niks3-server[1034]: 2026/09/21 13:22:51 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo4764worker2 # [ 70.295712] nix-grpc-daemon[772]: ts=2026-09-21T13:22:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:341944765worker2 # [ 70.301673] nix-daemon[860]: accepted connection from pid 772, user nix-grpc-daemon (trusted)4766worker2 # [ 70.306422] nix-daemon[860]: reaped child process 1037, status = succeeded4767client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051...4768worker2 # [ 70.318587] nix-daemon[860]: accepted connection from pid 772, user nix-grpc-daemon (trusted)4769client # uses-input-w1onlyc> worker2: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4770client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds4771worker2 # [ 71.800532] nix-daemon[860]: reaped child process 1051, status = succeeded4772worker2 # [ 71.903388] nix-daemon[860]: accepted connection from pid 1067, user nix-grpc-daemon (trusted)4773worker2 # [ 71.921265] nix-daemon[860]: reaped child process 1070, status = succeeded4774lb # [ 71.945011] niks3-server[1034]: 2026/09/21 13:22:53 INFO Received uploads request method=POST path=/api/pending_closures4775worker2 # [ 71.940436] nix-grpc-daemon[912]: time=2026-09-21T13:22:53.562Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"4776lb # [ 71.975290] niks3-server[1034]: 2026/09/21 13:22:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign4777worker2 # [ 71.967342] nix-grpc-daemon[912]: time=2026-09-21T13:22:53.589Z level=INFO msg="Uploading 1 narinfos"4778lb # [ 71.986348] niks3-server[1034]: 2026/09/21 13:22:53 INFO Signed narinfos id=24 count=14779lb # [ 71.996979] niks3-server[1034]: 2026/09/21 13:22:53 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls4780lb # [ 72.009325] niks3-server[1034]: 2026/09/21 13:22:53 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv4781worker2 # [ 71.999131] nix-grpc-daemon[912]: time=2026-09-21T13:22:53.621Z level=INFO msg="Upload complete. (270ms)"4782lb # [ 72.017896] niks3-server[1034]: 2026/09/21 13:22:53 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete4783lb # [ 72.023164] niks3-server[1034]: 2026/09/21 13:22:53 INFO Completed upload id=244784client # copying 1 paths...4785lb # [ 72.027534] niks3-server[1034]: 2026/09/21 13:22:53 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo4786worker2 # [ 72.010800] nix-grpc-daemon[772]: ts=2026-09-21T13:22:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34194 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=11 outputs=14787worker1 # [ 72.101609] nix-grpc-daemon[770]: ts=2026-09-21T13:22:53Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:365624788worker2 # [ 72.028087] nix-daemon[860]: reaped child process 1049, status = succeeded4789worker1 # [ 72.114258] nix-grpc-daemon[770]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'...4790client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'...4791worker1 # [ 72.157938] nix-grpc-daemon[770]: ts=2026-09-21T13:22:53Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:365624792worker1 # [ 72.163270] nix-grpc-daemon[770]: ts=2026-09-21T13:22:53Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:36562 duration_s=0 paths=1 nar_bytes_out=1204793client # [ 72.217625] nix-daemon[989]: reaped child process 1118, status = succeeded4794client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.73 seconds)4795(finished: subtest: an input only one worker has still reaches the builder, in 10.40 seconds)4796subtest: upload whose reference the worker lacks is completed from the cache4797client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache4798client # this derivation will be built:4799client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv4800client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'...4801client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache, in 0.33 seconds)4802client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache4803client # this derivation will be built:4804client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv4805client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'...4806client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache, in 0.28 seconds)4807client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure4808client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.12 seconds)4809worker1: must succeed: nix-store --import < /tmp/shared/ref.closure4810worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.19 seconds)4811worker2 # [ 73.203715] systemd[1]: Stopping Nix worker-protocol over gRPC...4812worker2 # [ 73.977058] nix-grpc-daemon[772]: ts=2026-09-21T13:22:55Z level=info event=signal_exit4813worker2 # [ 74.376094] nix-grpc-daemon[772]: ts=2026-09-21T13:22:55Z level=info event=scheduler_yield4814lb # [ 74.397575] niks3-server[1034]: 2026/09/21 13:22:56 INFO lead: released remote=192.168.1.4:560284815worker2 # [ 74.382379] nix-grpc-daemon[772]: ts=2026-09-21T13:22:56Z level=info event=scheduler_restarting addr=lb:500514816worker2 # [ 74.388185] nix-grpc-daemon[772]: ts=2026-09-21T13:22:56Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:341864817worker2 # [ 74.392671] nix-grpc-daemon[772]: ts=2026-09-21T13:22:56Z level=info event=worker_gone addr=192.168.1.4:500514818worker1 # [ 74.481520] nix-grpc-daemon[770]: ts=2026-09-21T13:22:56Z level=info event=scheduler_restarting addr=lb:500514819worker2 # [ 74.398382] nix-grpc-daemon[772]: ts=2026-09-21T13:22:56Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:341864820worker2 # [ 74.403210] nix-grpc-daemon[772]: ts=2026-09-21T13:22:56Z level=info event=worker_gone addr=192.168.1.3:500514821worker2 # [ 74.583956] nix-grpc-daemon[772]: ts=2026-09-21T13:22:56Z level=info event=scheduler_connected addr=lb:500514822worker2 # [ 74.589048] nix-grpc-daemon[772]: ts=2026-09-21T13:22:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4823worker1 # [ 74.681303] nix-grpc-daemon[770]: ts=2026-09-21T13:22:56Z level=info event=scheduler_connected addr=lb:500514824worker1 # [ 74.686352] nix-grpc-daemon[770]: ts=2026-09-21T13:22:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4825worker2 # [ 74.789970] nix-grpc-daemon[772]: ts=2026-09-21T13:22:56Z level=info event=scheduler_connected addr=lb:500514826worker2 # [ 74.795049] nix-grpc-daemon[772]: ts=2026-09-21T13:22:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4827worker1 # [ 74.887188] nix-grpc-daemon[770]: ts=2026-09-21T13:22:56Z level=info event=scheduler_connected addr=lb:500514828worker1 # [ 74.892911] nix-grpc-daemon[770]: ts=2026-09-21T13:22:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4829worker2 # [ 74.997869] nix-grpc-daemon[772]: ts=2026-09-21T13:22:56Z level=info event=scheduler_connected addr=lb:500514830worker2 # [ 75.002079] nix-grpc-daemon[772]: ts=2026-09-21T13:22:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4831worker1 # [ 75.093691] nix-grpc-daemon[770]: ts=2026-09-21T13:22:56Z level=info event=scheduler_connected addr=lb:500514832worker1 # [ 75.098833] nix-grpc-daemon[770]: ts=2026-09-21T13:22:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4833worker2 # [ 75.198815] nix-grpc-daemon[772]: ts=2026-09-21T13:22:56Z level=info event=scheduler_connected addr=lb:500514834worker2 # [ 75.205773] nix-grpc-daemon[772]: ts=2026-09-21T13:22:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4835worker1 # [ 75.299489] nix-grpc-daemon[770]: ts=2026-09-21T13:22:56Z level=info event=scheduler_connected addr=lb:500514836worker1 # [ 75.304754] nix-grpc-daemon[770]: ts=2026-09-21T13:22:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4837worker2 # [ 75.382061] nix-grpc-daemon[772]: ts=2026-09-21T13:22:57Z level=info event=scheduler_restarting4838worker1 # [ 75.507224] nix-grpc-daemon[770]: ts=2026-09-21T13:22:57Z level=info event=scheduler_connected addr=lb:500514839worker1 # [ 75.512100] nix-grpc-daemon[770]: ts=2026-09-21T13:22:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4840worker2 # [ 75.429140] nix-grpc-daemon[772]: ts=2026-09-21T13:22:57Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4841worker2 # [ 75.443543] nix-daemon[860]: reaped child process 862, status = succeeded4842worker1 # [ 75.707791] nix-grpc-daemon[770]: ts=2026-09-21T13:22:57Z level=info event=scheduler_connected addr=lb:500514843worker1 # [ 75.713109] nix-grpc-daemon[770]: ts=2026-09-21T13:22:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4844worker1 # [ 75.913846] nix-grpc-daemon[770]: ts=2026-09-21T13:22:57Z level=info event=scheduler_connected addr=lb:500514845worker1 # [ 75.918951] nix-grpc-daemon[770]: ts=2026-09-21T13:22:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4846worker1 # [ 76.119806] nix-grpc-daemon[770]: ts=2026-09-21T13:22:57Z level=info event=scheduler_connected addr=lb:500514847worker1 # [ 76.125060] nix-grpc-daemon[770]: ts=2026-09-21T13:22:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4848worker2 # [ 76.092102] nix-grpc-daemon[772]: E0000 00:00:1789996977.713514 817 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true4849worker2 # [ 76.134146] nix-grpc-daemon[772]: E0000 00:00:1789996977.756724 817 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true4850worker1 # [ 76.325649] nix-grpc-daemon[770]: ts=2026-09-21T13:22:57Z level=info event=scheduler_connected addr=lb:500514851worker1 # [ 76.330689] nix-grpc-daemon[770]: ts=2026-09-21T13:22:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4852worker1 # [ 76.455814] nix-grpc-daemon[770]: ts=2026-09-21T13:22:58Z level=info event=scheduler_take_over4853lb # [ 76.389554] niks3-server[1034]: 2026/09/21 13:22:58 INFO lead: acquired remote=192.168.1.3:373764854worker1 # [ 76.533268] nix-grpc-daemon[770]: ts=2026-09-21T13:22:58Z level=info event=scheduler_connected addr=lb:500514855worker1 # [ 76.537382] nix-grpc-daemon[770]: ts=2026-09-21T13:22:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4856worker2 # [ 76.494466] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4857worker2 # [ 76.501057] systemd[1]: Stopped Nix worker-protocol over gRPC.4858worker2 # [ 76.503593] systemd[1]: nix-grpc-daemon.service: Consumed 1.963s CPU time over 57.740s wall clock time, 57.3M memory peak, 58.8K incoming IP traffic, 169.9K outgoing IP traffic.4859worker2 # [ 76.525157] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.4860worker2 # [ 76.527535] systemd[1]: Closed Nix worker-protocol over gRPC.4861lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy4862lb: must succeed: curl -sf localhost:9901/clusters4863worker2 # [ 76.536925] systemd[1]: nix-grpc-daemon.socket: Consumed 584us CPU time over 1min 6.467s wall clock time, 256K memory peak, 74.6K incoming IP traffic, 64K outgoing IP traffic.4864lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)4865lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.07 seconds)4866client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&24867worker1 # [ 76.733947] nix-grpc-daemon[770]: ts=2026-09-21T13:22:58Z level=info event=scheduler_connected addr=lb:500514868worker1 # [ 76.739063] nix-grpc-daemon[770]: ts=2026-09-21T13:22:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4869worker1 # [ 76.901270] nix-grpc-daemon[770]: ts=2026-09-21T13:22:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:365624870client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4871worker1 # [ 76.909568] nix-grpc-daemon[770]: ts=2026-09-21T13:22:58Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:365624872client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.24 seconds)4873worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache4874worker1 # [ 76.939713] nix-grpc-daemon[770]: ts=2026-09-21T13:22:58Z level=info event=scheduler_connected addr=lb:500514875worker1 # [ 76.944929] nix-grpc-daemon[770]: ts=2026-09-21T13:22:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4876worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.03 seconds)4877worker2 # [ 76.942409] systemd[1]: Listening on Nix worker-protocol over gRPC.4878worker1 # [ 77.095872] systemd[1]: Stopping Nix worker-protocol over gRPC...4879worker1 # [ 77.145669] nix-grpc-daemon[770]: ts=2026-09-21T13:22:58Z level=info event=scheduler_connected addr=lb:500514880worker1 # [ 77.150717] nix-grpc-daemon[770]: ts=2026-09-21T13:22:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4881worker1 # [ 77.351569] nix-grpc-daemon[770]: ts=2026-09-21T13:22:58Z level=info event=scheduler_connected addr=lb:500514882worker1 # [ 77.369903] nix-grpc-daemon[770]: ts=2026-09-21T13:22:58Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:532104883worker1 # [ 77.375287] nix-grpc-daemon[770]: ts=2026-09-21T13:22:58Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=04884worker2 # [ 77.491384] systemd[1]: Starting scheduler-token.service...4885worker2 # [ 77.576834] systemd[1]: scheduler-token.service: Deactivated successfully.4886worker2 # [ 77.584436] systemd[1]: Finished scheduler-token.service.4887worker2 # [ 77.586614] systemd[1]: scheduler-token.service: Consumed 56ms CPU time over 88ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.4888worker2 # [ 77.609000] systemd[1]: Starting Nix worker-protocol over gRPC...4889worker2 # [ 77.729699] nix-grpc-daemon[1094]: ts=2026-09-21T13:22:59Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"4890worker2 # [ 77.731903] nix-grpc-daemon[1094]: ts=2026-09-21T13:22:59Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc4891worker2 # [ 77.735779] nix-grpc-daemon[1094]: ts=2026-09-21T13:22:59Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=14892worker2 # [ 77.742616] nix-grpc-daemon[1094]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR4893worker2 # [ 77.754106] nix-grpc-daemon[1094]: E0000 00:00:1789996979.363192 1094 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4894worker2 # [ 77.757655] nix-grpc-daemon[1094]: E0000 00:00:1789996979.365233 1094 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4895worker2 # [ 77.764900] nix-grpc-daemon[1094]: ts=2026-09-21T13:22:59Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket4896worker2 # [ 77.766526] nix-grpc-daemon[1094]: E0000 00:00:1789996979.384537 1136 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4897worker2 # [ 77.797860] systemd[1]: Started Nix worker-protocol over gRPC.4898worker2 # [ 77.799171] nix-grpc-daemon[1094]: E0000 00:00:1789996979.405368 1136 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4899worker2 # [ 77.801781] nix-grpc-daemon[1094]: E0000 00:00:1789996979.405503 1136 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.4900worker2 # [ 77.806792] nix-grpc-daemon[1094]: E0000 00:00:1789996979.419876 1136 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.4901worker2 # [ 77.811131] nix-grpc-daemon[1094]: ts=2026-09-21T13:22:59Z level=info event=scheduler_connected addr=lb:500514902worker1 # [ 77.898348] nix-grpc-daemon[770]: ts=2026-09-21T13:22:59Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:532104903worker1 # [ 77.904302] nix-grpc-daemon[770]: ts=2026-09-21T13:22:59Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=04904worker1 # [ 78.059916] nix-grpc-daemon[770]: ts=2026-09-21T13:22:59Z level=info event=signal_exit4905worker1 # [ 78.459244] nix-grpc-daemon[770]: ts=2026-09-21T13:23:00Z level=info event=scheduler_yield4906worker2 # [ 78.384959] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:00Z level=info event=scheduler_restarting addr=lb:500514907lb # [ 78.401890] niks3-server[1034]: 2026/09/21 13:23:00 INFO lead: released remote=192.168.1.3:373764908worker1 # [ 78.470574] nix-grpc-daemon[770]: ts=2026-09-21T13:23:00Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:532104909worker1 # [ 78.477646] nix-grpc-daemon[770]: ts=2026-09-21T13:23:00Z level=info event=worker_gone addr=192.168.1.4:500514910worker1 # [ 78.481774] nix-grpc-daemon[770]: ts=2026-09-21T13:23:00Z level=info event=scheduler_restarting addr=lb:500514911worker1 # [ 78.485790] nix-grpc-daemon[770]: ts=2026-09-21T13:23:00Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:532104912worker1 # [ 78.490279] nix-grpc-daemon[770]: ts=2026-09-21T13:23:00Z level=info event=worker_gone addr=192.168.1.3:500514913worker2 # [ 78.586473] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:00Z level=info event=scheduler_connected addr=lb:500514914worker2 # [ 78.591781] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4915worker1 # [ 78.677007] nix-grpc-daemon[770]: ts=2026-09-21T13:23:00Z level=info event=scheduler_connected addr=lb:500514916worker1 # [ 78.682183] nix-grpc-daemon[770]: ts=2026-09-21T13:23:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4917lb # [ 78.664965] envoy[897]: conn peer=[2001:db8:1::1]:35776 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3769 tx=1938 ms=36464918lb # [ 78.670073] envoy[897]: conn peer=[2001:db8:1::1]:35788 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=724919lb # [ 78.674935] envoy[897]: conn peer=[2001:db8:1::1]:35790 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3740 tx=1987 ms=23404920lb # [ 78.679805] envoy[897]: conn peer=[2001:db8:1::1]:55930 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=624921lb # [ 78.684802] envoy[897]: conn peer=[2001:db8:1::4]:52042 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=53490 tx=2760 ms=563884922lb # [ 78.689880] envoy[897]: conn peer=[2001:db8:1::1]:55936 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=214923worker2 # [ 78.794314] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:00Z level=info event=scheduler_connected addr=lb:500514924worker2 # [ 78.798469] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4925worker1 # [ 78.883128] nix-grpc-daemon[770]: ts=2026-09-21T13:23:00Z level=info event=scheduler_connected addr=lb:500514926worker1 # [ 78.888536] nix-grpc-daemon[770]: ts=2026-09-21T13:23:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4927worker2 # [ 78.995456] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:00Z level=info event=scheduler_connected addr=lb:500514928worker2 # [ 79.000694] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4929worker1 # [ 79.091044] nix-grpc-daemon[770]: ts=2026-09-21T13:23:00Z level=info event=scheduler_connected addr=lb:500514930worker1 # [ 79.095363] nix-grpc-daemon[770]: ts=2026-09-21T13:23:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4931worker2 # [ 79.201937] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:00Z level=info event=scheduler_connected addr=lb:500514932worker1 # [ 79.291333] nix-grpc-daemon[770]: ts=2026-09-21T13:23:00Z level=info event=scheduler_connected addr=lb:500514933worker2 # [ 79.207024] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4934worker1 # [ 79.296624] nix-grpc-daemon[770]: ts=2026-09-21T13:23:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4935worker1 # [ 79.464111] nix-grpc-daemon[770]: ts=2026-09-21T13:23:01Z level=info event=scheduler_restarting4936worker2 # [ 79.408095] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:01Z level=info event=scheduler_connected addr=lb:500514937worker2 # [ 79.413071] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4938worker1 # [ 79.521788] nix-grpc-daemon[770]: ts=2026-09-21T13:23:01Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"4939worker1 # [ 79.527116] nix-daemon[835]: reaped child process 837, status = succeeded4940worker2 # [ 79.613887] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:01Z level=info event=scheduler_connected addr=lb:500514941worker2 # [ 79.619008] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4942worker2 # [ 79.821819] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:01Z level=info event=scheduler_connected addr=lb:500514943worker2 # [ 79.825930] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4944worker2 # [ 80.022705] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:01Z level=info event=scheduler_connected addr=lb:500514945worker2 # [ 80.027822] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4946worker1 # [ 80.197263] nix-grpc-daemon[770]: E0000 00:00:1789996981.815322 815 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true4947worker2 # [ 80.228982] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:01Z level=info event=scheduler_connected addr=lb:500514948worker2 # [ 80.234111] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4949worker2 # [ 80.435265] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:02Z level=info event=scheduler_connected addr=lb:500514950worker2 # [ 80.441163] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4951worker1 # [ 80.203931] nix-grpc-daemon[770]: E0000 00:00:1789996981.815702 815 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true4952worker1 # [ 80.528124] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.4953worker1 # [ 80.546865] systemd[1]: Stopped Nix worker-protocol over gRPC.4954worker1 # [ 80.554817] systemd[1]: nix-grpc-daemon.service: Consumed 3.977s CPU time over 1min 1.796s wall clock time, 65.2M memory peak, 94.5K incoming IP traffic, 171.8K outgoing IP traffic.4955worker1 # [ 80.568731] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.4956worker1 # [ 80.570650] systemd[1]: Closed Nix worker-protocol over gRPC.4957lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy4958lb: must succeed: curl -sf localhost:9901/clusters4959worker1 # [ 80.574545] systemd[1]: nix-grpc-daemon.socket: Consumed 549us CPU time over 1min 11.315s wall clock time, 256K memory peak, 83.2K incoming IP traffic, 69.3K outgoing IP traffic.4960lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)4961lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.07 seconds)4962client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&24963worker2 # [ 80.641870] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:02Z level=info event=scheduler_connected addr=lb:500514964worker2 # [ 80.646684] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4965worker2 # [ 80.779076] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:510484966worker2 # [ 80.793858] nix-daemon[860]: accepted connection from pid 1094, user nix-grpc-daemon (trusted)4967worker2 # [ 80.819327] nix-grpc-daemon[1094]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 33 ms (attempt 1/5)4968worker2 # [ 80.848606] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:02Z level=info event=scheduler_connected addr=lb:500514969worker2 # [ 80.853232] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4970worker2 # [ 80.860458] nix-grpc-daemon[1094]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 161 ms (attempt 2/5)4971worker2 # [ 81.030150] nix-grpc-daemon[1094]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 128 ms (attempt 3/5)4972worker2 # [ 81.054251] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:02Z level=info event=scheduler_connected addr=lb:500514973worker2 # [ 81.059369] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4974worker2 # [ 81.159956] nix-grpc-daemon[1094]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 191 ms (attempt 4/5)4975worker2 # [ 81.260520] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:02Z level=info event=scheduler_connected addr=lb:500514976worker2 # [ 81.265356] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4977worker2 # [ 81.353731] nix-grpc-daemon[1094]: disabling binary cache 'https://cache.nixos.org' for 60 seconds4978worker2 # [ 81.360763] nix-grpc-daemon[1094]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org4979client # copying 2 paths...4980client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'...4981worker2 # [ 81.383435] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:510484982client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'...4983worker2 # [ 81.400802] nix-daemon[860]: accepted connection from pid 1094, user nix-grpc-daemon (trusted)4984worker2 # [ 81.430997] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:03Z level=info event=niks3_push_started pid=11544985worker2 # [ 81.467082] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:03Z level=info event=scheduler_connected addr=lb:500514986worker2 # [ 81.471993] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4987worker2 # [ 81.597318] nix-grpc-daemon[1154]: time=2026-09-21T13:23:03.219Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""4988worker2 # [ 81.672445] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:03Z level=info event=scheduler_connected addr=lb:500514989worker2 # [ 81.692175] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"4990worker2 # [ 81.707846] nix-daemon[860]: accepted connection from pid 1167, user nix-grpc-daemon (trusted)4991worker2 # [ 81.726899] nix-daemon[860]: reaped child process 1170, status = succeeded4992lb # [ 81.749026] niks3-server[1034]: 2026/09/21 13:23:03 INFO Received uploads request method=POST path=/api/pending_closures4993lb # [ 81.758315] niks3-server[1034]: 2026/09/21 13:23:03 INFO Received uploads request method=POST path=/api/pending_closures4994worker2 # [ 81.752071] nix-grpc-daemon[1154]: time=2026-09-21T13:23:03.373Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4995worker2 # [ 81.753582] nix-grpc-daemon[1154]: time=2026-09-21T13:23:03.373Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)"4996worker2 # [ 81.755365] nix-grpc-daemon[1154]: time=2026-09-21T13:23:03.373Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)"4997lb # [ 81.809673] niks3-server[1034]: 2026/09/21 13:23:03 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst4998lb # [ 81.825068] niks3-server[1034]: 2026/09/21 13:23:03 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst4999lb # [ 81.838084] niks3-server[1034]: 2026/09/21 13:23:03 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls5000lb # [ 81.846694] niks3-server[1034]: 2026/09/21 13:23:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign5001worker2 # [ 81.830228] nix-grpc-daemon[1154]: time=2026-09-21T13:23:03.451Z level=INFO msg="Uploading 3 narinfos"5002lb # [ 81.853457] niks3-server[1034]: 2026/09/21 13:23:03 INFO Signed narinfos id=25 count=15003lb # [ 81.855246] niks3-server[1034]: 2026/09/21 13:23:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign5004lb # [ 81.864501] niks3-server[1034]: 2026/09/21 13:23:03 INFO Signed narinfos id=26 count=25005lb # [ 81.866979] niks3-server[1034]: 2026/09/21 13:23:03 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls5006lb # [ 81.874727] niks3-server[1034]: 2026/09/21 13:23:03 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo5007lb # [ 81.886444] niks3-server[1034]: 2026/09/21 13:23:03 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete5008lb # [ 81.893501] niks3-server[1034]: 2026/09/21 13:23:03 INFO Completed upload id=255009worker2 # [ 81.877014] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:03Z level=info event=scheduler_connected addr=lb:500515010lb # [ 81.899846] niks3-server[1034]: 2026/09/21 13:23:03 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete5011worker2 # [ 81.884372] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5012lb # [ 81.907008] niks3-server[1034]: 2026/09/21 13:23:03 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5013lb # [ 81.911456] niks3-server[1034]: 2026/09/21 13:23:03 INFO Completed upload id=265014worker2 # [ 81.895797] nix-grpc-daemon[1154]: time=2026-09-21T13:23:03.504Z level=INFO msg="Upload complete. (280ms)"5015worker2 # [ 81.908379] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51048 duration_s=0 paths=2 nar_bytes_in=3125016worker2 # [ 81.913834] nix-grpc-daemon[1154]: time=2026-09-21T13:23:03.517Z level=WARN msg="Failed to register uploaded object" key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo error="server returned 404: object is not pending upload\n"5017client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.36 seconds)5018worker2 # [ 81.920458] nix-daemon[860]: reaped child process 1152, status = succeeded5019worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache5020worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.02 seconds)5021worker1 # [ 82.107189] systemd[1]: Listening on Nix worker-protocol over gRPC.5022worker1 # [ 82.117677] systemd[1]: Starting Nix worker-protocol over gRPC...5023worker2 # [ 82.077897] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:03Z level=info event=scheduler_connected addr=lb:500515024worker2 # [ 82.082998] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5025worker1 # [ 82.263412] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:03Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5026worker1 # [ 82.265845] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:03Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5027worker1 # [ 82.269825] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:03Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15028worker1 # [ 82.276059] nix-grpc-daemon[1213]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5029worker1 # [ 82.277529] nix-grpc-daemon[1213]: E0000 00:00:1789996983.894108 1213 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5030lb # [ 82.222682] niks3-server[1034]: 2026/09/21 13:23:03 INFO lead: acquired remote=192.168.1.3:462285031lb: waiting until exactly no endpoints of aarch64-linux are unhealthy5032lb: must succeed: curl -sf localhost:9901/clusters5033worker1 # [ 82.320104] nix-grpc-daemon[1213]: E0000 00:00:1789996983.900341 1213 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5034worker1 # [ 82.324128] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:03Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5035worker1 # [ 82.326285] nix-grpc-daemon[1213]: E0000 00:00:1789996983.907964 1252 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5036worker1 # [ 82.328888] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:03Z level=info event=scheduler_take_over5037worker1 # [ 82.329990] nix-grpc-daemon[1213]: E0000 00:00:1789996983.924845 1252 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5038worker1 # [ 82.332230] nix-grpc-daemon[1213]: E0000 00:00:1789996983.925003 1252 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5039worker1 # [ 82.337569] systemd[1]: Started Nix worker-protocol over gRPC.5040worker1 # [ 82.338451] nix-grpc-daemon[1213]: E0000 00:00:1789996983.942813 1252 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5041worker1 # [ 82.343677] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:03Z level=info event=scheduler_connected addr=lb:500515042worker1 # [ 82.345471] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5043worker2 # [ 82.284183] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:03Z level=info event=scheduler_connected addr=lb:500515044worker2 # [ 82.289146] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:03Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5045lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5046worker1 # [ 82.546315] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:04Z level=info event=scheduler_connected addr=lb:500515047worker1 # [ 82.555477] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:04Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:371985048worker1 # [ 82.561122] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:04Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05049worker2 # [ 82.490339] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:04Z level=info event=scheduler_connected addr=lb:500515050worker1 # [ 82.584133] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:04Z level=info event=oidc_jwks provider=mock keys=15051worker1 # [ 82.589464] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:04Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:371985052worker1 # [ 82.594789] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:04Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05053lb: must succeed: curl -sf localhost:9901/clusters5054lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5055lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.14 seconds)5056(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.15 seconds)5057subtest: scheduler down: the other node takes the lock and keeps it5058lb: must succeed: curl -sf localhost:9901/clusters5059lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5060lb: must succeed: curl -sf localhost:9901/clusters5061lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5062lb: must succeed: curl -sf localhost:9901/clusters5063lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5064lb: must succeed: curl -sf localhost:9901/clusters5065lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5066worker1 # [ 83.660753] systemd[1]: Stopping Nix worker-protocol over gRPC...5067worker1 # [ 84.283458] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:05Z level=info event=signal_exit5068worker1 # [ 84.298040] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:05Z level=info event=scheduler_yield5069worker2 # [ 84.218640] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:05Z level=info event=scheduler_restarting addr=lb:500515070lb # [ 84.237130] niks3-server[1034]: 2026/09/21 13:23:05 INFO lead: released remote=192.168.1.3:462285071worker1 # [ 84.304144] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:05Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:371985072worker1 # [ 84.310740] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:05Z level=info event=worker_gone addr=192.168.1.4:500515073worker1 # [ 84.315890] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:05Z level=info event=scheduler_restarting addr=lb:500515074worker1 # [ 84.320201] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:05Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:371985075worker1 # [ 84.324947] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:05Z level=info event=worker_gone addr=192.168.1.3:500515076worker2 # [ 84.418913] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:06Z level=info event=scheduler_connected addr=lb:500515077worker2 # [ 84.424300] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:06Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5078worker1 # [ 84.511392] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:06Z level=info event=scheduler_connected addr=lb:500515079worker1 # [ 84.518327] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:06Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"5080worker2 # [ 84.625353] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:06Z level=info event=scheduler_connected addr=lb:500515081worker2 # [ 84.630585] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5082worker1 # [ 84.719384] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:06Z level=info event=scheduler_connected addr=lb:500515083worker1 # [ 84.724802] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5084worker2 # [ 84.831649] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:06Z level=info event=scheduler_connected addr=lb:500515085worker2 # [ 84.836851] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5086worker1 # [ 84.925546] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:06Z level=info event=scheduler_connected addr=lb:500515087worker1 # [ 84.930827] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5088worker2 # [ 85.038010] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:06Z level=info event=scheduler_connected addr=lb:500515089worker2 # [ 85.043202] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5090worker1 # [ 85.131881] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:06Z level=info event=scheduler_connected addr=lb:500515091worker1 # [ 85.137091] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5092worker1 # [ 85.306837] nix-grpc-daemon[1213]: ts=2026-09-21T13:23:06Z level=info event=scheduler_restarting5093worker2 # [ 85.245917] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:06Z level=info event=scheduler_connected addr=lb:500515094worker2 # [ 85.249892] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:06Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5095worker2 # [ 85.446773] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:07Z level=info event=scheduler_connected addr=lb:500515096worker2 # [ 85.451773] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5097worker1 # [ 85.547088] nix-grpc-daemon[1213]: E0000 00:00:1789996987.166433 1258 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5098worker1 # [ 85.553419] nix-grpc-daemon[1213]: E0000 00:00:1789996987.166678 1258 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5099worker2 # [ 85.652874] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:07Z level=info event=scheduler_connected addr=lb:500515100worker2 # [ 85.657974] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5101worker2 # [ 85.858870] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:07Z level=info event=scheduler_connected addr=lb:500515102worker2 # [ 85.864135] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5103worker2 # [ 86.065185] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:07Z level=info event=scheduler_connected addr=lb:500515104worker2 # [ 86.070335] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5105worker1 # [ 86.305687] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5106worker1 # [ 86.310354] systemd[1]: Stopped Nix worker-protocol over gRPC.5107worker1 # [ 86.315421] systemd[1]: nix-grpc-daemon.service: Consumed 237ms CPU time over 4.192s wall clock time, 13.3M memory peak, 9.8K incoming IP traffic, 8.3K outgoing IP traffic.5108worker1 # [ 86.330721] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5109worker1 # [ 86.335851] systemd[1]: Closed Nix worker-protocol over gRPC.5110worker1 # [ 86.341422] systemd[1]: nix-grpc-daemon.socket: Consumed 1ms CPU time over 4.235s wall clock time, 256K memory peak, 15K incoming IP traffic, 10.9K outgoing IP traffic.5111worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5112worker2 # [ 86.274158] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:07Z level=info event=scheduler_connected addr=lb:500515113worker2 # [ 86.283428] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:07Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5114worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.09 seconds)5115worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5116worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5117worker2 # [ 86.475045] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:08Z level=info event=scheduler_connected addr=lb:500515118worker2 # [ 86.480419] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5119worker2 # [ 86.681571] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:08Z level=info event=scheduler_connected addr=lb:500515120worker2 # [ 86.686722] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5121worker2 # [ 86.887703] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:08Z level=info event=scheduler_connected addr=lb:500515122worker2 # [ 86.893128] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5123worker2 # [ 87.094108] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:08Z level=info event=scheduler_connected addr=lb:500515124worker2 # [ 87.099258] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5125worker2 # [ 87.302087] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:08Z level=info event=scheduler_connected addr=lb:500515126worker2 # [ 87.306243] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:08Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5127worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5128worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5129worker2 # [ 87.503066] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:09Z level=info event=scheduler_connected addr=lb:500515130worker2 # [ 87.508359] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5131worker2 # [ 87.709379] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:09Z level=info event=scheduler_connected addr=lb:500515132worker2 # [ 87.714523] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:09Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5133lb # [ 87.774493] niks3-server[1034]: 2026/09/21 13:23:09 INFO lead: acquired remote=192.168.1.4:394985134worker2 # [ 87.758314] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:09Z level=info event=scheduler_take_over5135worker2 # [ 87.915620] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:09Z level=info event=scheduler_connected addr=lb:500515136worker2 # [ 87.945653] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:09Z level=info event=oidc_jwks provider=mock keys=15137worker2 # [ 87.950108] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:09Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:539185138worker2 # [ 87.955443] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:09Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05139worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5140worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5141client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&25142lb # [ 88.668137] envoy[897]: conn peer=[2001:db8:1::3]:49678 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=13178 tx=3665 ms=603965143lb # [ 88.668384] envoy[897]: conn peer=[2001:db8:1::1]:55946 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1005 tx=305 ms=11285144lb # [ 88.668543] envoy[897]: conn peer=[2001:db8:1::3]:37020 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1163 tx=429 ms=30115145client # don't know how to build these paths:5146client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5147worker2 # [ 88.754520] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:510485148worker2 # [ 88.762063] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:510485149worker2 # [ 88.786065] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:10Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=05150worker2 # [ 88.794340] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:10Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.4:50051 clients=15151worker2 # [ 88.831336] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:510485152worker2 # [ 88.846981] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:510485153client # copying 1 paths to 'lb:50051'...5154client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'...5155worker2 # [ 88.862831] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:510485156worker2 # [ 88.878610] nix-daemon[860]: accepted connection from pid 1094, user nix-grpc-daemon (trusted)5157worker2 # [ 89.037305] nix-daemon[860]: accepted connection from pid 1194, user nix-grpc-daemon (trusted)5158worker2 # [ 89.063882] nix-daemon[860]: reaped child process 1197, status = succeeded5159worker2 # [ 89.156866] nix-daemon[860]: accepted connection from pid 1199, user nix-grpc-daemon (trusted)5160worker2 # [ 89.173573] nix-daemon[860]: reaped child process 1202, status = succeeded5161lb # [ 89.196993] niks3-server[1034]: 2026/09/21 13:23:10 INFO Received uploads request method=POST path=/api/pending_closures5162worker2 # [ 89.189029] nix-grpc-daemon[1154]: time=2026-09-21T13:23:10.811Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5163worker2 # [ 89.190468] nix-grpc-daemon[1154]: time=2026-09-21T13:23:10.811Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)"5164lb # [ 89.227308] niks3-server[1034]: 2026/09/21 13:23:10 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst5165lb # [ 89.246452] niks3-server[1034]: 2026/09/21 13:23:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign5166lb # [ 89.262511] niks3-server[1034]: 2026/09/21 13:23:10 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls5167lb # [ 89.273457] niks3-server[1034]: 2026/09/21 13:23:10 INFO Signed narinfos id=27 count=15168worker2 # [ 89.256908] nix-grpc-daemon[1154]: time=2026-09-21T13:23:10.879Z level=INFO msg="Uploading 1 narinfos"5169lb # [ 89.284227] niks3-server[1034]: 2026/09/21 13:23:10 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete5170worker2 # [ 89.273967] nix-grpc-daemon[1154]: time=2026-09-21T13:23:10.896Z level=INFO msg="Upload complete. (379ms)"5171lb # [ 89.293189] niks3-server[1034]: 2026/09/21 13:23:10 INFO Completed upload id=275172lb # [ 89.295340] niks3-server[1034]: 2026/09/21 13:23:10 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo5173worker2 # [ 89.284414] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51048 duration_s=0 paths=1 nar_bytes_in=5205174worker2 # [ 89.302327] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:510485175worker2 # [ 89.313034] nix-daemon[860]: accepted connection from pid 1094, user nix-grpc-daemon (trusted)5176worker2 # [ 89.319561] nix-daemon[860]: reaped child process 1192, status = succeeded5177client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051...5178worker2 # [ 89.329947] nix-daemon[860]: accepted connection from pid 1094, user nix-grpc-daemon (trusted)5179client # farm-a> worker2: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5180client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5181client # farm-a> LOG-farm-a-on-standby5182worker2 # [ 90.416617] nix-daemon[860]: reaped child process 1206, status = succeeded5183worker2 # [ 90.514807] nix-daemon[860]: accepted connection from pid 1222, user nix-grpc-daemon (trusted)5184worker2 # [ 90.533773] nix-daemon[860]: reaped child process 1225, status = succeeded5185lb # [ 90.557789] niks3-server[1034]: 2026/09/21 13:23:12 INFO Received uploads request method=POST path=/api/pending_closures5186worker2 # [ 90.550508] nix-grpc-daemon[1154]: time=2026-09-21T13:23:12.171Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5187worker2 # [ 90.552300] nix-grpc-daemon[1154]: time=2026-09-21T13:23:12.171Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)"5188lb # [ 90.593578] niks3-server[1034]: 2026/09/21 13:23:12 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5189lb # [ 90.610087] niks3-server[1034]: 2026/09/21 13:23:12 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5190lb # [ 90.619145] niks3-server[1034]: 2026/09/21 13:23:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign5191worker2 # [ 90.608129] nix-grpc-daemon[1154]: time=2026-09-21T13:23:12.229Z level=INFO msg="Uploading 1 narinfos"5192lb # [ 90.630795] niks3-server[1034]: 2026/09/21 13:23:12 INFO Signed narinfos id=28 count=15193lb # [ 90.641140] niks3-server[1034]: 2026/09/21 13:23:12 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls5194lb # [ 90.643714] niks3-server[1034]: 2026/09/21 13:23:12 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5195lb # [ 90.652723] niks3-server[1034]: 2026/09/21 13:23:12 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo5196lb # [ 90.656672] niks3-server[1034]: 2026/09/21 13:23:12 INFO Completed upload id=285197worker2 # [ 90.639968] nix-grpc-daemon[1154]: time=2026-09-21T13:23:12.262Z level=INFO msg="Upload complete. (296ms)"5198worker2 # [ 90.662271] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51048 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=15199worker2 # [ 90.678970] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:12Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=05200worker2 # [ 90.684497] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:12Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.4:50051 clients=15201worker2 # [ 90.689919] nix-daemon[860]: reaped child process 1204, status = succeeded5202worker2 # [ 90.707948] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:510485203worker2 # [ 90.722990] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:510485204client # copying 1 paths to 'lb:50051'...5205worker2 # [ 90.737690] nix-grpc-daemon[1154]: time=2026-09-21T13:23:12.360Z level=INFO msg="All 1 paths already cached"5206client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'...5207worker2 # [ 90.745039] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:510485208worker2 # [ 90.759063] nix-daemon[860]: accepted connection from pid 1094, user nix-grpc-daemon (trusted)5209worker2 # [ 90.909841] nix-daemon[860]: accepted connection from pid 1231, user nix-grpc-daemon (trusted)5210worker2 # [ 90.934240] nix-daemon[860]: reaped child process 1234, status = succeeded5211worker2 # [ 91.030564] nix-daemon[860]: accepted connection from pid 1236, user nix-grpc-daemon (trusted)5212worker2 # [ 91.045442] nix-daemon[860]: reaped child process 1239, status = succeeded5213lb # [ 91.069291] niks3-server[1034]: 2026/09/21 13:23:12 INFO Received uploads request method=POST path=/api/pending_closures5214worker2 # [ 91.063541] nix-grpc-daemon[1154]: time=2026-09-21T13:23:12.686Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5215worker2 # [ 91.065163] nix-grpc-daemon[1154]: time=2026-09-21T13:23:12.686Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)"5216lb # [ 91.099466] niks3-server[1034]: 2026/09/21 13:23:12 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst5217worker2 # [ 91.097249] nix-grpc-daemon[1154]: time=2026-09-21T13:23:12.719Z level=INFO msg="Uploading 1 narinfos"5218lb # [ 91.114750] niks3-server[1034]: 2026/09/21 13:23:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign5219lb # [ 91.126906] niks3-server[1034]: 2026/09/21 13:23:12 INFO Signed narinfos id=29 count=15220lb # [ 91.133255] niks3-server[1034]: 2026/09/21 13:23:12 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls5221lb # [ 91.143318] niks3-server[1034]: 2026/09/21 13:23:12 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5222lb # [ 91.152378] niks3-server[1034]: 2026/09/21 13:23:12 INFO Completed upload id=295223worker2 # [ 91.135632] nix-grpc-daemon[1154]: time=2026-09-21T13:23:12.758Z level=INFO msg="Upload complete. (361ms)"5224lb # [ 91.161396] niks3-server[1034]: 2026/09/21 13:23:12 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo5225worker2 # [ 91.145047] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51048 duration_s=0 paths=1 nar_bytes_in=6405226worker2 # [ 91.158855] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:510485227worker2 # [ 91.165268] nix-daemon[860]: accepted connection from pid 1094, user nix-grpc-daemon (trusted)5228worker2 # [ 91.170138] nix-daemon[860]: reaped child process 1229, status = succeeded5229client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051...5230worker2 # [ 91.182194] nix-daemon[860]: accepted connection from pid 1094, user nix-grpc-daemon (trusted)5231client # farm-b> worker2: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5232client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5233client # farm-b> LOG-farm-b-on-standby5234worker2 # [ 92.247083] nix-daemon[860]: reaped child process 1243, status = succeeded5235worker2 # [ 92.360161] nix-daemon[860]: accepted connection from pid 1259, user nix-grpc-daemon (trusted)5236worker2 # [ 92.384618] nix-daemon[860]: reaped child process 1262, status = succeeded5237lb # [ 92.401167] niks3-server[1034]: 2026/09/21 13:23:14 INFO Received uploads request method=POST path=/api/pending_closures5238worker2 # [ 92.397694] nix-grpc-daemon[1154]: time=2026-09-21T13:23:14.019Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5239worker2 # [ 92.399210] nix-grpc-daemon[1154]: time=2026-09-21T13:23:14.019Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)"5240lb # [ 92.439538] niks3-server[1034]: 2026/09/21 13:23:14 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5241lb # [ 92.465792] niks3-server[1034]: 2026/09/21 13:23:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5242worker2 # [ 92.454874] nix-grpc-daemon[1154]: time=2026-09-21T13:23:14.077Z level=INFO msg="Uploading 1 narinfos"5243lb # [ 92.480915] niks3-server[1034]: 2026/09/21 13:23:14 INFO Signed narinfos id=30 count=15244lb # [ 92.483434] niks3-server[1034]: 2026/09/21 13:23:14 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst5245lb # [ 92.498852] niks3-server[1034]: 2026/09/21 13:23:14 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls5246worker2 # [ 92.485281] nix-grpc-daemon[1154]: time=2026-09-21T13:23:14.108Z level=INFO msg="Upload complete. (293ms)"5247lb # [ 92.504950] niks3-server[1034]: 2026/09/21 13:23:14 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5248lb # [ 92.511853] niks3-server[1034]: 2026/09/21 13:23:14 INFO Completed upload id=305249lb # [ 92.514131] niks3-server[1034]: 2026/09/21 13:23:14 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo5250worker2 # [ 92.504748] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51048 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=15251worker2 # [ 92.520577] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:14Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=05252worker2 # [ 92.526456] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:14Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.4:50051 clients=15253worker2 # [ 92.530814] nix-daemon[860]: reaped child process 1241, status = succeeded5254worker2 # [ 92.552471] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:510485255worker2 # [ 92.569449] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:510485256worker2 # [ 92.587703] nix-grpc-daemon[1154]: time=2026-09-21T13:23:14.210Z level=INFO msg="All 2 paths already cached"5257client # copying 1 paths to 'lb:50051'...5258client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'...5259worker2 # [ 92.598880] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:510485260worker2 # [ 92.613526] nix-daemon[860]: accepted connection from pid 1094, user nix-grpc-daemon (trusted)5261worker2 # [ 92.812730] nix-daemon[860]: accepted connection from pid 1268, user nix-grpc-daemon (trusted)5262worker2 # [ 92.835578] nix-daemon[860]: reaped child process 1271, status = succeeded5263worker2 # [ 92.946513] nix-daemon[860]: accepted connection from pid 1273, user nix-grpc-daemon (trusted)5264worker2 # [ 92.971602] nix-daemon[860]: reaped child process 1276, status = succeeded5265lb # [ 92.988305] niks3-server[1034]: 2026/09/21 13:23:14 INFO Received uploads request method=POST path=/api/pending_closures5266worker2 # [ 92.990649] nix-grpc-daemon[1154]: time=2026-09-21T13:23:14.612Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5267worker2 # [ 92.992262] nix-grpc-daemon[1154]: time=2026-09-21T13:23:14.613Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)"5268lb # [ 93.034790] niks3-server[1034]: 2026/09/21 13:23:14 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst5269worker2 # [ 93.035174] nix-grpc-daemon[1154]: time=2026-09-21T13:23:14.657Z level=INFO msg="Uploading 1 narinfos"5270lb # [ 93.051677] niks3-server[1034]: 2026/09/21 13:23:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5271lb # [ 93.061431] niks3-server[1034]: 2026/09/21 13:23:14 INFO Signed narinfos id=31 count=15272lb # [ 93.063949] niks3-server[1034]: 2026/09/21 13:23:14 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls5273lb # [ 93.080785] niks3-server[1034]: 2026/09/21 13:23:14 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5274lb # [ 93.090495] niks3-server[1034]: 2026/09/21 13:23:14 INFO Completed upload id=315275worker2 # [ 93.077232] nix-grpc-daemon[1154]: time=2026-09-21T13:23:14.698Z level=INFO msg="Upload complete. (441ms)"5276lb # [ 93.096745] niks3-server[1034]: 2026/09/21 13:23:14 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo5277worker2 # [ 93.088628] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:14Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51048 duration_s=0 paths=1 nar_bytes_in=7685278worker2 # [ 93.103752] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:510485279worker2 # [ 93.118418] nix-daemon[860]: accepted connection from pid 1094, user nix-grpc-daemon (trusted)5280worker2 # [ 93.126483] nix-daemon[860]: reaped child process 1266, status = succeeded5281client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051...5282worker2 # [ 93.142151] nix-daemon[860]: accepted connection from pid 1094, user nix-grpc-daemon (trusted)5283client # farm-top> worker2: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5284client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5285client # farm-top> LOG-farm-top-on-standby5286worker2 # [ 94.417990] nix-daemon[860]: reaped child process 1280, status = succeeded5287worker2 # [ 94.535215] nix-daemon[860]: accepted connection from pid 1296, user nix-grpc-daemon (trusted)5288worker2 # [ 94.551439] nix-daemon[860]: reaped child process 1299, status = succeeded5289lb # [ 94.580966] niks3-server[1034]: 2026/09/21 13:23:16 INFO Received uploads request method=POST path=/api/pending_closures5290worker2 # [ 94.573989] nix-grpc-daemon[1154]: time=2026-09-21T13:23:16.194Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5291worker2 # [ 94.575806] nix-grpc-daemon[1154]: time=2026-09-21T13:23:16.194Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)"5292lb # [ 94.615871] niks3-server[1034]: 2026/09/21 13:23:16 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5293lb # [ 94.632354] niks3-server[1034]: 2026/09/21 13:23:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5294worker2 # [ 94.616339] nix-grpc-daemon[1154]: time=2026-09-21T13:23:16.238Z level=INFO msg="Uploading 1 narinfos"5295lb # [ 94.641996] niks3-server[1034]: 2026/09/21 13:23:16 INFO Signed narinfos id=32 count=15296lb # [ 94.647321] niks3-server[1034]: 2026/09/21 13:23:16 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst5297lb # [ 94.655889] niks3-server[1034]: 2026/09/21 13:23:16 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls5298lb # [ 94.664179] niks3-server[1034]: 2026/09/21 13:23:16 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5299lb # [ 94.671399] niks3-server[1034]: 2026/09/21 13:23:16 INFO Completed upload id=325300worker2 # [ 94.653484] nix-grpc-daemon[1154]: time=2026-09-21T13:23:16.276Z level=INFO msg="Upload complete. (289ms)"5301lb # [ 94.675106] niks3-server[1034]: 2026/09/21 13:23:16 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo5302worker2 # [ 94.666267] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51048 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=15303worker2 # [ 94.685077] nix-daemon[860]: reaped child process 1278, status = succeeded5304client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2, in 6.19 seconds)5305client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath5306client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath, in 0.18 seconds)5307worker1 # [ 95.065321] systemd[1]: Listening on Nix worker-protocol over gRPC.5308worker1 # [ 95.075092] systemd[1]: Starting Nix worker-protocol over gRPC...5309worker1 # [ 95.219997] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:16Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5310worker1 # [ 95.222345] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:16Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5311worker1 # [ 95.226445] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:16Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15312worker1 # [ 95.232046] nix-grpc-daemon[1280]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5313worker1 # [ 95.233496] nix-grpc-daemon[1280]: E0000 00:00:1789996996.850672 1280 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5314lb: must succeed: curl -sf localhost:9901/clusters5315worker1 # [ 95.277675] nix-grpc-daemon[1280]: E0000 00:00:1789996996.857015 1280 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5316worker1 # [ 95.281106] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:16Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5317worker1 # [ 95.283366] nix-grpc-daemon[1280]: E0000 00:00:1789996996.863538 1319 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5318worker1 # [ 95.286101] nix-grpc-daemon[1280]: E0000 00:00:1789996996.880844 1319 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5319worker1 # [ 95.288378] nix-grpc-daemon[1280]: E0000 00:00:1789996996.881034 1319 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5320worker1 # [ 95.290656] nix-grpc-daemon[1280]: E0000 00:00:1789996996.897292 1319 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5321worker1 # [ 95.295913] systemd[1]: Started Nix worker-protocol over gRPC.5322worker1 # [ 95.298585] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:16Z level=info event=scheduler_connected addr=lb:500515323worker2 # [ 95.219306] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:16Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:539185324worker2 # [ 95.224681] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:16Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05325lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.11 seconds)5326worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5327worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5328lb: must succeed: curl -sf localhost:9901/clusters5329lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5330lb: must succeed: curl -sf localhost:9901/clusters5331lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5332worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5333worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5334(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 12.16 seconds)5335subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5336client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs5337client # [ 95.605083] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs.5338client # Running as unit: rs.service; invocation ID: 1e4d226ce587483ba4c904379ecc348e5339client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs, in 0.08 seconds)5340worker2 # [ 95.789326] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:17Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:510485341client # [ 95.815168] nix[1217]: don't know how to build these paths:5342client # [ 95.816754] nix[1217]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5343worker2 # [ 95.807279] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:17Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=05344worker2 # [ 95.816599] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:17Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.4:50051 clients=15345worker2 # [ 95.855101] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:510485346worker2 # [ 95.871461] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:510485347client # [ 95.904228] nix[1217]: copying 1 paths to 'lb:50051'...5348worker2 # [ 95.886665] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:510485349client # [ 95.907974] nix[1217]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'...5350worker2 # [ 95.901941] nix-daemon[860]: accepted connection from pid 1094, user nix-grpc-daemon (trusted)5351worker2 # [ 96.057800] nix-daemon[860]: accepted connection from pid 1312, user nix-grpc-daemon (trusted)5352worker2 # [ 96.079094] nix-daemon[860]: reaped child process 1315, status = succeeded5353worker2 # [ 96.176931] nix-daemon[860]: accepted connection from pid 1317, user nix-grpc-daemon (trusted)5354worker2 # [ 96.197356] nix-daemon[860]: reaped child process 1320, status = succeeded5355lb # [ 96.218047] niks3-server[1034]: 2026/09/21 13:23:17 INFO Received uploads request method=POST path=/api/pending_closures5356worker2 # [ 96.211419] nix-grpc-daemon[1154]: time=2026-09-21T13:23:17.833Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5357worker2 # [ 96.213039] nix-grpc-daemon[1154]: time=2026-09-21T13:23:17.833Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)"5358worker2 # [ 96.241681] nix-grpc-daemon[1154]: time=2026-09-21T13:23:17.864Z level=INFO msg="Uploading 1 narinfos"5359lb # [ 96.249797] niks3-server[1034]: 2026/09/21 13:23:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5360lb # [ 96.267239] niks3-server[1034]: 2026/09/21 13:23:17 INFO Signed narinfos id=33 count=15361lb # [ 96.278628] niks3-server[1034]: 2026/09/21 13:23:17 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls5362lb # [ 96.292234] niks3-server[1034]: 2026/09/21 13:23:17 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst5363lb # [ 96.301169] niks3-server[1034]: 2026/09/21 13:23:17 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5364worker2 # [ 96.288264] nix-grpc-daemon[1154]: time=2026-09-21T13:23:17.910Z level=INFO msg="Upload complete. (367ms)"5365lb # [ 96.305424] niks3-server[1034]: 2026/09/21 13:23:17 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo5366lb # [ 96.309436] niks3-server[1034]: 2026/09/21 13:23:17 INFO Completed upload id=335367worker2 # [ 96.296164] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51048 duration_s=0 paths=1 nar_bytes_in=4165368worker2 # [ 96.311777] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:510485369worker2 # [ 96.322290] nix-daemon[860]: accepted connection from pid 1094, user nix-grpc-daemon (trusted)5370worker2 # [ 96.326018] nix-daemon[860]: reaped child process 1310, status = succeeded5371worker2 # [ 96.338366] nix-daemon[860]: accepted connection from pid 1094, user nix-grpc-daemon (trusted)5372client # [ 96.355109] nix[1217]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051...5373lb: must succeed: curl -sf localhost:9901/clusters5374lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5375lb: must succeed: curl -sf localhost:9901/clusters5376lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5377lb: must succeed: curl -sf localhost:9901/clusters5378lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5379lb: must succeed: curl -sf localhost:9901/clusters5380lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5381worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5382worker1 # [ 98.470881] systemd-journald[264]: Received client request to rotate journal, rotating.5383worker1 # [ 98.498333] systemd-journald[264]: Vacuuming done, freed 0B of archived journals from /var/log/journal/2afa59c5c45a42d9a62aea090e6ec737.5384worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.09 seconds)5385worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service5386worker2 # [ 98.462022] systemd[1]: Stopping Nix worker-protocol over gRPC...5387worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.05 seconds)5388client: must succeed: sleep 5.05389lb # [ 98.667533] envoy[897]: conn peer=[2001:db8:1::1]:33430 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9636 tx=4126 ms=59295390worker2 # [ 98.760740] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:20Z level=info event=draining5391client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5392worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5393worker2 # [ 104.036480] nix-daemon[860]: reaped child process 1324, status = succeeded5394worker2 # [ 104.128860] nix-daemon[860]: accepted connection from pid 1365, user nix-grpc-daemon (trusted)5395worker2 # [ 104.148106] nix-daemon[860]: reaped child process 1368, status = succeeded5396lb # [ 104.169082] niks3-server[1034]: 2026/09/21 13:23:25 INFO Received uploads request method=POST path=/api/pending_closures5397worker2 # [ 104.165661] nix-grpc-daemon[1154]: time=2026-09-21T13:23:25.787Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5398worker2 # [ 104.167425] nix-grpc-daemon[1154]: time=2026-09-21T13:23:25.787Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)"5399lb # [ 104.218639] niks3-server[1034]: 2026/09/21 13:23:25 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5400worker2 # [ 104.217668] nix-grpc-daemon[1154]: time=2026-09-21T13:23:25.840Z level=INFO msg="Uploading 1 narinfos"5401lb # [ 104.236842] niks3-server[1034]: 2026/09/21 13:23:25 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5402lb # [ 104.245871] niks3-server[1034]: 2026/09/21 13:23:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5403lb # [ 104.254080] niks3-server[1034]: 2026/09/21 13:23:25 INFO Signed narinfos id=34 count=15404lb # [ 104.258341] niks3-server[1034]: 2026/09/21 13:23:25 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls5405worker2 # [ 104.241274] nix-grpc-daemon[1154]: time=2026-09-21T13:23:25.864Z level=INFO msg="Upload complete. (284ms)"5406lb # [ 104.261397] niks3-server[1034]: 2026/09/21 13:23:25 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5407lb # [ 104.266446] niks3-server[1034]: 2026/09/21 13:23:25 INFO Completed upload id=345408lb # [ 104.269721] niks3-server[1034]: 2026/09/21 13:23:25 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo5409worker2 # [ 104.257134] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51048 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=15410worker2 # [ 104.273255] nix-daemon[860]: reaped child process 1322, status = succeeded5411client # [ 104.310137] systemd[1]: rs.service: Deactivated successfully.5412client # [ 104.314137] systemd[1]: rs.service: Consumed 195ms CPU time over 8.704s wall clock time, 39.2M memory peak, 7.3K incoming IP traffic, 8.6K outgoing IP traffic.5413worker2 # [ 104.762277] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:26Z level=info event=signal_exit5414worker2 # [ 105.760725] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:27Z level=info event=scheduler_yield5415lb # [ 105.781310] niks3-server[1034]: 2026/09/21 13:23:27 INFO lead: released remote=192.168.1.4:394985416worker2 # [ 105.766231] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:27Z level=info event=scheduler_restarting addr=lb:500515417worker1 # [ 105.851684] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:27Z level=info event=scheduler_restarting addr=lb:500515418worker2 # [ 105.773023] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:27Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:539185419worker2 # [ 105.777592] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:27Z level=info event=worker_gone addr=192.168.1.4:500515420worker2 # [ 105.781747] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:27Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:539185421worker2 # [ 105.786190] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:27Z level=info event=worker_gone addr=192.168.1.3:500515422worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.20 seconds)5423client: waiting for success: ! systemctl is-active rs5424worker1 # [ 106.051535] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:27Z level=info event=scheduler_connected addr=lb:500515425worker2 # [ 105.967592] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:27Z level=info event=scheduler_connected addr=lb:500515426worker1 # [ 106.055997] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5427worker2 # [ 105.973133] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5428client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds)5429client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5430client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds)5431client: must fail: journalctl -u rs | grep -q 'reconnecting'5432client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5433lb: must succeed: curl -sf localhost:9901/clusters5434lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5435worker1 # [ 106.256785] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:27Z level=info event=scheduler_connected addr=lb:500515436worker2 # [ 106.174086] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:27Z level=info event=scheduler_connected addr=lb:500515437worker1 # [ 106.262079] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5438worker2 # [ 106.179323] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5439worker1 # [ 106.462605] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:28Z level=info event=scheduler_connected addr=lb:500515440worker2 # [ 106.380445] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:28Z level=info event=scheduler_connected addr=lb:500515441worker1 # [ 106.467934] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5442worker2 # [ 106.385550] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5443worker1 # [ 106.668860] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:28Z level=info event=scheduler_connected addr=lb:500515444worker2 # [ 106.586537] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:28Z level=info event=scheduler_connected addr=lb:500515445worker1 # [ 106.674072] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5446worker2 # [ 106.591713] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5447worker2 # [ 106.770700] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:28Z level=info event=scheduler_restarting5448worker1 # [ 106.876762] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:28Z level=info event=scheduler_connected addr=lb:500515449worker1 # [ 106.880837] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5450worker2 # [ 106.821481] nix-grpc-daemon[1094]: ts=2026-09-21T13:23:28Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5451worker2 # [ 106.826743] nix-daemon[860]: reaped child process 1146, status = succeeded5452worker2 # [ 106.830055] nix-grpc-daemon[1094]: E0000 00:00:1789997008.452436 1139 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true5453worker2 # [ 106.835105] nix-grpc-daemon[1094]: E0000 00:00:1789997008.452667 1139 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5454worker1 # [ 107.077164] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:28Z level=info event=scheduler_connected addr=lb:500515455worker1 # [ 107.082287] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5456lb: must succeed: curl -sf localhost:9901/clusters5457worker1 # [ 107.283155] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:28Z level=info event=scheduler_connected addr=lb:500515458worker1 # [ 107.288379] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5459lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5460worker2 # [ 107.402149] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5461worker1 # [ 107.489098] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:29Z level=info event=scheduler_connected addr=lb:500515462worker2 # [ 107.408269] systemd[1]: Stopped Nix worker-protocol over gRPC.5463worker1 # [ 107.494219] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5464worker2 # [ 107.410914] systemd[1]: nix-grpc-daemon.service: Consumed 2.218s CPU time over 29.809s wall clock time, 38.9M memory peak, 62K incoming IP traffic, 157.7K outgoing IP traffic.5465worker2 # [ 107.434349] systemd[1]: Starting scheduler-token.service...5466worker2 # [ 107.511629] systemd[1]: scheduler-token.service: Deactivated successfully.5467worker2 # [ 107.516503] systemd[1]: Finished scheduler-token.service.5468worker2 # [ 107.518017] systemd[1]: scheduler-token.service: Consumed 49ms CPU time over 80ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5469worker2 # [ 107.526623] systemd[1]: Starting Nix worker-protocol over gRPC...5470worker1 # [ 107.694880] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:29Z level=info event=scheduler_connected addr=lb:500515471worker1 # [ 107.699989] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5472worker2 # [ 107.645675] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:29Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5473worker2 # [ 107.647917] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:29Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5474worker2 # [ 107.651878] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:29Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15475worker2 # [ 107.658766] nix-grpc-daemon[1378]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5476worker2 # [ 107.665882] nix-grpc-daemon[1378]: E0000 00:00:1789997009.279327 1378 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5477lb # [ 107.684715] niks3-server[1034]: 2026/09/21 13:23:29 INFO lead: acquired remote=192.168.1.4:593425478worker2 # [ 107.671988] nix-grpc-daemon[1378]: E0000 00:00:1789997009.281433 1378 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5479worker2 # [ 107.696168] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:29Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5480worker2 # [ 107.697851] nix-grpc-daemon[1378]: E0000 00:00:1789997009.291973 1417 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5481worker2 # [ 107.707066] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:29Z level=info event=scheduler_take_over5482worker2 # [ 107.708797] nix-grpc-daemon[1378]: E0000 00:00:1789997009.309943 1417 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5483worker2 # [ 107.711186] nix-grpc-daemon[1378]: E0000 00:00:1789997009.310113 1417 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5484worker2 # [ 107.716629] systemd[1]: Started Nix worker-protocol over gRPC.5485worker2 # [ 107.719640] nix-grpc-daemon[1378]: E0000 00:00:1789997009.328883 1417 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5486worker2 # [ 107.722134] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:29Z level=info event=scheduler_connected addr=lb:500515487worker2 # [ 107.723405] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5488worker1 # [ 107.902648] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:29Z level=info event=scheduler_connected addr=lb:500515489worker1 # [ 107.906773] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5490worker2 # [ 107.921208] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:29Z level=info event=scheduler_connected addr=lb:500515491worker2 # [ 107.926268] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5492worker1 # [ 108.103326] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:29Z level=info event=scheduler_connected addr=lb:500515493worker1 # [ 108.108577] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5494worker2 # [ 108.127396] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:29Z level=info event=scheduler_connected addr=lb:500515495worker2 # [ 108.132657] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5496worker1 # [ 108.309350] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:29Z level=info event=scheduler_connected addr=lb:500515497worker1 # [ 108.314449] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5498lb: must succeed: curl -sf localhost:9901/clusters5499lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5500worker2 # [ 108.333562] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:29Z level=info event=scheduler_connected addr=lb:500515501worker2 # [ 108.338419] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5502worker1 # [ 108.515174] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:30Z level=info event=scheduler_connected addr=lb:500515503worker2 # [ 108.436140] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:30Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:504385504worker2 # [ 108.441645] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:30Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05505worker2 # [ 108.539391] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:30Z level=info event=scheduler_connected addr=lb:500515506worker2 # [ 108.552154] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:30Z level=info event=oidc_jwks provider=mock keys=15507worker2 # [ 108.557089] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:30Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:504385508worker2 # [ 108.562431] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:30Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05509lb # [ 108.665629] envoy[897]: conn peer=[2001:db8:1::1]:35354 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2983 tx=1447 ms=84835510lb # [ 108.670654] envoy[897]: conn peer=[2001:db8:1::4]:58246 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=33956 tx=2022 ms=289975511lb: must succeed: curl -sf localhost:9901/clusters5512lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5513worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5514worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5515(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 13.93 seconds)5516subtest: scheduler restart mid-build: build finishes, no double build5517client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid5518client # [ 109.533193] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid.5519client # Running as unit: mid.service; invocation ID: 77f6967819a04a0c9ac18a9b40b9e2e75520client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid, in 0.09 seconds)5521worker2 # [ 109.719194] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:31Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:504505522worker2 # [ 109.731938] nix-daemon[860]: accepted connection from pid 1378, user nix-grpc-daemon (trusted)5523client # [ 109.765960] nix[1253]: don't know how to build these paths:5524client # [ 109.767201] nix[1253]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5525worker2 # [ 109.759930] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:31Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05526worker2 # [ 109.765498] nix-grpc-daemon[1378]: ts=2026-09-21T13:23:31Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.3:50051 clients=15527worker1 # [ 109.910111] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:508325528worker1 # [ 109.924337] nix-daemon[835]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5529worker1 # [ 109.947711] nix-grpc-daemon[1280]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 50 ms (attempt 1/5)5530worker1 # [ 110.005074] nix-grpc-daemon[1280]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 177 ms (attempt 2/5)5531worker1 # [ 110.192256] nix-grpc-daemon[1280]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 69 ms (attempt 3/5)5532worker1 # [ 110.263779] nix-grpc-daemon[1280]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 767 ms (attempt 4/5)5533worker1 # [ 111.035040] nix-grpc-daemon[1280]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5534worker1 # [ 111.035350] nix-grpc-daemon[1280]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5535worker1 # [ 111.062056] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:508325536client # [ 111.013457] nix[1253]: copying 1 paths to 'lb:50051'...5537worker1 # [ 111.078763] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:508325538client # [ 111.013732] nix[1253]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'...5539worker1 # [ 111.097735] nix-daemon[835]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5540worker1 # [ 111.127787] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:32Z level=info event=niks3_push_started pid=13985541worker1 # [ 111.274978] nix-grpc-daemon[1398]: time=2026-09-21T13:23:32.894Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""5542worker1 # [ 111.394576] nix-daemon[835]: accepted connection from pid 1411, user nix-grpc-daemon (trusted)5543worker1 # [ 111.418062] nix-daemon[835]: reaped child process 1414, status = succeeded5544worker1 # [ 111.523857] nix-daemon[835]: accepted connection from pid 1416, user nix-grpc-daemon (trusted)5545worker1 # [ 111.544972] nix-daemon[835]: reaped child process 1419, status = succeeded5546lb # [ 111.482914] niks3-server[1034]: 2026/09/21 13:23:33 INFO Received uploads request method=POST path=/api/pending_closures5547worker1 # [ 111.562092] nix-grpc-daemon[1398]: time=2026-09-21T13:23:33.180Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5548worker1 # [ 111.563604] nix-grpc-daemon[1398]: time=2026-09-21T13:23:33.180Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)"5549lb # [ 111.518253] niks3-server[1034]: 2026/09/21 13:23:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5550lb # [ 111.540098] niks3-server[1034]: 2026/09/21 13:23:33 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst5551worker1 # [ 111.616600] nix-grpc-daemon[1398]: time=2026-09-21T13:23:33.235Z level=INFO msg="Uploading 1 narinfos"5552lb # [ 111.552144] niks3-server[1034]: 2026/09/21 13:23:33 INFO Signed narinfos id=35 count=15553lb # [ 111.566693] niks3-server[1034]: 2026/09/21 13:23:33 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls5554lb # [ 111.572697] niks3-server[1034]: 2026/09/21 13:23:33 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5555lb # [ 111.581054] niks3-server[1034]: 2026/09/21 13:23:33 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo5556lb # [ 111.589420] niks3-server[1034]: 2026/09/21 13:23:33 INFO Completed upload id=355557worker1 # [ 111.656725] nix-grpc-daemon[1398]: time=2026-09-21T13:23:33.276Z level=INFO msg="Upload complete. (377ms)"5558worker1 # [ 111.668399] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50832 duration_s=0 paths=1 nar_bytes_in=4245559worker1 # [ 111.680986] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:508325560worker1 # [ 111.690786] nix-daemon[835]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5561worker1 # [ 111.697575] nix-daemon[835]: reaped child process 1396, status = succeeded5562client # [ 111.641950] nix[1253]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...5563worker1 # [ 111.711439] nix-daemon[835]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5564lb: must succeed: curl -sf localhost:9901/clusters5565lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5566lb: must succeed: curl -sf localhost:9901/clusters5567lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5568worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service5569worker2 # [ 113.385471] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1378 (nix-grpc-daemon) on client request.5570worker2 # [ 113.405448] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.5571worker1 # [ 113.495701] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:35Z level=info event=scheduler_disconnected addr=lb:50051 error="Stream removed (RST_STREAM (Received RST_STREAM with error code 2))"5572lb # [ 113.425977] niks3-server[1034]: 2026/09/21 13:23:35 INFO lead: released remote=192.168.1.4:593425573client # [ 113.431127] nix[1253]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting5574worker2 # [ 113.445910] systemd[1]: Starting scheduler-token.service...5575worker2 # [ 113.447394] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL5576worker2 # [ 113.473463] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.5577worker2 # [ 113.475143] systemd[1]: nix-grpc-daemon.service: Consumed 219ms CPU time over 5.915s wall clock time, 14.3M memory peak, 9.9K incoming IP traffic, 14.8K outgoing IP traffic.5578worker2 # [ 113.478921] nix-daemon[860]: reaped child process 1437, status = succeeded5579worker2 # [ 113.528105] systemd[1]: scheduler-token.service: Deactivated successfully.5580worker2 # [ 113.529370] systemd[1]: Finished scheduler-token.service.5581worker2 # [ 113.530246] systemd[1]: scheduler-token.service: Consumed 30ms CPU time over 86ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5582worker2 # [ 113.540158] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.5583worker2 # [ 113.542035] systemd[1]: Starting scheduler-token.service...5584worker2 # [ 113.592722] systemd[1]: scheduler-token.service: Deactivated successfully.5585worker2 # [ 113.598363] systemd[1]: Finished scheduler-token.service.5586worker2 # [ 113.599276] systemd[1]: scheduler-token.service: Consumed 28ms CPU time over 57ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5587worker2 # [ 113.608463] systemd[1]: Starting Nix worker-protocol over gRPC...5588worker1 # [ 113.695882] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:35Z level=info event=scheduler_connected addr=lb:500515589worker2 # [ 113.707932] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:35Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5590worker2 # [ 113.710593] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:35Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5591worker2 # [ 113.714705] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:35Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15592worker2 # [ 113.720102] nix-grpc-daemon[1471]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5593worker2 # [ 113.721610] nix-grpc-daemon[1471]: E0000 00:00:1789997015.342399 1471 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5594lb # [ 113.759371] niks3-server[1034]: 2026/09/21 13:23:35 INFO lead: acquired remote=192.168.1.4:601725595worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.42 seconds)5596client: must succeed: sleep 5.05597worker2 # [ 113.768116] nix-grpc-daemon[1471]: E0000 00:00:1789997015.348629 1471 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5598worker2 # [ 113.780719] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:35Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5599worker2 # [ 113.783077] nix-grpc-daemon[1471]: E0000 00:00:1789997015.363774 1510 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5600worker2 # [ 113.788839] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:35Z level=info event=scheduler_take_over5601worker2 # [ 113.793038] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:35Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:417625602worker2 # [ 113.795009] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:35Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=15603worker2 # [ 113.797583] nix-grpc-daemon[1471]: E0000 00:00:1789997015.385043 1510 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5604worker2 # [ 113.801361] nix-grpc-daemon[1471]: E0000 00:00:1789997015.385209 1510 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5605worker2 # [ 113.805867] systemd[1]: Started Nix worker-protocol over gRPC.5606worker2 # [ 113.806808] nix-grpc-daemon[1471]: E0000 00:00:1789997015.402418 1510 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5607worker2 # [ 113.809230] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:35Z level=info event=scheduler_connected addr=lb:500515608worker2 # [ 113.810481] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:35Z level=info event=oidc_jwks provider=mock keys=15609worker2 # [ 113.811730] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:35Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:417625610worker2 # [ 113.813522] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:35Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05611lb # [ 118.663124] envoy[897]: conn peer=[2001:db8:1::4]:43312 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=4102 tx=385 ms=57015612client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5613client: waiting for success: ! systemctl is-active mid5614worker1 # [ 119.153621] nix-daemon[835]: reaped child process 1423, status = succeeded5615worker1 # [ 119.269201] nix-daemon[835]: accepted connection from pid 1460, user nix-grpc-daemon (trusted)5616lb # [ 119.229544] niks3-server[1034]: 2026/09/21 13:23:40 INFO Received uploads request method=POST path=/api/pending_closures5617worker1 # [ 119.300113] nix-daemon[835]: reaped child process 1463, status = succeeded5618worker1 # [ 119.308707] nix-grpc-daemon[1398]: time=2026-09-21T13:23:40.926Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5619worker1 # [ 119.310662] nix-grpc-daemon[1398]: time=2026-09-21T13:23:40.926Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)"5620lb # [ 119.273299] niks3-server[1034]: 2026/09/21 13:23:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5621worker1 # [ 119.348165] nix-grpc-daemon[1398]: time=2026-09-21T13:23:40.967Z level=INFO msg="Uploading 1 narinfos"5622lb # [ 119.285480] niks3-server[1034]: 2026/09/21 13:23:40 INFO Signed narinfos id=36 count=15623lb # [ 119.287322] niks3-server[1034]: 2026/09/21 13:23:40 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst5624lb # [ 119.300184] niks3-server[1034]: 2026/09/21 13:23:40 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5625lb # [ 119.309707] niks3-server[1034]: 2026/09/21 13:23:40 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5626worker1 # [ 119.383652] nix-grpc-daemon[1398]: time=2026-09-21T13:23:41.003Z level=INFO msg="Upload complete. (250ms)"5627lb # [ 119.318848] niks3-server[1034]: 2026/09/21 13:23:40 INFO Completed upload id=365628lb # [ 119.324310] niks3-server[1034]: 2026/09/21 13:23:40 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo5629worker1 # [ 119.392795] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50832 duration_s=7 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=15630worker1 # [ 119.395760] nix-grpc-daemon[1398]: time=2026-09-21T13:23:41.006Z level=WARN msg="Failed to register uploaded object" key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls error="server returned 404: object is not pending upload\n"5631worker1 # [ 119.405646] nix-daemon[835]: reaped child process 1421, status = succeeded5632client # [ 119.353549] systemd[1]: mid.service: Deactivated successfully.5633client # [ 119.355351] systemd[1]: mid.service: Consumed 187ms CPU time over 9.822s wall clock time, 41.4M memory peak, 7.6K incoming IP traffic, 8.8K outgoing IP traffic.5634client: (finished: waiting for success: ! systemctl is-active mid, in 1.07 seconds)5635client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }5636client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.04 seconds)5637(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.66 seconds)5638subtest: interrupting the client stops the build on the worker5639client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr5640client # [ 120.168347] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr.5641client # Running as unit: intr.service; invocation ID: 49b1ad20195243989f3693d1ed5321fd5642client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr, in 0.06 seconds)5643worker2 # [ 120.378019] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:417785644worker2 # [ 120.387273] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5645client # [ 120.418487] nix[1289]: don't know how to build these paths:5646client # [ 120.420124] nix[1289]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5647worker2 # [ 120.415327] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:42Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=05648worker2 # [ 120.418568] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:42Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.4:50051 clients=15649worker2 # [ 120.464189] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417785650worker2 # [ 120.471942] nix-grpc-daemon[1471]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 73 ms (attempt 1/5)5651worker2 # [ 120.549277] nix-grpc-daemon[1471]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 132 ms (attempt 2/5)5652worker2 # [ 120.682986] nix-grpc-daemon[1471]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 110 ms (attempt 3/5)5653worker2 # [ 120.794536] nix-grpc-daemon[1471]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 525 ms (attempt 4/5)5654worker2 # [ 121.321760] nix-grpc-daemon[1471]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5655worker2 # [ 121.323197] nix-grpc-daemon[1471]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5656worker2 # [ 121.342693] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417785657worker2 # [ 121.357869] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417785658client # [ 121.373298] nix[1289]: copying 1 paths to 'lb:50051'...5659client # [ 121.375462] nix[1289]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'...5660worker2 # [ 121.368461] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5661worker2 # [ 121.406065] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:43Z level=info event=niks3_push_started pid=15495662worker2 # [ 121.554594] nix-grpc-daemon[1549]: time=2026-09-21T13:23:43.176Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""5663worker2 # [ 121.692814] nix-daemon[860]: accepted connection from pid 1561, user nix-grpc-daemon (trusted)5664worker2 # [ 121.717694] nix-daemon[860]: reaped child process 1564, status = succeeded5665worker2 # [ 121.849451] nix-daemon[860]: accepted connection from pid 1566, user nix-grpc-daemon (trusted)5666worker2 # [ 121.871649] nix-daemon[860]: reaped child process 1569, status = succeeded5667lb # [ 121.893599] niks3-server[1034]: 2026/09/21 13:23:43 INFO Received uploads request method=POST path=/api/pending_closures5668worker2 # [ 121.886144] nix-grpc-daemon[1549]: time=2026-09-21T13:23:43.507Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5669worker2 # [ 121.887720] nix-grpc-daemon[1549]: time=2026-09-21T13:23:43.507Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)"5670lb # [ 121.919597] niks3-server[1034]: 2026/09/21 13:23:43 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst5671lb # [ 121.926338] niks3-server[1034]: 2026/09/21 13:23:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5672worker2 # [ 121.914017] nix-grpc-daemon[1549]: time=2026-09-21T13:23:43.536Z level=INFO msg="Uploading 1 narinfos"5673lb # [ 121.933164] niks3-server[1034]: 2026/09/21 13:23:43 INFO Signed narinfos id=37 count=15674lb # [ 121.948212] niks3-server[1034]: 2026/09/21 13:23:43 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5675lb # [ 121.955589] niks3-server[1034]: 2026/09/21 13:23:43 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls5676lb # [ 121.960855] niks3-server[1034]: 2026/09/21 13:23:43 INFO Completed upload id=375677worker2 # [ 121.944219] nix-grpc-daemon[1549]: time=2026-09-21T13:23:43.566Z level=INFO msg="Upload complete. (384ms)"5678lb # [ 121.965111] niks3-server[1034]: 2026/09/21 13:23:43 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo5679worker2 # [ 121.952883] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41778 duration_s=0 paths=1 nar_bytes_in=4245680worker2 # [ 121.955088] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417785681worker2 # [ 121.962691] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5682worker2 # [ 121.966101] nix-daemon[860]: reaped child process 1547, status = succeeded5683worker2 # [ 121.973988] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5684client # [ 121.993181] nix[1289]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051...5685client: must succeed: systemctl kill -s INT intr5686client # [ 123.656471] systemd[1]: intr.service: Sent signal SIGINT to main process 1289 (nix) on client request.5687client # [ 123.660851] nix[1289]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)5688worker2 # [ 123.644446] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:45Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv5689client # [ 123.665423] nix[1289]: error: interrupted by the user5690client: (finished: must succeed: systemctl kill -s INT intr, in 0.04 seconds)5691client # [ 123.683697] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE5692client # [ 123.687087] systemd[1]: intr.service: Failed with result 'exit-code'.5693client # [ 123.688862] systemd[1]: intr.service: Consumed 197ms CPU time over 3.517s wall clock time, 41.4M memory peak, 6.4K incoming IP traffic, 8.3K outgoing IP traffic.5694worker2 # [ 123.802495] nix-daemon[860]: reaped child process 1571, status = succeeded5695worker2 # [ 123.808633] nix-daemon[1573]: unexpected Nix daemon error: error: interrupted by the user5696worker2 # [ 123.814683] nix-daemon[860]: reaped child process 1573, status = failed with exit code 15697(finished: subtest: interrupting the client stops the build on the worker, in 4.81 seconds)5698subtest: a deploy mid-build drains: the build finishes, the new generation takes over5699client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw5700client # [ 125.002717] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw.5701client # Running as unit: sw.service; invocation ID: bdc7746b861741b19bd0883cdf0c0d285702client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw, in 0.08 seconds)5703client # [ 125.209361] nix[1310]: don't know how to build these paths:5704client # [ 125.210396] nix[1310]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv5705worker1 # [ 125.273775] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:508325706worker2 # [ 125.205241] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:46Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=05707worker2 # [ 125.211285] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:46Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.3:50051 clients=15708worker1 # [ 125.344831] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:508325709worker1 # [ 125.359940] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:508325710client # [ 125.308317] nix[1310]: copying 1 paths to 'lb:50051'...5711worker1 # [ 125.376778] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:508325712client # [ 125.313337] nix[1310]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'...5713worker1 # [ 125.393643] nix-daemon[835]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5714worker1 # [ 125.558998] nix-daemon[835]: accepted connection from pid 1495, user nix-grpc-daemon (trusted)5715worker1 # [ 125.583088] nix-daemon[835]: reaped child process 1498, status = succeeded5716worker1 # [ 125.691439] nix-daemon[835]: accepted connection from pid 1500, user nix-grpc-daemon (trusted)5717worker1 # [ 125.713486] nix-daemon[835]: reaped child process 1503, status = succeeded5718lb # [ 125.651298] niks3-server[1034]: 2026/09/21 13:23:47 INFO Received uploads request method=POST path=/api/pending_closures5719worker1 # [ 125.735597] nix-grpc-daemon[1398]: time=2026-09-21T13:23:47.354Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5720worker1 # [ 125.737418] nix-grpc-daemon[1398]: time=2026-09-21T13:23:47.354Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)"5721lb # [ 125.689241] niks3-server[1034]: 2026/09/21 13:23:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign5722worker1 # [ 125.761443] nix-grpc-daemon[1398]: time=2026-09-21T13:23:47.380Z level=INFO msg="Uploading 1 narinfos"5723lb # [ 125.695044] niks3-server[1034]: 2026/09/21 13:23:47 INFO Signed narinfos id=38 count=15724lb # [ 125.701906] niks3-server[1034]: 2026/09/21 13:23:47 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst5725lb # [ 125.707779] niks3-server[1034]: 2026/09/21 13:23:47 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls5726lb # [ 125.718400] niks3-server[1034]: 2026/09/21 13:23:47 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete5727lb # [ 125.724728] niks3-server[1034]: 2026/09/21 13:23:47 INFO Completed upload id=385728worker1 # [ 125.791649] nix-grpc-daemon[1398]: time=2026-09-21T13:23:47.411Z level=INFO msg="Upload complete. (377ms)"5729lb # [ 125.727384] niks3-server[1034]: 2026/09/21 13:23:47 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo5730worker1 # [ 125.800379] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:50832 duration_s=0 paths=1 nar_bytes_in=4165731worker1 # [ 125.808694] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:508325732worker1 # [ 125.819235] nix-daemon[835]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5733worker1 # [ 125.823958] nix-daemon[835]: reaped child process 1493, status = succeeded5734client # [ 125.767901] nix[1310]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051...5735worker1 # [ 125.836741] nix-daemon[835]: accepted connection from pid 1280, user nix-grpc-daemon (trusted)5736worker1: must succeed: systemctl reload nix-grpc-daemon.service5737worker1 # [ 127.727773] systemd[1]: Reloading Nix worker-protocol over gRPC...5738worker1 # [ 127.762249] systemd[1]: Reloaded Nix worker-protocol over gRPC.5739worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.09 seconds)5740worker1: must succeed: pgrep -f 'read -t [9]0 x'5741worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds)5742client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&25743client # don't know how to build these paths:5744client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5745worker2 # [ 127.972420] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:49Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:417785746worker2 # [ 127.979263] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417785747worker2 # [ 128.011885] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:49Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=05748worker2 # [ 128.019414] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:49Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.4:50051 clients=15749worker2 # [ 128.070841] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417785750worker2 # [ 128.088389] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417785751client # copying 1 paths to 'lb:50051'...5752worker2 # [ 128.105560] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417785753client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'...5754worker2 # [ 128.130220] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5755worker1 # [ 128.257445] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:49Z level=info event=draining5756worker2 # [ 128.312329] nix-daemon[860]: accepted connection from pid 1628, user nix-grpc-daemon (trusted)5757worker2 # [ 128.334632] nix-daemon[860]: reaped child process 1631, status = succeeded5758worker2 # [ 128.446886] nix-daemon[860]: accepted connection from pid 1633, user nix-grpc-daemon (trusted)5759worker2 # [ 128.468547] nix-daemon[860]: reaped child process 1636, status = succeeded5760lb # [ 128.490014] niks3-server[1034]: 2026/09/21 13:23:50 INFO Received uploads request method=POST path=/api/pending_closures5761worker2 # [ 128.484770] nix-grpc-daemon[1549]: time=2026-09-21T13:23:50.107Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5762worker2 # [ 128.487383] nix-grpc-daemon[1549]: time=2026-09-21T13:23:50.109Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)"5763lb # [ 128.528823] niks3-server[1034]: 2026/09/21 13:23:50 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst5764worker2 # [ 128.526151] nix-grpc-daemon[1549]: time=2026-09-21T13:23:50.148Z level=INFO msg="Uploading 1 narinfos"5765lb # [ 128.544194] niks3-server[1034]: 2026/09/21 13:23:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign5766lb # [ 128.558750] niks3-server[1034]: 2026/09/21 13:23:50 INFO Signed narinfos id=39 count=15767lb # [ 128.566382] niks3-server[1034]: 2026/09/21 13:23:50 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls5768lb # [ 128.583519] niks3-server[1034]: 2026/09/21 13:23:50 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete5769lb # [ 128.600137] niks3-server[1034]: 2026/09/21 13:23:50 INFO Completed upload id=395770worker2 # [ 128.583669] nix-grpc-daemon[1549]: time=2026-09-21T13:23:50.206Z level=INFO msg="Upload complete. (434ms)"5771lb # [ 128.603481] niks3-server[1034]: 2026/09/21 13:23:50 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo5772worker2 # [ 128.595457] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41778 duration_s=0 paths=1 nar_bytes_in=5285773worker2 # [ 128.612711] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417785774worker2 # [ 128.627121] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5775worker2 # [ 128.631357] nix-daemon[860]: reaped child process 1626, status = succeeded5776client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051...5777worker2 # [ 128.644248] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5778client # farm-a> worker2: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5779lb # [ 128.665632] envoy[897]: conn peer=[2001:db8:1::1]:42106 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3078 tx=1494 ms=96185780lb # [ 128.670809] envoy[897]: conn peer=[2001:db8:1::1]:37662 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2987 tx=853 ms=32835781client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5782client # farm-a> LOG-farm-a-during-drain5783worker2 # [ 129.519192] nix-daemon[860]: reaped child process 1640, status = succeeded5784worker2 # [ 129.630905] nix-daemon[860]: accepted connection from pid 1656, user nix-grpc-daemon (trusted)5785worker2 # [ 129.654121] nix-daemon[860]: reaped child process 1659, status = succeeded5786lb # [ 129.670188] niks3-server[1034]: 2026/09/21 13:23:51 INFO Received uploads request method=POST path=/api/pending_closures5787worker2 # [ 129.667446] nix-grpc-daemon[1549]: time=2026-09-21T13:23:51.286Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5788worker2 # [ 129.670360] nix-grpc-daemon[1549]: time=2026-09-21T13:23:51.286Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)"5789lb # [ 129.722124] niks3-server[1034]: 2026/09/21 13:23:51 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv5790lb # [ 129.741838] niks3-server[1034]: 2026/09/21 13:23:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign5791lb # [ 129.752481] niks3-server[1034]: 2026/09/21 13:23:51 INFO Signed narinfos id=40 count=15792worker2 # [ 129.733945] nix-grpc-daemon[1549]: time=2026-09-21T13:23:51.355Z level=INFO msg="Uploading 1 narinfos"5793lb # [ 129.754426] niks3-server[1034]: 2026/09/21 13:23:51 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst5794lb # [ 129.762498] niks3-server[1034]: 2026/09/21 13:23:51 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls5795lb # [ 129.769853] niks3-server[1034]: 2026/09/21 13:23:51 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete5796lb # [ 129.773968] niks3-server[1034]: 2026/09/21 13:23:51 INFO Completed upload id=405797worker2 # [ 129.757085] nix-grpc-daemon[1549]: time=2026-09-21T13:23:51.379Z level=INFO msg="Upload complete. (299ms)"5798lb # [ 129.776468] niks3-server[1034]: 2026/09/21 13:23:51 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo5799worker2 # [ 129.775475] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41778 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=3 outputs=15800worker2 # [ 129.795937] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:51Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=05801worker2 # [ 129.799951] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:51Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.4:50051 clients=15802worker2 # [ 129.805071] nix-daemon[860]: reaped child process 1638, status = succeeded5803worker2 # [ 129.819926] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417785804worker2 # [ 129.835433] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417785805worker2 # [ 129.850666] nix-grpc-daemon[1549]: time=2026-09-21T13:23:51.473Z level=INFO msg="All 1 paths already cached"5806client # copying 1 paths to 'lb:50051'...5807worker2 # [ 129.862021] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417785808client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'...5809worker2 # [ 129.884744] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5810worker2 # [ 130.048832] nix-daemon[860]: accepted connection from pid 1665, user nix-grpc-daemon (trusted)5811worker2 # [ 130.069778] nix-daemon[860]: reaped child process 1668, status = succeeded5812worker2 # [ 130.183921] nix-daemon[860]: accepted connection from pid 1670, user nix-grpc-daemon (trusted)5813worker2 # [ 130.206312] nix-daemon[860]: reaped child process 1673, status = succeeded5814lb # [ 130.221593] niks3-server[1034]: 2026/09/21 13:23:51 INFO Received uploads request method=POST path=/api/pending_closures5815worker2 # [ 130.216997] nix-grpc-daemon[1549]: time=2026-09-21T13:23:51.839Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5816worker2 # [ 130.218512] nix-grpc-daemon[1549]: time=2026-09-21T13:23:51.839Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)"5817lb # [ 130.255259] niks3-server[1034]: 2026/09/21 13:23:51 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst5818worker2 # [ 130.253778] nix-grpc-daemon[1549]: time=2026-09-21T13:23:51.876Z level=INFO msg="Uploading 1 narinfos"5819lb # [ 130.273612] niks3-server[1034]: 2026/09/21 13:23:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign5820lb # [ 130.285054] niks3-server[1034]: 2026/09/21 13:23:51 INFO Signed narinfos id=41 count=15821lb # [ 130.291345] niks3-server[1034]: 2026/09/21 13:23:51 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls5822lb # [ 130.304233] niks3-server[1034]: 2026/09/21 13:23:51 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete5823lb # [ 130.316175] niks3-server[1034]: 2026/09/21 13:23:51 INFO Completed upload id=415824worker2 # [ 130.299618] nix-grpc-daemon[1549]: time=2026-09-21T13:23:51.922Z level=INFO msg="Upload complete. (397ms)"5825lb # [ 130.321963] niks3-server[1034]: 2026/09/21 13:23:51 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo5826worker2 # [ 130.308368] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41778 duration_s=0 paths=1 nar_bytes_in=6405827worker2 # [ 130.324888] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417785828worker2 # [ 130.332813] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5829worker2 # [ 130.338995] nix-daemon[860]: reaped child process 1663, status = succeeded5830client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051...5831client # farm-b> worker2: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv5832worker2 # [ 130.356093] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5833client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5834client # farm-b> LOG-farm-b-during-drain5835worker2 # [ 132.036932] nix-daemon[860]: reaped child process 1677, status = succeeded5836worker2 # [ 132.149002] nix-daemon[860]: accepted connection from pid 1694, user nix-grpc-daemon (trusted)5837worker2 # [ 132.166290] nix-daemon[860]: reaped child process 1697, status = succeeded5838lb # [ 132.192719] niks3-server[1034]: 2026/09/21 13:23:53 INFO Received uploads request method=POST path=/api/pending_closures5839worker2 # [ 132.187358] nix-grpc-daemon[1549]: time=2026-09-21T13:23:53.809Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5840worker2 # [ 132.189163] nix-grpc-daemon[1549]: time=2026-09-21T13:23:53.809Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)"5841lb # [ 132.232144] niks3-server[1034]: 2026/09/21 13:23:53 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst5842lb # [ 132.246990] niks3-server[1034]: 2026/09/21 13:23:53 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv5843worker2 # [ 132.244598] nix-grpc-daemon[1549]: time=2026-09-21T13:23:53.867Z level=INFO msg="Uploading 1 narinfos"5844lb # [ 132.255516] niks3-server[1034]: 2026/09/21 13:23:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign5845lb # [ 132.266021] niks3-server[1034]: 2026/09/21 13:23:53 INFO Signed narinfos id=42 count=15846lb # [ 132.272971] niks3-server[1034]: 2026/09/21 13:23:53 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls5847lb # [ 132.289149] niks3-server[1034]: 2026/09/21 13:23:53 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete5848worker2 # [ 132.278526] nix-grpc-daemon[1549]: time=2026-09-21T13:23:53.901Z level=INFO msg="Upload complete. (309ms)"5849lb # [ 132.298895] niks3-server[1034]: 2026/09/21 13:23:53 INFO Completed upload id=425850worker2 # [ 132.284443] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41778 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=4 outputs=15851lb # [ 132.303755] niks3-server[1034]: 2026/09/21 13:23:53 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo5852worker2 # [ 132.293710] nix-daemon[860]: reaped child process 1675, status = succeeded5853worker2 # [ 132.297910] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:53Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=05854worker2 # [ 132.300536] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:53Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.4:50051 clients=15855worker2 # [ 132.347657] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417785856worker2 # [ 132.364598] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417785857worker2 # [ 132.378021] nix-grpc-daemon[1549]: time=2026-09-21T13:23:54.000Z level=INFO msg="All 2 paths already cached"5858client # copying 1 paths to 'lb:50051'...5859worker2 # [ 132.388397] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417785860client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'...5861worker2 # [ 132.411958] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5862worker2 # [ 132.574520] nix-daemon[860]: accepted connection from pid 1703, user nix-grpc-daemon (trusted)5863worker2 # [ 132.598782] nix-daemon[860]: reaped child process 1706, status = succeeded5864worker2 # [ 132.703460] nix-daemon[860]: accepted connection from pid 1708, user nix-grpc-daemon (trusted)5865worker2 # [ 132.722589] nix-daemon[860]: reaped child process 1711, status = succeeded5866lb # [ 132.744597] niks3-server[1034]: 2026/09/21 13:23:54 INFO Received uploads request method=POST path=/api/pending_closures5867worker2 # [ 132.736799] nix-grpc-daemon[1549]: time=2026-09-21T13:23:54.359Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5868worker2 # [ 132.738328] nix-grpc-daemon[1549]: time=2026-09-21T13:23:54.359Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)"5869lb # [ 132.776185] niks3-server[1034]: 2026/09/21 13:23:54 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst5870worker2 # [ 132.774015] nix-grpc-daemon[1549]: time=2026-09-21T13:23:54.396Z level=INFO msg="Uploading 1 narinfos"5871lb # [ 132.790647] niks3-server[1034]: 2026/09/21 13:23:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign5872lb # [ 132.802169] niks3-server[1034]: 2026/09/21 13:23:54 INFO Signed narinfos id=43 count=15873worker2 # [ 132.796782] nix-grpc-daemon[1549]: time=2026-09-21T13:23:54.419Z level=WARN msg="Failed to register uploaded object" key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo error="server returned 404: object is not pending upload\n"5874lb # [ 132.815173] niks3-server[1034]: 2026/09/21 13:23:54 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete5875lb # [ 132.821935] niks3-server[1034]: 2026/09/21 13:23:54 INFO Completed upload id=435876worker2 # [ 132.808261] nix-grpc-daemon[1549]: time=2026-09-21T13:23:54.419Z level=INFO msg="Upload complete. (367ms)"5877lb # [ 132.825830] niks3-server[1034]: 2026/09/21 13:23:54 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls5878worker2 # [ 132.818590] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41778 duration_s=0 paths=1 nar_bytes_in=7685879worker2 # [ 132.822473] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417785880worker2 # [ 132.835840] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5881worker2 # [ 132.837758] nix-daemon[860]: reaped child process 1701, status = succeeded5882worker2 # [ 132.842587] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)5883client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051...5884client # farm-top> worker2: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5885client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5886client # farm-top> LOG-farm-top-during-drain5887worker2 # [ 133.896957] nix-daemon[860]: reaped child process 1715, status = succeeded5888worker2 # [ 134.037037] nix-daemon[860]: accepted connection from pid 1731, user nix-grpc-daemon (trusted)5889worker2 # [ 134.060760] nix-daemon[860]: reaped child process 1734, status = succeeded5890lb # [ 134.084973] niks3-server[1034]: 2026/09/21 13:23:55 INFO Received uploads request method=POST path=/api/pending_closures5891worker2 # [ 134.080502] nix-grpc-daemon[1549]: time=2026-09-21T13:23:55.702Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5892worker2 # [ 134.082073] nix-grpc-daemon[1549]: time=2026-09-21T13:23:55.702Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)"5893lb # [ 134.132893] niks3-server[1034]: 2026/09/21 13:23:55 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst5894worker2 # [ 134.134058] nix-grpc-daemon[1549]: time=2026-09-21T13:23:55.756Z level=INFO msg="Uploading 1 narinfos"5895lb # [ 134.150382] niks3-server[1034]: 2026/09/21 13:23:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign5896lb # [ 134.160354] niks3-server[1034]: 2026/09/21 13:23:55 INFO Signed narinfos id=44 count=15897lb # [ 134.168161] niks3-server[1034]: 2026/09/21 13:23:55 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls5898lb # [ 134.171805] niks3-server[1034]: 2026/09/21 13:23:55 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv5899lb # [ 134.183333] niks3-server[1034]: 2026/09/21 13:23:55 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete5900lb # [ 134.194665] niks3-server[1034]: 2026/09/21 13:23:55 INFO Completed upload id=445901worker2 # [ 134.175980] nix-grpc-daemon[1549]: time=2026-09-21T13:23:55.798Z level=INFO msg="Upload complete. (318ms)"5902lb # [ 134.197043] niks3-server[1034]: 2026/09/21 13:23:55 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo5903worker2 # [ 134.189972] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41778 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=5 outputs=15904worker2 # [ 134.211373] nix-daemon[860]: reaped child process 1713, status = succeeded5905client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2, in 6.50 seconds)5906client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath5907client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath, in 0.20 seconds)5908worker1: must succeed: pkill -f 'read -t [9]0 x'5909worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.06 seconds)5910client: waiting for success: ! systemctl is-active sw5911worker1 # [ 134.672806] nix-daemon[835]: reaped child process 1507, status = succeeded5912worker1 # [ 134.782325] nix-daemon[835]: accepted connection from pid 1548, user nix-grpc-daemon (trusted)5913worker1 # [ 134.804612] nix-daemon[835]: reaped child process 1551, status = succeeded5914lb # [ 134.736136] niks3-server[1034]: 2026/09/21 13:23:56 INFO Received uploads request method=POST path=/api/pending_closures5915worker1 # [ 134.817473] nix-grpc-daemon[1398]: time=2026-09-21T13:23:56.436Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5916worker1 # [ 134.819106] nix-grpc-daemon[1398]: time=2026-09-21T13:23:56.436Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)"5917lb # [ 134.777852] niks3-server[1034]: 2026/09/21 13:23:56 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv5918lb # [ 134.789892] niks3-server[1034]: 2026/09/21 13:23:56 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst5919worker1 # [ 134.867379] nix-grpc-daemon[1398]: time=2026-09-21T13:23:56.486Z level=INFO msg="Uploading 1 narinfos"5920lb # [ 134.802809] niks3-server[1034]: 2026/09/21 13:23:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign5921lb # [ 134.810625] niks3-server[1034]: 2026/09/21 13:23:56 INFO Signed narinfos id=45 count=15922lb # [ 134.817633] niks3-server[1034]: 2026/09/21 13:23:56 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls5923lb # [ 134.825362] niks3-server[1034]: 2026/09/21 13:23:56 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete5924lb # [ 134.832226] niks3-server[1034]: 2026/09/21 13:23:56 INFO Completed upload id=455925worker1 # [ 134.898586] nix-grpc-daemon[1398]: time=2026-09-21T13:23:56.518Z level=INFO msg="Upload complete. (245ms)"5926lb # [ 134.835483] niks3-server[1034]: 2026/09/21 13:23:56 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo5927worker1 # [ 134.910148] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:50832 duration_s=9 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=2 outputs=15928worker1 # [ 134.927992] nix-daemon[835]: reaped child process 1505, status = succeeded5929client # [ 134.881759] systemd[1]: sw.service: Deactivated successfully.5930client # [ 134.887196] systemd[1]: sw.service: Consumed 202ms CPU time over 9.883s wall clock time, 41.2M memory peak, 7.1K incoming IP traffic, 8.3K outgoing IP traffic.5931worker1 # [ 135.261084] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:56Z level=info event=signal_exit5932client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds)5933client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }5934client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.05 seconds)5935worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service5936worker1 # [ 137.249330] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:58Z level=info event=scheduler_restarting5937worker2 # [ 137.177300] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:58Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:417625938worker2 # [ 137.182681] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:58Z level=info event=worker_gone addr=192.168.1.3:500515939worker1 # [ 137.281579] nix-grpc-daemon[1280]: ts=2026-09-21T13:23:58Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5940worker1 # [ 137.312335] nix-daemon[835]: reaped child process 1386, status = succeeded5941worker1 # [ 137.425688] nix-grpc-daemon[1280]: E0000 00:00:1789997039.044972 1325 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true5942worker1 # [ 137.434179] nix-grpc-daemon[1280]: E0000 00:00:1789997039.045242 1325 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5943worker1 # [ 137.676886] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5944worker1 # [ 137.685271] systemd[1]: nix-grpc-daemon.service: Consumed 1.203s CPU time over 42.604s wall clock time, 36M memory peak, 27.7K incoming IP traffic, 40.7K outgoing IP traffic.5945worker1 # [ 137.714949] systemd[1]: Starting Nix worker-protocol over gRPC...5946worker1 # [ 137.861536] nix-grpc-daemon[1565]: ts=2026-09-21T13:23:59Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5947worker1 # [ 137.864107] nix-grpc-daemon[1565]: ts=2026-09-21T13:23:59Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5948worker1 # [ 137.868120] nix-grpc-daemon[1565]: ts=2026-09-21T13:23:59Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15949worker1 # [ 137.874889] nix-grpc-daemon[1565]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5950worker1 # [ 137.896220] nix-grpc-daemon[1565]: E0000 00:00:1789997039.492182 1565 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5951worker1 # [ 137.898646] nix-grpc-daemon[1565]: E0000 00:00:1789997039.494394 1565 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5952worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 2.19 seconds)5953lb: waiting until exactly no endpoints of aarch64-linux are unhealthy5954lb: must succeed: curl -sf localhost:9901/clusters5955worker1 # [ 137.925472] nix-grpc-daemon[1565]: ts=2026-09-21T13:23:59Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5956worker1 # [ 137.927207] nix-grpc-daemon[1565]: E0000 00:00:1789997039.502691 1604 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5957worker1 # [ 137.933214] nix-grpc-daemon[1565]: E0000 00:00:1789997039.525424 1604 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5958worker1 # [ 137.937949] nix-grpc-daemon[1565]: E0000 00:00:1789997039.525689 1604 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5959worker1 # [ 137.940534] nix-grpc-daemon[1565]: E0000 00:00:1789997039.542492 1604 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5960worker1 # [ 137.942870] systemd[1]: Started Nix worker-protocol over gRPC.5961worker2 # [ 137.854691] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:59Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:417625962worker1 # [ 137.943744] nix-grpc-daemon[1565]: ts=2026-09-21T13:23:59Z level=info event=scheduler_connected addr=lb:500515963worker2 # [ 137.861154] nix-grpc-daemon[1471]: ts=2026-09-21T13:23:59Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05964lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5965lb # [ 138.666221] envoy[897]: conn peer=[2001:db8:1::1]:42306 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9638 tx=4114 ms=62395966lb # [ 138.671529] envoy[897]: conn peer=[2001:db8:1::1]:42298 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2965 tx=1428 ms=96545967lb # [ 138.676586] envoy[897]: conn peer=[2001:db8:1::3]:38208 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2850 tx=873 ms=419765968lb: must succeed: curl -sf localhost:9901/clusters5969lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5970lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.15 seconds)5971lb: must succeed: curl -sf localhost:9901/clusters5972lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5973worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5974worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5975(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 14.18 seconds)5976subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does5977client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&15978worker1 # [ 139.430873] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:01Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:447565979worker1 # [ 139.446756] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)5980worker2 # [ 139.390624] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417785981worker2 # [ 139.438961] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:01Z level=debug event=want client=5 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=05982worker2 # [ 139.444970] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:01Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=15983worker2 # [ 139.451307] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:01Z level=debug event=want client=5 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=05984worker2 # [ 139.456638] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:01Z level=debug event=want client=5 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=05985worker1 # [ 139.572155] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447565986worker1 # [ 139.588635] nix-grpc-daemon[1565]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 16 ms (attempt 1/5)5987worker1 # [ 139.614090] nix-grpc-daemon[1565]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 112 ms (attempt 2/5)5988worker1 # [ 139.728603] nix-grpc-daemon[1565]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 275 ms (attempt 3/5)5989worker1 # [ 140.013642] nix-grpc-daemon[1565]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 715 ms (attempt 4/5)5990worker1 # [ 140.733896] nix-grpc-daemon[1565]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5991worker1 # [ 140.738465] nix-grpc-daemon[1565]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5992worker1 # [ 140.760237] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447565993worker1 # [ 140.777416] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:447565994worker1 # [ 140.794077] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)5995worker1 # [ 140.832828] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:02Z level=info event=niks3_push_started pid=16255996worker1 # [ 140.982312] nix-grpc-daemon[1625]: time=2026-09-21T13:24:02.601Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""5997worker1 # [ 141.091168] nix-daemon[835]: accepted connection from pid 1637, user nix-grpc-daemon (trusted)5998worker1 # [ 141.115835] nix-daemon[835]: reaped child process 1640, status = succeeded5999worker1 # [ 141.216110] nix-daemon[835]: accepted connection from pid 1642, user nix-grpc-daemon (trusted)6000worker1 # [ 141.238172] nix-daemon[835]: reaped child process 1645, status = succeeded6001lb # [ 141.170958] niks3-server[1034]: 2026/09/21 13:24:02 INFO Received uploads request method=POST path=/api/pending_closures6002worker1 # [ 141.250448] nix-grpc-daemon[1625]: time=2026-09-21T13:24:02.868Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6003worker1 # [ 141.251974] nix-grpc-daemon[1625]: time=2026-09-21T13:24:02.868Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)"6004lb # [ 141.209706] niks3-server[1034]: 2026/09/21 13:24:02 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst6005worker1 # [ 141.302772] nix-grpc-daemon[1625]: time=2026-09-21T13:24:02.921Z level=INFO msg="Uploading 1 narinfos"6006lb # [ 141.234924] niks3-server[1034]: 2026/09/21 13:24:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign6007lb # [ 141.244291] niks3-server[1034]: 2026/09/21 13:24:02 INFO Signed narinfos id=46 count=16008lb # [ 141.259544] niks3-server[1034]: 2026/09/21 13:24:02 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete6009worker1 # [ 141.334660] nix-grpc-daemon[1625]: time=2026-09-21T13:24:02.954Z level=INFO msg="Upload complete. (348ms)"6010lb # [ 141.271467] niks3-server[1034]: 2026/09/21 13:24:02 INFO Completed upload id=466011worker1 # [ 141.344320] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=0 paths=1 nar_bytes_in=5126012lb # [ 141.280136] niks3-server[1034]: 2026/09/21 13:24:02 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls6013lb # [ 141.285201] niks3-server[1034]: 2026/09/21 13:24:02 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo6014worker1 # [ 141.360344] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566015worker1 # [ 141.367988] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6016worker1 # [ 141.373846] nix-daemon[835]: reaped child process 1623, status = succeeded6017worker1 # [ 141.388454] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6018worker1 # [ 142.045047] nix-daemon[835]: reaped child process 1649, status = succeeded6019worker1 # [ 142.182202] nix-daemon[835]: accepted connection from pid 1665, user nix-grpc-daemon (trusted)6020worker1 # [ 142.202183] nix-daemon[835]: reaped child process 1668, status = succeeded6021lb # [ 142.139508] niks3-server[1034]: 2026/09/21 13:24:03 INFO Received uploads request method=POST path=/api/pending_closures6022worker1 # [ 142.220473] nix-grpc-daemon[1625]: time=2026-09-21T13:24:03.836Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6023worker1 # [ 142.222258] nix-grpc-daemon[1625]: time=2026-09-21T13:24:03.836Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)"6024lb # [ 142.185016] niks3-server[1034]: 2026/09/21 13:24:03 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv6025lb # [ 142.203095] niks3-server[1034]: 2026/09/21 13:24:03 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6026lb # [ 142.213060] niks3-server[1034]: 2026/09/21 13:24:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign6027worker1 # [ 142.285459] nix-grpc-daemon[1625]: time=2026-09-21T13:24:03.903Z level=INFO msg="Uploading 1 narinfos"6028lb # [ 142.219425] niks3-server[1034]: 2026/09/21 13:24:03 INFO Signed narinfos id=47 count=16029lb # [ 142.225722] niks3-server[1034]: 2026/09/21 13:24:03 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls6030lb # [ 142.234421] niks3-server[1034]: 2026/09/21 13:24:03 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete6031lb # [ 142.240711] niks3-server[1034]: 2026/09/21 13:24:03 INFO Completed upload id=476032worker1 # [ 142.308142] nix-grpc-daemon[1625]: time=2026-09-21T13:24:03.927Z level=INFO msg="Upload complete. (309ms)"6033lb # [ 142.244452] niks3-server[1034]: 2026/09/21 13:24:03 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo6034worker2 # [ 142.234590] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:03Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=16035worker1 # [ 142.321186] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=0 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=6 outputs=16036worker2 # [ 142.248740] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:03Z level=debug event=want client=5 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=06037worker1 # [ 142.343063] nix-daemon[835]: reaped child process 1647, status = succeeded6038worker1 # [ 142.376747] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566039worker1 # [ 142.394295] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447566040worker1 # [ 142.408209] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:447566041worker1 # [ 142.414743] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6042worker1 # [ 142.527651] nix-daemon[835]: accepted connection from pid 1672, user nix-grpc-daemon (trusted)6043worker1 # [ 142.547366] nix-daemon[835]: reaped child process 1675, status = succeeded6044worker1 # [ 142.656769] nix-daemon[835]: accepted connection from pid 1677, user nix-grpc-daemon (trusted)6045worker1 # [ 142.677690] nix-daemon[835]: reaped child process 1680, status = succeeded6046lb # [ 142.608510] niks3-server[1034]: 2026/09/21 13:24:04 INFO Received uploads request method=POST path=/api/pending_closures6047worker1 # [ 142.686116] nix-grpc-daemon[1625]: time=2026-09-21T13:24:04.305Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6048worker1 # [ 142.687602] nix-grpc-daemon[1625]: time=2026-09-21T13:24:04.305Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)"6049lb # [ 142.641173] niks3-server[1034]: 2026/09/21 13:24:04 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst6050worker1 # [ 142.723717] nix-grpc-daemon[1625]: time=2026-09-21T13:24:04.343Z level=INFO msg="Uploading 1 narinfos"6051lb # [ 142.656521] niks3-server[1034]: 2026/09/21 13:24:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign6052lb # [ 142.666929] niks3-server[1034]: 2026/09/21 13:24:04 INFO Signed narinfos id=48 count=16053lb # [ 142.674338] niks3-server[1034]: 2026/09/21 13:24:04 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls6054worker1 # [ 142.756538] nix-grpc-daemon[1625]: time=2026-09-21T13:24:04.376Z level=INFO msg="Upload complete. (334ms)"6055lb # [ 142.689520] niks3-server[1034]: 2026/09/21 13:24:04 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete6056worker1 # [ 142.764262] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=0 paths=1 nar_bytes_in=5126057lb # [ 142.700702] niks3-server[1034]: 2026/09/21 13:24:04 INFO Completed upload id=486058lb # [ 142.703934] niks3-server[1034]: 2026/09/21 13:24:04 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo6059worker1 # [ 142.779877] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566060worker1 # [ 142.786043] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6061worker1 # [ 142.790514] nix-daemon[835]: reaped child process 1670, status = succeeded6062worker1 # [ 142.802913] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6063worker1 # [ 144.468475] nix-daemon[835]: reaped child process 1684, status = succeeded6064worker1 # [ 144.609662] nix-daemon[835]: accepted connection from pid 1700, user nix-grpc-daemon (trusted)6065worker1 # [ 144.633171] nix-daemon[835]: reaped child process 1703, status = succeeded6066lb # [ 144.565873] niks3-server[1034]: 2026/09/21 13:24:06 INFO Received uploads request method=POST path=/api/pending_closures6067worker1 # [ 144.644292] nix-grpc-daemon[1625]: time=2026-09-21T13:24:06.262Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6068worker1 # [ 144.645801] nix-grpc-daemon[1625]: time=2026-09-21T13:24:06.262Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)"6069lb # [ 144.604328] niks3-server[1034]: 2026/09/21 13:24:06 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6070worker1 # [ 144.693742] nix-grpc-daemon[1625]: time=2026-09-21T13:24:06.313Z level=INFO msg="Uploading 1 narinfos"6071lb # [ 144.621540] niks3-server[1034]: 2026/09/21 13:24:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign6072lb # [ 144.634931] niks3-server[1034]: 2026/09/21 13:24:06 INFO Signed narinfos id=49 count=16073lb # [ 144.641178] niks3-server[1034]: 2026/09/21 13:24:06 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv6074lb # [ 144.650251] niks3-server[1034]: 2026/09/21 13:24:06 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls6075lb # [ 144.658960] niks3-server[1034]: 2026/09/21 13:24:06 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete6076lb # [ 144.665696] niks3-server[1034]: 2026/09/21 13:24:06 INFO Completed upload id=496077worker1 # [ 144.731231] nix-grpc-daemon[1625]: time=2026-09-21T13:24:06.350Z level=INFO msg="Upload complete. (317ms)"6078lb # [ 144.668979] niks3-server[1034]: 2026/09/21 13:24:06 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo6079worker2 # [ 144.661312] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:06Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=16080worker1 # [ 144.746430] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=7 outputs=16081worker2 # [ 144.674705] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:06Z level=debug event=want client=5 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=06082worker1 # [ 144.764605] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566083worker1 # [ 144.770417] nix-daemon[835]: reaped child process 1682, status = succeeded6084worker1 # [ 144.778751] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447566085worker1 # [ 144.792922] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:447566086worker1 # [ 144.806095] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6087worker1 # [ 144.950672] nix-daemon[835]: accepted connection from pid 1707, user nix-grpc-daemon (trusted)6088worker1 # [ 144.970537] nix-daemon[835]: reaped child process 1710, status = succeeded6089worker1 # [ 145.084167] nix-daemon[835]: accepted connection from pid 1712, user nix-grpc-daemon (trusted)6090worker1 # [ 145.106149] nix-daemon[835]: reaped child process 1715, status = succeeded6091lb # [ 145.040248] niks3-server[1034]: 2026/09/21 13:24:06 INFO Received uploads request method=POST path=/api/pending_closures6092worker1 # [ 145.117718] nix-grpc-daemon[1625]: time=2026-09-21T13:24:06.737Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6093worker1 # [ 145.119298] nix-grpc-daemon[1625]: time=2026-09-21T13:24:06.737Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)"6094lb # [ 145.070888] niks3-server[1034]: 2026/09/21 13:24:06 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst6095worker1 # [ 145.152494] nix-grpc-daemon[1625]: time=2026-09-21T13:24:06.771Z level=INFO msg="Uploading 1 narinfos"6096lb # [ 145.086393] niks3-server[1034]: 2026/09/21 13:24:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign6097lb # [ 145.098371] niks3-server[1034]: 2026/09/21 13:24:06 INFO Signed narinfos id=50 count=16098lb # [ 145.103661] niks3-server[1034]: 2026/09/21 13:24:06 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls6099worker1 # [ 145.180513] nix-grpc-daemon[1625]: time=2026-09-21T13:24:06.800Z level=INFO msg="Upload complete. (362ms)"6100lb # [ 145.113480] niks3-server[1034]: 2026/09/21 13:24:06 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete6101lb # [ 145.117323] niks3-server[1034]: 2026/09/21 13:24:06 INFO Completed upload id=506102lb # [ 145.121289] niks3-server[1034]: 2026/09/21 13:24:06 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo6103worker1 # [ 145.192209] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=0 paths=1 nar_bytes_in=5126104worker1 # [ 145.202267] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566105worker1 # [ 145.208548] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6106worker1 # [ 145.213452] nix-daemon[835]: reaped child process 1705, status = succeeded6107worker1 # [ 145.225835] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6108worker1 # [ 146.089636] nix-daemon[835]: reaped child process 1719, status = succeeded6109worker1 # [ 146.215227] nix-daemon[835]: accepted connection from pid 1735, user nix-grpc-daemon (trusted)6110worker1 # [ 146.238144] nix-daemon[835]: reaped child process 1738, status = succeeded6111lb # [ 146.175792] niks3-server[1034]: 2026/09/21 13:24:07 INFO Received uploads request method=POST path=/api/pending_closures6112worker1 # [ 146.253984] nix-grpc-daemon[1625]: time=2026-09-21T13:24:07.871Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6113worker1 # [ 146.255562] nix-grpc-daemon[1625]: time=2026-09-21T13:24:07.871Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)"6114lb # [ 146.214034] niks3-server[1034]: 2026/09/21 13:24:07 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6115lb # [ 146.227744] niks3-server[1034]: 2026/09/21 13:24:07 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv6116worker1 # [ 146.306822] nix-grpc-daemon[1625]: time=2026-09-21T13:24:07.926Z level=INFO msg="Uploading 1 narinfos"6117lb # [ 146.242360] niks3-server[1034]: 2026/09/21 13:24:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign6118lb # [ 146.252279] niks3-server[1034]: 2026/09/21 13:24:07 INFO Signed narinfos id=51 count=16119lb # [ 146.254729] niks3-server[1034]: 2026/09/21 13:24:07 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls6120lb # [ 146.266291] niks3-server[1034]: 2026/09/21 13:24:07 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6121lb # [ 146.271511] niks3-server[1034]: 2026/09/21 13:24:07 INFO Completed upload id=516122lb # [ 146.274934] niks3-server[1034]: 2026/09/21 13:24:07 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo6123worker1 # [ 146.337868] nix-grpc-daemon[1625]: time=2026-09-21T13:24:07.957Z level=INFO msg="Upload complete. (302ms)"6124worker2 # [ 146.265046] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:07Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=16125worker2 # [ 146.278104] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:07Z level=debug event=want client=5 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=06126worker1 # [ 146.350608] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=8 outputs=16127worker1 # [ 146.371378] nix-daemon[835]: reaped child process 1717, status = succeeded6128worker1 # [ 146.395273] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566129worker1 # [ 146.412323] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447566130worker1 # [ 146.427736] nix-grpc-daemon[1625]: time=2026-09-21T13:24:08.047Z level=INFO msg="All 1 paths already cached"6131worker1 # [ 146.437256] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:447566132worker1 # [ 146.451554] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6133worker1 # [ 146.608918] nix-daemon[835]: accepted connection from pid 1742, user nix-grpc-daemon (trusted)6134worker1 # [ 146.627609] nix-daemon[835]: reaped child process 1745, status = succeeded6135worker1 # [ 146.743121] nix-daemon[835]: accepted connection from pid 1747, user nix-grpc-daemon (trusted)6136worker1 # [ 146.765296] nix-daemon[835]: reaped child process 1750, status = succeeded6137lb # [ 146.699539] niks3-server[1034]: 2026/09/21 13:24:08 INFO Received uploads request method=POST path=/api/pending_closures6138worker1 # [ 146.776215] nix-grpc-daemon[1625]: time=2026-09-21T13:24:08.395Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6139worker1 # [ 146.778162] nix-grpc-daemon[1625]: time=2026-09-21T13:24:08.395Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)"6140lb # [ 146.731284] niks3-server[1034]: 2026/09/21 13:24:08 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst6141worker1 # [ 146.817877] nix-grpc-daemon[1625]: time=2026-09-21T13:24:08.437Z level=INFO msg="Uploading 1 narinfos"6142lb # [ 146.749487] niks3-server[1034]: 2026/09/21 13:24:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6143lb # [ 146.763190] niks3-server[1034]: 2026/09/21 13:24:08 INFO Signed narinfos id=52 count=16144lb # [ 146.771222] niks3-server[1034]: 2026/09/21 13:24:08 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls6145lb # [ 146.784170] niks3-server[1034]: 2026/09/21 13:24:08 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6146lb # [ 146.791987] niks3-server[1034]: 2026/09/21 13:24:08 INFO Completed upload id=526147worker1 # [ 146.858250] nix-grpc-daemon[1625]: time=2026-09-21T13:24:08.477Z level=INFO msg="Upload complete. (388ms)"6148lb # [ 146.795508] niks3-server[1034]: 2026/09/21 13:24:08 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo6149worker1 # [ 146.868411] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=0 paths=1 nar_bytes_in=6246150worker1 # [ 146.886810] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566151worker1 # [ 146.894716] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6152worker1 # [ 146.899946] nix-daemon[835]: reaped child process 1740, status = succeeded6153worker1 # [ 146.912618] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6154worker1 # [ 148.223655] nix-daemon[835]: reaped child process 1754, status = succeeded6155worker1 # [ 148.299850] nix-daemon[835]: accepted connection from pid 1770, user nix-grpc-daemon (trusted)6156worker1 # [ 148.322273] nix-daemon[835]: reaped child process 1773, status = succeeded6157lb # [ 148.261029] niks3-server[1034]: 2026/09/21 13:24:09 INFO Received uploads request method=POST path=/api/pending_closures6158worker1 # [ 148.339934] nix-grpc-daemon[1625]: time=2026-09-21T13:24:09.959Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6159worker1 # [ 148.344786] nix-grpc-daemon[1625]: time=2026-09-21T13:24:09.961Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)"6160lb # [ 148.300162] niks3-server[1034]: 2026/09/21 13:24:09 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv6161lb # [ 148.315927] niks3-server[1034]: 2026/09/21 13:24:09 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst6162lb # [ 148.329308] niks3-server[1034]: 2026/09/21 13:24:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6163worker1 # [ 148.396893] nix-grpc-daemon[1625]: time=2026-09-21T13:24:10.016Z level=INFO msg="Uploading 1 narinfos"6164lb # [ 148.334641] niks3-server[1034]: 2026/09/21 13:24:09 INFO Signed narinfos id=53 count=16165lb # [ 148.341048] niks3-server[1034]: 2026/09/21 13:24:09 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls6166lb # [ 148.353104] niks3-server[1034]: 2026/09/21 13:24:09 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6167lb # [ 148.360394] niks3-server[1034]: 2026/09/21 13:24:09 INFO Completed upload id=536168worker1 # [ 148.425736] nix-grpc-daemon[1625]: time=2026-09-21T13:24:10.045Z level=INFO msg="Upload complete. (298ms)"6169lb # [ 148.363653] niks3-server[1034]: 2026/09/21 13:24:09 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo6170worker1 # [ 148.437143] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=9 outputs=16171worker2 # [ 148.357785] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:09Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=16172worker2 # [ 148.365166] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:09Z level=debug event=want client=5 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=06173worker1 # [ 148.459079] nix-daemon[835]: reaped child process 1752, status = succeeded6174worker1 # [ 148.483589] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566175worker1 # [ 148.499822] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447566176worker1 # [ 148.514352] nix-grpc-daemon[1625]: time=2026-09-21T13:24:10.133Z level=INFO msg="All 1 paths already cached"6177worker1 # [ 148.523661] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:447566178worker1 # [ 148.537102] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6179worker1 # [ 148.690456] nix-daemon[835]: accepted connection from pid 1777, user nix-grpc-daemon (trusted)6180worker1 # [ 148.711744] nix-daemon[835]: reaped child process 1780, status = succeeded6181worker1 # [ 148.820530] nix-daemon[835]: accepted connection from pid 1782, user nix-grpc-daemon (trusted)6182worker1 # [ 148.839705] nix-daemon[835]: reaped child process 1785, status = succeeded6183lb # [ 148.777156] niks3-server[1034]: 2026/09/21 13:24:10 INFO Received uploads request method=POST path=/api/pending_closures6184worker1 # [ 148.853177] nix-grpc-daemon[1625]: time=2026-09-21T13:24:10.472Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6185worker1 # [ 148.854663] nix-grpc-daemon[1625]: time=2026-09-21T13:24:10.472Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)"6186lb # [ 148.804675] niks3-server[1034]: 2026/09/21 13:24:10 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst6187worker1 # [ 148.885548] nix-grpc-daemon[1625]: time=2026-09-21T13:24:10.505Z level=INFO msg="Uploading 1 narinfos"6188lb # [ 148.819822] niks3-server[1034]: 2026/09/21 13:24:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6189lb # [ 148.831085] niks3-server[1034]: 2026/09/21 13:24:10 INFO Signed narinfos id=54 count=16190lb # [ 148.837277] niks3-server[1034]: 2026/09/21 13:24:10 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls6191lb # [ 148.846477] niks3-server[1034]: 2026/09/21 13:24:10 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6192worker1 # [ 148.919144] nix-grpc-daemon[1625]: time=2026-09-21T13:24:10.538Z level=INFO msg="Upload complete. (366ms)"6193lb # [ 148.854070] niks3-server[1034]: 2026/09/21 13:24:10 INFO Completed upload id=546194lb # [ 148.857053] niks3-server[1034]: 2026/09/21 13:24:10 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo6195worker1 # [ 148.928319] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=0 paths=1 nar_bytes_in=6246196worker1 # [ 148.938961] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566197worker1 # [ 148.945323] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6198worker1 # [ 148.949797] nix-daemon[835]: reaped child process 1775, status = succeeded6199worker1 # [ 148.962853] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6200worker1 # [ 151.483649] nix-daemon[835]: reaped child process 1789, status = succeeded6201worker1 # [ 151.622486] nix-daemon[835]: accepted connection from pid 1805, user nix-grpc-daemon (trusted)6202worker1 # [ 151.644582] nix-daemon[835]: reaped child process 1808, status = succeeded6203lb # [ 151.581076] niks3-server[1034]: 2026/09/21 13:24:13 INFO Received uploads request method=POST path=/api/pending_closures6204worker1 # [ 151.664524] nix-grpc-daemon[1625]: time=2026-09-21T13:24:13.280Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6205worker1 # [ 151.666642] nix-grpc-daemon[1625]: time=2026-09-21T13:24:13.280Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)"6206lb # [ 151.639110] niks3-server[1034]: 2026/09/21 13:24:13 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst6207worker1 # [ 151.715058] nix-grpc-daemon[1625]: time=2026-09-21T13:24:13.334Z level=INFO msg="Uploading 1 narinfos"6208lb # [ 151.654954] niks3-server[1034]: 2026/09/21 13:24:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6209lb # [ 151.664696] niks3-server[1034]: 2026/09/21 13:24:13 INFO Signed narinfos id=55 count=16210lb # [ 151.670574] niks3-server[1034]: 2026/09/21 13:24:13 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls6211lb # [ 151.680249] niks3-server[1034]: 2026/09/21 13:24:13 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv6212worker1 # [ 151.754202] nix-grpc-daemon[1625]: time=2026-09-21T13:24:13.373Z level=INFO msg="Upload complete. (320ms)"6213lb # [ 151.687774] niks3-server[1034]: 2026/09/21 13:24:13 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6214lb # [ 151.695392] niks3-server[1034]: 2026/09/21 13:24:13 INFO Completed upload id=556215worker2 # [ 151.679144] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:13Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=16216lb # [ 151.699977] niks3-server[1034]: 2026/09/21 13:24:13 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo6217worker1 # [ 151.769198] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=2 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=10 outputs=16218worker2 # [ 151.693032] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:13Z level=debug event=want client=5 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=06219worker1 # [ 151.788521] nix-daemon[835]: reaped child process 1787, status = succeeded6220worker1 # [ 151.820190] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566221worker1 # [ 151.837060] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447566222worker1 # [ 151.852069] nix-grpc-daemon[1625]: time=2026-09-21T13:24:13.471Z level=INFO msg="All 1 paths already cached"6223worker1 # [ 151.862270] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:447566224worker1 # [ 151.876724] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6225worker1 # [ 152.042130] nix-daemon[835]: accepted connection from pid 1813, user nix-grpc-daemon (trusted)6226worker1 # [ 152.070332] nix-daemon[835]: reaped child process 1816, status = succeeded6227worker1 # [ 152.171468] nix-daemon[835]: accepted connection from pid 1818, user nix-grpc-daemon (trusted)6228worker1 # [ 152.190914] nix-daemon[835]: reaped child process 1821, status = succeeded6229lb # [ 152.128516] niks3-server[1034]: 2026/09/21 13:24:13 INFO Received uploads request method=POST path=/api/pending_closures6230worker1 # [ 152.205326] nix-grpc-daemon[1625]: time=2026-09-21T13:24:13.824Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6231worker1 # [ 152.206837] nix-grpc-daemon[1625]: time=2026-09-21T13:24:13.824Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)"6232lb # [ 152.160177] niks3-server[1034]: 2026/09/21 13:24:13 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst6233worker1 # [ 152.249178] nix-grpc-daemon[1625]: time=2026-09-21T13:24:13.868Z level=INFO msg="Uploading 1 narinfos"6234lb # [ 152.175812] niks3-server[1034]: 2026/09/21 13:24:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6235lb # [ 152.190138] niks3-server[1034]: 2026/09/21 13:24:13 INFO Signed narinfos id=56 count=16236lb # [ 152.197243] niks3-server[1034]: 2026/09/21 13:24:13 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls6237lb # [ 152.208740] niks3-server[1034]: 2026/09/21 13:24:13 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6238lb # [ 152.215280] niks3-server[1034]: 2026/09/21 13:24:13 INFO Completed upload id=566239worker1 # [ 152.282544] nix-grpc-daemon[1625]: time=2026-09-21T13:24:13.902Z level=INFO msg="Upload complete. (388ms)"6240lb # [ 152.220932] niks3-server[1034]: 2026/09/21 13:24:13 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo6241worker1 # [ 152.292517] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=0 paths=1 nar_bytes_in=6246242worker1 # [ 152.306083] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566243worker1 # [ 152.312820] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6244worker1 # [ 152.318033] nix-daemon[835]: reaped child process 1811, status = succeeded6245worker1 # [ 152.331892] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6246worker1 # [ 153.816923] nix-daemon[835]: reaped child process 1825, status = succeeded6247worker1 # [ 153.906949] nix-daemon[835]: accepted connection from pid 1841, user nix-grpc-daemon (trusted)6248worker1 # [ 153.930705] nix-daemon[835]: reaped child process 1844, status = succeeded6249lb # [ 153.863403] niks3-server[1034]: 2026/09/21 13:24:15 INFO Received uploads request method=POST path=/api/pending_closures6250worker1 # [ 153.943486] nix-grpc-daemon[1625]: time=2026-09-21T13:24:15.561Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6251worker1 # [ 153.945300] nix-grpc-daemon[1625]: time=2026-09-21T13:24:15.561Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)"6252lb # [ 153.909131] niks3-server[1034]: 2026/09/21 13:24:15 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv6253worker1 # [ 153.988005] nix-grpc-daemon[1625]: time=2026-09-21T13:24:15.607Z level=INFO msg="Uploading 1 narinfos"6254lb # [ 153.925296] niks3-server[1034]: 2026/09/21 13:24:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6255lb # [ 153.932367] niks3-server[1034]: 2026/09/21 13:24:15 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst6256lb # [ 153.941526] niks3-server[1034]: 2026/09/21 13:24:15 INFO Signed narinfos id=57 count=16257lb # [ 153.944677] niks3-server[1034]: 2026/09/21 13:24:15 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls6258lb # [ 153.947328] niks3-server[1034]: 2026/09/21 13:24:15 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6259lb # [ 153.953228] niks3-server[1034]: 2026/09/21 13:24:15 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo6260worker1 # [ 154.019699] nix-grpc-daemon[1625]: time=2026-09-21T13:24:15.639Z level=INFO msg="Upload complete. (276ms)"6261lb # [ 153.956375] niks3-server[1034]: 2026/09/21 13:24:15 INFO Completed upload id=576262worker2 # [ 153.948105] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:15Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=16263worker2 # [ 153.957013] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:15Z level=debug event=want client=5 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=06264worker1 # [ 154.032917] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=11 outputs=16265worker1 # [ 154.053186] nix-daemon[835]: reaped child process 1823, status = succeeded6266worker1 # [ 154.075892] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566267worker1 # [ 154.091601] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447566268worker1 # [ 154.106468] nix-grpc-daemon[1625]: time=2026-09-21T13:24:15.726Z level=INFO msg="All 2 paths already cached"6269worker1 # [ 154.115628] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:447566270worker1 # [ 154.129826] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6271worker1 # [ 154.277062] nix-daemon[835]: accepted connection from pid 1848, user nix-grpc-daemon (trusted)6272worker1 # [ 154.297592] nix-daemon[835]: reaped child process 1851, status = succeeded6273worker1 # [ 154.407997] nix-daemon[835]: accepted connection from pid 1853, user nix-grpc-daemon (trusted)6274worker1 # [ 154.428819] nix-daemon[835]: reaped child process 1856, status = succeeded6275lb # [ 154.360344] niks3-server[1034]: 2026/09/21 13:24:15 INFO Received uploads request method=POST path=/api/pending_closures6276worker1 # [ 154.436108] nix-grpc-daemon[1625]: time=2026-09-21T13:24:16.055Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6277worker1 # [ 154.437619] nix-grpc-daemon[1625]: time=2026-09-21T13:24:16.055Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)"6278lb # [ 154.391629] niks3-server[1034]: 2026/09/21 13:24:16 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst6279worker1 # [ 154.472483] nix-grpc-daemon[1625]: time=2026-09-21T13:24:16.091Z level=INFO msg="Uploading 1 narinfos"6280lb # [ 154.407095] niks3-server[1034]: 2026/09/21 13:24:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6281lb # [ 154.418680] niks3-server[1034]: 2026/09/21 13:24:16 INFO Signed narinfos id=58 count=16282lb # [ 154.424712] niks3-server[1034]: 2026/09/21 13:24:16 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls6283lb # [ 154.434388] niks3-server[1034]: 2026/09/21 13:24:16 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6284lb # [ 154.439987] niks3-server[1034]: 2026/09/21 13:24:16 INFO Completed upload id=586285worker1 # [ 154.506183] nix-grpc-daemon[1625]: time=2026-09-21T13:24:16.125Z level=INFO msg="Upload complete. (361ms)"6286lb # [ 154.442711] niks3-server[1034]: 2026/09/21 13:24:16 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo6287worker1 # [ 154.516972] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=0 paths=1 nar_bytes_in=7526288worker1 # [ 154.527387] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566289worker1 # [ 154.534231] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6290worker1 # [ 154.538969] nix-daemon[835]: reaped child process 1846, status = succeeded6291worker1 # [ 154.552217] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6292worker1 # [ 155.833234] nix-daemon[835]: reaped child process 1860, status = succeeded6293worker1 # [ 155.942813] nix-daemon[835]: accepted connection from pid 1876, user nix-grpc-daemon (trusted)6294worker1 # [ 155.965683] nix-daemon[835]: reaped child process 1879, status = succeeded6295lb # [ 155.903460] niks3-server[1034]: 2026/09/21 13:24:17 INFO Received uploads request method=POST path=/api/pending_closures6296worker1 # [ 155.988515] nix-grpc-daemon[1625]: time=2026-09-21T13:24:17.604Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6297worker1 # [ 155.990516] nix-grpc-daemon[1625]: time=2026-09-21T13:24:17.604Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)"6298lb # [ 155.950043] niks3-server[1034]: 2026/09/21 13:24:17 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv6299lb # [ 155.965999] niks3-server[1034]: 2026/09/21 13:24:17 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst6300lb # [ 155.980749] niks3-server[1034]: 2026/09/21 13:24:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6301worker1 # [ 156.053502] nix-grpc-daemon[1625]: time=2026-09-21T13:24:17.672Z level=INFO msg="Uploading 1 narinfos"6302lb # [ 155.993048] niks3-server[1034]: 2026/09/21 13:24:17 INFO Signed narinfos id=59 count=16303lb # [ 156.001490] niks3-server[1034]: 2026/09/21 13:24:17 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls6304lb # [ 156.016818] niks3-server[1034]: 2026/09/21 13:24:17 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6305lb # [ 156.025270] niks3-server[1034]: 2026/09/21 13:24:17 INFO Completed upload id=596306worker1 # [ 156.090501] nix-grpc-daemon[1625]: time=2026-09-21T13:24:17.709Z level=INFO msg="Upload complete. (323ms)"6307lb # [ 156.029816] niks3-server[1034]: 2026/09/21 13:24:17 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo6308worker2 # [ 156.015468] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:17Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=16309worker1 # [ 156.102462] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=12 outputs=16310worker1 # [ 156.124341] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566311worker1 # [ 156.135369] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447566312worker1 # [ 156.141769] nix-daemon[835]: reaped child process 1858, status = succeeded6313worker1 # [ 156.149030] nix-grpc-daemon[1625]: time=2026-09-21T13:24:17.768Z level=INFO msg="All 2 paths already cached"6314worker1 # [ 156.157099] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:447566315worker1 # [ 156.173185] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6316worker1 # [ 156.328762] nix-daemon[835]: accepted connection from pid 1883, user nix-grpc-daemon (trusted)6317worker1 # [ 156.356742] nix-daemon[835]: reaped child process 1886, status = succeeded6318worker1 # [ 156.460875] nix-daemon[835]: accepted connection from pid 1888, user nix-grpc-daemon (trusted)6319worker1 # [ 156.477929] nix-daemon[835]: reaped child process 1891, status = succeeded6320lb # [ 156.418161] niks3-server[1034]: 2026/09/21 13:24:18 INFO Received uploads request method=POST path=/api/pending_closures6321worker1 # [ 156.494124] nix-grpc-daemon[1625]: time=2026-09-21T13:24:18.113Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6322worker1 # [ 156.495658] nix-grpc-daemon[1625]: time=2026-09-21T13:24:18.113Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)"6323lb # [ 156.443833] niks3-server[1034]: 2026/09/21 13:24:18 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst6324worker1 # [ 156.524613] nix-grpc-daemon[1625]: time=2026-09-21T13:24:18.144Z level=INFO msg="Uploading 1 narinfos"6325lb # [ 156.458714] niks3-server[1034]: 2026/09/21 13:24:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6326lb # [ 156.470617] niks3-server[1034]: 2026/09/21 13:24:18 INFO Signed narinfos id=60 count=16327lb # [ 156.476483] niks3-server[1034]: 2026/09/21 13:24:18 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls6328lb # [ 156.487509] niks3-server[1034]: 2026/09/21 13:24:18 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6329lb # [ 156.493024] niks3-server[1034]: 2026/09/21 13:24:18 INFO Completed upload id=606330worker1 # [ 156.554028] nix-grpc-daemon[1625]: time=2026-09-21T13:24:18.173Z level=INFO msg="Upload complete. (368ms)"6331lb # [ 156.498393] niks3-server[1034]: 2026/09/21 13:24:18 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo6332worker1 # [ 156.564610] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=0 paths=1 nar_bytes_in=7526333worker1 # [ 156.574651] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566334worker1 # [ 156.580057] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6335worker1 # [ 156.585538] nix-daemon[835]: reaped child process 1881, status = succeeded6336worker1 # [ 156.596170] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6337worker1 # [ 158.048610] nix-daemon[835]: reaped child process 1895, status = succeeded6338worker1 # [ 158.245373] nix-daemon[835]: accepted connection from pid 1911, user nix-grpc-daemon (trusted)6339worker1 # [ 158.275298] nix-daemon[835]: reaped child process 1914, status = succeeded6340lb # [ 158.206700] niks3-server[1034]: 2026/09/21 13:24:19 INFO Received uploads request method=POST path=/api/pending_closures6341worker1 # [ 158.283035] nix-grpc-daemon[1625]: time=2026-09-21T13:24:19.901Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6342worker1 # [ 158.285094] nix-grpc-daemon[1625]: time=2026-09-21T13:24:19.901Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)"6343lb # [ 158.238339] niks3-server[1034]: 2026/09/21 13:24:19 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv6344lb # [ 158.244534] niks3-server[1034]: 2026/09/21 13:24:19 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst6345lb # [ 158.254488] niks3-server[1034]: 2026/09/21 13:24:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6346lb # [ 158.258310] niks3-server[1034]: 2026/09/21 13:24:19 INFO Signed narinfos id=61 count=16347worker1 # [ 158.324852] nix-grpc-daemon[1625]: time=2026-09-21T13:24:19.944Z level=INFO msg="Uploading 1 narinfos"6348lb # [ 158.263270] niks3-server[1034]: 2026/09/21 13:24:19 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls6349lb # [ 158.272260] niks3-server[1034]: 2026/09/21 13:24:19 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6350lb # [ 158.277266] niks3-server[1034]: 2026/09/21 13:24:19 INFO Completed upload id=616351worker1 # [ 158.343998] nix-grpc-daemon[1625]: time=2026-09-21T13:24:19.963Z level=INFO msg="Upload complete. (326ms)"6352worker2 # [ 158.269887] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:19Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=16353worker1 # [ 158.355274] nix-grpc-daemon[1625]: time=2026-09-21T13:24:19.963Z level=WARN msg="Failed to register uploaded object" key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo error="server returned 404: object is not pending upload\n"6354worker1 # [ 158.371265] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=13 outputs=16355worker1 # [ 158.376725] nix-daemon[835]: reaped child process 1893, status = succeeded6356worker1 # [ 158.399877] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566357worker1 # [ 158.415371] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447566358worker1 # [ 158.431706] nix-grpc-daemon[1625]: time=2026-09-21T13:24:20.051Z level=INFO msg="All 2 paths already cached"6359worker1 # [ 158.441782] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:447566360worker1 # [ 158.456256] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6361worker1 # [ 158.607430] nix-daemon[835]: accepted connection from pid 1918, user nix-grpc-daemon (trusted)6362worker1 # [ 158.629001] nix-daemon[835]: reaped child process 1921, status = succeeded6363worker1 # [ 158.746185] nix-daemon[835]: accepted connection from pid 1923, user nix-grpc-daemon (trusted)6364worker1 # [ 158.771427] nix-daemon[835]: reaped child process 1926, status = succeeded6365lb # [ 158.701756] niks3-server[1034]: 2026/09/21 13:24:20 INFO Received uploads request method=POST path=/api/pending_closures6366worker1 # [ 158.786961] nix-grpc-daemon[1625]: time=2026-09-21T13:24:20.406Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6367worker1 # [ 158.788678] nix-grpc-daemon[1625]: time=2026-09-21T13:24:20.406Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)"6368lb # [ 158.756009] niks3-server[1034]: 2026/09/21 13:24:20 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst6369worker1 # [ 158.834708] nix-grpc-daemon[1625]: time=2026-09-21T13:24:20.453Z level=INFO msg="Uploading 1 narinfos"6370lb # [ 158.768731] niks3-server[1034]: 2026/09/21 13:24:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6371lb # [ 158.771850] niks3-server[1034]: 2026/09/21 13:24:20 INFO Signed narinfos id=62 count=16372lb # [ 158.785128] niks3-server[1034]: 2026/09/21 13:24:20 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls6373lb # [ 158.792968] niks3-server[1034]: 2026/09/21 13:24:20 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6374lb # [ 158.799715] niks3-server[1034]: 2026/09/21 13:24:20 INFO Completed upload id=626375worker1 # [ 158.866754] nix-grpc-daemon[1625]: time=2026-09-21T13:24:20.486Z level=INFO msg="Upload complete. (394ms)"6376lb # [ 158.802125] niks3-server[1034]: 2026/09/21 13:24:20 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo6377worker1 # [ 158.878281] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=0 paths=1 nar_bytes_in=7526378worker1 # [ 158.892733] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:447566379worker1 # [ 158.903066] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6380worker1 # [ 158.910437] nix-daemon[835]: reaped child process 1916, status = succeeded6381worker1 # [ 158.921246] nix-daemon[835]: accepted connection from pid 1565, user nix-grpc-daemon (trusted)6382worker1 # [ 159.823375] nix-daemon[835]: reaped child process 1930, status = succeeded6383worker1 # [ 159.908856] nix-daemon[835]: accepted connection from pid 1946, user nix-grpc-daemon (trusted)6384worker1 # [ 159.931870] nix-daemon[835]: reaped child process 1949, status = succeeded6385lb # [ 159.867486] niks3-server[1034]: 2026/09/21 13:24:21 INFO Received uploads request method=POST path=/api/pending_closures6386worker1 # [ 159.948431] nix-grpc-daemon[1625]: time=2026-09-21T13:24:21.564Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6387worker1 # [ 159.950031] nix-grpc-daemon[1625]: time=2026-09-21T13:24:21.564Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)"6388lb # [ 159.916141] niks3-server[1034]: 2026/09/21 13:24:21 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv6389worker1 # [ 159.998105] nix-grpc-daemon[1625]: time=2026-09-21T13:24:21.617Z level=INFO msg="Uploading 1 narinfos"6390lb # [ 159.931795] niks3-server[1034]: 2026/09/21 13:24:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6391lb # [ 159.941423] niks3-server[1034]: 2026/09/21 13:24:21 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst6392lb # [ 159.948776] niks3-server[1034]: 2026/09/21 13:24:21 INFO Signed narinfos id=63 count=16393worker1 # [ 160.021579] nix-grpc-daemon[1625]: time=2026-09-21T13:24:21.641Z level=INFO msg="Upload complete. (288ms)"6394lb # [ 159.956156] niks3-server[1034]: 2026/09/21 13:24:21 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls6395lb # [ 159.960361] niks3-server[1034]: 2026/09/21 13:24:21 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6396lb # [ 159.964167] niks3-server[1034]: 2026/09/21 13:24:21 INFO Completed upload id=636397lb # [ 159.969036] niks3-server[1034]: 2026/09/21 13:24:21 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo6398worker1 # [ 160.033197] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44756 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=14 outputs=16399worker1 # [ 160.052623] nix-daemon[835]: reaped child process 1928, status = succeeded6400client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 20.91 seconds)6401worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6402worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6403client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]'6404client # [ 160.163326] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features "[\"gpu\"]".6405client # Running as unit: gpu.service; invocation ID: 14f5b089e43e4b92ad62ac8a4fb3e4726406client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds)6407lb: must succeed: curl -sf localhost:9901/clusters6408lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6409lb: must succeed: curl -sf localhost:9901/clusters6410lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6411worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6412worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)6413client # [ 160.385872] nix[1389]: don't know how to build these paths:6414client # [ 160.386900] nix[1389]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv6415worker1 # [ 160.448921] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:22Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:447566416worker1 # [ 160.458728] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447566417worker2 # [ 160.397393] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:22Z level=debug event=want client=6 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=06418client # [ 160.422475] nix[1389]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, waiting up to 30s for a worker6419lb: must succeed: curl -sf localhost:9901/clusters6420lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)6421lb: must succeed: curl -sf localhost:9901/clusters6422lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)6423worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6424worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6425client: waiting for success: systemctl show -p Result gpu | grep -q exit-code6426lb # [ 168.667296] envoy[897]: conn peer=[2001:db8:1::1]:34284 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27157 tx=11439 ms=206286427client # [ 190.429946] nix[1389]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}6428client # [ 190.441722] nix[1389]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed6429client # [ 190.473699] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE6430client # [ 190.481827] systemd[1]: gpu.service: Failed with result 'exit-code'.6431client # [ 190.485678] systemd[1]: gpu.service: Consumed 213ms CPU time over 30.315s wall clock time, 35.4M memory peak, 5.1K incoming IP traffic, 5K outgoing IP traffic.6432client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.11 seconds)6433client: must succeed: journalctl -u gpu -o cat6434client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)6435lb: must succeed: curl -sf localhost:9901/clusters6436lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6437lb: must succeed: curl -sf localhost:9901/clusters6438lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6439worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics6440worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6441(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 51.67 seconds)6442subtest: low disk drains a worker and builds go to the other6443worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill6444worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.18 seconds)6445worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'6446worker1 # [ 192.057684] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:53Z level=info event=unhealthy reason=min_free6447worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 0.03 seconds)6448client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&26449client # don't know how to build these paths:6450client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6451worker1 # [ 192.286558] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:53Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:447566452worker1 # [ 192.295163] nix-grpc-daemon[1565]: ts=2026-09-21T13:24:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:447566453worker2 # [ 192.233511] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:53Z level=debug event=want client=7 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=06454worker2 # [ 192.240259] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:53Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=16455worker2 # [ 192.249931] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417786456worker2 # [ 192.263272] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417786457client # copying 1 paths to 'lb:50051'...6458client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'...6459worker2 # [ 192.276213] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417786460worker2 # [ 192.294345] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)6461worker2 # [ 192.449445] nix-daemon[860]: accepted connection from pid 1768, user nix-grpc-daemon (trusted)6462worker2 # [ 192.474082] nix-daemon[860]: reaped child process 1771, status = succeeded6463worker2 # [ 192.575957] nix-daemon[860]: accepted connection from pid 1773, user nix-grpc-daemon (trusted)6464worker2 # [ 192.595344] nix-daemon[860]: reaped child process 1776, status = succeeded6465lb # [ 192.617175] niks3-server[1034]: 2026/09/21 13:24:54 INFO Received uploads request method=POST path=/api/pending_closures6466worker2 # [ 192.610766] nix-grpc-daemon[1549]: time=2026-09-21T13:24:54.232Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6467worker2 # [ 192.612370] nix-grpc-daemon[1549]: time=2026-09-21T13:24:54.233Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)"6468lb # [ 192.660215] niks3-server[1034]: 2026/09/21 13:24:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6469worker2 # [ 192.651331] nix-grpc-daemon[1549]: time=2026-09-21T13:24:54.273Z level=INFO msg="Uploading 1 narinfos"6470lb # [ 192.674181] niks3-server[1034]: 2026/09/21 13:24:54 INFO Signed narinfos id=64 count=16471lb # [ 192.680236] niks3-server[1034]: 2026/09/21 13:24:54 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls6472lb # [ 192.687336] niks3-server[1034]: 2026/09/21 13:24:54 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst6473lb # [ 192.699109] niks3-server[1034]: 2026/09/21 13:24:54 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6474lb # [ 192.704230] niks3-server[1034]: 2026/09/21 13:24:54 INFO Completed upload id=646475worker2 # [ 192.686652] nix-grpc-daemon[1549]: time=2026-09-21T13:24:54.309Z level=INFO msg="Upload complete. (376ms)"6476lb # [ 192.709426] niks3-server[1034]: 2026/09/21 13:24:54 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo6477worker2 # [ 192.696784] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41778 duration_s=0 paths=1 nar_bytes_in=5126478worker2 # [ 192.711461] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417786479worker2 # [ 192.718282] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)6480worker2 # [ 192.723286] nix-daemon[860]: reaped child process 1766, status = succeeded6481client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051...6482client # farm-a> worker2: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6483worker2 # [ 192.737389] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)6484client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6485client # farm-a> LOG-farm-a-drain6486worker2 # [ 193.848278] nix-daemon[860]: reaped child process 1780, status = succeeded6487worker2 # [ 193.919176] nix-daemon[860]: accepted connection from pid 1796, user nix-grpc-daemon (trusted)6488worker2 # [ 193.938611] nix-daemon[860]: reaped child process 1799, status = succeeded6489lb # [ 193.963051] niks3-server[1034]: 2026/09/21 13:24:55 INFO Received uploads request method=POST path=/api/pending_closures6490worker2 # [ 193.958875] nix-grpc-daemon[1549]: time=2026-09-21T13:24:55.580Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6491worker2 # [ 193.963719] nix-grpc-daemon[1549]: time=2026-09-21T13:24:55.580Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)"6492lb # [ 194.007451] niks3-server[1034]: 2026/09/21 13:24:55 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst6493worker2 # [ 194.004720] nix-grpc-daemon[1549]: time=2026-09-21T13:24:55.627Z level=INFO msg="Uploading 1 narinfos"6494lb # [ 194.021676] niks3-server[1034]: 2026/09/21 13:24:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6495lb # [ 194.030965] niks3-server[1034]: 2026/09/21 13:24:55 INFO Signed narinfos id=65 count=16496lb # [ 194.037252] niks3-server[1034]: 2026/09/21 13:24:55 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls6497worker2 # [ 194.032554] nix-grpc-daemon[1549]: time=2026-09-21T13:24:55.655Z level=INFO msg="Upload complete. (282ms)"6498lb # [ 194.050258] niks3-server[1034]: 2026/09/21 13:24:55 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv6499lb # [ 194.057671] niks3-server[1034]: 2026/09/21 13:24:55 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6500lb # [ 194.061891] niks3-server[1034]: 2026/09/21 13:24:55 INFO Completed upload id=656501lb # [ 194.067487] niks3-server[1034]: 2026/09/21 13:24:55 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo6502worker2 # [ 194.050185] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41778 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=15 outputs=16503worker2 # [ 194.068509] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:55Z level=debug event=want client=7 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=06504worker2 # [ 194.074009] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:55Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=16505worker2 # [ 194.079090] nix-daemon[860]: reaped child process 1778, status = succeeded6506worker2 # [ 194.095594] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417786507worker2 # [ 194.118636] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417786508worker2 # [ 194.134622] nix-grpc-daemon[1549]: time=2026-09-21T13:24:55.757Z level=INFO msg="All 1 paths already cached"6509client # copying 1 paths to 'lb:50051'...6510client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'...6511worker2 # [ 194.146098] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417786512worker2 # [ 194.160118] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)6513worker2 # [ 194.312546] nix-daemon[860]: accepted connection from pid 1805, user nix-grpc-daemon (trusted)6514worker2 # [ 194.331167] nix-daemon[860]: reaped child process 1808, status = succeeded6515worker2 # [ 194.435626] nix-daemon[860]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)6516worker2 # [ 194.453557] nix-daemon[860]: reaped child process 1813, status = succeeded6517lb # [ 194.475772] niks3-server[1034]: 2026/09/21 13:24:56 INFO Received uploads request method=POST path=/api/pending_closures6518worker2 # [ 194.468086] nix-grpc-daemon[1549]: time=2026-09-21T13:24:56.090Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6519worker2 # [ 194.469686] nix-grpc-daemon[1549]: time=2026-09-21T13:24:56.090Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)"6520lb # [ 194.507721] niks3-server[1034]: 2026/09/21 13:24:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign6521worker2 # [ 194.497127] nix-grpc-daemon[1549]: time=2026-09-21T13:24:56.119Z level=INFO msg="Uploading 1 narinfos"6522lb # [ 194.523373] niks3-server[1034]: 2026/09/21 13:24:56 INFO Signed narinfos id=66 count=16523lb # [ 194.531146] niks3-server[1034]: 2026/09/21 13:24:56 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls6524lb # [ 194.544545] niks3-server[1034]: 2026/09/21 13:24:56 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst6525worker2 # [ 194.534026] nix-grpc-daemon[1549]: time=2026-09-21T13:24:56.156Z level=INFO msg="Upload complete. (357ms)"6526lb # [ 194.554110] niks3-server[1034]: 2026/09/21 13:24:56 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete6527lb # [ 194.559368] niks3-server[1034]: 2026/09/21 13:24:56 INFO Completed upload id=666528worker2 # [ 194.544311] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:56Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41778 duration_s=0 paths=1 nar_bytes_in=6326529lb # [ 194.563565] niks3-server[1034]: 2026/09/21 13:24:56 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo6530worker2 # [ 194.555557] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417786531worker2 # [ 194.562627] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)6532worker2 # [ 194.567695] nix-daemon[860]: reaped child process 1803, status = succeeded6533client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051...6534client # farm-b> worker2: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6535worker2 # [ 194.581753] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)6536client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6537client # farm-b> LOG-farm-b-drain6538worker2 # [ 196.018566] nix-daemon[860]: reaped child process 1817, status = succeeded6539worker2 # [ 196.134458] nix-daemon[860]: accepted connection from pid 1833, user nix-grpc-daemon (trusted)6540worker2 # [ 196.155206] nix-daemon[860]: reaped child process 1836, status = succeeded6541lb # [ 196.177401] niks3-server[1034]: 2026/09/21 13:24:57 INFO Received uploads request method=POST path=/api/pending_closures6542worker2 # [ 196.171100] nix-grpc-daemon[1549]: time=2026-09-21T13:24:57.793Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6543worker2 # [ 196.175942] nix-grpc-daemon[1549]: time=2026-09-21T13:24:57.795Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)"6544lb # [ 196.225153] niks3-server[1034]: 2026/09/21 13:24:57 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst6545lb # [ 196.237353] niks3-server[1034]: 2026/09/21 13:24:57 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv6546worker2 # [ 196.230311] nix-grpc-daemon[1549]: time=2026-09-21T13:24:57.852Z level=INFO msg="Uploading 1 narinfos"6547lb # [ 196.247593] niks3-server[1034]: 2026/09/21 13:24:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign6548lb # [ 196.256695] niks3-server[1034]: 2026/09/21 13:24:57 INFO Signed narinfos id=67 count=16549lb # [ 196.261989] niks3-server[1034]: 2026/09/21 13:24:57 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls6550worker2 # [ 196.252644] nix-grpc-daemon[1549]: time=2026-09-21T13:24:57.875Z level=INFO msg="Upload complete. (259ms)"6551lb # [ 196.272384] niks3-server[1034]: 2026/09/21 13:24:57 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete6552lb # [ 196.278756] niks3-server[1034]: 2026/09/21 13:24:57 INFO Completed upload id=676553lb # [ 196.284122] niks3-server[1034]: 2026/09/21 13:24:57 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo6554worker2 # [ 196.271733] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41778 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=16 outputs=16555worker2 # [ 196.290599] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:57Z level=debug event=want client=7 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=06556worker2 # [ 196.295971] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:57Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=16557worker2 # [ 196.300989] nix-daemon[860]: reaped child process 1815, status = succeeded6558worker2 # [ 196.336084] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417786559worker2 # [ 196.352978] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:417786560worker2 # [ 196.368748] nix-grpc-daemon[1549]: time=2026-09-21T13:24:57.991Z level=INFO msg="All 2 paths already cached"6561client # copying 1 paths to 'lb:50051'...6562client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'...6563worker2 # [ 196.380197] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:417786564worker2 # [ 196.391871] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)6565worker2 # [ 196.552818] nix-daemon[860]: accepted connection from pid 1842, user nix-grpc-daemon (trusted)6566worker2 # [ 196.579067] nix-daemon[860]: reaped child process 1845, status = succeeded6567worker2 # [ 196.681742] nix-daemon[860]: accepted connection from pid 1847, user nix-grpc-daemon (trusted)6568worker2 # [ 196.699931] nix-daemon[860]: reaped child process 1850, status = succeeded6569lb # [ 196.722921] niks3-server[1034]: 2026/09/21 13:24:58 INFO Received uploads request method=POST path=/api/pending_closures6570worker2 # [ 196.715035] nix-grpc-daemon[1549]: time=2026-09-21T13:24:58.337Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6571worker2 # [ 196.716721] nix-grpc-daemon[1549]: time=2026-09-21T13:24:58.337Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)"6572lb # [ 196.751334] niks3-server[1034]: 2026/09/21 13:24:58 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst6573lb # [ 196.768394] niks3-server[1034]: 2026/09/21 13:24:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign6574worker2 # [ 196.757765] nix-grpc-daemon[1549]: time=2026-09-21T13:24:58.380Z level=INFO msg="Uploading 1 narinfos"6575lb # [ 196.780812] niks3-server[1034]: 2026/09/21 13:24:58 INFO Signed narinfos id=68 count=16576lb # [ 196.786538] niks3-server[1034]: 2026/09/21 13:24:58 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls6577lb # [ 196.795962] niks3-server[1034]: 2026/09/21 13:24:58 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete6578lb # [ 196.803194] niks3-server[1034]: 2026/09/21 13:24:58 INFO Completed upload id=686579worker2 # [ 196.786221] nix-grpc-daemon[1549]: time=2026-09-21T13:24:58.408Z level=INFO msg="Upload complete. (372ms)"6580lb # [ 196.805940] niks3-server[1034]: 2026/09/21 13:24:58 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo6581worker2 # [ 196.796761] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41778 duration_s=0 paths=1 nar_bytes_in=7606582worker2 # [ 196.811511] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:417786583worker2 # [ 196.817369] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)6584worker2 # [ 196.822613] nix-daemon[860]: reaped child process 1840, status = succeeded6585worker2 # [ 196.833503] nix-daemon[860]: accepted connection from pid 1471, user nix-grpc-daemon (trusted)6586client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051...6587client # farm-top> worker2: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6588client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6589client # farm-top> LOG-farm-top-drain6590worker2 # [ 198.103619] nix-daemon[860]: reaped child process 1854, status = succeeded6591worker2 # [ 198.236813] nix-daemon[860]: accepted connection from pid 1870, user nix-grpc-daemon (trusted)6592worker2 # [ 198.256928] nix-daemon[860]: reaped child process 1873, status = succeeded6593lb # [ 198.281960] niks3-server[1034]: 2026/09/21 13:24:59 INFO Received uploads request method=POST path=/api/pending_closures6594worker2 # [ 198.275919] nix-grpc-daemon[1549]: time=2026-09-21T13:24:59.897Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6595worker2 # [ 198.278227] nix-grpc-daemon[1549]: time=2026-09-21T13:24:59.897Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)"6596lb # [ 198.321023] niks3-server[1034]: 2026/09/21 13:24:59 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv6597lb # [ 198.336240] niks3-server[1034]: 2026/09/21 13:24:59 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst6598lb # [ 198.350328] niks3-server[1034]: 2026/09/21 13:24:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign6599worker2 # [ 198.333310] nix-grpc-daemon[1549]: time=2026-09-21T13:24:59.955Z level=INFO msg="Uploading 1 narinfos"6600lb # [ 198.355892] niks3-server[1034]: 2026/09/21 13:24:59 INFO Signed narinfos id=69 count=16601lb # [ 198.362642] niks3-server[1034]: 2026/09/21 13:24:59 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls6602lb # [ 198.371047] niks3-server[1034]: 2026/09/21 13:24:59 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete6603lb # [ 198.377921] niks3-server[1034]: 2026/09/21 13:25:00 INFO Completed upload id=696604worker2 # [ 198.358524] nix-grpc-daemon[1549]: time=2026-09-21T13:24:59.981Z level=INFO msg="Upload complete. (305ms)"6605lb # [ 198.382742] niks3-server[1034]: 2026/09/21 13:25:00 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo6606worker2 # [ 198.373290] nix-grpc-daemon[1471]: ts=2026-09-21T13:24:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41778 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=17 outputs=16607worker2 # [ 198.393061] nix-daemon[860]: reaped child process 1852, status = succeeded6608client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2, in 6.43 seconds)6609client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath6610client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath, in 0.19 seconds)6611lb # [ 198.665713] envoy[897]: conn peer=[2001:db8:1::1]:35072 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=897 tx=499 ms=300676612lb # [ 198.670896] envoy[897]: conn peer=[2001:db8:1::1]:55122 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9549 tx=4114 ms=61846613worker1: must succeed: rm /nix/.rw-store/fill6614worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.17 seconds)6615worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'6616worker1 # [ 199.063082] nix-grpc-daemon[1565]: ts=2026-09-21T13:25:00Z level=info event=healthy reason=min_free6617worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.13 seconds)6618(finished: subtest: low disk drains a worker and builds go to the other, in 9.24 seconds)6619(finished: run the VM test script, in 200.89 seconds)6620test script finished in 201.14s6621cleanup6622kill QemuMachine (pid 50)6623client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6624client # [2026-09-21T13:25:02Z INFO virtiofsd] Client disconnected, shutting down6625client # [2026-09-21T13:25:02Z INFO virtiofsd] Client disconnected, shutting down6626client # [2026-09-21T13:25:02Z INFO virtiofsd] Client disconnected, shutting down6627kill QemuMachine (pid 49)6628lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6629lb # [2026-09-21T13:25:02Z INFO virtiofsd] Client disconnected, shutting down6630lb # [2026-09-21T13:25:02Z INFO virtiofsd] Client disconnected, shutting down6631lb # [2026-09-21T13:25:02Z INFO virtiofsd] Client disconnected, shutting down6632kill QemuMachine (pid 51)6633worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6634worker1 # [2026-09-21T13:25:02Z INFO virtiofsd] Client disconnected, shutting down6635worker1 # [2026-09-21T13:25:02Z INFO virtiofsd] Client disconnected, shutting down6636worker1 # [2026-09-21T13:25:02Z INFO virtiofsd] Client disconnected, shutting down6637kill QemuMachine (pid 52)6638worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)6639worker2 # [2026-09-21T13:25:03Z INFO virtiofsd] Client disconnected, shutting down6640worker2 # [2026-09-21T13:25:03Z INFO virtiofsd] Client disconnected, shutting down6641worker2 # [2026-09-21T13:25:03Z INFO virtiofsd] Client disconnected, shutting down6642(finished: cleanup, in 1.08 seconds)6643additionally exposed symbols:6644 client, lb, worker1, worker2,6645 vlan1,6646 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh