vm-test-run-nix-grpc-farm
checks.aarch64-linux.farm
· build #513
· raw
1tribuchet: building on eliza2Machine state will be reset. To keep it, pass --keep-machine-state3start all VLans4(finished: start all VLans, in 0.00 seconds)5Test will time out and terminate in 900.0 seconds6run the VM test script7start all VMs8client: starting vm9lb: starting vm10worker1: starting vm11worker2: starting vm12worker2 # Disk image does not exist, creating the virtualisation disk image...13worker2: QEMU running (pid 51)14worker2 # Formatting '/build/vm-state-worker2/tmp.tyDu2ijOG0', fmt=raw size=107374182415lb: QEMU running (pid 50)16client # Disk image does not exist, creating the virtualisation disk image...17worker2 # mke2fs 1.47.4 (6-Mar-2025)18lb # Disk image does not exist, creating the virtualisation disk image...19client: QEMU running (pid 49)20worker1 # Disk image does not exist, creating the virtualisation disk image...21worker1: QEMU running (pid 52)22client # Formatting '/build/vm-state-client/tmp.ab6azs0AGm', fmt=raw size=107374182423worker1 # Formatting '/build/vm-state-worker1/tmp.SJuop7CSUX', fmt=raw size=107374182424worker2 # Discarding device blocks: 0/262144 done25lb # Formatting '/build/vm-state-lb/tmp.FWQ74XQFjC', fmt=raw size=107374182426lb # mke2fs 1.47.4 (6-Mar-2025)27client # mke2fs 1.47.4 (6-Mar-2025)28(finished: start all VMs, in 0.60 seconds)29worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes30client: waiting for unit network-addresses-eth1.service31client # Discarding device blocks: 0/262144 done32worker2 # Filesystem UUID: 40ecc9c7-9fd9-4070-b4d7-d1545491025733client: waiting for the VM to finish booting34client # Creating filesystem with 262144 4k blocks and 65536 inodes35worker2 # Superblock backups stored on blocks:36lb # Discarding device blocks: 0/262144 done37worker1 # mke2fs 1.47.4 (6-Mar-2025)38client # Filesystem UUID: abf982be-913c-4750-bc0e-a2ce8cd2e84039worker2 # 32768, 98304, 163840, 22937640lb # Creating filesystem with 262144 4k blocks and 65536 inodes41worker1 # Discarding device blocks: 0/262144 done42client # Superblock backups stored on blocks:43worker2 # 44lb # Filesystem UUID: 446074de-18b4-468d-886e-480134b8b75c45worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes46client # 32768, 98304, 163840, 22937647worker2 # Allocating group tables: 0/8 done48lb # Superblock backups stored on blocks:49worker1 # Filesystem UUID: 21758fb7-bf7e-44b7-a633-61a5ce34fe3550client # 51worker2 # Writing inode tables: 0/8 done52lb # 32768, 98304, 163840, 22937653worker1 # Superblock backups stored on blocks:54client # Allocating group tables: 0/8 done55worker2 # Creating journal (8192 blocks): done56lb # 57worker1 # 32768, 98304, 163840, 22937658client # Writing inode tables: 0/8 done59worker2 # Writing superblocks and filesystem accounting information: 0/8 done60lb # Allocating group tables: 0/8 done61worker1 # 62client # Creating journal (8192 blocks): done63worker2 # 64lb # Writing inode tables: 0/8 done65worker1 # Allocating group tables: 0/8 done66client # Writing superblocks and filesystem accounting information: 0/8 done67worker2 # Virtualisation disk image created.68lb # Creating journal (8192 blocks): done69worker1 # Writing inode tables: 0/8 done70client # 71worker2 # Starting virtiofs daemons...72lb # Writing superblocks and filesystem accounting information: 0/8 done73worker1 # Creating journal (8192 blocks): done74client # Virtualisation disk image created.75worker2 # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)76lb # 77lb # Virtualisation disk image created.78client # Starting virtiofs daemons...79worker2 # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether80worker1 # Writing superblocks and filesystem accounting information: 0/8 done81lb # Starting virtiofs daemons...82client # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)83worker2 # [2026-09-23T04:42:49Z INFO virtiofsd] Waiting for vhost-user socket connection...84worker1 # 85lb # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)86client # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether87worker2 # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)88worker1 # Virtualisation disk image created.89lb # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether90client # [2026-09-23T04:42:49Z INFO virtiofsd] Waiting for vhost-user socket connection...91worker2 # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether92worker1 # Starting virtiofs daemons...93worker1 # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)94worker2 # [2026-09-23T04:42:49Z INFO virtiofsd] Waiting for vhost-user socket connection...95lb # [2026-09-23T04:42:49Z INFO virtiofsd] Waiting for vhost-user socket connection...96worker1 # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether97client # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)98worker2 # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)99worker1 # [2026-09-23T04:42:49Z INFO virtiofsd] Waiting for vhost-user socket connection...100lb # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)101client # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether102worker2 # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether103worker1 # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)104lb # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether105client # [2026-09-23T04:42:49Z INFO virtiofsd] Waiting for vhost-user socket connection...106worker2 # [2026-09-23T04:42:49Z INFO virtiofsd] Waiting for vhost-user socket connection...107worker1 # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether108lb # [2026-09-23T04:42:49Z INFO virtiofsd] Waiting for vhost-user socket connection...109client # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)110worker2 # [2026-09-23T04:42:49Z INFO virtiofsd] Client connected, servicing requests111worker1 # [2026-09-23T04:42:49Z INFO virtiofsd] Waiting for vhost-user socket connection...112lb # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)113client # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether114worker2 # [2026-09-23T04:42:49Z INFO virtiofsd] Client connected, servicing requests115worker1 # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1)116lb # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether117client # [2026-09-23T04:42:49Z INFO virtiofsd] Waiting for vhost-user socket connection...118worker2 # [2026-09-23T04:42:49Z INFO virtiofsd] Client connected, servicing requests119worker1 # [2026-09-23T04:42:49Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether120lb # [2026-09-23T04:42:49Z INFO virtiofsd] Waiting for vhost-user socket connection...121worker1 # [2026-09-23T04:42:49Z INFO virtiofsd] Waiting for vhost-user socket connection...122lb # [2026-09-23T04:42:49Z INFO virtiofsd] Client connected, servicing requests123worker1 # [2026-09-23T04:42:49Z INFO virtiofsd] Client connected, servicing requests124lb # [2026-09-23T04:42:49Z INFO virtiofsd] Client connected, servicing requests125worker1 # [2026-09-23T04:42:49Z INFO virtiofsd] Client connected, servicing requests126lb # [2026-09-23T04:42:49Z INFO virtiofsd] Client connected, servicing requests127worker1 # [2026-09-23T04:42:49Z INFO virtiofsd] Client connected, servicing requests128client # [2026-09-23T04:42:49Z INFO virtiofsd] Client connected, servicing requests129client # [2026-09-23T04:42:49Z INFO virtiofsd] Client connected, servicing requests130client # [2026-09-23T04:42:49Z INFO virtiofsd] Client connected, servicing requests131worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]132worker2 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026133worker2 # [ 0.000000] KASLR enabled134worker2 # [ 0.000000] random: crng init done135worker2 # [ 0.000000] Machine model: linux,dummy-virt136worker2 # [ 0.000000] efi: UEFI not found.137worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT138worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]139worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]140worker2 # [ 0.000000] Zone ranges:141worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]142worker2 # [ 0.000000] DMA32 empty143worker2 # [ 0.000000] Normal empty144worker2 # [ 0.000000] Device empty145worker2 # [ 0.000000] Movable zone start for each node146worker2 # [ 0.000000] Early memory node ranges147worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]148worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]149worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000150worker2 # [ 0.000000] psci: probing for conduit method from DT.151worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware.152worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs153worker2 # [ 0.000000] psci: Trusted OS migration not required154worker2 # [ 0.000000] psci: SMC Calling Convention v1.1155worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)156worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296157worker2 # [ 0.000000] Detected PIPT I-cache on CPU0158worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)159worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface160worker2 # [ 0.000000] CPU features: detected: Spectre-v4161worker2 # [ 0.000000] CPU features: detected: Spectre-BHB162worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38163worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23164worker2 # [ 0.000000] alternatives: applying boot alternatives165worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/pqhzhyfwqnllrlnwlcjj16jzjgklgsi9-nixos-system-worker2-test/init regInfo=/nix/store/wpdxdy4v7zk476k55q7hr105szmpb4hx-closure-info/registration console=ttyAMA0,115200n8 console=tty0166worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/wpdxdy4v7zk476k55q7hr105szmpb4hx-closure-info/registration", will be passed to user space.167worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes168worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)169worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)170worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB171worker2 # [ 0.000000] software IO TLB: area num 1.172worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB173worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)174worker2 # [ 0.000000] Fallback order for Node 0: 0175worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216176worker2 # [ 0.000000] Policy zone: DMA177worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off178worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1179worker2 # [ 0.000000] allocated 3145728 bytes of page_ext180worker2 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages181worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups182worker2 # [ 0.000000] rcu: Hierarchical RCU implementation.183worker2 # [ 0.000000] rcu: RCU event tracing is enabled.184worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.185worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled.186worker2 # [ 0.000000] Rude variant of Tasks RCU enabled.187worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled.188worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.189worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1190worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.191worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.192worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.193worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0194worker2 # [ 0.000000] GICv3: 256 SPIs implemented195worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented196worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq197worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI198worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0199worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000200worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]201worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1)202worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1)203worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a00000204worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a10000205worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.206worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns207worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).208worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns209lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]210worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns211worker2 # [ 0.000036] arm-pv: using stolen time PV212lb # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026213worker2 # [ 0.000704] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)214lb # [ 0.000000] KASLR enabled215worker2 # [ 0.000911] Console: colour dummy device 80x25216lb # [ 0.000000] random: crng init done217lb # [ 0.000000] Machine model: linux,dummy-virt218worker2 # [ 0.000919] printk: legacy console [tty0] enabled219lb # [ 0.000000] efi: UEFI not found.220lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT221worker2 # [ 0.001149] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)222worker2 # [ 0.001156] pid_max: default: 32768 minimum: 301223lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]224worker2 # [ 0.001240] LSM: initializing lsm=capability,landlock,yama,bpf,ima225lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]226worker2 # [ 0.001417] landlock: Up and running.227lb # [ 0.000000] Zone ranges:228worker2 # [ 0.001420] Yama: becoming mindful.229lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]230worker2 # [ 0.002112] LSM support for eBPF active231lb # [ 0.000000] DMA32 empty232lb # [ 0.000000] Normal empty233worker2 # [ 0.002305] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)234lb # [ 0.000000] Device empty235lb # [ 0.000000] Movable zone start for each node236worker2 # [ 0.002339] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)237lb # [ 0.000000] Early memory node ranges238worker2 # [ 0.004398] rcu: Hierarchical SRCU implementation.239lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]240worker2 # [ 0.004405] rcu: Max phase no-delay instances is 1000.241lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]242worker2 # [ 0.005748] fsl-mc MSI: its@8080000 domain created243worker2 # [ 0.005843] EFI services will not be available.244lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000245worker2 # [ 0.005929] smp: Bringing up secondary CPUs ...246lb # [ 0.000000] psci: probing for conduit method from DT.247worker2 # [ 0.005938] smp: Brought up 1 node, 1 CPU248lb # [ 0.000000] psci: PSCIv1.3 detected in firmware.249worker2 # [ 0.005941] SMP: Total of 1 processors activated.250lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs251worker2 # [ 0.005947] CPU: All CPU(s) started at EL1252lb # [ 0.000000] psci: Trusted OS migration not required253worker2 # [ 0.005959] CPU features: detected: Branch Target Identification254lb # [ 0.000000] psci: SMC Calling Convention v1.1255worker2 # [ 0.005963] CPU features: detected: ARMv8.4 Translation Table Level256lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)257worker2 # [ 0.005967] CPU features: detected: Instruction cache invalidation not required for I/D coherence258lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296259lb # [ 0.000000] Detected PIPT I-cache on CPU0260worker2 # [ 0.005971] CPU features: detected: Data cache clean to the PoU not required for I/D coherence261lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)262worker2 # [ 0.005975] CPU features: detected: Common not Private translations263lb # [ 0.000000] CPU features: detected: GICv3 CPU interface264worker2 # [ 0.005978] CPU features: detected: CRC32 instructions265lb # [ 0.000000] CPU features: detected: Spectre-v4266client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]267worker2 # [ 0.005981] CPU features: detected: Data cache clean to Point of Deep Persistence268lb # [ 0.000000] CPU features: detected: Spectre-BHB269lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38270worker2 # [ 0.005984] CPU features: detected: Data cache clean to Point of Persistence271client # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026272client # [ 0.000000] KASLR enabled273lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23274worker2 # [ 0.005987] CPU features: detected: Data independent timing control (DIT)275client # [ 0.000000] random: crng init done276lb # [ 0.000000] alternatives: applying boot alternatives277worker2 # [ 0.005990] CPU features: detected: E0PD278client # [ 0.000000] Machine model: linux,dummy-virt279client # [ 0.000000] efi: UEFI not found.280worker2 # [ 0.005993] CPU features: detected: Enhanced Counter Virtualization281client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT282worker2 # [ 0.005996] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)283client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]284worker2 # [ 0.005999] CPU features: detected: Enhanced Virtualization Traps285worker2 # [ 0.006002] CPU features: detected: Fine Grained Traps286client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]287client # [ 0.000000] Zone ranges:288lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/kgjzg78npi7z07am2szqxkr41f06jp5w-nixos-system-lb-test/init regInfo=/nix/store/lcw0xq38k7svc2ifmp8wza1s56mbyq61-closure-info/registration console=ttyAMA0,115200n8 console=tty0289worker2 # [ 0.006006] CPU features: detected: Generic authentication (architected QARMA5 algorithm)290client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]291client # [ 0.000000] DMA32 empty292worker2 # [ 0.006011] CPU features: detected: RCpc load-acquire (LDAPR)293client # [ 0.000000] Normal empty294client # [ 0.000000] Device empty295worker2 # [ 0.006014] CPU features: detected: LSE atomic instructions296lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/lcw0xq38k7svc2ifmp8wza1s56mbyq61-closure-info/registration", will be passed to user space.297client # [ 0.000000] Movable zone start for each node298client # [ 0.000000] Early memory node ranges299lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes300worker2 # [ 0.006017] CPU features: detected: Privileged Access Never301client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]302lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)303worker2 # [ 0.006020] CPU features: detected: PMUv3304client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]305worker2 # [ 0.006022] CPU features: detected: RAS Extension Support306lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)307client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000308worker2 # [ 0.006025] CPU features: detected: RASv1p1 Extension Support309lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB310client # [ 0.000000] psci: probing for conduit method from DT.311worker2 # [ 0.006028] CPU features: detected: Random Number Generator312lb # [ 0.000000] software IO TLB: area num 1.313client # [ 0.000000] psci: PSCIv1.3 detected in firmware.314worker2 # [ 0.006030] CPU features: detected: Speculation barrier (SB)315client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs316lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB317client # [ 0.000000] psci: Trusted OS migration not required318worker2 # [ 0.006033] CPU features: detected: Stage-2 Force Write-Back319lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)320client # [ 0.000000] psci: SMC Calling Convention v1.1321worker2 # [ 0.006036] CPU features: detected: TLB range maintenance instructions322lb # [ 0.000000] Fallback order for Node 0: 0323client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)324worker2 # [ 0.006041] CPU features: detected: Speculative Store Bypassing Safe (SSBS)325lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216326lb # [ 0.000000] Policy zone: DMA327worker2 # [ 0.006091] alternatives: applying system-wide alternatives328client # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296329lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off330client # [ 0.000000] Detected PIPT I-cache on CPU0331worker2 # [ 0.009175] CPU features: detected: BBM Level 2 without TLB conflict abort332lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1333client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)334lb # [ 0.000000] allocated 3145728 bytes of page_ext335client # [ 0.000000] CPU features: detected: GICv3 CPU interface336lb # [ 0.000000] ftrace: allocating 74950 entries in 294 pages337worker2 # [ 0.009404] Memory: 1405688K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124632K reserved, 32768K cma-reserved)338client # [ 0.000000] CPU features: detected: Spectre-v4339lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups340worker2 # [ 0.009871] devtmpfs: initialized341client # [ 0.000000] CPU features: detected: Spectre-BHB342lb # [ 0.000000] rcu: Hierarchical RCU implementation.343worker2 # [ 0.011832] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)344client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38345lb # [ 0.000000] rcu: RCU event tracing is enabled.346client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23347worker2 # [ 0.011856] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).348lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.349client # [ 0.000000] alternatives: applying boot alternatives350lb # [ 0.000000] Trampoline variant of Tasks RCU enabled.351worker2 # [ 0.012043] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL352lb # [ 0.000000] Rude variant of Tasks RCU enabled.353worker2 # [ 0.012048] 0 pages in range for non-PLT usage354lb # [ 0.000000] Tracing variant of Tasks RCU enabled.355worker2 # [ 0.012049] 508272 pages in range for PLT usage356worker2 # [ 0.012177] pinctrl core: initialized pinctrl subsystem357lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.358worker2 # [ 0.012951] DMI not present or invalid.359lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1360worker2 # [ 0.016931] NET: Registered PF_NETLINK/PF_ROUTE protocol family361client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/vvnvhrfyszz76jc10b9zglzsybdh2zp5-nixos-system-client-test/init regInfo=/nix/store/2z9knvlsizfqhy7lglh3byxddflyimf2-closure-info/registration console=ttyAMA0,115200n8 console=tty0362lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.363worker2 # [ 0.019298] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations364worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40]365worker2 # [ 0.019507] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations366lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.367client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/2z9knvlsizfqhy7lglh3byxddflyimf2-closure-info/registration", will be passed to user space.368worker2 # [ 0.019723] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations369lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.370client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes371worker1 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026372worker2 # [ 0.019750] audit: initializing netlink subsys (disabled)373worker1 # [ 0.000000] KASLR enabled374lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0375lb # [ 0.000000] GICv3: 256 SPIs implemented376worker1 # [ 0.000000] random: crng init done377client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)378worker2 # [ 0.020377] thermal_sys: Registered thermal governor 'fair_share'379worker1 # [ 0.000000] Machine model: linux,dummy-virt380lb # [ 0.000000] GICv3: 0 Extended SPIs implemented381worker2 # [ 0.020379] thermal_sys: Registered thermal governor 'bang_bang'382worker1 # [ 0.000000] efi: UEFI not found.383client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)384lb # [ 0.000000] Root IRQ handler: gic_handle_irq385worker2 # [ 0.020383] thermal_sys: Registered thermal governor 'step_wise'386lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI387client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB388worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT389lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0390client # [ 0.000000] software IO TLB: area num 1.391worker2 # [ 0.020386] thermal_sys: Registered thermal governor 'user_space'392worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff]393lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000394client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB395worker2 # [ 0.020391] thermal_sys: Registered thermal governor 'power_allocator'396worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f]397lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]398worker1 # [ 0.000000] Zone ranges:399client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)400worker2 # [ 0.020426] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1401client # [ 0.000000] Fallback order for Node 0: 0402worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff]403worker2 # [ 0.020435] cpuidle: using governor ladder404lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1)405worker1 # [ 0.000000] DMA32 empty406worker2 # [ 0.020441] cpuidle: using governor menu407worker1 # [ 0.000000] Normal empty408client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216409worker1 # [ 0.000000] Device empty410client # [ 0.000000] Policy zone: DMA411worker2 # [ 0.020653] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.412lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1)413worker1 # [ 0.000000] Movable zone start for each node414client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off415worker2 # [ 0.020671] ASID allocator initialised with 65536 entries416worker1 # [ 0.000000] Early memory node ranges417lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a00000418worker2 # [ 0.021985] Serial: AMBA PL011 UART driver419client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1420worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff]421lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a10000422client # [ 0.000000] allocated 3145728 bytes of page_ext423worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff]424worker2 # [ 0.027548] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1425lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.426client # [ 0.000000] ftrace: allocating 74950 entries in 294 pages427worker2 # [ 0.027743] printk: console [ttyAMA0] enabled428worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000429client # [ 0.000000] ftrace: allocated 294 pages with 4 groups430worker1 # [ 0.000000] psci: probing for conduit method from DT.431lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns432client # [ 0.000000] rcu: Hierarchical RCU implementation.433worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware.434client # [ 0.000000] rcu: RCU event tracing is enabled.435lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).436worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs437client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.438worker1 # [ 0.000000] psci: Trusted OS migration not required439client # [ 0.000000] Trampoline variant of Tasks RCU enabled.440lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns441worker1 # [ 0.000000] psci: SMC Calling Convention v1.1442client # [ 0.000000] Rude variant of Tasks RCU enabled.443lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns444client # [ 0.000000] Tracing variant of Tasks RCU enabled.445worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003)446lb # [ 0.000035] arm-pv: using stolen time PV447worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296448client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.449worker1 # [ 0.000000] Detected PIPT I-cache on CPU0450lb # [ 0.000552] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)451client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1452lb # [ 0.000732] Console: colour dummy device 80x25453worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)454lb # [ 0.000740] printk: legacy console [tty0] enabled455client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.456worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface457worker1 # [ 0.000000] CPU features: detected: Spectre-v4458client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.459lb # [ 0.000963] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)460worker1 # [ 0.000000] CPU features: detected: Spectre-BHB461lb # [ 0.000970] pid_max: default: 32768 minimum: 301462worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38463client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.464worker2 # [ 0.151625] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages465lb # [ 0.001048] LSM: initializing lsm=capability,landlock,yama,bpf,ima466client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0467worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23468lb # [ 0.001209] landlock: Up and running.469worker2 # [ 0.151646] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page470client # [ 0.000000] GICv3: 256 SPIs implemented471lb # [ 0.001212] Yama: becoming mindful.472worker1 # [ 0.000000] alternatives: applying boot alternatives473worker2 # [ 0.151652] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages474lb # [ 0.001822] LSM support for eBPF active475client # [ 0.000000] GICv3: 0 Extended SPIs implemented476client # [ 0.000000] Root IRQ handler: gic_handle_irq477worker2 # [ 0.151656] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page478lb # [ 0.001977] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)479client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI480worker2 # [ 0.151660] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages481client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0482lb # [ 0.002018] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)483worker2 # [ 0.151665] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page484lb # [ 0.003986] rcu: Hierarchical SRCU implementation.485client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000486lb # [ 0.003993] rcu: Max phase no-delay instances is 1000.487client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]488worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/43arb2c1nvqnmh278pvp5s0pfn2v8h50-nixos-system-worker1-test/init regInfo=/nix/store/qbcjs6rnszmz4aiwyb3i6y07ims347sb-closure-info/registration console=ttyAMA0,115200n8 console=tty0489worker2 # [ 0.151669] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages490lb # [ 0.005314] fsl-mc MSI: its@8080000 domain created491worker2 # [ 0.151673] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page492client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)493lb # [ 0.005414] EFI services will not be available.494lb # [ 0.005504] smp: Bringing up secondary CPUs ...495worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/qbcjs6rnszmz4aiwyb3i6y07ims347sb-closure-info/registration", will be passed to user space.496lb # [ 0.005512] smp: Brought up 1 node, 1 CPU497client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)498lb # [ 0.005516] SMP: Total of 1 processors activated.499worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes500client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000501lb # [ 0.005523] CPU: All CPU(s) started at EL1502worker2 # [ 0.159149] fbcon: Taking over console503worker2 # [ 0.159167] ACPI: Interpreter disabled.504worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear)505client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000506lb # [ 0.005534] CPU features: detected: Branch Target Identification507lb # [ 0.005539] CPU features: detected: ARMv8.4 Translation Table Level508client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.509worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear)510worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB511lb # [ 0.005542] CPU features: detected: Instruction cache invalidation not required for I/D coherence512client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns513worker1 # [ 0.000000] software IO TLB: area num 1.514client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).515lb # [ 0.005546] CPU features: detected: Data cache clean to the PoU not required for I/D coherence516worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB517lb # [ 0.005550] CPU features: detected: Common not Private translations518worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB)519client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns520lb # [ 0.005553] CPU features: detected: CRC32 instructions521worker1 # [ 0.000000] Fallback order for Node 0: 0522client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns523lb # [ 0.005556] CPU features: detected: Data cache clean to Point of Deep Persistence524worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216525worker1 # [ 0.000000] Policy zone: DMA526client # [ 0.000035] arm-pv: using stolen time PV527lb # [ 0.005559] CPU features: detected: Data cache clean to Point of Persistence528worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off529client # [ 0.000475] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)530lb # [ 0.005563] CPU features: detected: Data independent timing control (DIT)531worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1532lb # [ 0.005566] CPU features: detected: E0PD533client # [ 0.000679] Console: colour dummy device 80x25534worker1 # [ 0.000000] allocated 3145728 bytes of page_ext535client # [ 0.000687] printk: legacy console [tty0] enabled536lb # [ 0.005568] CPU features: detected: Enhanced Counter Virtualization537worker2 # [ 0.167857] iommu: Default domain type: Translated538worker1 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages539lb # [ 0.005571] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)540worker2 # [ 0.167869] iommu: DMA domain TLB invalidation policy: strict mode541worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups542client # [ 0.000875] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)543worker2 # [ 0.168260] SCSI subsystem initialized544worker1 # [ 0.000000] rcu: Hierarchical RCU implementation.545lb # [ 0.005575] CPU features: detected: Enhanced Virtualization Traps546client # [ 0.000882] pid_max: default: 32768 minimum: 301547worker1 # [ 0.000000] rcu: RCU event tracing is enabled.548lb # [ 0.005578] CPU features: detected: Fine Grained Traps549client # [ 0.000962] LSM: initializing lsm=capability,landlock,yama,bpf,ima550worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1.551client # [ 0.001116] landlock: Up and running.552worker2 # [ 0.170233] usbcore: registered new interface driver usbfs553lb # [ 0.005581] CPU features: detected: Generic authentication (architected QARMA5 algorithm)554client # [ 0.001119] Yama: becoming mindful.555worker2 # [ 0.170273] usbcore: registered new interface driver hub556worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled.557client # [ 0.001593] LSM support for eBPF active558lb # [ 0.005586] CPU features: detected: RCpc load-acquire (LDAPR)559worker2 # [ 0.170301] usbcore: registered new device driver usb560worker1 # [ 0.000000] Rude variant of Tasks RCU enabled.561worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled.562lb # [ 0.005589] CPU features: detected: LSE atomic instructions563worker2 # [ 0.170582] pps_core: LinuxPPS API ver. 1 registered564client # [ 0.001739] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)565lb # [ 0.005592] CPU features: detected: Privileged Access Never566worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.567client # [ 0.001773] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)568lb # [ 0.005594] CPU features: detected: PMUv3569worker2 # [ 0.170588] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>570client # [ 0.003717] rcu: Hierarchical SRCU implementation.571worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1572worker2 # [ 0.170599] PTP clock support registered573lb # [ 0.005597] CPU features: detected: RAS Extension Support574worker2 # [ 0.170662] EDAC MC: Ver: 3.0.0575client # [ 0.003723] rcu: Max phase no-delay instances is 1000.576lb # [ 0.005600] CPU features: detected: RASv1p1 Extension Support577client # [ 0.005064] fsl-mc MSI: its@8080000 domain created578worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.579worker2 # [ 0.175284] scmi_core: SCMI protocol bus registered580lb # [ 0.005603] CPU features: detected: Random Number Generator581client # [ 0.005158] EFI services will not be available.582worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.583client # [ 0.005247] smp: Bringing up secondary CPUs ...584lb # [ 0.005605] CPU features: detected: Speculation barrier (SB)585worker2 # [ 0.176294] FPGA manager framework586client # [ 0.005255] smp: Brought up 1 node, 1 CPU587lb # [ 0.005608] CPU features: detected: Stage-2 Force Write-Back588worker2 # [ 0.177289] vgaarb: loaded589worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1.590client # [ 0.005258] SMP: Total of 1 processors activated.591lb # [ 0.005611] CPU features: detected: TLB range maintenance instructions592client # [ 0.005264] CPU: All CPU(s) started at EL1593worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0594worker2 # [ 0.177924] clocksource: Switched to clocksource arch_sys_counter595worker1 # [ 0.000000] GICv3: 256 SPIs implemented596lb # [ 0.005616] CPU features: detected: Speculative Store Bypassing Safe (SSBS)597client # [ 0.005274] CPU features: detected: Branch Target Identification598worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented599lb # [ 0.005671] alternatives: applying system-wide alternatives600client # [ 0.005279] CPU features: detected: ARMv8.4 Translation Table Level601worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq602lb # [ 0.008734] CPU features: detected: BBM Level 2 without TLB conflict abort603worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI604client # [ 0.005282] CPU features: detected: Instruction cache invalidation not required for I/D coherence605worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0606client # [ 0.005286] CPU features: detected: Data cache clean to the PoU not required for I/D coherence607worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000608lb # [ 0.008964] Memory: 1405716K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124636K reserved, 32768K cma-reserved)609worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff]610lb # [ 0.009373] devtmpfs: initialized611client # [ 0.005290] CPU features: detected: Common not Private translations612worker2 # [ 0.182536] VFS: Disk quotas dquot_6.6.0613client # [ 0.005293] CPU features: detected: CRC32 instructions614lb # [ 0.011259] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)615worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1)616worker2 # [ 0.182598] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)617client # [ 0.005296] CPU features: detected: Data cache clean to Point of Deep Persistence618lb # [ 0.011286] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).619worker2 # [ 0.184271] netfs: FS-Cache loaded620client # [ 0.005299] CPU features: detected: Data cache clean to Point of Persistence621worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1)622lb # [ 0.011476] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL623worker2 # [ 0.184406] pnp: PnP ACPI: disabled624worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000625lb # [ 0.011480] 0 pages in range for non-PLT usage626client # [ 0.005303] CPU features: detected: Data independent timing control (DIT)627client # [ 0.005306] CPU features: detected: E0PD628lb # [ 0.011482] 508272 pages in range for PLT usage629worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000630lb # [ 0.011597] pinctrl core: initialized pinctrl subsystem631client # [ 0.005309] CPU features: detected: Enhanced Counter Virtualization632worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.633lb # [ 0.012439] DMI not present or invalid.634client # [ 0.005312] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)635lb # [ 0.015857] NET: Registered PF_NETLINK/PF_ROUTE protocol family636worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns637client # [ 0.005315] CPU features: detected: Enhanced Virtualization Traps638lb # [ 0.018340] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations639worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt).640client # [ 0.005318] CPU features: detected: Fine Grained Traps641lb # [ 0.018549] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations642client # [ 0.005322] CPU features: detected: Generic authentication (architected QARMA5 algorithm)643worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns644worker2 # [ 0.188356] NET: Registered PF_INET protocol family645lb # [ 0.018766] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations646client # [ 0.005327] CPU features: detected: RCpc load-acquire (LDAPR)647lb # [ 0.018793] audit: initializing netlink subsys (disabled)648worker2 # [ 0.188633] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)649client # [ 0.005330] CPU features: detected: LSE atomic instructions650worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns651worker1 # [ 0.000032] arm-pv: using stolen time PV652lb # [ 0.019384] thermal_sys: Registered thermal governor 'fair_share'653client # [ 0.005333] CPU features: detected: Privileged Access Never654client # [ 0.005335] CPU features: detected: PMUv3655lb # [ 0.019386] thermal_sys: Registered thermal governor 'bang_bang'656worker1 # [ 0.000402] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____)657client # [ 0.005338] CPU features: detected: RAS Extension Support658lb # [ 0.019390] thermal_sys: Registered thermal governor 'step_wise'659worker1 # [ 0.000659] Console: colour dummy device 80x25660client # [ 0.005341] CPU features: detected: RASv1p1 Extension Support661worker1 # [ 0.000667] printk: legacy console [tty0] enabled662lb # [ 0.019393] thermal_sys: Registered thermal governor 'user_space'663client # [ 0.005343] CPU features: detected: Random Number Generator664lb # [ 0.019399] thermal_sys: Registered thermal governor 'power_allocator'665client # [ 0.005346] CPU features: detected: Speculation barrier (SB)666worker1 # [ 0.000865] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000)667client # [ 0.005349] CPU features: detected: Stage-2 Force Write-Back668lb # [ 0.019425] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1669worker1 # [ 0.000872] pid_max: default: 32768 minimum: 301670lb # [ 0.019434] cpuidle: using governor ladder671worker1 # [ 0.000948] LSM: initializing lsm=capability,landlock,yama,bpf,ima672client # [ 0.005351] CPU features: detected: TLB range maintenance instructions673lb # [ 0.019440] cpuidle: using governor menu674worker1 # [ 0.001084] landlock: Up and running.675client # [ 0.005356] CPU features: detected: Speculative Store Bypassing Safe (SSBS)676worker1 # [ 0.001088] Yama: becoming mindful.677lb # [ 0.019632] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.678worker1 # [ 0.001540] LSM support for eBPF active679client # [ 0.005396] alternatives: applying system-wide alternatives680lb # [ 0.019648] ASID allocator initialised with 65536 entries681lb # [ 0.020923] Serial: AMBA PL011 UART driver682worker1 # [ 0.001674] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)683client # [ 0.008395] CPU features: detected: BBM Level 2 without TLB conflict abort684worker1 # [ 0.001706] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)685lb # [ 0.026488] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1686lb # [ 0.026630] printk: console [ttyAMA0] enabled687worker1 # [ 0.003660] rcu: Hierarchical SRCU implementation.688client # [ 0.008606] Memory: 1405640K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124660K reserved, 32768K cma-reserved)689worker1 # [ 0.003666] rcu: Max phase no-delay instances is 1000.690client # [ 0.009020] devtmpfs: initialized691worker1 # [ 0.005022] fsl-mc MSI: its@8080000 domain created692client # [ 0.011213] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)693worker1 # [ 0.005116] EFI services will not be available.694worker1 # [ 0.005200] smp: Bringing up secondary CPUs ...695client # [ 0.011238] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).696worker1 # [ 0.005209] smp: Brought up 1 node, 1 CPU697client # [ 0.011421] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL698worker1 # [ 0.005213] SMP: Total of 1 processors activated.699worker1 # [ 0.005218] CPU: All CPU(s) started at EL1700client # [ 0.011425] 0 pages in range for non-PLT usage701client # [ 0.011426] 508272 pages in range for PLT usage702worker1 # [ 0.005230] CPU features: detected: Branch Target Identification703client # [ 0.011553] pinctrl core: initialized pinctrl subsystem704worker1 # [ 0.005235] CPU features: detected: ARMv8.4 Translation Table Level705client # [ 0.012520] DMI not present or invalid.706client # [ 0.017478] NET: Registered PF_NETLINK/PF_ROUTE protocol family707worker1 # [ 0.005238] CPU features: detected: Instruction cache invalidation not required for I/D coherence708client # [ 0.019899] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations709lb # [ 0.145290] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages710worker1 # [ 0.005242] CPU features: detected: Data cache clean to the PoU not required for I/D coherence711client # [ 0.020105] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations712lb # [ 0.145311] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page713worker1 # [ 0.005246] CPU features: detected: Common not Private translations714worker1 # [ 0.005250] CPU features: detected: CRC32 instructions715lb # [ 0.145318] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages716client # [ 0.020331] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations717lb # [ 0.145322] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page718client # [ 0.020354] audit: initializing netlink subsys (disabled)719worker1 # [ 0.005252] CPU features: detected: Data cache clean to Point of Deep Persistence720client # [ 0.021022] thermal_sys: Registered thermal governor 'fair_share'721lb # [ 0.145326] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages722worker1 # [ 0.005256] CPU features: detected: Data cache clean to Point of Persistence723client # [ 0.021024] thermal_sys: Registered thermal governor 'bang_bang'724lb # [ 0.145331] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page725worker1 # [ 0.005259] CPU features: detected: Data independent timing control (DIT)726client # [ 0.021027] thermal_sys: Registered thermal governor 'step_wise'727worker1 # [ 0.005262] CPU features: detected: E0PD728lb # [ 0.145335] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages729client # [ 0.021030] thermal_sys: Registered thermal governor 'user_space'730worker1 # [ 0.005265] CPU features: detected: Enhanced Counter Virtualization731lb # [ 0.145339] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page732worker1 # [ 0.005268] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF)733client # [ 0.021038] thermal_sys: Registered thermal governor 'power_allocator'734worker1 # [ 0.005271] CPU features: detected: Enhanced Virtualization Traps735worker1 # [ 0.005274] CPU features: detected: Fine Grained Traps736lb # [ 0.152731] fbcon: Taking over console737client # [ 0.021063] audit: type=2000 audit(0.020:1): state=initialized audit_enabled=0 res=1738lb # [ 0.152746] ACPI: Interpreter disabled.739client # [ 0.021075] cpuidle: using governor ladder740worker1 # [ 0.005278] CPU features: detected: Generic authentication (architected QARMA5 algorithm)741client # [ 0.021080] cpuidle: using governor menu742worker1 # [ 0.005282] CPU features: detected: RCpc load-acquire (LDAPR)743client # [ 0.021268] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.744worker1 # [ 0.005285] CPU features: detected: LSE atomic instructions745client # [ 0.021285] ASID allocator initialised with 65536 entries746worker1 # [ 0.005288] CPU features: detected: Privileged Access Never747client # [ 0.022863] Serial: AMBA PL011 UART driver748worker1 # [ 0.005291] CPU features: detected: PMUv3749worker1 # [ 0.005294] CPU features: detected: RAS Extension Support750client # [ 0.028161] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1751client # [ 0.028316] printk: console [ttyAMA0] enabled752worker1 # [ 0.005297] CPU features: detected: RASv1p1 Extension Support753worker1 # [ 0.005299] CPU features: detected: Random Number Generator754worker1 # [ 0.005302] CPU features: detected: Speculation barrier (SB)755worker1 # [ 0.005305] CPU features: detected: Stage-2 Force Write-Back756worker1 # [ 0.005308] CPU features: detected: TLB range maintenance instructions757lb # [ 0.161801] iommu: Default domain type: Translated758worker1 # [ 0.005313] CPU features: detected: Speculative Store Bypassing Safe (SSBS)759lb # [ 0.161812] iommu: DMA domain TLB invalidation policy: strict mode760worker1 # [ 0.005353] alternatives: applying system-wide alternatives761lb # [ 0.162176] SCSI subsystem initialized762worker1 # [ 0.008465] CPU features: detected: BBM Level 2 without TLB conflict abort763client # [ 0.147101] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages764lb # [ 0.164285] usbcore: registered new interface driver usbfs765worker2 # [ 0.218661] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)766client # [ 0.147121] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page767lb # [ 0.164317] usbcore: registered new interface driver hub768worker1 # [ 0.008682] Memory: 1405608K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124664K reserved, 32768K cma-reserved)769worker2 # [ 0.218721] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)770lb # [ 0.164332] usbcore: registered new device driver usb771client # [ 0.147127] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages772worker1 # [ 0.009109] devtmpfs: initialized773lb # [ 0.164615] pps_core: LinuxPPS API ver. 1 registered774client # [ 0.147131] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page775worker2 # [ 0.218755] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)776worker1 # [ 0.010992] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear)777lb # [ 0.164622] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>778client # [ 0.147136] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages779worker2 # [ 0.218840] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)780worker1 # [ 0.011018] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear).781lb # [ 0.164631] PTP clock support registered782client # [ 0.147140] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page783lb # [ 0.164687] EDAC MC: Ver: 3.0.0784worker2 # [ 0.219004] TCP: Hash tables configured (established 16384 bind 16384)785worker1 # [ 0.011235] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL786client # [ 0.147144] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages787worker1 # [ 0.011240] 0 pages in range for non-PLT usage788worker2 # [ 0.219091] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)789lb # [ 0.169182] scmi_core: SCMI protocol bus registered790worker1 # [ 0.011241] 508272 pages in range for PLT usage791client # [ 0.147149] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page792worker2 # [ 0.219128] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)793lb # [ 0.170142] FPGA manager framework794worker1 # [ 0.011371] pinctrl core: initialized pinctrl subsystem795lb # [ 0.171106] vgaarb: loaded796worker1 # [ 0.012118] DMI not present or invalid.797worker2 # [ 0.219165] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)798worker1 # [ 0.015522] NET: Registered PF_NETLINK/PF_ROUTE protocol family799worker2 # [ 0.219260] NET: Registered PF_UNIX/PF_LOCAL protocol family800lb # [ 0.171747] clocksource: Switched to clocksource arch_sys_counter801client # [ 0.154468] fbcon: Taking over console802worker2 # [ 0.219281] NET: Registered PF_XDP protocol family803worker1 # [ 0.017856] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations804client # [ 0.154485] ACPI: Interpreter disabled.805worker2 # [ 0.219300] PCI: CLS 0 bytes, default 64806worker1 # [ 0.018057] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations807worker2 # [ 0.219558] Trying to unpack rootfs image as initramfs...808worker1 # [ 0.018279] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations809worker1 # [ 0.018306] audit: initializing netlink subsys (disabled)810worker1 # [ 0.018939] thermal_sys: Registered thermal governor 'fair_share'811worker1 # [ 0.018942] thermal_sys: Registered thermal governor 'bang_bang'812worker1 # [ 0.018946] thermal_sys: Registered thermal governor 'step_wise'813worker1 # [ 0.018948] thermal_sys: Registered thermal governor 'user_space'814lb # [ 0.178348] VFS: Disk quotas dquot_6.6.0815worker1 # [ 0.018954] thermal_sys: Registered thermal governor 'power_allocator'816lb # [ 0.178394] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)817worker1 # [ 0.018989] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1818lb # [ 0.180113] netfs: FS-Cache loaded819worker1 # [ 0.018997] cpuidle: using governor ladder820lb # [ 0.180219] pnp: PnP ACPI: disabled821worker2 # [ 0.235491] kvm [1]: HYP mode not available822client # [ 0.163536] iommu: Default domain type: Translated823worker1 # [ 0.019003] cpuidle: using governor menu824client # [ 0.163547] iommu: DMA domain TLB invalidation policy: strict mode825worker1 # [ 0.019195] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers.826client # [ 0.165230] SCSI subsystem initialized827worker1 # [ 0.019213] ASID allocator initialised with 65536 entries828worker1 # [ 0.020438] Serial: AMBA PL011 UART driver829client # [ 0.165940] usbcore: registered new interface driver usbfs830worker1 # [ 0.025869] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1831client # [ 0.165972] usbcore: registered new interface driver hub832worker1 # [ 0.026021] printk: console [ttyAMA0] enabled833client # [ 0.165988] usbcore: registered new device driver usb834client # [ 0.166269] pps_core: LinuxPPS API ver. 1 registered835client # [ 0.166276] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>836client # [ 0.166296] PTP clock support registered837client # [ 0.166346] EDAC MC: Ver: 3.0.0838lb # [ 0.184182] NET: Registered PF_INET protocol family839client # [ 0.171121] scmi_core: SCMI protocol bus registered840lb # [ 0.184453] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)841client # [ 0.172159] FPGA manager framework842client # [ 0.173145] vgaarb: loaded843client # [ 0.173776] clocksource: Switched to clocksource arch_sys_counter844worker1 # [ 0.147878] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages845worker1 # [ 0.147900] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page846worker1 # [ 0.147907] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages847worker1 # [ 0.147911] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page848worker1 # [ 0.147915] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages849worker1 # [ 0.147920] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page850client # [ 0.180351] VFS: Disk quotas dquot_6.6.0851worker1 # [ 0.147924] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages852client # [ 0.180396] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)853worker1 # [ 0.147929] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page854client # [ 0.182177] netfs: FS-Cache loaded855client # [ 0.182283] pnp: PnP ACPI: disabled856worker1 # [ 0.155809] fbcon: Taking over console857worker1 # [ 0.155826] ACPI: Interpreter disabled.858client # [ 0.186156] NET: Registered PF_INET protocol family859client # [ 0.186434] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)860worker1 # [ 0.164346] iommu: Default domain type: Translated861worker1 # [ 0.164358] iommu: DMA domain TLB invalidation policy: strict mode862worker1 # [ 0.164740] SCSI subsystem initialized863worker1 # [ 0.166784] usbcore: registered new interface driver usbfs864worker1 # [ 0.166822] usbcore: registered new interface driver hub865worker1 # [ 0.166846] usbcore: registered new device driver usb866worker1 # [ 0.167127] pps_core: LinuxPPS API ver. 1 registered867worker1 # [ 0.167133] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>868worker1 # [ 0.167144] PTP clock support registered869worker1 # [ 0.167191] EDAC MC: Ver: 3.0.0870worker1 # [ 0.171947] scmi_core: SCMI protocol bus registered871worker1 # [ 0.173024] FPGA manager framework872worker1 # [ 0.173974] vgaarb: loaded873worker1 # [ 0.174640] clocksource: Switched to clocksource arch_sys_counter874lb # [ 0.214983] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)875worker1 # [ 0.179604] VFS: Disk quotas dquot_6.6.0876lb # [ 0.215034] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)877worker1 # [ 0.179646] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)878lb # [ 0.215065] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)879worker1 # [ 0.181405] netfs: FS-Cache loaded880worker1 # [ 0.181512] pnp: PnP ACPI: disabled881lb # [ 0.215143] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)882lb # [ 0.215288] TCP: Hash tables configured (established 16384 bind 16384)883lb # [ 0.215380] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)884lb # [ 0.215419] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)885lb # [ 0.215457] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)886lb # [ 0.215578] NET: Registered PF_UNIX/PF_LOCAL protocol family887worker1 # [ 0.185504] NET: Registered PF_INET protocol family888lb # [ 0.215605] NET: Registered PF_XDP protocol family889lb # [ 0.215624] PCI: CLS 0 bytes, default 64890worker1 # [ 0.185780] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear)891lb # [ 0.215859] Trying to unpack rootfs image as initramfs...892lb # [ 0.233854] kvm [1]: HYP mode not available893client # [ 0.216233] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)894client # [ 0.216289] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)895client # [ 0.216323] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)896client # [ 0.216411] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)897client # [ 0.216630] TCP: Hash tables configured (established 16384 bind 16384)898client # [ 0.216729] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)899client # [ 0.216765] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)900client # [ 0.216801] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)901client # [ 0.216896] NET: Registered PF_UNIX/PF_LOCAL protocol family902client # [ 0.216950] NET: Registered PF_XDP protocol family903client # [ 0.216970] PCI: CLS 0 bytes, default 64904client # [ 0.217233] Trying to unpack rootfs image as initramfs...905client # [ 0.235623] kvm [1]: HYP mode not available906worker1 # [ 0.216484] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear)907worker1 # [ 0.216542] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear)908worker1 # [ 0.216575] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear)909worker1 # [ 0.216684] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear)910worker1 # [ 0.216827] TCP: Hash tables configured (established 16384 bind 16384)911worker1 # [ 0.216921] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear)912worker1 # [ 0.216964] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear)913worker1 # [ 0.217001] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear)914worker1 # [ 0.217096] NET: Registered PF_UNIX/PF_LOCAL protocol family915worker1 # [ 0.217123] NET: Registered PF_XDP protocol family916worker1 # [ 0.217143] PCI: CLS 0 bytes, default 64917worker1 # [ 0.217385] Trying to unpack rootfs image as initramfs...918worker1 # [ 0.232628] kvm [1]: HYP mode not available919worker2 # [ 0.348336] Initialise system trusted keyrings920worker2 # [ 0.349099] workingset: timestamp_bits=42 max_order=19 bucket_order=0921worker2 # [ 0.350411] squashfs: version 4.0 (2009/01/31) Phillip Lougher922worker2 # [ 0.351205] 9p: Installing v9fs 9p2000 file system support923worker2 # [ 0.379343] Key type asymmetric registered924worker2 # [ 0.379375] Asymmetric key parser 'x509' registered925worker2 # [ 0.379456] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)926worker2 # [ 0.381636] io scheduler mq-deadline registered927worker2 # [ 0.381649] io scheduler kyber registered928worker2 # [ 0.395225] pl061_gpio 9030000.pl061: PL061 GPIO chip registered929worker2 # [ 0.395825] ledtrig-cpu: registered to indicate activity on CPUs930worker2 # [ 0.396231] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:931worker2 # [ 0.396248] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000932worker2 # [ 0.396261] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000933worker2 # [ 0.396270] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000934worker2 # [ 0.396290] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits935worker2 # [ 0.396313] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]936worker2 # [ 0.396399] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00937worker2 # [ 0.396410] pci_bus 0000:00: root bus resource [bus 00-ff]938worker2 # [ 0.396416] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]939worker2 # [ 0.396421] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]940worker2 # [ 0.396426] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]941worker2 # [ 0.396492] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint942worker2 # [ 0.396940] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint943worker2 # [ 0.397126] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]944worker2 # [ 0.397143] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]945worker2 # [ 0.397172] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]946worker2 # [ 0.397187] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]947worker2 # [ 0.397632] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint948worker2 # [ 0.397818] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]949worker2 # [ 0.397833] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]950worker2 # [ 0.397863] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]951lb # [ 0.360405] Initialise system trusted keyrings952worker2 # [ 0.398349] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint953lb # [ 0.361184] workingset: timestamp_bits=42 max_order=19 bucket_order=0954worker2 # [ 0.398532] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]955worker2 # [ 0.398548] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]956lb # [ 0.362577] squashfs: version 4.0 (2009/01/31) Phillip Lougher957worker2 # [ 0.398577] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]958lb # [ 0.363398] 9p: Installing v9fs 9p2000 file system support959worker2 # [ 0.399067] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint960worker2 # [ 0.399249] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]961worker2 # [ 0.399264] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]962worker2 # [ 0.399293] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]963worker2 # [ 0.399311] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]964worker2 # [ 0.399765] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint965worker2 # [ 0.399949] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]966worker2 # [ 0.399978] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]967worker2 # [ 0.400437] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint968worker2 # [ 0.400622] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]969worker2 # [ 0.400651] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]970worker2 # [ 0.401077] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint971worker2 # [ 0.401260] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]972worker2 # [ 0.401506] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint973client # [ 0.358324] Initialise system trusted keyrings974worker2 # [ 0.401693] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]975client # [ 0.359108] workingset: timestamp_bits=42 max_order=19 bucket_order=0976worker2 # [ 0.401723] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]977client # [ 0.360453] squashfs: version 4.0 (2009/01/31) Phillip Lougher978client # [ 0.361274] 9p: Installing v9fs 9p2000 file system support979worker2 # [ 0.442302] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint980worker2 # [ 0.442516] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]981worker2 # [ 0.442548] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]982worker2 # [ 0.443043] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint983worker2 # [ 0.443233] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]984worker2 # [ 0.443263] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]985worker2 # [ 0.443723] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint986worker2 # [ 0.444033] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]987worker2 # [ 0.444051] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]988lb # [ 0.392217] Key type asymmetric registered989worker2 # [ 0.444080] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]990lb # [ 0.392249] Asymmetric key parser 'x509' registered991worker2 # [ 0.444550] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint992lb # [ 0.392335] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)993worker2 # [ 0.444737] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]994lb # [ 0.394620] io scheduler mq-deadline registered995worker2 # [ 0.444753] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]996lb # [ 0.394634] io scheduler kyber registered997worker2 # [ 0.444782] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]998worker2 # [ 0.445377] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned999worker2 # [ 0.445390] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1000worker2 # [ 0.445395] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1001worker2 # [ 0.445443] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1002worker2 # [ 0.445489] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1003worker2 # [ 0.445538] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1004worker2 # [ 0.445585] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1005worker2 # [ 0.445635] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1006worker1 # [ 0.359278] Initialise system trusted keyrings1007worker2 # [ 0.445682] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1008worker1 # [ 0.360091] workingset: timestamp_bits=42 max_order=19 bucket_order=01009worker2 # [ 0.445728] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1010worker2 # [ 0.445773] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1011worker1 # [ 0.361438] squashfs: version 4.0 (2009/01/31) Phillip Lougher1012client # [ 0.390131] Key type asymmetric registered1013worker2 # [ 0.445821] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1014worker1 # [ 0.362267] 9p: Installing v9fs 9p2000 file system support1015client # [ 0.390160] Asymmetric key parser 'x509' registered1016worker2 # [ 0.445906] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1017client # [ 0.390245] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)1018client # [ 0.392407] io scheduler mq-deadline registered1019client # [ 0.392420] io scheduler kyber registered1020lb # [ 0.407972] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1021lb # [ 0.408796] ledtrig-cpu: registered to indicate activity on CPUs1022lb # [ 0.409202] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1023lb # [ 0.409222] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001024lb # [ 0.409236] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001025client # [ 0.401949] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1026lb # [ 0.409244] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001027lb # [ 0.409278] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1028client # [ 0.403467] ledtrig-cpu: registered to indicate activity on CPUs1029lb # [ 0.409325] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1030client # [ 0.403877] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1031lb # [ 0.409404] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001032worker2 # [ 0.482092] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1033lb # [ 0.409413] pci_bus 0000:00: root bus resource [bus 00-ff]1034client # [ 0.403895] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001035lb # [ 0.409419] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1036worker2 # [ 0.482142] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1037client # [ 0.403908] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001038worker2 # [ 0.482166] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1039lb # [ 0.409425] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1040worker2 # [ 0.482189] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1041lb # [ 0.409430] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1042client # [ 0.403916] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001043worker2 # [ 0.482212] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1044lb # [ 0.409510] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1045client # [ 0.403939] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1046worker2 # [ 0.482235] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1047lb # [ 0.410009] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1048client # [ 0.403967] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1049worker2 # [ 0.482258] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1050lb # [ 0.410200] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1051client # [ 0.404047] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001052worker2 # [ 0.482287] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1053lb # [ 0.410217] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1054client # [ 0.404057] pci_bus 0000:00: root bus resource [bus 00-ff]1055worker2 # [ 0.482310] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1056lb # [ 0.410248] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1057client # [ 0.404064] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1058worker2 # [ 0.482333] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1059lb # [ 0.410266] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1060client # [ 0.404069] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1061worker2 # [ 0.482356] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1062lb # [ 0.410724] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1063client # [ 0.404074] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1064lb # [ 0.410904] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1065worker2 # [ 0.482380] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1066client # [ 0.404131] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1067lb # [ 0.410920] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1068worker2 # [ 0.482402] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1069worker2 # [ 0.482423] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1070lb # [ 0.410951] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1071client # [ 0.404604] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1072client # [ 0.404795] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1073worker2 # [ 0.482443] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1074lb # [ 0.411398] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1075worker1 # [ 0.391062] Key type asymmetric registered1076client # [ 0.404813] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1077worker2 # [ 0.482464] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1078lb # [ 0.411580] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1079worker1 # [ 0.391095] Asymmetric key parser 'x509' registered1080worker2 # [ 0.482487] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1081client # [ 0.404843] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1082lb # [ 0.411597] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1083worker1 # [ 0.391177] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242)1084worker2 # [ 0.482509] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1085client # [ 0.404860] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1086lb # [ 0.411626] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1087worker1 # [ 0.393382] io scheduler mq-deadline registered1088worker2 # [ 0.482543] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1089worker1 # [ 0.393395] io scheduler kyber registered1090client # [ 0.405333] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1091lb # [ 0.412099] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1092worker2 # [ 0.482553] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1093client # [ 0.405520] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1094lb # [ 0.412286] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1095worker2 # [ 0.482558] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1096client # [ 0.405535] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1097lb # [ 0.412302] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1098worker2 # [ 0.483453] pci 0000:00:07.0: enabling device (0000 -> 0002)1099client # [ 0.405567] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1100lb # [ 0.412331] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1101lb # [ 0.412350] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1102lb # [ 0.412820] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1103client # [ 0.425194] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1104client # [ 0.425384] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1105lb # [ 0.413015] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1106client # [ 0.425400] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1107lb # [ 0.413046] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1108client # [ 0.425429] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1109lb # [ 0.413490] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1110lb # [ 0.413690] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1111lb # [ 0.413719] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1112lb # [ 0.414134] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1113worker1 # [ 0.402813] pl061_gpio 9030000.pl061: PL061 GPIO chip registered1114lb # [ 0.414315] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1115lb # [ 0.414564] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1116lb # [ 0.414748] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1117worker1 # [ 0.404318] ledtrig-cpu: registered to indicate activity on CPUs1118lb # [ 0.414777] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1119worker1 # [ 0.404764] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges:1120lb # [ 0.415225] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1121worker2 # [ 0.513282] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1122lb # [ 0.415408] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1123worker1 # [ 0.404784] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x00000000001124lb # [ 0.415437] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1125worker1 # [ 0.404795] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x00100000001126worker1 # [ 0.404804] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x80000000001127worker1 # [ 0.404827] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits1128worker1 # [ 0.404862] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff]1129worker1 # [ 0.404952] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:001130client # [ 0.434009] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1131worker1 # [ 0.404962] pci_bus 0000:00: root bus resource [bus 00-ff]1132client # [ 0.434221] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1133client # [ 0.434239] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1134worker1 # [ 0.404968] pci_bus 0000:00: root bus resource [io 0x0000-0xffff]1135client # [ 0.434271] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1136worker1 # [ 0.404973] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff]1137client # [ 0.434291] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1138worker1 # [ 0.404978] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff]1139client # [ 0.434793] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1140worker1 # [ 0.405039] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint1141client # [ 0.434986] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1142worker1 # [ 0.405491] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1143client # [ 0.435016] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1144worker1 # [ 0.405680] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f]1145client # [ 0.435484] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1146worker1 # [ 0.405696] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff]1147client # [ 0.435674] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1148worker1 # [ 0.405725] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1149client # [ 0.435704] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1150worker1 # [ 0.405742] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref]1151client # [ 0.436095] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1152worker1 # [ 0.406216] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1153worker1 # [ 0.406404] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f]1154client # [ 0.436279] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1155worker1 # [ 0.406420] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff]1156client # [ 0.436587] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1157lb # [ 0.464045] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1158worker1 # [ 0.406449] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1159client # [ 0.436792] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1160lb # [ 0.464267] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1161client # [ 0.436823] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1162lb # [ 0.464298] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1163client # [ 0.437285] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1164worker2 # [ 0.531882] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1165lb # [ 0.464795] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1166client # [ 0.437474] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1167lb # [ 0.465098] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1168client # [ 0.437503] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1169lb # [ 0.465116] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1170client # [ 0.437985] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1171lb # [ 0.465145] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1172client # [ 0.438175] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1173lb # [ 0.465611] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1174client # [ 0.438205] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1175lb # [ 0.465817] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1176lb # [ 0.465833] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1177client # [ 0.438664] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1178client # [ 0.438959] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1179lb # [ 0.465863] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1180client # [ 0.438975] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1181lb # [ 0.466467] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1182client # [ 0.439004] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1183lb # [ 0.466478] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1184client # [ 0.439479] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1185lb # [ 0.466483] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1186client # [ 0.439663] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1187worker2 # [ 0.535259] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1188lb # [ 0.466529] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1189client # [ 0.439678] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1190worker2 # [ 0.537636] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1191client # [ 0.439709] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1192lb # [ 0.466579] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1193worker2 # [ 0.540171] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1194client # [ 0.440306] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1195lb # [ 0.466627] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1196client # [ 0.440317] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1197lb # [ 0.466676] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1198client # [ 0.440323] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1199lb # [ 0.466723] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1200client # [ 0.440368] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1201lb # [ 0.466772] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1202worker1 # [ 0.435109] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint1203client # [ 0.440414] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1204lb # [ 0.466819] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1205worker1 # [ 0.435327] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f]1206client # [ 0.440465] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1207worker1 # [ 0.435344] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff]1208lb # [ 0.466869] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1209client # [ 0.440511] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1210worker1 # [ 0.435374] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1211lb # [ 0.466915] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1212client # [ 0.440577] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1213worker1 # [ 0.435852] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint1214lb # [ 0.466991] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1215worker1 # [ 0.436043] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f]1216client # [ 0.440624] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1217lb # [ 0.467037] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1218worker1 # [ 0.436060] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff]1219client # [ 0.440669] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1220lb # [ 0.467060] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1221worker1 # [ 0.436089] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1222lb # [ 0.467083] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1223client # [ 0.440716] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1224worker1 # [ 0.436108] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref]1225lb # [ 0.467104] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1226client # [ 0.440762] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1227worker1 # [ 0.436579] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint1228lb # [ 0.467125] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1229worker1 # [ 0.436791] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff]1230client # [ 0.440835] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1231lb # [ 0.467146] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1232worker1 # [ 0.436821] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1233client # [ 0.440881] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1234lb # [ 0.467168] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1235worker1 # [ 0.437284] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint1236client # [ 0.440902] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1237lb # [ 0.467198] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1238worker1 # [ 0.437473] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff]1239client # [ 0.440923] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1240lb # [ 0.467221] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1241worker1 # [ 0.437503] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1242client # [ 0.440944] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1243lb # [ 0.467243] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1244client # [ 0.440967] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1245worker1 # [ 0.437946] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint1246lb # [ 0.467265] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1247worker1 # [ 0.438132] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff]1248client # [ 0.440989] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1249lb # [ 0.467288] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1250client # [ 0.441011] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1251worker1 # [ 0.438382] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1252lb # [ 0.467309] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1253client # [ 0.441036] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1254worker1 # [ 0.438570] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff]1255lb # [ 0.467330] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1256client # [ 0.441057] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1257lb # [ 0.467350] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1258worker1 # [ 0.438600] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1259lb # [ 0.467371] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1260client # [ 0.441079] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1261worker1 # [ 0.439093] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1262lb # [ 0.467392] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1263client # [ 0.441101] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1264worker1 # [ 0.439286] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff]1265worker2 # [ 0.558232] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1266lb # [ 0.467413] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1267client # [ 0.441123] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1268worker2 # [ 0.560403] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1269worker1 # [ 0.439315] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1270lb # [ 0.467441] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1271client # [ 0.441145] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1272worker2 # [ 0.562427] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1273lb # [ 0.467451] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1274worker1 # [ 0.439781] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint1275client # [ 0.441166] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1276worker1 # [ 0.439976] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff]1277lb # [ 0.467456] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1278client # [ 0.441186] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1279lb # [ 0.468310] pci 0000:00:07.0: enabling device (0000 -> 0002)1280worker1 # [ 0.440009] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1281client # [ 0.441208] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1282worker2 # [ 0.564500] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1283client # [ 0.441229] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1284worker1 # [ 0.440483] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint1285worker1 # [ 0.440830] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f]1286client # [ 0.441249] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1287worker1 # [ 0.440848] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff]1288client # [ 0.441277] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1289client # [ 0.441287] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1290worker1 # [ 0.440878] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1291client # [ 0.441291] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1292worker1 # [ 0.441336] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint1293worker1 # [ 0.441518] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f]1294worker1 # [ 0.441534] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff]1295worker1 # [ 0.441563] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref]1296worker1 # [ 0.442175] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned1297worker1 # [ 0.442187] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned1298worker2 # [ 0.574107] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1299worker2 # [ 0.576613] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1300worker1 # [ 0.442192] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned1301worker1 # [ 0.442238] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned1302worker1 # [ 0.442284] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned1303worker1 # [ 0.442332] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned1304client # [ 0.502228] pci 0000:00:07.0: enabling device (0000 -> 0002)1305worker1 # [ 0.442380] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned1306worker1 # [ 0.442426] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned1307worker1 # [ 0.442472] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned1308worker1 # [ 0.442518] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned1309worker1 # [ 0.442563] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned1310worker1 # [ 0.442610] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned1311lb # [ 0.527694] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1312worker2 # [ 0.591059] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1313worker1 # [ 0.486772] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned1314worker1 # [ 0.486895] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned1315worker1 # [ 0.486921] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned1316worker1 # [ 0.486945] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned1317worker1 # [ 0.486969] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned1318worker1 # [ 0.486993] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned1319lb # [ 0.537245] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1320worker2 # [ 0.599366] msm_serial: driver initialized1321worker1 # [ 0.487017] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned1322worker2 # [ 0.599517] SuperH (H)SCI(F) driver initialized1323worker2 # [ 0.599571] STM32 USART driver initialized1324worker1 # [ 0.487040] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned1325worker1 # [ 0.487068] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned1326worker1 # [ 0.487091] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned1327worker1 # [ 0.487116] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned1328worker1 # [ 0.487143] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned1329worker1 # [ 0.487166] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned1330worker1 # [ 0.487187] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned1331client # [ 0.528719] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1332worker1 # [ 0.487208] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned1333lb # [ 0.543963] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1334worker1 # [ 0.487229] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned1335lb # [ 0.546173] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1336worker1 # [ 0.487250] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned1337worker1 # [ 0.487271] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned1338worker1 # [ 0.487292] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned1339client # [ 0.531878] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1340worker1 # [ 0.487324] pci_bus 0000:00: resource 4 [io 0x0000-0xffff]1341worker1 # [ 0.487333] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff]1342worker1 # [ 0.487338] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff]1343worker1 # [ 0.488190] pci 0000:00:07.0: enabling device (0000 -> 0002)1344client # [ 0.542069] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1345lb # [ 0.556428] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1346client # [ 0.544293] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1347lb # [ 0.558956] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1348lb # [ 0.560859] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1349lb # [ 0.562768] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1350client # [ 0.548149] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1351worker2 # [ 0.629863] loop: module loaded1352worker2 # [ 0.634105] virtio_blk virtio2: 1/0/0 default/read/poll queues1353client # [ 0.553958] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1354worker2 # [ 0.635109] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1355client # [ 0.557298] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1356lb # [ 0.573005] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1357worker1 # [ 0.528288] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003)1358lb # [ 0.575141] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1359worker1 # [ 0.530582] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003)1360worker2 # [ 0.642662] megasas: 07.734.00.00-rc11361worker2 # [ 0.643512] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1362worker2 # [ 0.645908] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001363client # [ 0.568722] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1364worker2 # [ 0.648823] Intel/Sharp Extended Query Table at 0x00311365lb # [ 0.587396] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1366worker2 # [ 0.654610] Using buffer write method1367worker1 # [ 0.548242] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003)1368client # [ 0.575266] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1369worker1 # [ 0.550427] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003)1370worker2 # [ 0.654695] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1371client # [ 0.577506] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1372worker1 # [ 0.554316] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002)1373lb # [ 0.600041] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1374lb # [ 0.602750] msm_serial: driver initialized1375lb # [ 0.602888] SuperH (H)SCI(F) driver initialized1376lb # [ 0.602943] STM32 USART driver initialized1377worker1 # [ 0.557531] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002)1378worker2 # [ 0.665986] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001379worker2 # [ 0.666030] Intel/Sharp Extended Query Table at 0x00311380worker2 # [ 0.667807] Using buffer write method1381worker2 # [ 0.667844] Concatenating MTD devices:1382worker2 # [ 0.667848] (0): "0.flash"1383worker2 # [ 0.667852] (1): "0.flash"1384worker2 # [ 0.667864] into device "0.flash"1385client # [ 0.591808] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1386worker1 # [ 0.568182] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002)1387worker1 # [ 0.570337] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002)1388client # [ 0.609525] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1389worker1 # [ 0.581608] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002)1390worker1 # [ 0.584691] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003)1391client # [ 0.613952] msm_serial: driver initialized1392client # [ 0.614275] SuperH (H)SCI(F) driver initialized1393client # [ 0.614368] STM32 USART driver initialized1394lb # [ 0.641789] loop: module loaded1395lb # [ 0.642007] virtio_blk virtio2: 1/0/0 default/read/poll queues1396worker1 # [ 0.594948] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003)1397lb # [ 0.642832] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1398worker1 # [ 0.601092] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled1399worker1 # [ 0.608066] msm_serial: driver initialized1400worker1 # [ 0.608216] SuperH (H)SCI(F) driver initialized1401worker1 # [ 0.608273] STM32 USART driver initialized1402lb # [ 0.656539] megasas: 07.734.00.00-rc11403lb # [ 0.657336] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1404lb # [ 0.659868] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001405lb # [ 0.659940] Intel/Sharp Extended Query Table at 0x00311406lb # [ 0.661764] Using buffer write method1407lb # [ 0.661844] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1408lb # [ 0.663725] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001409lb # [ 0.663751] Intel/Sharp Extended Query Table at 0x00311410lb # [ 0.673075] Using buffer write method1411lb # [ 0.673109] Concatenating MTD devices:1412lb # [ 0.673114] (0): "0.flash"1413lb # [ 0.673117] (1): "0.flash"1414lb # [ 0.673129] into device "0.flash"1415client # [ 0.667482] loop: module loaded1416client # [ 0.667747] virtio_blk virtio2: 1/0/0 default/read/poll queues1417client # [ 0.668971] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1418worker1 # [ 0.642944] loop: module loaded1419worker1 # [ 0.643172] virtio_blk virtio2: 1/0/0 default/read/poll queues1420worker1 # [ 0.644025] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB)1421worker1 # [ 0.651395] megasas: 07.734.00.00-rc11422worker1 # [ 0.652204] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1423worker1 # [ 0.654452] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001424client # [ 0.683085] megasas: 07.734.00.00-rc11425worker1 # [ 0.654484] Intel/Sharp Extended Query Table at 0x00311426worker1 # [ 0.656317] Using buffer write method1427client # [ 0.684491] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff]1428worker1 # [ 0.656403] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1429client # [ 0.686758] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001430client # [ 0.686809] Intel/Sharp Extended Query Table at 0x00311431worker1 # [ 0.658364] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001432client # [ 0.688592] Using buffer write method1433worker1 # [ 0.658391] Intel/Sharp Extended Query Table at 0x00311434client # [ 0.688749] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff]1435worker1 # [ 0.678515] Using buffer write method1436client # [ 0.702633] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x0000001437worker1 # [ 0.678610] Concatenating MTD devices:1438worker1 # [ 0.678615] (0): "0.flash"1439client # [ 0.702697] Intel/Sharp Extended Query Table at 0x00311440worker1 # [ 0.678619] (1): "0.flash"1441client # [ 0.704371] Using buffer write method1442worker1 # [ 0.678628] into device "0.flash"1443client # [ 0.704426] Concatenating MTD devices:1444client # [ 0.704432] (0): "0.flash"1445client # [ 0.704438] (1): "0.flash"1446client # [ 0.704453] into device "0.flash"1447worker2 # [ 0.966776] Freeing initrd memory: 26944K1448worker2 # [ 0.973138] tun: Universal TUN/TAP device driver, 1.61449worker2 # [ 0.977141] thunder_xcv, ver 1.01450worker2 # [ 0.977192] thunder_bgx, ver 1.01451worker2 # [ 0.977215] nicpf, ver 1.01452worker2 # [ 0.977772] e1000: Intel(R) PRO/1000 Network Driver1453worker2 # [ 0.977780] e1000: Copyright (c) 1999-2006 Intel Corporation.1454worker2 # [ 0.977803] e1000e: Intel(R) PRO/1000 Network Driver1455worker2 # [ 0.977812] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1456worker2 # [ 0.977842] igb: Intel(R) Gigabit Ethernet Network Driver1457worker2 # [ 0.977848] igb: Copyright (c) 2007-2014 Intel Corporation.1458worker2 # [ 0.977889] igbvf: Intel(R) Gigabit Virtual Function Network Driver1459worker2 # [ 0.977896] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1460worker2 # [ 0.985029] sky2: driver version 1.301461worker2 # [ 0.986694] usbcore: registered new interface driver usb-storage1462worker2 # [ 0.986789] usbcore: registered new interface driver usbserial_generic1463worker2 # [ 0.986804] usbserial: USB Serial support registered for generic1464worker2 # [ 0.987503] hv_vmbus: registering driver hyperv_keyboard1465worker2 # [ 0.988929] ehci-pci 0000:00:07.0: EHCI Host Controller1466worker2 # [ 0.988960] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11467worker2 # [ 0.989255] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001468worker2 # [ 0.995683] rtc-pl031 9010000.pl031: registered as rtc01469worker2 # [ 0.995718] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T04:42:50 UTC (1790138570)1470worker2 # [ 0.996108] i2c_dev: i2c /dev entries driver1471worker2 # [ 0.999355] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001472worker2 # [ 0.999658] hub 1-0:1.0: USB hub found1473worker2 # [ 0.999679] hub 1-0:1.0: 6 ports detected1474worker2 # [ 1.004275] sdhci: Secure Digital Host Controller Interface driver1475worker2 # [ 1.004287] sdhci: Copyright(c) Pierre Ossman1476worker2 # [ 1.004557] Synopsys Designware Multimedia Card Interface Driver1477worker2 # [ 1.004937] sdhci-pltfm: SDHCI platform and OF driver helper1478worker2 # [ 1.009365] hid: raw HID events driver (C) Jiri Kosina1479worker2 # [ 1.009635] usbcore: registered new interface driver usbhid1480worker2 # [ 1.009642] usbhid: USB HID core driver1481worker2 # [ 1.012664] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1482worker2 # [ 1.015447] drop_monitor: Initializing network drop monitor service1483worker2 # [ 1.015611] NET: Registered PF_INET6 protocol family1484worker2 # [ 1.017720] Segment Routing with IPv61485worker2 # [ 1.017740] In-situ OAM (IOAM) with IPv61486worker2 # [ 1.017770] NET: Registered PF_PACKET protocol family1487worker2 # [ 1.019548] 9pnet: Installing 9P2000 support1488worker2 # [ 1.019597] Key type dns_resolver registered1489lb # [ 0.966884] Freeing initrd memory: 26948K1490worker2 # [ 1.027017] registered taskstats version 11491worker2 # [ 1.027173] Loading compiled-in X.509 certificates1492lb # [ 0.973194] tun: Universal TUN/TAP device driver, 1.61493worker2 # [ 1.036116] Demotion targets for Node 0: null1494worker2 # [ 1.036244] Key type .fscrypt registered1495worker2 # [ 1.036252] Key type fscrypt-provisioning registered1496worker2 # [ 1.036353] ima: No TPM chip found, activating TPM-bypass!1497worker2 # [ 1.036372] ima: Allocated hash algorithm: sha11498worker2 # [ 1.036396] ima: No architecture policies found1499worker2 # [ 1.040718] input: gpio-keys as /devices/platform/gpio-keys/input/input01500lb # [ 0.977262] thunder_xcv, ver 1.01501lb # [ 0.977311] thunder_bgx, ver 1.01502lb # [ 0.977335] nicpf, ver 1.01503lb # [ 0.977986] e1000: Intel(R) PRO/1000 Network Driver1504lb # [ 0.977994] e1000: Copyright (c) 1999-2006 Intel Corporation.1505lb # [ 0.978019] e1000e: Intel(R) PRO/1000 Network Driver1506lb # [ 0.978027] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1507lb # [ 0.978056] igb: Intel(R) Gigabit Ethernet Network Driver1508lb # [ 0.978062] igb: Copyright (c) 2007-2014 Intel Corporation.1509lb # [ 0.978086] igbvf: Intel(R) Gigabit Virtual Function Network Driver1510lb # [ 0.978092] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1511lb # [ 0.978228] sky2: driver version 1.301512lb # [ 0.980623] ehci-pci 0000:00:07.0: EHCI Host Controller1513lb # [ 0.980652] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11514lb # [ 0.980967] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001515lb # [ 0.990770] usbcore: registered new interface driver usb-storage1516lb # [ 0.990913] usbcore: registered new interface driver usbserial_generic1517lb # [ 0.990928] usbserial: USB Serial support registered for generic1518lb # [ 0.991564] hv_vmbus: registering driver hyperv_keyboard1519lb # [ 0.994777] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001520lb # [ 0.995873] hub 1-0:1.0: USB hub found1521lb # [ 0.995907] hub 1-0:1.0: 6 ports detected1522lb # [ 0.998012] rtc-pl031 9010000.pl031: registered as rtc01523worker2 # [ 1.059422] clk: Disabling unused clocks1524lb # [ 0.998047] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T04:42:50 UTC (1790138570)1525lb # [ 0.998364] i2c_dev: i2c /dev entries driver1526worker2 # [ 1.059459] PM: genpd: Disabling unused power domains1527worker2 # [ 1.063729] Freeing unused kernel memory: 4736K1528worker2 # [ 1.063938] Run /init as init process1529lb # [ 1.003925] sdhci: Secure Digital Host Controller Interface driver1530lb # [ 1.003937] sdhci: Copyright(c) Pierre Ossman1531lb # [ 1.004206] Synopsys Designware Multimedia Card Interface Driver1532lb # [ 1.004582] sdhci-pltfm: SDHCI platform and OF driver helper1533lb # [ 1.006170] hid: raw HID events driver (C) Jiri Kosina1534lb # [ 1.006421] usbcore: registered new interface driver usbhid1535lb # [ 1.006428] usbhid: USB HID core driver1536lb # [ 1.012196] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1537lb # [ 1.013787] drop_monitor: Initializing network drop monitor service1538lb # [ 1.013952] NET: Registered PF_INET6 protocol family1539lb # [ 1.017089] Segment Routing with IPv61540lb # [ 1.017121] In-situ OAM (IOAM) with IPv61541lb # [ 1.017155] NET: Registered PF_PACKET protocol family1542lb # [ 1.018859] 9pnet: Installing 9P2000 support1543lb # [ 1.018907] Key type dns_resolver registered1544worker2 # [ 1.082527] systemd[1]: Successfully made /usr/ read-only.1545lb # [ 1.026040] registered taskstats version 11546lb # [ 1.026194] Loading compiled-in X.509 certificates1547worker1 # [ 0.983752] Freeing initrd memory: 26976K1548worker1 # [ 0.990086] tun: Universal TUN/TAP device driver, 1.61549lb # [ 1.035154] Demotion targets for Node 0: null1550lb # [ 1.035290] Key type .fscrypt registered1551lb # [ 1.035298] Key type fscrypt-provisioning registered1552lb # [ 1.035401] ima: No TPM chip found, activating TPM-bypass!1553lb # [ 1.035421] ima: Allocated hash algorithm: sha11554lb # [ 1.035443] ima: No architecture policies found1555worker1 # [ 0.994239] thunder_xcv, ver 1.01556worker1 # [ 0.994290] thunder_bgx, ver 1.01557lb # [ 1.039817] input: gpio-keys as /devices/platform/gpio-keys/input/input01558worker1 # [ 0.994312] nicpf, ver 1.01559worker1 # [ 0.996219] e1000: Intel(R) PRO/1000 Network Driver1560worker1 # [ 0.996232] e1000: Copyright (c) 1999-2006 Intel Corporation.1561worker1 # [ 0.996258] e1000e: Intel(R) PRO/1000 Network Driver1562worker1 # [ 0.996268] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1563worker1 # [ 0.996298] igb: Intel(R) Gigabit Ethernet Network Driver1564worker1 # [ 0.996303] igb: Copyright (c) 2007-2014 Intel Corporation.1565worker1 # [ 0.996327] igbvf: Intel(R) Gigabit Virtual Function Network Driver1566worker1 # [ 0.996333] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1567worker1 # [ 0.996473] sky2: driver version 1.301568worker1 # [ 0.998109] usbcore: registered new interface driver usb-storage1569worker1 # [ 0.998163] usbcore: registered new interface driver usbserial_generic1570worker1 # [ 0.998177] usbserial: USB Serial support registered for generic1571worker1 # [ 0.999557] ehci-pci 0000:00:07.0: EHCI Host Controller1572worker1 # [ 0.999588] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11573worker1 # [ 1.000097] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001574worker1 # [ 1.011272] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001575worker1 # [ 1.011654] hub 1-0:1.0: USB hub found1576worker1 # [ 1.011689] hub 1-0:1.0: 6 ports detected1577lb # [ 1.059072] clk: Disabling unused clocks1578worker1 # [ 1.014051] hv_vmbus: registering driver hyperv_keyboard1579lb # [ 1.059104] PM: genpd: Disabling unused power domains1580worker1 # [ 1.015760] rtc-pl031 9010000.pl031: registered as rtc01581worker1 # [ 1.015799] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T04:42:50 UTC (1790138570)1582worker1 # [ 1.016119] i2c_dev: i2c /dev entries driver1583lb # [ 1.063361] Freeing unused kernel memory: 4736K1584lb # [ 1.063560] Run /init as init process1585worker1 # [ 1.021820] sdhci: Secure Digital Host Controller Interface driver1586worker1 # [ 1.021842] sdhci: Copyright(c) Pierre Ossman1587worker1 # [ 1.022112] Synopsys Designware Multimedia Card Interface Driver1588worker1 # [ 1.022486] sdhci-pltfm: SDHCI platform and OF driver helper1589worker1 # [ 1.026978] hid: raw HID events driver (C) Jiri Kosina1590client # [ 1.054943] Freeing initrd memory: 26972K1591worker1 # [ 1.027228] usbcore: registered new interface driver usbhid1592worker1 # [ 1.027238] usbhid: USB HID core driver1593worker1 # [ 1.030398] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1594lb # [ 1.078149] systemd[1]: Successfully made /usr/ read-only.1595client # [ 1.061486] tun: Universal TUN/TAP device driver, 1.61596worker1 # [ 1.033229] drop_monitor: Initializing network drop monitor service1597worker1 # [ 1.033430] NET: Registered PF_INET6 protocol family1598worker1 # [ 1.035847] Segment Routing with IPv61599worker1 # [ 1.035872] In-situ OAM (IOAM) with IPv61600worker1 # [ 1.035904] NET: Registered PF_PACKET protocol family1601client # [ 1.065541] thunder_xcv, ver 1.01602client # [ 1.065596] thunder_bgx, ver 1.01603client # [ 1.065620] nicpf, ver 1.01604worker1 # [ 1.037684] 9pnet: Installing 9P2000 support1605worker1 # [ 1.037741] Key type dns_resolver registered1606client # [ 1.067569] e1000: Intel(R) PRO/1000 Network Driver1607client # [ 1.067584] e1000: Copyright (c) 1999-2006 Intel Corporation.1608client # [ 1.067610] e1000e: Intel(R) PRO/1000 Network Driver1609client # [ 1.067620] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.1610client # [ 1.067650] igb: Intel(R) Gigabit Ethernet Network Driver1611client # [ 1.067656] igb: Copyright (c) 2007-2014 Intel Corporation.1612worker1 # [ 1.044694] registered taskstats version 11613client # [ 1.067680] igbvf: Intel(R) Gigabit Virtual Function Network Driver1614worker1 # [ 1.044854] Loading compiled-in X.509 certificates1615client # [ 1.067686] igbvf: Copyright (c) 2009 - 2012 Intel Corporation.1616client # [ 1.067828] sky2: driver version 1.301617client # [ 1.069518] usbcore: registered new interface driver usb-storage1618client # [ 1.069655] usbcore: registered new interface driver usbserial_generic1619client # [ 1.069670] usbserial: USB Serial support registered for generic1620client # [ 1.070506] ehci-pci 0000:00:07.0: EHCI Host Controller1621client # [ 1.070536] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 11622client # [ 1.070703] ehci-pci 0000:00:07.0: irq 16, io mem 0x100860001623worker1 # [ 1.054024] Demotion targets for Node 0: null1624client # [ 1.081877] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.001625worker1 # [ 1.054175] Key type .fscrypt registered1626client # [ 1.082269] hub 1-0:1.0: USB hub found1627worker1 # [ 1.054184] Key type fscrypt-provisioning registered1628client # [ 1.082291] hub 1-0:1.0: 6 ports detected1629worker1 # [ 1.054304] ima: No TPM chip found, activating TPM-bypass!1630worker1 # [ 1.054325] ima: Allocated hash algorithm: sha11631worker1 # [ 1.054352] ima: No architecture policies found1632client # [ 1.085063] hv_vmbus: registering driver hyperv_keyboard1633worker1 # [ 1.058950] input: gpio-keys as /devices/platform/gpio-keys/input/input01634client # [ 1.086721] rtc-pl031 9010000.pl031: registered as rtc01635client # [ 1.086754] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T04:42:51 UTC (1790138571)1636client # [ 1.087070] i2c_dev: i2c /dev entries driver1637client # [ 1.092663] sdhci: Secure Digital Host Controller Interface driver1638client # [ 1.092678] sdhci: Copyright(c) Pierre Ossman1639client # [ 1.092953] Synopsys Designware Multimedia Card Interface Driver1640client # [ 1.093326] sdhci-pltfm: SDHCI platform and OF driver helper1641client # [ 1.097756] hid: raw HID events driver (C) Jiri Kosina1642client # [ 1.098716] usbcore: registered new interface driver usbhid1643client # [ 1.098727] usbhid: USB HID core driver1644client # [ 1.101165] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available1645client # [ 1.103927] drop_monitor: Initializing network drop monitor service1646client # [ 1.104115] NET: Registered PF_INET6 protocol family1647client # [ 1.106354] Segment Routing with IPv61648client # [ 1.106378] In-situ OAM (IOAM) with IPv61649client # [ 1.106420] NET: Registered PF_PACKET protocol family1650worker1 # [ 1.079537] clk: Disabling unused clocks1651client # [ 1.108100] 9pnet: Installing 9P2000 support1652client # [ 1.108147] Key type dns_resolver registered1653worker1 # [ 1.079583] PM: genpd: Disabling unused power domains1654worker1 # [ 1.083963] Freeing unused kernel memory: 4736K1655worker1 # [ 1.084183] Run /init as init process1656client # [ 1.115097] registered taskstats version 11657client # [ 1.115269] Loading compiled-in X.509 certificates1658client # [ 1.124291] Demotion targets for Node 0: null1659client # [ 1.124421] Key type .fscrypt registered1660client # [ 1.124428] Key type fscrypt-provisioning registered1661client # [ 1.124529] ima: No TPM chip found, activating TPM-bypass!1662client # [ 1.124578] ima: Allocated hash algorithm: sha11663client # [ 1.124601] ima: No architecture policies found1664client # [ 1.128994] input: gpio-keys as /devices/platform/gpio-keys/input/input01665worker1 # [ 1.101675] systemd[1]: Successfully made /usr/ read-only.1666client # [ 1.147665] clk: Disabling unused clocks1667client # [ 1.147703] PM: genpd: Disabling unused power domains1668client # [ 1.151953] Freeing unused kernel memory: 4736K1669client # [ 1.152162] Run /init as init process1670client # [ 1.168878] systemd[1]: Successfully made /usr/ read-only.1671worker2 # [ 1.246203] usb 1-1: new high-speed USB device number 2 using ehci-pci1672lb # [ 1.243836] usb 1-1: new high-speed USB device number 2 using ehci-pci1673worker1 # [ 1.258810] usb 1-1: new high-speed USB device number 2 using ehci-pci1674worker2 # [ 1.396031] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11675client # [ 1.329889] usb 1-1: new high-speed USB device number 2 using ehci-pci1676worker2 # [ 1.417764] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1677worker2 # [ 1.430746] systemd[1]: Detected virtualization qemu.1678worker2 # [ 1.432910] systemd[1]: Detected architecture arm64.1679worker2 # [ 1.435024] systemd[1]: Running in initrd.1680worker2 # [ 1.437767] systemd[1]: Initializing machine ID from random generator.1681worker2 # [ 1.441005] systemd[1]: Hostname set to <worker2>.1682lb # [ 1.391606] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11683lb # [ 1.413121] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1684worker2 # [ 1.486167] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01685lb # [ 1.426002] systemd[1]: Detected virtualization qemu.1686lb # [ 1.428272] systemd[1]: Detected architecture arm64.1687lb # [ 1.430290] systemd[1]: Running in initrd.1688lb # [ 1.433063] systemd[1]: Initializing machine ID from random generator.1689lb # [ 1.436275] systemd[1]: Hostname set to <lb>.1690worker1 # [ 1.413572] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11691lb # [ 1.484067] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01692worker1 # [ 1.436704] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1693worker1 # [ 1.443508] systemd[1]: Detected virtualization qemu.1694worker1 # [ 1.444588] systemd[1]: Detected architecture arm64.1695worker1 # [ 1.445711] systemd[1]: Running in initrd.1696worker1 # [ 1.447197] systemd[1]: Initializing machine ID from random generator.1697worker1 # [ 1.448913] systemd[1]: Hostname set to <worker1>.1698client # [ 1.482686] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input11699client # [ 1.504128] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE)1700client # [ 1.509221] systemd[1]: Detected virtualization qemu.1701client # [ 1.510163] systemd[1]: Detected architecture arm64.1702client # [ 1.510183] systemd[1]: Running in initrd.1703client # [ 1.510805] systemd[1]: Initializing machine ID from random generator.1704client # [ 1.511005] systemd[1]: Hostname set to <client>.1705worker2 # [ 1.606032] usb 1-2: new high-speed USB device number 3 using ehci-pci1706worker1 # [ 1.502891] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01707client # [ 1.578136] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input01708lb # [ 1.607886] usb 1-2: new high-speed USB device number 3 using ehci-pci1709worker2 # [ 1.716758] systemd[1]: bpf-restrict-fs: LSM BPF program attached1710worker1 # [ 1.622703] usb 1-2: new high-speed USB device number 3 using ehci-pci1711worker2 # [ 1.765339] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21712worker2 # [ 1.768499] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01713client # [ 1.697820] usb 1-2: new high-speed USB device number 3 using ehci-pci1714worker2 # [ 1.814759] systemd[1]: Queued start job for default target Initrd Default Target.1715lb # [ 1.753003] systemd[1]: bpf-restrict-fs: LSM BPF program attached1716worker2 # [ 1.828061] systemd[1]: Created slice Slice /system/modprobe.1717worker2 # [ 1.829380] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1718worker2 # [ 1.831182] systemd[1]: Expecting device /dev/disk/by-label/nixos...1719worker1 # [ 1.725115] systemd[1]: bpf-restrict-fs: LSM BPF program attached1720worker2 # [ 1.832316] systemd[1]: Reached target Path Units.1721worker2 # [ 1.833171] systemd[1]: Reached target Slice Units.1722lb # [ 1.770609] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21723worker2 # [ 1.834052] systemd[1]: Reached target Swaps.1724worker2 # [ 1.834081] systemd[1]: Reached target Timer Units.1725worker2 # [ 1.834334] systemd[1]: Listening on D-Bus System Message Bus Socket.1726worker2 # [ 1.834502] systemd[1]: Listening on Journal Socket (/dev/log).1727worker2 # [ 1.834670] systemd[1]: Listening on Journal Sockets.1728worker2 # [ 1.834813] systemd[1]: Listening on udev Control Socket.1729lb # [ 1.776112] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01730worker2 # [ 1.834933] systemd[1]: Listening on udev Kernel Socket.1731worker2 # [ 1.834983] systemd[1]: Reached target Socket Units.1732worker2 # [ 1.844164] systemd[1]: Starting Create List of Static Device Nodes...1733worker2 # [ 1.845377] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1734worker2 # [ 1.857454] systemd[1]: Mounting Kernel Configuration File System...1735worker2 # [ 1.887033] systemd[1]: Starting Journal Service...1736worker1 # [ 1.780534] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21737worker2 # [ 1.889915] systemd[1]: Starting Load Kernel Modules...1738worker2 # [ 1.890966] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1739worker1 # [ 1.786821] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01740worker2 # [ 1.902324] systemd[1]: Starting Coldplug All udev Devices...1741client # [ 1.828827] systemd[1]: bpf-restrict-fs: LSM BPF program attached1742worker2 # [ 1.918338] systemd[1]: Finished Create List of Static Device Nodes.1743worker2 # [ 1.919339] systemd[1]: Mounted Kernel Configuration File System.1744worker1 # [ 1.818940] systemd[1]: Queued start job for default target Initrd Default Target.1745lb # [ 1.867236] systemd[1]: Queued start job for default target Initrd Default Target.1746worker2 # [ 1.931815] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1747lb # [ 1.874160] systemd[1]: Created slice Slice /system/modprobe.1748worker1 # [ 1.829239] systemd[1]: Created slice Slice /system/modprobe.1749worker1 # [ 1.830337] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1750lb # [ 1.875308] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1751lb # [ 1.876582] systemd[1]: Expecting device /dev/disk/by-label/nixos...1752lb # [ 1.877686] systemd[1]: Reached target Path Units.1753lb # [ 1.878433] systemd[1]: Reached target Slice Units.1754worker1 # [ 1.831545] systemd[1]: Expecting device /dev/disk/by-label/nixos...1755lb # [ 1.879175] systemd[1]: Reached target Swaps.1756worker1 # [ 1.831585] systemd[1]: Reached target Path Units.1757lb # [ 1.879867] systemd[1]: Reached target Timer Units.1758client # [ 1.858585] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input21759worker1 # [ 1.831605] systemd[1]: Reached target Slice Units.1760worker1 # [ 1.831623] systemd[1]: Reached target Swaps.1761lb # [ 1.880825] systemd[1]: Listening on D-Bus System Message Bus Socket.1762worker1 # [ 1.831640] systemd[1]: Reached target Timer Units.1763lb # [ 1.881964] systemd[1]: Listening on Journal Socket (/dev/log).1764worker1 # [ 1.831845] systemd[1]: Listening on D-Bus System Message Bus Socket.1765worker1 # [ 1.831983] systemd[1]: Listening on Journal Socket (/dev/log).1766lb # [ 1.882996] systemd[1]: Listening on Journal Sockets.1767worker1 # [ 1.832110] systemd[1]: Listening on Journal Sockets.1768lb # [ 1.883920] systemd[1]: Listening on udev Control Socket.1769worker1 # [ 1.832224] systemd[1]: Listening on udev Control Socket.1770worker2 # [ 1.941357] systemd-journald[72]: Collecting audit messages is disabled.1771lb # [ 1.884828] systemd[1]: Listening on udev Kernel Socket.1772worker1 # [ 1.832325] systemd[1]: Listening on udev Kernel Socket.1773worker1 # [ 1.832345] systemd[1]: Reached target Socket Units.1774lb # [ 1.885606] systemd[1]: Reached target Socket Units.1775client # [ 1.866260] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input01776worker1 # [ 1.841737] systemd[1]: Starting Create List of Static Device Nodes...1777lb # [ 1.888184] systemd[1]: Starting Create List of Static Device Nodes...1778worker1 # [ 1.841834] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1779lb # [ 1.889271] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1780worker1 # [ 1.847756] systemd[1]: Mounting Kernel Configuration File System...1781lb # [ 1.896880] systemd[1]: Mounting Kernel Configuration File System...1782worker1 # [ 1.858046] systemd[1]: Starting Journal Service...1783lb # [ 1.908136] systemd[1]: Starting Journal Service...1784worker1 # [ 1.875302] systemd[1]: Starting Load Kernel Modules...1785worker1 # [ 1.875450] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1786worker2 # [ 1.986819] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1787lb # [ 1.928105] systemd[1]: Starting Load Kernel Modules...1788lb # [ 1.929050] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1789worker2 # [ 1.994401] systemd[1]: Starting Create Static Device Nodes in /dev...1790worker1 # [ 1.889892] systemd[1]: Starting Coldplug All udev Devices...1791lb # [ 1.936146] systemd[1]: Starting Coldplug All udev Devices...1792worker2 # [ 2.003613] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1793worker1 # [ 1.902873] systemd[1]: Finished Create List of Static Device Nodes.1794worker1 # [ 1.903786] systemd[1]: Mounted Kernel Configuration File System.1795worker2 # [ 2.014123] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1796lb # [ 1.955967] systemd[1]: Finished Create List of Static Device Nodes.1797lb # [ 1.956795] systemd[1]: Mounted Kernel Configuration File System.1798client # [ 1.946437] systemd[1]: Queued start job for default target Initrd Default Target.1799worker1 # [ 1.919337] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1800lb # [ 1.968552] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1801worker2 # [ 2.032234] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01802worker2 # [ 2.032508] [drm] features: -virgl +edid -resource_blob -host_visible1803worker2 # [ 2.032522] [drm] features: -context_init1804worker2 # [ 2.033291] [drm] number of scanouts: 11805worker2 # [ 2.033314] [drm] number of cap sets: 01806client # [ 1.958329] systemd[1]: Created slice Slice /system/modprobe.1807client # [ 1.959759] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.1808client # [ 1.961187] systemd[1]: Expecting device /dev/disk/by-label/nixos...1809client # [ 1.962425] systemd[1]: Reached target Path Units.1810client # [ 1.963286] systemd[1]: Reached target Slice Units.1811client # [ 1.964143] systemd[1]: Reached target Swaps.1812client # [ 1.965091] systemd[1]: Reached target Timer Units.1813client # [ 1.966311] systemd[1]: Listening on D-Bus System Message Bus Socket.1814client # [ 1.967725] systemd[1]: Listening on Journal Socket (/dev/log).1815client # [ 1.969021] systemd[1]: Listening on Journal Sockets.1816client # [ 1.970187] systemd[1]: Listening on udev Control Socket.1817client # [ 1.971315] systemd[1]: Listening on udev Kernel Socket.1818client # [ 1.972257] systemd[1]: Reached target Socket Units.1819client # [ 1.975836] systemd[1]: Starting Create List of Static Device Nodes...1820lb # [ 1.995013] systemd-journald[72]: Collecting audit messages is disabled.1821client # [ 1.977157] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs1822worker1 # [ 1.956986] systemd-journald[72]: Collecting audit messages is disabled.1823worker2 # [ 2.059782] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1824worker2 # [ 2.059815] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01825worker2 # [ 2.068938] systemd[1]: Finished Create Static Device Nodes in /dev.1826worker2 # [ 2.069398] systemd[1]: Reached target Preparation for Local File Systems.1827worker2 # [ 2.069433] systemd[1]: Reached target Local File Systems.1828client # [ 1.990128] systemd[1]: Mounting Kernel Configuration File System...1829lb # [ 2.013487] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1830client # [ 2.002789] systemd[1]: Starting Journal Service...1831worker2 # [ 2.078520] systemd[1]: Starting Rule-based Manager for Device Events and Files...1832worker1 # [ 1.973789] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1833worker1 # [ 1.980891] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1834lb # [ 2.032782] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1835worker2 # [ 2.090377] Console: switching to colour frame buffer device 160x501836worker1 # [ 1.991160] systemd[1]: Starting Create Static Device Nodes in /dev...1837lb # [ 2.045084] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01838client # [ 2.030072] systemd[1]: Starting Load Kernel Modules...1839lb # [ 2.045323] [drm] features: -virgl +edid -resource_blob -host_visible1840lb # [ 2.045333] [drm] features: -context_init1841client # [ 2.030245] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os1842lb # [ 2.046066] [drm] number of scanouts: 11843lb # [ 2.046085] [drm] number of cap sets: 01844client # [ 2.033658] systemd[1]: Starting Coldplug All udev Devices...1845lb # [ 2.047022] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1846worker1 # [ 2.002818] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1847worker1 # [ 2.009299] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01848worker1 # [ 2.009556] [drm] features: -virgl +edid -resource_blob -host_visible1849worker1 # [ 2.009570] [drm] features: -context_init1850worker1 # [ 2.010359] [drm] number of scanouts: 11851worker1 # [ 2.010382] [drm] number of cap sets: 01852worker2 # [ 2.118579] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1853lb # [ 2.056301] systemd[1]: Starting Create Static Device Nodes in /dev...1854lb # [ 2.068777] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1855lb # [ 2.068798] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01856worker2 # [ 2.134708] systemd[1]: Finished Load Kernel Modules.1857client # [ 2.057592] systemd-journald[72]: Collecting audit messages is disabled.1858worker2 # [ 2.125885] systemd-modules-load[73]: Inserted module 'dm_mod'1859worker1 # [ 2.035074] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1860worker1 # [ 2.035101] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01861worker2 # [ 2.145245] systemd[1]: Starting Apply Kernel Variables...1862client # [ 2.066039] systemd[1]: Finished Create List of Static Device Nodes.1863client # [ 2.069253] systemd[1]: Starting Create Static Device Nodes in /dev gracefully...1864worker2 # [ 2.147758] systemd[1]: Started Journal Service.1865client # [ 2.070582] systemd[1]: Mounted Kernel Configuration File System.1866worker2 # [ 2.136382] systemd-modules-load[73]: Module 'virtio_balloon' is built in1867worker2 # [ 2.137629] systemd-modules-load[73]: Module 'virtio_console' is built in1868worker2 # [ 2.138779] systemd-modules-load[73]: Inserted module 'virtio_gpu'1869worker2 # [ 2.139765] systemd-modules-load[73]: Module 'virtio_rng' is built in1870lb # [ 2.091849] Console: switching to colour frame buffer device 160x501871worker1 # [ 2.063472] Console: switching to colour frame buffer device 160x501872worker1 # [ 2.070248] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1873lb # [ 2.116480] systemd[1]: Finished Create Static Device Nodes in /dev.1874lb # [ 2.116880] systemd[1]: Reached target Preparation for Local File Systems.1875lb # [ 2.116911] systemd[1]: Reached target Local File Systems.1876worker1 # [ 2.075499] systemd[1]: Finished Create Static Device Nodes in /dev.1877worker2 # [ 2.167375] systemd[1]: Starting Create System Files and Directories...1878worker1 # [ 2.075965] systemd[1]: Reached target Preparation for Local File Systems.1879worker1 # [ 2.076009] systemd[1]: Reached target Local File Systems.1880lb # [ 2.124198] systemd[1]: Starting Rule-based Manager for Device Events and Files...1881lb # [ 2.129608] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1882worker2 # [ 2.175982] systemd-udevd[79]: Using default interface naming scheme 'v261'.1883client # [ 2.117391] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log.1884worker1 # [ 2.087255] systemd[1]: Starting Rule-based Manager for Device Events and Files...1885lb # [ 2.147964] systemd[1]: Finished Load Kernel Modules.1886worker1 # [ 2.105972] systemd[1]: Finished Load Kernel Modules.1887lb # [ 2.153231] systemd[1]: Starting Apply Kernel Variables...1888client # [ 2.133894] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev1889client # [ 2.140040] [drm] pci: virtio-gpu-pci detected at 0000:00:05.01890client # [ 2.140339] [drm] features: -virgl +edid -resource_blob -host_visible1891client # [ 2.140349] [drm] features: -context_init1892worker1 # [ 2.115145] systemd[1]: Starting Apply Kernel Variables...1893client # [ 2.141196] [drm] number of scanouts: 11894client # [ 2.141216] [drm] number of cap sets: 01895worker2 # [ 2.213334] systemd[1]: Finished Apply Kernel Variables.1896client # [ 2.154473] systemd[1]: Finished Create Static Device Nodes in /dev gracefully.1897client # [ 2.157141] systemd[1]: Starting Create Static Device Nodes in /dev...1898lb # [ 2.169277] systemd-modules-load[73]: Inserted module 'dm_mod'1899worker2 # [ 2.228748] systemd[1]: Finished Create System Files and Directories.1900lb # [ 2.186280] systemd[1]: Started Journal Service.1901lb # [ 2.173994] systemd-modules-load[73]: Module 'virtio_balloon' is built in1902lb # [ 2.175190] systemd-modules-load[73]: Module 'virtio_console' is built in1903worker2 # [ 2.234685] systemd[1]: Started Rule-based Manager for Device Events and Files.1904client # [ 2.170161] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic1905client # [ 2.170184] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 01906worker1 # [ 2.140420] systemd-modules-load[73]: Inserted module 'dm_mod'1907worker1 # [ 2.156271] systemd[1]: Started Journal Service.1908worker1 # [ 2.145434] systemd-modules-load[73]: Module 'virtio_balloon' is built in1909lb # [ 2.192201] systemd-modules-load[73]: Inserted module 'virtio_gpu'1910lb # [ 2.193302] systemd-modules-load[73]: Module 'virtio_rng' is built in1911lb # [ 2.194341] systemd[1]: Starting Create System Files and Directories...1912worker1 # [ 2.156103] systemd-modules-load[73]: Module 'virtio_console' is built in1913client # [ 2.198110] Console: switching to colour frame buffer device 160x501914worker1 # [ 2.163517] systemd-modules-load[73]: Inserted module 'virtio_gpu'1915client # [ 2.204935] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device1916worker1 # [ 2.172476] systemd-modules-load[73]: Module 'virtio_rng' is built in1917worker1 # [ 2.173562] systemd[1]: Starting Create System Files and Directories...1918client # [ 2.218518] systemd[1]: Finished Create Static Device Nodes in /dev.1919client # [ 2.218751] systemd[1]: Reached target Preparation for Local File Systems.1920client # [ 2.218785] systemd[1]: Reached target Local File Systems.1921lb # [ 2.225281] systemd[1]: Finished Apply Kernel Variables.1922client # [ 2.223986] systemd[1]: Starting Rule-based Manager for Device Events and Files...1923worker1 # [ 2.184174] systemd[1]: Finished Apply Kernel Variables.1924lb # [ 2.229491] systemd-udevd[78]: Using default interface naming scheme 'v261'.1925worker2 # [ 2.296504] systemd[1]: Starting Virtual Console Setup...1926client # [ 2.218333] systemd-modules-load[73]: Inserted module 'dm_mod'1927client # [ 2.219476] systemd-modules-load[73]: Module 'virtio_balloon' is built in1928client # [ 2.220833] systemd-modules-load[73]: Module 'virtio_console' is built in1929client # [ 2.239884] systemd[1]: Started Journal Service.1930lb # [ 2.242350] systemd[1]: Finished Create System Files and Directories.1931worker1 # [ 2.201104] systemd-udevd[79]: Using default interface naming scheme 'v261'.1932client # [ 2.228376] systemd-modules-load[73]: Inserted module 'virtio_gpu'1933client # [ 2.229664] systemd-modules-load[73]: Module 'virtio_rng' is built in1934worker1 # [ 2.206437] systemd[1]: Finished Create System Files and Directories.1935client # [ 2.237200] systemd[1]: Finished Load Kernel Modules.1936lb # [ 2.262191] systemd[1]: Started Rule-based Manager for Device Events and Files.1937worker1 # [ 2.230670] systemd[1]: Started Rule-based Manager for Device Events and Files.1938client # [ 2.261343] systemd[1]: Starting Apply Kernel Variables...1939client # [ 2.264611] systemd[1]: Starting Create System Files and Directories...1940worker2 # [ 2.352595] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1941worker2 # [ 2.356486] systemd[1]: Finished Virtual Console Setup.1942client # [ 2.303286] systemd-udevd[79]: Using default interface naming scheme 'v261'.1943lb # [ 2.328233] systemd[1]: Starting Virtual Console Setup...1944worker1 # [ 2.297321] systemd[1]: Starting Virtual Console Setup...1945client # [ 2.328222] systemd[1]: Finished Create System Files and Directories.1946client # [ 2.331691] systemd[1]: Finished Apply Kernel Variables.1947client # [ 2.349293] systemd[1]: Started Rule-based Manager for Device Events and Files.1948lb # [ 2.376530] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1949lb # [ 2.379377] systemd[1]: Finished Virtual Console Setup.1950worker1 # [ 2.352617] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1951worker1 # [ 2.356417] systemd[1]: Finished Virtual Console Setup.1952client # [ 2.408308] systemd[1]: Starting Virtual Console Setup...1953client # [ 2.472513] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones.1954client # [ 2.476068] systemd[1]: Finished Virtual Console Setup.1955worker2 # [ 2.954399] systemd[1]: Finished Coldplug All udev Devices.1956worker2 # [ 2.955411] systemd[1]: Reached target System Initialization.1957worker2 # [ 2.956915] systemd[1]: Reached target Basic System.1958lb # [ 2.996456] systemd[1]: Finished Coldplug All udev Devices.1959lb # [ 2.998115] systemd[1]: Reached target System Initialization.1960lb # [ 2.998992] systemd[1]: Reached target Basic System.1961worker1 # [ 2.993676] systemd[1]: Finished Coldplug All udev Devices.1962worker1 # [ 2.994656] systemd[1]: Reached target System Initialization.1963worker1 # [ 2.995488] systemd[1]: Reached target Basic System.1964worker2 # [ 3.106102] (udev-worker)[95]: Network interface NamePolicy= disabled on kernel command line.1965worker2 # [ 3.146323] (udev-worker)[90]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1966client # [ 3.070192] systemd[1]: Finished Coldplug All udev Devices.1967client # [ 3.071300] systemd[1]: Reached target System Initialization.1968client # [ 3.072398] systemd[1]: Reached target Basic System.1969worker2 # [ 3.150470] (udev-worker)[90]: Network interface NamePolicy= disabled on kernel command line.1970lb # [ 3.139285] (udev-worker)[104]: Network interface NamePolicy= disabled on kernel command line.1971worker2 # [ 3.206531] systemd[1]: Found device /dev/disk/by-label/nixos.1972worker2 # [ 3.209001] systemd[1]: Reached target Initrd Root Device.1973lb # [ 3.152338] (udev-worker)[93]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1974worker2 # [ 3.213687] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1975lb # [ 3.174221] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line.1976worker1 # [ 3.144235] (udev-worker)[100]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1977worker2 # [ 3.272489] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks1978worker2 # [ 3.280092] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.1979worker1 # [ 3.183857] (udev-worker)[100]: Network interface NamePolicy= disabled on kernel command line.1980worker2 # [ 3.288373] systemd[1]: Mounting /sysroot...1981worker1 # [ 3.187091] (udev-worker)[97]: Network interface NamePolicy= disabled on kernel command line.1982lb # [ 3.257729] systemd[1]: Found device /dev/disk/by-label/nixos.1983lb # [ 3.264172] systemd[1]: Reached target Initrd Root Device.1984lb # [ 3.269361] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1985worker2 # [ 3.346190] EXT4-fs (vda): mounted filesystem 40ecc9c7-9fd9-4070-b4d7-d15454910257 r/w with ordered data mode. Quota mode: none.1986worker2 # [ 3.334544] systemd[1]: Mounted /sysroot.1987worker2 # [ 3.337846] systemd[1]: Reached target Initrd Root File System.1988worker2 # [ 3.344161] systemd[1]: Starting Mountpoints Configured in the Real Root...1989worker1 # [ 3.243887] systemd[1]: Found device /dev/disk/by-label/nixos.1990worker1 # [ 3.248503] systemd[1]: Reached target Initrd Root Device.1991worker1 # [ 3.256162] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos...1992client # [ 3.283796] (udev-worker)[93]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name.1993worker2 # [ 3.375009] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload.1994client # [ 3.297768] (udev-worker)[91]: Network interface NamePolicy= disabled on kernel command line.1995lb # [ 3.321043] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks1996client # [ 3.301884] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line.1997worker2 # [ 3.380317] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)...1998worker2 # [ 3.385313] systemd[1]: Reloading...1999lb # [ 3.328184] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos.2000lb # [ 3.332497] systemd[1]: Mounting /sysroot...
loading 2,381 hidden lines…
4382lb # [ 53.294386] niks3-server[973]: 2026/09/23 04:43:42 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete4383worker1 # [ 53.250578] nix-grpc-daemon[861]: time=2026-09-23T04:43:42.749Z level=INFO msg="Upload complete. (277ms)"4384lb # [ 53.298134] niks3-server[973]: 2026/09/23 04:43:42 INFO Completed upload id=104385lb # [ 53.303143] niks3-server[973]: 2026/09/23 04:43:42 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.narinfo4386lb # [ 53.306886] niks3-server[973]: 2026/09/23 04:43:42 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.ls4387worker1 # [ 53.264251] nix-grpc-daemon[767]: ts=2026-09-23T04:43:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=1 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv assign_id=5 outputs=14388worker2 # [ 53.373218] nix-grpc-daemon[770]: ts=2026-09-23T04:43:42Z level=debug event=want client=4 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv cached=04389worker2 # [ 53.378811] nix-grpc-daemon[770]: ts=2026-09-23T04:43:42Z level=debug event=assigned drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker=192.168.1.3:50051 clients=14390worker1 # [ 53.282334] nix-daemon[834]: reaped child process 879, status = succeeded4391worker1 # [ 53.316987] nix-grpc-daemon[767]: ts=2026-09-23T04:43:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:571384392worker1 # [ 53.332951] nix-grpc-daemon[767]: ts=2026-09-23T04:43:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:571384393worker1 # [ 53.346164] nix-grpc-daemon[861]: time=2026-09-23T04:43:42.844Z level=INFO msg="All 1 paths already cached"4394client # copying 1 paths to 'lb:50051'...4395client # copying path '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' to 'lb:50051'...4396worker1 # [ 53.358171] nix-grpc-daemon[767]: ts=2026-09-23T04:43:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:571384397worker1 # [ 53.372140] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)4398worker1 # [ 53.523134] nix-daemon[834]: accepted connection from pid 904, user nix-grpc-daemon (trusted)4399worker1 # [ 53.544809] nix-daemon[834]: reaped child process 907, status = succeeded4400lb # [ 53.593153] niks3-server[973]: 2026/09/23 04:43:43 INFO Received uploads request method=POST path=/api/pending_closures4401worker1 # [ 53.559095] nix-grpc-daemon[861]: time=2026-09-23T04:43:43.057Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4402worker1 # [ 53.560842] nix-grpc-daemon[861]: time=2026-09-23T04:43:43.057Z level=INFO msg="Uploading z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv (624B)"4403worker1 # [ 53.591071] nix-grpc-daemon[861]: time=2026-09-23T04:43:43.089Z level=INFO msg="Uploading 1 narinfos"4404lb # [ 53.625324] niks3-server[973]: 2026/09/23 04:43:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign4405lb # [ 53.639481] niks3-server[973]: 2026/09/23 04:43:43 INFO Signed narinfos id=11 count=14406lb # [ 53.647916] niks3-server[973]: 2026/09/23 04:43:43 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.ls4407lb # [ 53.662606] niks3-server[973]: 2026/09/23 04:43:43 INFO Registered completed upload object_key=nar/0cnhkgsbmxv5hj7zw3fa59qfxikw6251mlhz2gz1v6pgn8c8shwp.nar.zst4408worker1 # [ 53.626259] nix-grpc-daemon[861]: time=2026-09-23T04:43:43.124Z level=INFO msg="Upload complete. (239ms)"4409lb # [ 53.671187] niks3-server[973]: 2026/09/23 04:43:43 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete4410lb # [ 53.680856] niks3-server[973]: 2026/09/23 04:43:43 INFO Completed upload id=114411worker1 # [ 53.636421] nix-grpc-daemon[767]: ts=2026-09-23T04:43:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=0 paths=1 nar_bytes_in=6244412lb # [ 53.683707] niks3-server[973]: 2026/09/23 04:43:43 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.narinfo4413worker1 # [ 53.654947] nix-grpc-daemon[767]: ts=2026-09-23T04:43:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:571384414worker1 # [ 53.661837] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)4415worker1 # [ 53.667105] nix-daemon[834]: reaped child process 902, status = succeeded4416client # building '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' on lb:50051...4417worker1 # [ 53.681910] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)4418client # farm-b> node-a: building z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv4419client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds4420client # farm-b> LOG-farm-b-sig4421worker1 # [ 55.358997] nix-daemon[834]: reaped child process 911, status = succeeded4422worker1 # [ 55.488144] nix-daemon[834]: accepted connection from pid 927, user nix-grpc-daemon (trusted)4423worker1 # [ 55.510746] nix-daemon[834]: reaped child process 930, status = succeeded4424lb # [ 55.553180] niks3-server[973]: 2026/09/23 04:43:45 INFO Received uploads request method=POST path=/api/pending_closures4425worker1 # [ 55.520519] nix-grpc-daemon[861]: time=2026-09-23T04:43:45.018Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4426worker1 # [ 55.522065] nix-grpc-daemon[861]: time=2026-09-23T04:43:45.018Z level=INFO msg="Uploading h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b (176B)"4427lb # [ 55.593386] niks3-server[973]: 2026/09/23 04:43:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign4428worker1 # [ 55.558658] nix-grpc-daemon[861]: time=2026-09-23T04:43:45.057Z level=INFO msg="Uploading 1 narinfos"4429lb # [ 55.607177] niks3-server[973]: 2026/09/23 04:43:45 INFO Signed narinfos id=12 count=14430lb # [ 55.615262] niks3-server[973]: 2026/09/23 04:43:45 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.ls4431lb # [ 55.624323] niks3-server[973]: 2026/09/23 04:43:45 INFO Registered completed upload object_key=log/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv4432lb # [ 55.632162] niks3-server[973]: 2026/09/23 04:43:45 INFO Registered completed upload object_key=nar/0f75gpfkjj0g9xi2113yvvpdw0l16spc2f1xik1ifnzfbrvx4pfk.nar.zst4433lb # [ 55.639624] niks3-server[973]: 2026/09/23 04:43:45 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete4434worker1 # [ 55.595695] nix-grpc-daemon[861]: time=2026-09-23T04:43:45.094Z level=INFO msg="Upload complete. (297ms)"4435lb # [ 55.643337] niks3-server[973]: 2026/09/23 04:43:45 INFO Completed upload id=124436lb # [ 55.647172] niks3-server[973]: 2026/09/23 04:43:45 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.narinfo4437worker1 # [ 55.610784] nix-grpc-daemon[767]: ts=2026-09-23T04:43:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=1 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv assign_id=6 outputs=14438worker2 # [ 55.719278] nix-grpc-daemon[770]: ts=2026-09-23T04:43:45Z level=debug event=want client=4 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv cached=04439worker2 # [ 55.725327] nix-grpc-daemon[770]: ts=2026-09-23T04:43:45Z level=debug event=assigned drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker=192.168.1.3:50051 clients=14440worker1 # [ 55.632866] nix-daemon[834]: reaped child process 909, status = succeeded4441worker1 # [ 55.661020] nix-grpc-daemon[767]: ts=2026-09-23T04:43:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:571384442worker1 # [ 55.678498] nix-grpc-daemon[767]: ts=2026-09-23T04:43:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:571384443worker1 # [ 55.694768] nix-grpc-daemon[861]: time=2026-09-23T04:43:45.193Z level=INFO msg="All 2 paths already cached"4444client # copying 1 paths to 'lb:50051'...4445client # copying path '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' to 'lb:50051'...4446worker1 # [ 55.706732] nix-grpc-daemon[767]: ts=2026-09-23T04:43:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:571384447worker1 # [ 55.721241] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)4448worker1 # [ 55.877723] nix-daemon[834]: accepted connection from pid 934, user nix-grpc-daemon (trusted)4449worker1 # [ 55.900849] nix-daemon[834]: reaped child process 937, status = succeeded4450lb # [ 55.947648] niks3-server[973]: 2026/09/23 04:43:45 INFO Received uploads request method=POST path=/api/pending_closures4451worker1 # [ 55.912618] nix-grpc-daemon[861]: time=2026-09-23T04:43:45.411Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4452worker1 # [ 55.914840] nix-grpc-daemon[861]: time=2026-09-23T04:43:45.413Z level=INFO msg="Uploading knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv (752B)"4453lb # [ 55.980679] niks3-server[973]: 2026/09/23 04:43:45 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.ls4454lb # [ 55.993984] niks3-server[973]: 2026/09/23 04:43:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign4455worker1 # [ 55.956193] nix-grpc-daemon[861]: time=2026-09-23T04:43:45.454Z level=INFO msg="Uploading 1 narinfos"4456lb # [ 56.006419] niks3-server[973]: 2026/09/23 04:43:45 INFO Signed narinfos id=13 count=14457lb # [ 56.012112] niks3-server[973]: 2026/09/23 04:43:45 INFO Registered completed upload object_key=nar/0lpagyrngav67x80d1hq0njjlkmqdkvqawraz8dsp36ag50v6iwf.nar.zst4458lb # [ 56.022767] niks3-server[973]: 2026/09/23 04:43:45 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete4459lb # [ 56.029548] niks3-server[973]: 2026/09/23 04:43:45 INFO Completed upload id=134460worker1 # [ 55.983993] nix-grpc-daemon[861]: time=2026-09-23T04:43:45.482Z level=INFO msg="Upload complete. (245ms)"4461lb # [ 56.031640] niks3-server[973]: 2026/09/23 04:43:45 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.narinfo4462worker1 # [ 55.995387] nix-grpc-daemon[767]: ts=2026-09-23T04:43:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=0 paths=1 nar_bytes_in=7524463worker1 # [ 56.011967] nix-grpc-daemon[767]: ts=2026-09-23T04:43:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:571384464worker1 # [ 56.018270] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)4465worker1 # [ 56.024320] nix-daemon[834]: reaped child process 932, status = succeeded4466client # building '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' on lb:50051...4467worker1 # [ 56.037355] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)4468client # farm-top> node-a: building knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv4469client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds4470client # farm-top> LOG-farm-top-sig4471worker1 # [ 57.740315] nix-daemon[834]: reaped child process 941, status = succeeded4472worker1 # [ 57.811305] nix-daemon[834]: accepted connection from pid 957, user nix-grpc-daemon (trusted)4473worker1 # [ 57.831130] nix-daemon[834]: reaped child process 960, status = succeeded4474lb # [ 57.880848] niks3-server[973]: 2026/09/23 04:43:47 INFO Received uploads request method=POST path=/api/pending_closures4475worker1 # [ 57.849302] nix-grpc-daemon[861]: time=2026-09-23T04:43:47.346Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4476worker1 # [ 57.850964] nix-grpc-daemon[861]: time=2026-09-23T04:43:47.346Z level=INFO msg="Uploading 4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top (232B)"4477lb # [ 57.933655] niks3-server[973]: 2026/09/23 04:43:47 INFO Registered completed upload object_key=log/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv4478lb # [ 57.964682] niks3-server[973]: 2026/09/23 04:43:47 INFO Registered completed upload object_key=nar/07ar364rwqqniwi2nkdpqlgv7ws7702n5lqz703ackfmssrh40v8.nar.zst4479worker1 # [ 57.931997] nix-grpc-daemon[861]: time=2026-09-23T04:43:47.430Z level=INFO msg="Uploading 1 narinfos"4480lb # [ 57.980037] niks3-server[973]: 2026/09/23 04:43:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign4481lb # [ 57.983597] niks3-server[973]: 2026/09/23 04:43:47 INFO Signed narinfos id=14 count=14482lb # [ 57.992293] niks3-server[973]: 2026/09/23 04:43:47 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.ls4483lb # [ 58.004149] niks3-server[973]: 2026/09/23 04:43:47 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete4484lb # [ 58.011022] niks3-server[973]: 2026/09/23 04:43:47 INFO Completed upload id=144485worker1 # [ 57.967784] nix-grpc-daemon[861]: time=2026-09-23T04:43:47.466Z level=INFO msg="Upload complete. (318ms)"4486lb # [ 58.014989] niks3-server[973]: 2026/09/23 04:43:47 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.narinfo4487worker1 # [ 57.981491] nix-grpc-daemon[767]: ts=2026-09-23T04:43:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=1 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv assign_id=7 outputs=14488worker1 # [ 58.004289] nix-daemon[834]: reaped child process 939, status = succeeded4489client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2, in 7.26 seconds)4490client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath4491client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath, in 0.19 seconds)4492client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top4493worker1 # [ 58.377213] nix-grpc-daemon[767]: ts=2026-09-23T04:43:47Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:451004494worker1 # [ 58.389618] nix-grpc-daemon[767]: ts=2026-09-23T04:43:47Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:451004495client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.23 seconds)4496client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top4497worker2 # [ 58.762874] nix-grpc-daemon[770]: ts=2026-09-23T04:43:48Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:600584498client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked):4499client # /nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a4500client # /nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b4501client # /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top4502worker2 # [ 58.804833] nix-grpc-daemon[770]: ts=2026-09-23T04:43:48Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:600584503worker2 # [ 58.815437] nix-grpc-daemon[770]: copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'http://lb:5751'...4504worker2 # [ 58.837977] nix-grpc-daemon[770]: copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'http://lb:5751'...4505worker2 # [ 58.854392] nix-grpc-daemon[770]: copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'http://lb:5751'...4506client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.35 seconds)4507client: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top4508client # finding garbage collector roots...4509client # removing stale temporary roots file "/nix/var/nix/temproots/940"4510client # removing stale temporary roots file "/nix/var/nix/temproots/896"4511client # removing stale temporary roots file "/nix/var/nix/temproots/884"4512client # removing stale temporary roots file "/nix/var/nix/temproots/925"4513client # removing stale temporary roots file "/nix/var/nix/temproots/917"4514client # removing stale temporary roots file "/nix/var/nix/temproots/904"4515client # removing stale temporary roots file "/nix/var/nix/temproots/846"4516client # removing stale temporary roots file "/nix/var/nix/temproots/863"4517client # deleting specified paths...4518client # deleting '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top'4519client # deleting unused links...4520client # note: hard linking is currently saving 0.0 KiB4521worker2 # [ 59.292673] nix-grpc-daemon[770]: ts=2026-09-23T04:43:48Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:516484522worker1 # [ 59.206310] nix-grpc-daemon[767]: ts=2026-09-23T04:43:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:571384523worker2 # [ 59.325121] nix-grpc-daemon[770]: ts=2026-09-23T04:43:48Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:516484524worker2 # [ 59.335977] nix-grpc-daemon[770]: ts=2026-09-23T04:43:48Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:516484525client # copying 3 paths...4526worker2 # [ 59.351753] nix-grpc-daemon[770]: ts=2026-09-23T04:43:48Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:516484527client # copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'grpc://lb:50051'...4528worker2 # [ 59.384686] nix-grpc-daemon[770]: ts=2026-09-23T04:43:48Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:516484529worker2 # [ 59.394340] nix-grpc-daemon[770]: ts=2026-09-23T04:43:48Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=0 paths=1 nar_bytes_out=2324530worker2 # [ 59.403585] nix-grpc-daemon[770]: ts=2026-09-23T04:43:48Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:516484531worker2 # [ 59.412120] nix-grpc-daemon[770]: ts=2026-09-23T04:43:48Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=0 paths=1 nar_bytes_out=1764532worker1 # [ 59.213801] nix-grpc-daemon[861]: time=2026-09-23T04:43:48.711Z level=INFO msg="All 1 paths already cached"4533worker1 # [ 59.317942] nix-grpc-daemon[767]: ts=2026-09-23T04:43:48Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:571384534worker1 # [ 59.330190] nix-grpc-daemon[767]: ts=2026-09-23T04:43:48Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=0 paths=1 nar_bytes_out=1284535client # copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'grpc://lb:50051'...4536client # copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'grpc://lb:50051'...4537lb # [ 59.418564] envoy[900]: conn peer=[2001:db8:1::1]:59244 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4198 tx=1896 ms=23914538client: (finished: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.60 seconds)4539(finished: subtest: outputs read back through any worker carry the cache signature, in 8.63 seconds)4540subtest: two clients wanting the same drv build it once4541client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup4542client # [ 59.477678] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup.4543client # Running as unit: dedup1.service; invocation ID: d8b140bc51344b3386e2f7f3421d3c134544client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 0.07 seconds)4545client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup4546worker1 # [ 59.667991] nix-grpc-daemon[767]: ts=2026-09-23T04:43:49Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:571384547client # [ 59.701575] nix[993]: don't know how to build these paths:4548client # [ 59.702736] nix[993]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4549worker2 # [ 59.797576] nix-grpc-daemon[770]: ts=2026-09-23T04:43:49Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04550worker2 # [ 59.803877] nix-grpc-daemon[770]: ts=2026-09-23T04:43:49Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.4:50051 clients=14551worker2 # [ 59.842010] nix-grpc-daemon[770]: ts=2026-09-23T04:43:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:516484552worker2 # [ 59.862371] nix-grpc-daemon[770]: ts=2026-09-23T04:43:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:516484553client # [ 59.800383] nix[993]: copying 1 paths to 'lb:50051'...4554client # [ 59.806202] nix[993]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'...4555worker2 # [ 59.885515] nix-grpc-daemon[770]: ts=2026-09-23T04:43:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:516484556worker2 # [ 59.910977] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4557lb # [ 59.424370] envoy[900]: conn peer=[2001:db8:1::1]:53408 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9571 tx=4391 ms=70084558lb # [ 59.424660] envoy[900]: conn peer=[2001:db8:1::1]:53414 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1051 tx=2890 ms=1834559lb # [ 59.424845] envoy[900]: conn peer=[2001:db8:1::1]:53444 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=330 ms=574560lb # [ 59.425055] envoy[900]: conn peer=[2001:db8:1::1]:53438 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=376 ms=794561lb # [ 59.425160] envoy[900]: conn peer=[2001:db8:1::1]:53430 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=410 ms=974562worker2 # [ 60.135554] nix-daemon[856]: accepted connection from pid 1051, user nix-grpc-daemon (trusted)4563worker2 # [ 60.161785] nix-daemon[856]: reaped child process 1054, status = succeeded4564lb # [ 60.110394] niks3-server[973]: 2026/09/23 04:43:49 INFO Received uploads request method=POST path=/api/pending_closures4565worker2 # [ 60.186740] nix-grpc-daemon[871]: time=2026-09-23T04:43:49.707Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4566worker2 # [ 60.189743] nix-grpc-daemon[871]: time=2026-09-23T04:43:49.707Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)"4567worker2 # [ 60.222932] nix-grpc-daemon[871]: time=2026-09-23T04:43:49.744Z level=INFO msg="Uploading 1 narinfos"4568lb # [ 60.155949] niks3-server[973]: 2026/09/23 04:43:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign4569lb # [ 60.171267] niks3-server[973]: 2026/09/23 04:43:49 INFO Signed narinfos id=15 count=14570lb # [ 60.178653] niks3-server[973]: 2026/09/23 04:43:49 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls4571lb # [ 60.187594] niks3-server[973]: 2026/09/23 04:43:49 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst4572lb # [ 60.205820] niks3-server[973]: 2026/09/23 04:43:49 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete4573lb # [ 60.210259] niks3-server[973]: 2026/09/23 04:43:49 INFO Completed upload id=154574worker2 # [ 60.263755] nix-grpc-daemon[871]: time=2026-09-23T04:43:49.785Z level=INFO msg="Upload complete. (333ms)"4575lb # [ 60.213280] niks3-server[973]: 2026/09/23 04:43:49 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo4576worker2 # [ 60.278373] nix-grpc-daemon[770]: ts=2026-09-23T04:43:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=0 paths=1 nar_bytes_in=4324577worker2 # [ 60.293320] nix-grpc-daemon[770]: ts=2026-09-23T04:43:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:516484578worker2 # [ 60.309006] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4579worker2 # [ 60.319631] nix-daemon[856]: reaped child process 1049, status = succeeded4580client # [ 60.256349] nix[993]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4581worker2 # [ 60.337206] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4582client # [ 60.591266] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup.4583client # Running as unit: dedup2.service; invocation ID: d71b9e6348f640fcbd0c74308507d1a94584client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 1.12 seconds)4585worker1 # [ 60.827605] nix-grpc-daemon[767]: ts=2026-09-23T04:43:50Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:571384586client # [ 60.861583] nix[1009]: don't know how to build these paths:4587client # [ 60.862549] nix[1009]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4588worker2 # [ 60.953537] nix-grpc-daemon[770]: ts=2026-09-23T04:43:50Z level=debug event=want client=6 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=04589client # [ 60.884723] nix[1009]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051...4590worker2 # [ 60.968760] nix-grpc-daemon[770]: ts=2026-09-23T04:43:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:516484591client: must succeed: sleep 24592client: (finished: must succeed: sleep 2, in 2.08 seconds)4593client: waiting for success: ! systemctl is-active dedup1 dedup24594worker2 # [ 64.660096] nix-daemon[856]: reaped child process 1058, status = succeeded4595worker2 # [ 64.726504] nix-daemon[856]: accepted connection from pid 1090, user nix-grpc-daemon (trusted)4596worker2 # [ 64.750035] nix-daemon[856]: reaped child process 1093, status = succeeded4597lb # [ 64.690531] niks3-server[973]: 2026/09/23 04:43:54 INFO Received uploads request method=POST path=/api/pending_closures4598worker2 # [ 64.760663] nix-grpc-daemon[871]: time=2026-09-23T04:43:54.282Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4599worker2 # [ 64.765931] nix-grpc-daemon[871]: time=2026-09-23T04:43:54.284Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)"4600lb # [ 64.745010] niks3-server[973]: 2026/09/23 04:43:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign4601worker2 # [ 64.815731] nix-grpc-daemon[871]: time=2026-09-23T04:43:54.337Z level=INFO msg="Uploading 1 narinfos"4602lb # [ 64.757789] niks3-server[973]: 2026/09/23 04:43:54 INFO Signed narinfos id=16 count=14603lb # [ 64.766397] niks3-server[973]: 2026/09/23 04:43:54 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst4604lb # [ 64.775796] niks3-server[973]: 2026/09/23 04:43:54 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv4605lb # [ 64.783004] niks3-server[973]: 2026/09/23 04:43:54 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls4606lb # [ 64.791745] niks3-server[973]: 2026/09/23 04:43:54 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete4607lb # [ 64.799506] niks3-server[973]: 2026/09/23 04:43:54 INFO Completed upload id=164608worker2 # [ 64.857485] nix-grpc-daemon[871]: time=2026-09-23T04:43:54.379Z level=INFO msg="Upload complete. (296ms)"4609lb # [ 64.803032] niks3-server[973]: 2026/09/23 04:43:54 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo4610worker2 # [ 64.871869] nix-grpc-daemon[770]: ts=2026-09-23T04:43:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=14611worker2 # [ 64.892386] nix-grpc-daemon[770]: ts=2026-09-23T04:43:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=14612worker2 # [ 64.901149] nix-daemon[856]: reaped child process 1056, status = succeeded4613client # [ 64.862309] systemd[1]: dedup1.service: Deactivated successfully.4614client # [ 64.864099] systemd[1]: dedup1.service: Consumed 202ms CPU time over 5.386s wall clock time, 41.4M memory peak, 7.2K incoming IP traffic, 8.3K outgoing IP traffic.4615client # [ 64.878745] systemd[1]: dedup2.service: Deactivated successfully.4616client # [ 64.881778] systemd[1]: dedup2.service: Consumed 239ms CPU time over 4.292s wall clock time, 33.2M memory peak, 6K incoming IP traffic, 5.6K outgoing IP traffic.4617client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.13 seconds)4618client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }4619client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.06 seconds)4620worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4621worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4622worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true4623worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.07 seconds)4624worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics4625worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)4626worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics4627worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)4628(finished: subtest: two clients wanting the same drv build it once, in 6.39 seconds)4629subtest: --timeout and --max-silent-time reach the builder4630worker1 # [ 66.017740] nix-grpc-daemon[767]: ts=2026-09-23T04:43:55Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:571384631worker2 # [ 66.140844] nix-grpc-daemon[770]: ts=2026-09-23T04:43:55Z level=debug event=want client=7 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv cached=04632worker1 # [ 66.041893] nix-grpc-daemon[767]: ts=2026-09-23T04:43:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:571384633worker2 # [ 66.141225] nix-grpc-daemon[770]: ts=2026-09-23T04:43:55Z level=debug event=assigned drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv worker=192.168.1.3:50051 clients=14634worker1 # [ 66.059446] nix-grpc-daemon[767]: ts=2026-09-23T04:43:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:571384635worker1 # [ 66.074565] nix-grpc-daemon[767]: ts=2026-09-23T04:43:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:571384636worker1 # [ 66.092125] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)4637worker1 # [ 66.260090] nix-daemon[834]: accepted connection from pid 985, user nix-grpc-daemon (trusted)4638worker1 # [ 66.282968] nix-daemon[834]: reaped child process 988, status = succeeded4639lb # [ 66.330816] niks3-server[973]: 2026/09/23 04:43:55 INFO Received uploads request method=POST path=/api/pending_closures4640worker1 # [ 66.298434] nix-grpc-daemon[861]: time=2026-09-23T04:43:55.796Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4641worker1 # [ 66.299953] nix-grpc-daemon[861]: time=2026-09-23T04:43:55.796Z level=INFO msg="Uploading sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv (424B)"4642lb # [ 66.360354] niks3-server[973]: 2026/09/23 04:43:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign4643worker1 # [ 66.325683] nix-grpc-daemon[861]: time=2026-09-23T04:43:55.824Z level=INFO msg="Uploading 1 narinfos"4644lb # [ 66.371281] niks3-server[973]: 2026/09/23 04:43:55 INFO Signed narinfos id=17 count=14645lb # [ 66.381949] niks3-server[973]: 2026/09/23 04:43:55 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.ls4646lb # [ 66.391144] niks3-server[973]: 2026/09/23 04:43:55 INFO Registered completed upload object_key=nar/1m12xmj6yzmb9y2j8ly0yvh5fyzy1nqk7ajhqwr4wn3b5h0bsch5.nar.zst4647lb # [ 66.403132] niks3-server[973]: 2026/09/23 04:43:55 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete4648worker1 # [ 66.364668] nix-grpc-daemon[861]: time=2026-09-23T04:43:55.863Z level=INFO msg="Upload complete. (254ms)"4649lb # [ 66.411178] niks3-server[973]: 2026/09/23 04:43:55 INFO Completed upload id=174650lb # [ 66.415744] niks3-server[973]: 2026/09/23 04:43:55 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.narinfo4651worker1 # [ 66.376144] nix-grpc-daemon[767]: ts=2026-09-23T04:43:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=0 paths=1 nar_bytes_in=4244652worker1 # [ 66.390651] nix-grpc-daemon[767]: ts=2026-09-23T04:43:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:571384653worker1 # [ 66.398060] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)4654worker1 # [ 66.403737] nix-daemon[834]: reaped child process 983, status = succeeded4655worker1 # [ 66.416222] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)4656lb # [ 69.414460] envoy[900]: conn peer=[2001:db8:1::1]:53448 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3005 tx=1565 ms=51184657lb # [ 69.419606] envoy[900]: conn peer=[2001:db8:1::1]:56148 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1430 tx=1198 ms=40054658worker1 # [ 72.234948] nix-grpc-daemon[767]: ts=2026-09-23T04:44:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=5 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv assign_id=9 outputs=04659worker1 # [ 72.275824] nix-daemon[834]: reaped child process 990, status = succeeded4660worker1 # [ 72.279382] nix-daemon[834]: reaped child process 992, status = succeeded4661worker1 # [ 72.519756] nix-grpc-daemon[767]: ts=2026-09-23T04:44:02Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:571384662worker2 # [ 72.643786] nix-grpc-daemon[770]: ts=2026-09-23T04:44:02Z level=debug event=want client=8 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv cached=04663worker2 # [ 72.650825] nix-grpc-daemon[770]: ts=2026-09-23T04:44:02Z level=debug event=assigned drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv worker=192.168.1.4:50051 clients=14664worker2 # [ 72.658184] nix-grpc-daemon[770]: ts=2026-09-23T04:44:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:516484665worker2 # [ 72.670749] nix-grpc-daemon[770]: ts=2026-09-23T04:44:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:516484666worker2 # [ 72.686004] nix-grpc-daemon[770]: ts=2026-09-23T04:44:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:516484667worker2 # [ 72.701205] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4668worker2 # [ 72.853485] nix-daemon[856]: accepted connection from pid 1110, user nix-grpc-daemon (trusted)4669worker2 # [ 72.876057] nix-daemon[856]: reaped child process 1113, status = succeeded4670lb # [ 72.818507] niks3-server[973]: 2026/09/23 04:44:02 INFO Received uploads request method=POST path=/api/pending_closures4671worker2 # [ 72.885495] nix-grpc-daemon[871]: time=2026-09-23T04:44:02.406Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4672worker2 # [ 72.886969] nix-grpc-daemon[871]: time=2026-09-23T04:44:02.406Z level=INFO msg="Uploading pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv (424B)"4673lb # [ 72.850307] niks3-server[973]: 2026/09/23 04:44:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign4674worker2 # [ 72.919063] nix-grpc-daemon[871]: time=2026-09-23T04:44:02.440Z level=INFO msg="Uploading 1 narinfos"4675lb # [ 72.865126] niks3-server[973]: 2026/09/23 04:44:02 INFO Signed narinfos id=18 count=14676lb # [ 72.873701] niks3-server[973]: 2026/09/23 04:44:02 INFO Registered completed upload object_key=nar/14f5yrnllfav3svmw4v4svw15r4zlf5yv1p972jy14mmq6gdw943.nar.zst4677lb # [ 72.885055] niks3-server[973]: 2026/09/23 04:44:02 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.ls4678lb # [ 72.896757] niks3-server[973]: 2026/09/23 04:44:02 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete4679worker2 # [ 72.962526] nix-grpc-daemon[871]: time=2026-09-23T04:44:02.484Z level=INFO msg="Upload complete. (245ms)"4680lb # [ 72.907413] niks3-server[973]: 2026/09/23 04:44:02 INFO Completed upload id=184681lb # [ 72.911678] niks3-server[973]: 2026/09/23 04:44:02 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.narinfo4682worker2 # [ 72.972437] nix-grpc-daemon[770]: ts=2026-09-23T04:44:02Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=0 paths=1 nar_bytes_in=4244683worker2 # [ 72.992813] nix-grpc-daemon[770]: ts=2026-09-23T04:44:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:516484684worker2 # [ 72.999860] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4685worker2 # [ 73.005531] nix-daemon[856]: reaped child process 1108, status = succeeded4686worker2 # [ 73.019241] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4687worker2 # [ 79.236815] nix-grpc-daemon[770]: ts=2026-09-23T04:44:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=6 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv assign_id=10 outputs=04688worker2 # [ 79.277422] nix-daemon[856]: reaped child process 1117, status = succeeded4689worker2 # [ 79.281139] nix-daemon[856]: reaped child process 1115, status = succeeded4690(finished: subtest: --timeout and --max-silent-time reach the builder, in 13.58 seconds)4691subtest: the first client leaving does not fail the second4692client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share4693lb # [ 79.415327] envoy[900]: conn peer=[2001:db8:1::1]:56158 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3014 tx=953 ms=62834694client # [ 79.462131] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share.4695client # Running as unit: share1.service; invocation ID: 387e3433360048c38007bf6d7998e5c94696client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 0.09 seconds)4697client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share4698worker2 # [ 79.756259] nix-grpc-daemon[770]: ts=2026-09-23T04:44:09Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:516484699client # [ 79.685217] nix[1081]: don't know how to build these paths:4700client # [ 79.686217] nix[1081]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4701worker2 # [ 79.775314] nix-grpc-daemon[770]: ts=2026-09-23T04:44:09Z level=debug event=want client=9 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=04702worker1 # [ 79.677109] nix-grpc-daemon[767]: ts=2026-09-23T04:44:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:571384703worker2 # [ 79.781819] nix-grpc-daemon[770]: ts=2026-09-23T04:44:09Z level=debug event=assigned drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker=192.168.1.3:50051 clients=14704worker1 # [ 79.694784] nix-grpc-daemon[767]: ts=2026-09-23T04:44:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:571384705client # [ 79.732557] nix[1081]: copying 1 paths to 'lb:50051'...4706client # [ 79.735285] nix[1081]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' to 'lb:50051'...4707worker1 # [ 79.712116] nix-grpc-daemon[767]: ts=2026-09-23T04:44:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:571384708worker1 # [ 79.727355] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)4709worker1 # [ 79.892914] nix-daemon[834]: accepted connection from pid 1015, user nix-grpc-daemon (trusted)4710worker1 # [ 79.913250] nix-daemon[834]: reaped child process 1018, status = succeeded4711lb # [ 79.420373] envoy[900]: conn peer=[2001:db8:1::1]:60300 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2972 tx=937 ms=66724712lb # [ 79.965870] niks3-server[973]: 2026/09/23 04:44:09 INFO Received uploads request method=POST path=/api/pending_closures4713worker1 # [ 79.936605] nix-grpc-daemon[861]: time=2026-09-23T04:44:09.434Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4714worker1 # [ 79.938419] nix-grpc-daemon[861]: time=2026-09-23T04:44:09.434Z level=INFO msg="Uploading rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv (432B)"4715worker1 # [ 79.968776] nix-grpc-daemon[861]: time=2026-09-23T04:44:09.467Z level=INFO msg="Uploading 1 narinfos"4716lb # [ 80.005934] niks3-server[973]: 2026/09/23 04:44:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign4717lb # [ 80.019350] niks3-server[973]: 2026/09/23 04:44:09 INFO Signed narinfos id=19 count=14718lb # [ 80.029763] niks3-server[973]: 2026/09/23 04:44:09 INFO Registered completed upload object_key=nar/0chdjpc9an2ckdblh08lnyrmzvw3h8qs76cn5ydwjwvgwkrm731l.nar.zst4719lb # [ 80.043462] niks3-server[973]: 2026/09/23 04:44:09 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.ls4720lb # [ 80.058332] niks3-server[973]: 2026/09/23 04:44:09 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete4721worker1 # [ 80.019786] nix-grpc-daemon[861]: time=2026-09-23T04:44:09.518Z level=INFO msg="Upload complete. (272ms)"4722lb # [ 80.066639] niks3-server[973]: 2026/09/23 04:44:09 INFO Completed upload id=194723lb # [ 80.068819] niks3-server[973]: 2026/09/23 04:44:09 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.narinfo4724worker1 # [ 80.032442] nix-grpc-daemon[767]: ts=2026-09-23T04:44:09Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=0 paths=1 nar_bytes_in=4324725worker1 # [ 80.051439] nix-grpc-daemon[767]: ts=2026-09-23T04:44:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:571384726worker1 # [ 80.059753] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)4727worker1 # [ 80.066818] nix-daemon[834]: reaped child process 1013, status = succeeded4728worker1 # [ 80.079551] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)4729client # [ 80.103562] nix[1081]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051...4730client # [ 80.574290] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share.4731client # Running as unit: share2.service; invocation ID: 1e0414c1a5d34cd8832fd5acbcbf49524732client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 1.12 seconds)4733worker1 # [ 80.782414] nix-grpc-daemon[767]: ts=2026-09-23T04:44:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:571384734client # [ 80.823178] nix[1097]: this derivation will be built:4735worker2 # [ 80.906281] nix-grpc-daemon[770]: ts=2026-09-23T04:44:10Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:516484736worker2 # [ 80.914452] nix-grpc-daemon[770]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' from 'http://lb:5751'...4737client # [ 80.857670] nix[1097]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4738worker2 # [ 80.947818] nix-grpc-daemon[770]: ts=2026-09-23T04:44:10Z level=debug event=want client=10 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=04739worker1 # [ 80.864345] nix-grpc-daemon[767]: ts=2026-09-23T04:44:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:571384740client # [ 80.898626] nix[1097]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051...4741client: must succeed: sleep 2; systemctl kill -s INT share14742client # [ 83.928397] systemd[1]: share1.service: Sent signal SIGINT to main process 1081 (nix) on client request.4743client # [ 83.937536] nix[1081]: shutting downrxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)4744client # [ 83.950406] nix[1081]: error: interrupted by the user4745client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.09 seconds)4746client: waiting for success: ! systemctl is-active share14747client # [ 84.008260] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE4748client # [ 84.011221] systemd[1]: share1.service: Failed with result 'exit-code'.4749client # [ 84.013779] systemd[1]: share1.service: Consumed 182ms CPU time over 4.549s wall clock time, 41.4M memory peak, 6.2K incoming IP traffic, 8.2K outgoing IP traffic.4750client: (finished: waiting for success: ! systemctl is-active share1, in 1.11 seconds)4751client: waiting for success: ! systemctl is-active share24752worker1 # [ 85.312332] nix-daemon[834]: reaped child process 1022, status = succeeded4753worker1 # [ 85.418040] nix-daemon[834]: accepted connection from pid 1056, user nix-grpc-daemon (trusted)4754worker1 # [ 85.438141] nix-daemon[834]: reaped child process 1059, status = succeeded4755lb # [ 85.486476] niks3-server[973]: 2026/09/23 04:44:15 INFO Received uploads request method=POST path=/api/pending_closures4756worker1 # [ 85.456439] nix-grpc-daemon[861]: time=2026-09-23T04:44:14.952Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4757worker1 # [ 85.457996] nix-grpc-daemon[861]: time=2026-09-23T04:44:14.952Z level=INFO msg="Uploading fjww0fd4rwk1s35jav3cyal62l24n356-slow-share (120B)"4758lb # [ 85.530503] niks3-server[973]: 2026/09/23 04:44:15 INFO Registered completed upload object_key=log/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv4759worker1 # [ 85.497661] nix-grpc-daemon[861]: time=2026-09-23T04:44:14.996Z level=INFO msg="Uploading 1 narinfos"4760lb # [ 85.543315] niks3-server[973]: 2026/09/23 04:44:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign4761lb # [ 85.551998] niks3-server[973]: 2026/09/23 04:44:15 INFO Signed narinfos id=20 count=14762lb # [ 85.560097] niks3-server[973]: 2026/09/23 04:44:15 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete4763lb # [ 85.746064] niks3-server[973]: 2026/09/23 04:44:15 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.ls4764lb # [ 85.768709] niks3-server[973]: 2026/09/23 04:44:15 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst4765worker1 # [ 85.739060] nix-grpc-daemon[861]: time=2026-09-23T04:44:15.237Z level=INFO msg="Upload complete. (438ms)"4766lb # [ 85.782458] niks3-server[973]: 2026/09/23 04:44:15 INFO Completed upload id=204767lb # [ 85.788202] niks3-server[973]: 2026/09/23 04:44:15 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.narinfo4768worker1 # [ 85.753220] nix-grpc-daemon[767]: ts=2026-09-23T04:44:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=5 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=14769worker1 # [ 85.773436] nix-grpc-daemon[767]: ts=2026-09-23T04:44:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=4 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=14770worker1 # [ 85.781810] nix-daemon[834]: reaped child process 1020, status = succeeded4771client # [ 85.856391] systemd[1]: share2.service: Deactivated successfully.4772client # [ 85.860738] systemd[1]: share2.service: Consumed 207ms CPU time over 5.286s wall clock time, 35.3M memory peak, 6.6K incoming IP traffic, 5.9K outgoing IP traffic.4773client: (finished: waiting for success: ! systemctl is-active share2, in 1.12 seconds)4774client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }4775client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.05 seconds)4776(finished: subtest: the first client leaving does not fail the second, in 7.10 seconds)4777subtest: build hook: nix-daemon with builders = grpc://lb4778client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log4779client # [ 86.624707] systemd[1]: Started Nix Daemon.4780client # [ 86.741418] nix-daemon[1149]: accepted connection from pid 1146, user root (trusted)4781worker2 # [ 86.964344] nix-grpc-daemon[770]: ts=2026-09-23T04:44:16Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:516484782worker2 # [ 86.970807] nix-grpc-daemon[770]: ts=2026-09-23T04:44:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:516484783worker2 # [ 86.989430] nix-grpc-daemon[770]: ts=2026-09-23T04:44:16Z level=debug event=want client=11 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=04784worker2 # [ 86.996627] nix-grpc-daemon[770]: ts=2026-09-23T04:44:16Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.4:50051 clients=14785worker2 # [ 87.005527] nix-grpc-daemon[770]: ts=2026-09-23T04:44:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:516484786worker2 # [ 87.018037] nix-grpc-daemon[770]: ts=2026-09-23T04:44:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:516484787worker2 # [ 87.032937] nix-grpc-daemon[770]: ts=2026-09-23T04:44:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:516484788worker2 # [ 87.047876] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4789worker2 # [ 87.208115] nix-daemon[856]: accepted connection from pid 1163, user nix-grpc-daemon (trusted)4790worker2 # [ 87.230258] nix-daemon[856]: reaped child process 1166, status = succeeded4791lb # [ 87.170213] niks3-server[973]: 2026/09/23 04:44:16 INFO Received uploads request method=POST path=/api/pending_closures4792worker2 # [ 87.239577] nix-grpc-daemon[871]: time=2026-09-23T04:44:16.760Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4793worker2 # [ 87.241226] nix-grpc-daemon[871]: time=2026-09-23T04:44:16.761Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)"4794lb # [ 87.202159] niks3-server[973]: 2026/09/23 04:44:16 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls4795worker2 # [ 87.274716] nix-grpc-daemon[871]: time=2026-09-23T04:44:16.796Z level=INFO msg="Uploading 1 narinfos"4796lb # [ 87.215430] niks3-server[973]: 2026/09/23 04:44:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign4797lb # [ 87.228249] niks3-server[973]: 2026/09/23 04:44:16 INFO Signed narinfos id=21 count=14798lb # [ 87.231211] niks3-server[973]: 2026/09/23 04:44:16 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst4799lb # [ 87.242538] niks3-server[973]: 2026/09/23 04:44:16 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete4800worker2 # [ 87.310412] nix-grpc-daemon[871]: time=2026-09-23T04:44:16.832Z level=INFO msg="Upload complete. (247ms)"4801lb # [ 87.253995] niks3-server[973]: 2026/09/23 04:44:16 INFO Completed upload id=214802lb # [ 87.258678] niks3-server[973]: 2026/09/23 04:44:16 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo4803worker2 # [ 87.320358] nix-grpc-daemon[770]: ts=2026-09-23T04:44:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=0 paths=1 nar_bytes_in=5124804worker2 # [ 87.338907] nix-grpc-daemon[770]: ts=2026-09-23T04:44:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:516484805worker2 # [ 87.346363] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4806worker2 # [ 87.351937] nix-daemon[856]: reaped child process 1161, status = succeeded4807worker2 # [ 87.367817] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4808worker2 # [ 89.234298] nix-daemon[856]: reaped child process 1170, status = succeeded4809worker2 # [ 89.374454] nix-daemon[856]: accepted connection from pid 1186, user nix-grpc-daemon (trusted)4810worker2 # [ 89.394363] nix-daemon[856]: reaped child process 1189, status = succeeded4811lb # [ 89.343536] niks3-server[973]: 2026/09/23 04:44:18 INFO Received uploads request method=POST path=/api/pending_closures4812worker2 # [ 89.412643] nix-grpc-daemon[871]: time=2026-09-23T04:44:18.934Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4813worker2 # [ 89.416324] nix-grpc-daemon[871]: time=2026-09-23T04:44:18.936Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)"4814lb # [ 89.387560] niks3-server[973]: 2026/09/23 04:44:18 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv4815lb # [ 89.399902] niks3-server[973]: 2026/09/23 04:44:18 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls4816worker2 # [ 89.470739] nix-grpc-daemon[871]: time=2026-09-23T04:44:18.992Z level=INFO msg="Uploading 1 narinfos"4817lb # [ 89.412243] niks3-server[973]: 2026/09/23 04:44:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign4818lb # [ 89.424556] niks3-server[973]: 2026/09/23 04:44:18 INFO Signed narinfos id=22 count=14819lb # [ 89.430648] niks3-server[973]: 2026/09/23 04:44:18 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst4820lb # [ 89.445034] envoy[900]: conn peer=[2001:db8:1::1]:60314 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3003 tx=788 ms=42784821worker2 # [ 89.510410] nix-grpc-daemon[871]: time=2026-09-23T04:44:19.032Z level=INFO msg="Upload complete. (328ms)"4822lb # [ 89.453082] envoy[900]: conn peer=[2001:db8:1::1]:60322 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1605 tx=1600 ms=50284823lb # [ 89.457279] niks3-server[973]: 2026/09/23 04:44:18 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete4824lb # [ 89.462943] niks3-server[973]: 2026/09/23 04:44:18 INFO Completed upload id=224825lb # [ 89.466393] niks3-server[973]: 2026/09/23 04:44:18 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo4826worker2 # [ 89.523117] nix-grpc-daemon[770]: ts=2026-09-23T04:44:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=2 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=12 outputs=14827worker2 # [ 89.545685] nix-daemon[856]: reaped child process 1168, status = succeeded4828worker1 # [ 89.441136] nix-grpc-daemon[767]: ts=2026-09-23T04:44:18Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:571384829worker1 # [ 89.455109] nix-grpc-daemon[767]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'...4830worker1 # [ 89.497183] nix-grpc-daemon[767]: ts=2026-09-23T04:44:18Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:571384831worker1 # [ 89.504198] nix-grpc-daemon[767]: ts=2026-09-23T04:44:19Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=0 paths=1 nar_bytes_out=1284832worker1 # [ 89.785643] nix-grpc-daemon[767]: ts=2026-09-23T04:44:19Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:571384833worker1 # [ 89.794060] nix-grpc-daemon[767]: ts=2026-09-23T04:44:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:571384834worker1 # [ 89.801626] nix-grpc-daemon[861]: time=2026-09-23T04:44:19.298Z level=INFO msg="All 1 paths already cached"4835worker2 # [ 89.916459] nix-grpc-daemon[770]: ts=2026-09-23T04:44:19Z level=debug event=want client=12 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=04836worker2 # [ 89.923827] nix-grpc-daemon[770]: ts=2026-09-23T04:44:19Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.4:50051 clients=14837worker2 # [ 89.967776] nix-grpc-daemon[770]: ts=2026-09-23T04:44:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:516484838worker2 # [ 89.985525] nix-grpc-daemon[770]: ts=2026-09-23T04:44:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:516484839worker2 # [ 90.000039] nix-grpc-daemon[871]: time=2026-09-23T04:44:19.522Z level=INFO msg="All 1 paths already cached"4840worker2 # [ 90.010613] nix-grpc-daemon[770]: ts=2026-09-23T04:44:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:516484841worker2 # [ 90.026410] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4842worker2 # [ 90.181799] nix-daemon[856]: accepted connection from pid 1195, user nix-grpc-daemon (trusted)4843worker2 # [ 90.202424] nix-daemon[856]: reaped child process 1198, status = succeeded4844lb # [ 90.144243] niks3-server[973]: 2026/09/23 04:44:19 INFO Received uploads request method=POST path=/api/pending_closures4845worker2 # [ 90.212871] nix-grpc-daemon[871]: time=2026-09-23T04:44:19.734Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4846worker2 # [ 90.215072] nix-grpc-daemon[871]: time=2026-09-23T04:44:19.734Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)"4847lb # [ 90.174610] niks3-server[973]: 2026/09/23 04:44:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign4848worker2 # [ 90.240366] nix-grpc-daemon[871]: time=2026-09-23T04:44:19.762Z level=INFO msg="Uploading 1 narinfos"4849lb # [ 90.190246] niks3-server[973]: 2026/09/23 04:44:19 INFO Signed narinfos id=23 count=14850lb # [ 90.198692] niks3-server[973]: 2026/09/23 04:44:19 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls4851worker2 # [ 90.270274] nix-grpc-daemon[871]: time=2026-09-23T04:44:19.792Z level=INFO msg="Upload complete. (228ms)"4852lb # [ 90.210898] niks3-server[973]: 2026/09/23 04:44:19 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst4853worker2 # [ 90.280309] nix-grpc-daemon[770]: ts=2026-09-23T04:44:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=0 paths=1 nar_bytes_in=6244854lb # [ 90.224526] niks3-server[973]: 2026/09/23 04:44:19 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete4855lb # [ 90.229169] niks3-server[973]: 2026/09/23 04:44:19 INFO Completed upload id=234856lb # [ 90.232148] niks3-server[973]: 2026/09/23 04:44:19 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo4857worker2 # [ 90.296381] nix-grpc-daemon[770]: ts=2026-09-23T04:44:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:516484858worker2 # [ 90.303217] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4859worker2 # [ 90.308713] nix-daemon[856]: reaped child process 1193, status = succeeded4860worker2 # [ 90.321278] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4861worker2 # [ 91.372810] nix-daemon[856]: reaped child process 1202, status = succeeded4862worker2 # [ 91.452446] nix-daemon[856]: accepted connection from pid 1218, user nix-grpc-daemon (trusted)4863worker2 # [ 91.473093] nix-daemon[856]: reaped child process 1221, status = succeeded4864lb # [ 91.413021] niks3-server[973]: 2026/09/23 04:44:20 INFO Received uploads request method=POST path=/api/pending_closures4865worker2 # [ 91.484421] nix-grpc-daemon[871]: time=2026-09-23T04:44:21.003Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"4866worker2 # [ 91.485906] nix-grpc-daemon[871]: time=2026-09-23T04:44:21.003Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)"4867lb # [ 91.458423] niks3-server[973]: 2026/09/23 04:44:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign4868worker2 # [ 91.527019] nix-grpc-daemon[871]: time=2026-09-23T04:44:21.048Z level=INFO msg="Uploading 1 narinfos"4869lb # [ 91.472103] niks3-server[973]: 2026/09/23 04:44:20 INFO Signed narinfos id=24 count=14870lb # [ 91.482660] niks3-server[973]: 2026/09/23 04:44:20 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls4871lb # [ 91.492336] niks3-server[973]: 2026/09/23 04:44:20 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst4872lb # [ 91.504059] niks3-server[973]: 2026/09/23 04:44:21 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv4873lb # [ 91.509239] niks3-server[973]: 2026/09/23 04:44:21 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete4874worker2 # [ 91.572080] nix-grpc-daemon[871]: time=2026-09-23T04:44:21.092Z level=INFO msg="Upload complete. (240ms)"4875lb # [ 91.520242] niks3-server[973]: 2026/09/23 04:44:21 INFO Completed upload id=244876lb # [ 91.526053] niks3-server[973]: 2026/09/23 04:44:21 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo4877worker2 # [ 91.584822] nix-grpc-daemon[770]: ts=2026-09-23T04:44:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=13 outputs=14878worker2 # [ 91.599487] nix-grpc-daemon[770]: ts=2026-09-23T04:44:21Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:516484879worker2 # [ 91.604919] nix-daemon[856]: reaped child process 1200, status = succeeded4880worker2 # [ 91.627457] nix-grpc-daemon[770]: ts=2026-09-23T04:44:21Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:516484881worker2 # [ 91.634525] nix-grpc-daemon[770]: ts=2026-09-23T04:44:21Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=0 paths=1 nar_bytes_out=1764882worker1 # [ 91.803947] nix-grpc-daemon[767]: ts=2026-09-23T04:44:21Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:571384883worker2 # [ 91.910499] nix-grpc-daemon[770]: ts=2026-09-23T04:44:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:516484884worker2 # [ 91.923506] nix-grpc-daemon[871]: time=2026-09-23T04:44:21.445Z level=INFO msg="All 2 paths already cached"4885worker2 # [ 91.944117] nix-grpc-daemon[770]: ts=2026-09-23T04:44:21Z level=debug event=want client=13 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=04886worker2 # [ 91.950793] nix-grpc-daemon[770]: ts=2026-09-23T04:44:21Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.4:50051 clients=14887worker2 # [ 91.960613] nix-grpc-daemon[770]: ts=2026-09-23T04:44:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:516484888worker2 # [ 91.972120] nix-grpc-daemon[770]: ts=2026-09-23T04:44:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:516484889worker2 # [ 91.986543] nix-grpc-daemon[871]: time=2026-09-23T04:44:21.508Z level=INFO msg="All 2 paths already cached"4890worker2 # [ 91.997065] nix-grpc-daemon[770]: ts=2026-09-23T04:44:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:516484891worker2 # [ 92.010845] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4892worker2 # [ 92.165247] nix-daemon[856]: accepted connection from pid 1227, user nix-grpc-daemon (trusted)4893worker2 # [ 92.184376] nix-daemon[856]: reaped child process 1230, status = succeeded4894lb # [ 92.131228] niks3-server[973]: 2026/09/23 04:44:21 INFO Received uploads request method=POST path=/api/pending_closures4895worker2 # [ 92.200083] nix-grpc-daemon[871]: time=2026-09-23T04:44:21.722Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4896worker2 # [ 92.202167] nix-grpc-daemon[871]: time=2026-09-23T04:44:21.722Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)"4897lb # [ 92.162334] niks3-server[973]: 2026/09/23 04:44:21 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst4898lb # [ 92.176799] niks3-server[973]: 2026/09/23 04:44:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign4899worker2 # [ 92.244323] nix-grpc-daemon[871]: time=2026-09-23T04:44:21.766Z level=INFO msg="Uploading 1 narinfos"4900lb # [ 92.190622] niks3-server[973]: 2026/09/23 04:44:21 INFO Signed narinfos id=25 count=14901lb # [ 92.198137] niks3-server[973]: 2026/09/23 04:44:21 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls4902lb # [ 92.211054] niks3-server[973]: 2026/09/23 04:44:21 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete4903worker2 # [ 92.276243] nix-grpc-daemon[871]: time=2026-09-23T04:44:21.798Z level=INFO msg="Upload complete. (248ms)"4904lb # [ 92.221778] niks3-server[973]: 2026/09/23 04:44:21 INFO Completed upload id=254905lb # [ 92.225814] niks3-server[973]: 2026/09/23 04:44:21 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo4906worker2 # [ 92.288255] nix-grpc-daemon[770]: ts=2026-09-23T04:44:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=0 paths=1 nar_bytes_in=7524907worker2 # [ 92.304387] nix-grpc-daemon[770]: ts=2026-09-23T04:44:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:516484908worker2 # [ 92.316888] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4909worker2 # [ 92.320498] nix-daemon[856]: reaped child process 1225, status = succeeded4910worker2 # [ 92.335215] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)4911worker2 # [ 93.847372] nix-daemon[856]: reaped child process 1234, status = succeeded4912worker2 # [ 93.922280] nix-daemon[856]: accepted connection from pid 1250, user nix-grpc-daemon (trusted)4913worker2 # [ 93.942303] nix-daemon[856]: reaped child process 1253, status = succeeded4914lb # [ 93.888734] niks3-server[973]: 2026/09/23 04:44:23 INFO Received uploads request method=POST path=/api/pending_closures4915worker2 # [ 93.962227] nix-grpc-daemon[871]: time=2026-09-23T04:44:23.482Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"4916worker2 # [ 93.964098] nix-grpc-daemon[871]: time=2026-09-23T04:44:23.482Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)"4917lb # [ 93.927905] niks3-server[973]: 2026/09/23 04:44:23 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls4918worker2 # [ 94.001218] nix-grpc-daemon[871]: time=2026-09-23T04:44:23.523Z level=INFO msg="Uploading 1 narinfos"4919lb # [ 93.941516] niks3-server[973]: 2026/09/23 04:44:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign4920lb # [ 93.953218] niks3-server[973]: 2026/09/23 04:44:23 INFO Signed narinfos id=26 count=14921lb # [ 93.955949] niks3-server[973]: 2026/09/23 04:44:23 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst4922lb # [ 93.974707] niks3-server[973]: 2026/09/23 04:44:23 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete4923worker2 # [ 94.040931] nix-grpc-daemon[871]: time=2026-09-23T04:44:23.562Z level=INFO msg="Upload complete. (290ms)"4924lb # [ 93.982168] niks3-server[973]: 2026/09/23 04:44:23 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv4925lb # [ 93.985970] niks3-server[973]: 2026/09/23 04:44:23 INFO Completed upload id=264926lb # [ 93.989697] niks3-server[973]: 2026/09/23 04:44:23 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo4927worker2 # [ 94.054145] nix-grpc-daemon[770]: ts=2026-09-23T04:44:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=14 outputs=14928worker2 # [ 94.075570] nix-grpc-daemon[770]: ts=2026-09-23T04:44:23Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:516484929worker2 # [ 94.081725] nix-daemon[856]: reaped child process 1232, status = succeeded4930worker2 # [ 94.104244] nix-grpc-daemon[770]: ts=2026-09-23T04:44:23Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:516484931worker2 # [ 94.111135] nix-grpc-daemon[770]: ts=2026-09-23T04:44:23Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=0 paths=1 nar_bytes_out=2324932client # [ 94.157282] nix-daemon[1149]: reaped child process 1151, status = succeeded4933client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log, in 7.69 seconds)4934client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top4935client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.03 seconds)4936client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }4937client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.04 seconds)4938client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }4939client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds)4940(finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.79 seconds)4941subtest: an input only one worker has still reaches the builder4942client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only4943client # this derivation will be built:4944client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv4945client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'...4946client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only, in 0.29 seconds)4947client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure4948client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.12 seconds)4949worker1: must succeed: nix-store --import < /tmp/shared/inp.closure4950worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.20 seconds)4951worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only4952worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.03 seconds)4953client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&24954client # [ 95.104252] nix-daemon[1149]: accepted connection from pid 1232, user root (trusted)4955client # this derivation will be built:4956client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv4957client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...4958worker1 # [ 95.230221] nix-grpc-daemon[767]: ts=2026-09-23T04:44:24Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:571384959worker1 # [ 95.236795] nix-grpc-daemon[767]: ts=2026-09-23T04:44:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:571384960worker1 # [ 95.401345] nix-daemon[834]: accepted connection from pid 1067, user nix-grpc-daemon (trusted)4961worker1 # [ 95.417122] nix-daemon[834]: reaped child process 1070, status = succeeded4962lb # [ 95.469831] niks3-server[973]: 2026/09/23 04:44:24 INFO Received uploads request method=POST path=/api/pending_closures4963worker1 # [ 95.435981] nix-grpc-daemon[861]: time=2026-09-23T04:44:24.934Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"4964worker1 # [ 95.438369] nix-grpc-daemon[861]: time=2026-09-23T04:44:24.936Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)"4965lb # [ 95.503938] niks3-server[973]: 2026/09/23 04:44:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign4966worker1 # [ 95.472094] nix-grpc-daemon[861]: time=2026-09-23T04:44:24.970Z level=INFO msg="Uploading 1 narinfos"4967lb # [ 95.519339] niks3-server[973]: 2026/09/23 04:44:25 INFO Signed narinfos id=27 count=14968lb # [ 95.526848] niks3-server[973]: 2026/09/23 04:44:25 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst4969lb # [ 95.542600] niks3-server[973]: 2026/09/23 04:44:25 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls4970worker1 # [ 95.503628] nix-grpc-daemon[861]: time=2026-09-23T04:44:25.002Z level=INFO msg="Upload complete. (261ms)"4971lb # [ 95.551190] niks3-server[973]: 2026/09/23 04:44:25 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete4972lb # [ 95.556571] niks3-server[973]: 2026/09/23 04:44:25 INFO Completed upload id=274973lb # [ 95.559745] niks3-server[973]: 2026/09/23 04:44:25 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo4974worker2 # [ 95.633420] nix-grpc-daemon[770]: ts=2026-09-23T04:44:25Z level=debug event=want client=14 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=04975worker2 # [ 95.640339] nix-grpc-daemon[770]: ts=2026-09-23T04:44:25Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.3:50051 clients=14976worker1 # [ 95.585209] nix-grpc-daemon[767]: ts=2026-09-23T04:44:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:571384977worker1 # [ 95.601812] nix-grpc-daemon[767]: ts=2026-09-23T04:44:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:571384978client # copying 2 paths to 'lb:50051'...4979client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'...4980worker1 # [ 95.621765] nix-grpc-daemon[767]: ts=2026-09-23T04:44:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:571384981client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'...4982worker1 # [ 95.636321] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)4983worker1 # [ 95.803615] nix-daemon[834]: accepted connection from pid 1074, user nix-grpc-daemon (trusted)4984worker1 # [ 95.822156] nix-daemon[834]: reaped child process 1077, status = succeeded4985lb # [ 95.871183] niks3-server[973]: 2026/09/23 04:44:25 INFO Received uploads request method=POST path=/api/pending_closures4986lb # [ 95.884637] niks3-server[973]: 2026/09/23 04:44:25 INFO Received uploads request method=POST path=/api/pending_closures4987worker1 # [ 95.852098] nix-grpc-daemon[861]: time=2026-09-23T04:44:25.349Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"4988worker1 # [ 95.854630] nix-grpc-daemon[861]: time=2026-09-23T04:44:25.349Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)"4989worker1 # [ 95.859424] nix-grpc-daemon[861]: time=2026-09-23T04:44:25.349Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)"4990lb # [ 95.924639] niks3-server[973]: 2026/09/23 04:44:25 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst4991lb # [ 95.943580] niks3-server[973]: 2026/09/23 04:44:25 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls4992worker1 # [ 95.911355] nix-grpc-daemon[861]: time=2026-09-23T04:44:25.409Z level=INFO msg="Uploading 3 narinfos"4993lb # [ 95.957405] niks3-server[973]: 2026/09/23 04:44:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign4994lb # [ 95.964596] niks3-server[973]: 2026/09/23 04:44:25 INFO Signed narinfos id=28 count=14995lb # [ 95.966929] niks3-server[973]: 2026/09/23 04:44:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign4996lb # [ 95.975720] niks3-server[973]: 2026/09/23 04:44:25 INFO Signed narinfos id=29 count=24997lb # [ 95.981292] niks3-server[973]: 2026/09/23 04:44:25 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst4998lb # [ 95.989203] niks3-server[973]: 2026/09/23 04:44:25 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls4999lb # [ 95.998251] niks3-server[973]: 2026/09/23 04:44:25 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete5000lb # [ 96.006770] niks3-server[973]: 2026/09/23 04:44:25 INFO Completed upload id=285001lb # [ 96.010983] niks3-server[973]: 2026/09/23 04:44:25 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete5002lb # [ 96.015859] niks3-server[973]: 2026/09/23 04:44:25 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo5003lb # [ 96.021126] niks3-server[973]: 2026/09/23 04:44:25 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo5004worker1 # [ 95.976435] nix-grpc-daemon[861]: time=2026-09-23T04:44:25.475Z level=INFO msg="Upload complete. (321ms)"5005lb # [ 96.023695] niks3-server[973]: 2026/09/23 04:44:25 INFO Completed upload id=295006lb # [ 96.028679] niks3-server[973]: 2026/09/23 04:44:25 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo5007worker1 # [ 95.989407] nix-grpc-daemon[767]: ts=2026-09-23T04:44:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=0 paths=2 nar_bytes_in=9845008worker1 # [ 96.004585] nix-grpc-daemon[767]: ts=2026-09-23T04:44:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:571385009worker1 # [ 96.017017] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)5010worker1 # [ 96.024380] nix-daemon[834]: reaped child process 1072, status = succeeded5011client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051...5012client # uses-input-w1onlya> node-a: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv5013worker1 # [ 96.043157] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)5014client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds5015worker1 # [ 97.105081] nix-daemon[834]: reaped child process 1081, status = succeeded5016worker1 # [ 97.232461] nix-daemon[834]: accepted connection from pid 1097, user nix-grpc-daemon (trusted)5017worker1 # [ 97.248660] nix-daemon[834]: reaped child process 1100, status = succeeded5018lb # [ 97.300986] niks3-server[973]: 2026/09/23 04:44:26 INFO Received uploads request method=POST path=/api/pending_closures5019worker1 # [ 97.268550] nix-grpc-daemon[861]: time=2026-09-23T04:44:26.765Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5020lb # [ 97.331043] niks3-server[973]: 2026/09/23 04:44:26 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv5021worker1 # [ 97.305747] nix-grpc-daemon[861]: time=2026-09-23T04:44:26.804Z level=INFO msg="Uploading 1 narinfos"5022lb # [ 97.342738] niks3-server[973]: 2026/09/23 04:44:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign5023lb # [ 97.354519] niks3-server[973]: 2026/09/23 04:44:26 INFO Signed narinfos id=30 count=15024lb # [ 97.361566] niks3-server[973]: 2026/09/23 04:44:26 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls5025lb # [ 97.374214] niks3-server[973]: 2026/09/23 04:44:26 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete5026lb # [ 97.383043] niks3-server[973]: 2026/09/23 04:44:26 INFO Completed upload id=305027worker1 # [ 97.338732] nix-grpc-daemon[861]: time=2026-09-23T04:44:26.837Z level=INFO msg="Upload complete. (281ms)"5028lb # [ 97.386292] niks3-server[973]: 2026/09/23 04:44:26 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo5029worker1 # [ 97.353235] nix-grpc-daemon[767]: ts=2026-09-23T04:44:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=15 outputs=15030worker1 # [ 97.369485] nix-daemon[834]: reaped child process 1079, status = succeeded5031client # copying 1 paths...5032worker1 # [ 97.397938] nix-grpc-daemon[767]: ts=2026-09-23T04:44:26Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:571385033client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'...5034worker1 # [ 97.430985] nix-grpc-daemon[767]: ts=2026-09-23T04:44:26Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:571385035worker1 # [ 97.437826] nix-grpc-daemon[767]: ts=2026-09-23T04:44:26Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=0 paths=1 nar_bytes_out=1205036client # [ 97.587626] nix-daemon[1149]: reaped child process 1235, status = succeeded5037client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.68 seconds)5038client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&25039client # [ 97.764506] nix-daemon[1149]: accepted connection from pid 1254, user root (trusted)5040client # this derivation will be built:5041client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv5042worker1 # [ 97.883491] nix-grpc-daemon[767]: ts=2026-09-23T04:44:27Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:571385043client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...5044worker1 # [ 97.891461] nix-grpc-daemon[767]: ts=2026-09-23T04:44:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:571385045worker1 # [ 97.898875] nix-grpc-daemon[861]: time=2026-09-23T04:44:27.396Z level=INFO msg="All 1 paths already cached"5046worker2 # [ 98.020466] nix-grpc-daemon[770]: ts=2026-09-23T04:44:27Z level=debug event=want client=15 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=05047worker2 # [ 98.028581] nix-grpc-daemon[770]: ts=2026-09-23T04:44:27Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.4:50051 clients=15048worker2 # [ 98.037206] nix-grpc-daemon[770]: ts=2026-09-23T04:44:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:516485049worker2 # [ 98.043296] nix-grpc-daemon[770]: ts=2026-09-23T04:44:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:516485050client # copying 1 paths to 'lb:50051'...5051client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'...5052worker2 # [ 98.055052] nix-grpc-daemon[770]: ts=2026-09-23T04:44:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:516485053worker2 # [ 98.069916] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)5054worker2 # [ 98.089758] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 66 ms (attempt 1/5)5055worker2 # [ 98.162941] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 60 ms (attempt 2/5)5056worker2 # [ 98.224850] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 38 ms (attempt 3/5)5057worker2 # [ 98.264556] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 520 ms (attempt 4/5)5058worker2 # [ 98.787578] nix-grpc-daemon[770]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5059worker2 # [ 98.795741] nix-grpc-daemon[770]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5060worker2 # [ 98.803687] nix-grpc-daemon[770]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'...5061worker2 # [ 98.981712] nix-daemon[856]: accepted connection from pid 1269, user nix-grpc-daemon (trusted)5062worker2 # [ 99.004399] nix-daemon[856]: reaped child process 1272, status = succeeded5063lb # [ 98.949616] niks3-server[973]: 2026/09/23 04:44:28 INFO Received uploads request method=POST path=/api/pending_closures5064worker2 # [ 99.019486] nix-grpc-daemon[871]: time=2026-09-23T04:44:28.540Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5065worker2 # [ 99.021280] nix-grpc-daemon[871]: time=2026-09-23T04:44:28.540Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)"5066lb # [ 98.981810] niks3-server[973]: 2026/09/23 04:44:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign5067worker2 # [ 99.052677] nix-grpc-daemon[871]: time=2026-09-23T04:44:28.574Z level=INFO msg="Uploading 1 narinfos"5068lb # [ 98.997718] niks3-server[973]: 2026/09/23 04:44:28 INFO Signed narinfos id=31 count=15069lb # [ 99.004537] niks3-server[973]: 2026/09/23 04:44:28 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls5070lb # [ 99.018041] niks3-server[973]: 2026/09/23 04:44:28 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst5071worker2 # [ 99.085773] nix-grpc-daemon[871]: time=2026-09-23T04:44:28.607Z level=INFO msg="Upload complete. (267ms)"5072lb # [ 99.027510] niks3-server[973]: 2026/09/23 04:44:28 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete5073lb # [ 99.038496] niks3-server[973]: 2026/09/23 04:44:28 INFO Completed upload id=315074worker2 # [ 99.098608] nix-grpc-daemon[770]: ts=2026-09-23T04:44:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=1 paths=1 nar_bytes_in=5685075lb # [ 99.043018] niks3-server[973]: 2026/09/23 04:44:28 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo5076worker2 # [ 99.110601] nix-grpc-daemon[770]: ts=2026-09-23T04:44:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:516485077worker2 # [ 99.123423] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)5078worker2 # [ 99.128292] nix-daemon[856]: reaped child process 1262, status = succeeded5079worker2 # [ 99.143178] nix-grpc-daemon[770]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'...5080client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051...5081worker2 # [ 99.157366] nix-daemon[856]: accepted connection from pid 770, user nix-grpc-daemon (trusted)5082client # uses-input-w1onlyb> worker2: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv5083lb # [ 99.418337] envoy[900]: conn peer=[2001:db8:1::1]:44840 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3444 tx=1961 ms=27015084lb # [ 99.423478] envoy[900]: conn peer=[2001:db8:1::1]:44844 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=333 ms=705085lb # [ 99.429123] envoy[900]: conn peer=[2001:db8:1::1]:44852 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3888 tx=2157 ms=18035086lb # [ 99.433891] envoy[900]: conn peer=[2001:db8:1::1]:36388 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=375 ms=685087lb # [ 99.438548] envoy[900]: conn peer=[2001:db8:1::1]:36398 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4378 tx=2392 ms=22615088lb # [ 99.443451] envoy[900]: conn peer=[2001:db8:1::1]:36404 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=438 tx=406 ms=685089lb # [ 99.448216] envoy[900]: conn peer=[2001:db8:1::1]:36414 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3871 tx=2066 ms=22685090lb # [ 99.452972] envoy[900]: conn peer=[2001:db8:1::1]:36428 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=715091client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds5092worker2 # [ 100.264384] nix-daemon[856]: reaped child process 1277, status = succeeded5093worker2 # [ 100.348712] nix-daemon[856]: accepted connection from pid 1293, user nix-grpc-daemon (trusted)5094worker2 # [ 100.371338] nix-daemon[856]: reaped child process 1296, status = succeeded5095lb # [ 100.311177] niks3-server[973]: 2026/09/23 04:44:29 INFO Received uploads request method=POST path=/api/pending_closures5096worker2 # [ 100.382638] nix-grpc-daemon[871]: time=2026-09-23T04:44:29.903Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5097lb # [ 100.345129] niks3-server[973]: 2026/09/23 04:44:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign5098worker2 # [ 100.414352] nix-grpc-daemon[871]: time=2026-09-23T04:44:29.936Z level=INFO msg="Uploading 1 narinfos"5099lb # [ 100.360395] niks3-server[973]: 2026/09/23 04:44:29 INFO Signed narinfos id=32 count=15100lb # [ 100.368252] niks3-server[973]: 2026/09/23 04:44:29 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls5101lb # [ 100.376255] niks3-server[973]: 2026/09/23 04:44:29 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv5102worker2 # [ 100.445114] nix-grpc-daemon[871]: time=2026-09-23T04:44:29.967Z level=INFO msg="Upload complete. (265ms)"5103lb # [ 100.386597] niks3-server[973]: 2026/09/23 04:44:29 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete5104lb # [ 100.393165] niks3-server[973]: 2026/09/23 04:44:29 INFO Completed upload id=325105lb # [ 100.396805] niks3-server[973]: 2026/09/23 04:44:29 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo5106worker2 # [ 100.457346] nix-grpc-daemon[770]: ts=2026-09-23T04:44:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=16 outputs=15107worker2 # [ 100.474594] nix-daemon[856]: reaped child process 1274, status = succeeded5108client # copying 1 paths...5109worker2 # [ 100.505397] nix-grpc-daemon[770]: ts=2026-09-23T04:44:30Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:516485110client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'...5111worker2 # [ 100.543890] nix-grpc-daemon[770]: ts=2026-09-23T04:44:30Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:516485112worker2 # [ 100.550817] nix-grpc-daemon[770]: ts=2026-09-23T04:44:30Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:51648 duration_s=0 paths=1 nar_bytes_out=1205113client # [ 100.607066] nix-daemon[1149]: reaped child process 1257, status = succeeded5114client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.02 seconds)5115client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&25116client # [ 100.802754] nix-daemon[1149]: accepted connection from pid 1276, user root (trusted)5117client # this derivation will be built:5118client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv5119client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'...5120worker2 # [ 101.050782] nix-grpc-daemon[770]: ts=2026-09-23T04:44:30Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:516485121worker2 # [ 101.059278] nix-grpc-daemon[770]: ts=2026-09-23T04:44:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:516485122worker2 # [ 101.066517] nix-grpc-daemon[871]: time=2026-09-23T04:44:30.587Z level=INFO msg="All 1 paths already cached"5123worker2 # [ 101.079660] nix-grpc-daemon[770]: ts=2026-09-23T04:44:30Z level=debug event=want client=16 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=05124worker2 # [ 101.086766] nix-grpc-daemon[770]: ts=2026-09-23T04:44:30Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.3:50051 clients=15125worker1 # [ 100.988941] nix-grpc-daemon[767]: ts=2026-09-23T04:44:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:571385126worker1 # [ 101.007903] nix-grpc-daemon[767]: ts=2026-09-23T04:44:30Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:571385127worker1 # [ 101.026252] nix-grpc-daemon[861]: time=2026-09-23T04:44:30.523Z level=INFO msg="All 1 paths already cached"5128client # copying 1 paths to 'lb:50051'...5129client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'...5130worker1 # [ 101.039870] nix-grpc-daemon[767]: ts=2026-09-23T04:44:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:571385131worker1 # [ 101.059821] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)5132worker1 # [ 101.266181] nix-daemon[834]: accepted connection from pid 1104, user nix-grpc-daemon (trusted)5133worker1 # [ 101.293289] nix-daemon[834]: reaped child process 1107, status = succeeded5134lb # [ 101.347253] niks3-server[973]: 2026/09/23 04:44:30 INFO Received uploads request method=POST path=/api/pending_closures5135worker1 # [ 101.311262] nix-grpc-daemon[861]: time=2026-09-23T04:44:30.809Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5136worker1 # [ 101.313121] nix-grpc-daemon[861]: time=2026-09-23T04:44:30.809Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)"5137lb # [ 101.383081] niks3-server[973]: 2026/09/23 04:44:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign5138worker1 # [ 101.351059] nix-grpc-daemon[861]: time=2026-09-23T04:44:30.849Z level=INFO msg="Uploading 1 narinfos"5139lb # [ 101.400296] niks3-server[973]: 2026/09/23 04:44:30 INFO Signed narinfos id=33 count=15140lb # [ 101.403535] niks3-server[973]: 2026/09/23 04:44:30 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst5141lb # [ 101.419247] niks3-server[973]: 2026/09/23 04:44:30 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls5142worker1 # [ 101.390300] nix-grpc-daemon[861]: time=2026-09-23T04:44:30.888Z level=INFO msg="Upload complete. (307ms)"5143lb # [ 101.434685] niks3-server[973]: 2026/09/23 04:44:30 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete5144lb # [ 101.446335] niks3-server[973]: 2026/09/23 04:44:30 INFO Completed upload id=335145worker1 # [ 101.400471] nix-grpc-daemon[767]: ts=2026-09-23T04:44:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=0 paths=1 nar_bytes_in=5685146lb # [ 101.449493] niks3-server[973]: 2026/09/23 04:44:30 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo5147worker1 # [ 101.416913] nix-grpc-daemon[767]: ts=2026-09-23T04:44:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:571385148worker1 # [ 101.436271] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)5149worker1 # [ 101.444493] nix-daemon[834]: reaped child process 1102, status = succeeded5150worker1 # [ 101.458330] nix-daemon[834]: accepted connection from pid 767, user nix-grpc-daemon (trusted)5151client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051...5152client # uses-input-w1onlyc> node-a: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv5153client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds5154worker1 # [ 102.956977] nix-daemon[834]: reaped child process 1111, status = succeeded5155worker1 # [ 103.093634] nix-daemon[834]: accepted connection from pid 1127, user nix-grpc-daemon (trusted)5156worker1 # [ 103.118190] nix-daemon[834]: reaped child process 1130, status = succeeded5157lb # [ 103.161599] niks3-server[973]: 2026/09/23 04:44:32 INFO Received uploads request method=POST path=/api/pending_closures5158worker1 # [ 103.130760] nix-grpc-daemon[861]: time=2026-09-23T04:44:32.628Z level=INFO msg="Uploading 0 paths to lb (1 already cached)"5159lb # [ 103.206663] niks3-server[973]: 2026/09/23 04:44:32 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv5160worker1 # [ 103.177948] nix-grpc-daemon[861]: time=2026-09-23T04:44:32.676Z level=INFO msg="Uploading 1 narinfos"5161lb # [ 103.221467] niks3-server[973]: 2026/09/23 04:44:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign5162lb # [ 103.228490] niks3-server[973]: 2026/09/23 04:44:32 INFO Signed narinfos id=34 count=15163lb # [ 103.233204] niks3-server[973]: 2026/09/23 04:44:32 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls5164lb # [ 103.246547] niks3-server[973]: 2026/09/23 04:44:32 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete5165lb # [ 103.254465] niks3-server[973]: 2026/09/23 04:44:32 INFO Completed upload id=345166lb # [ 103.261229] niks3-server[973]: 2026/09/23 04:44:32 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo5167worker1 # [ 103.209756] nix-grpc-daemon[861]: time=2026-09-23T04:44:32.708Z level=INFO msg="Upload complete. (334ms)"5168worker1 # [ 103.225822] nix-grpc-daemon[767]: ts=2026-09-23T04:44:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=17 outputs=15169worker1 # [ 103.242895] nix-daemon[834]: reaped child process 1109, status = succeeded5170client # copying 1 paths...5171worker2 # [ 103.377824] nix-grpc-daemon[770]: ts=2026-09-23T04:44:32Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:516485172worker2 # [ 103.396779] nix-grpc-daemon[770]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'...5173client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'...5174worker1 # [ 103.343843] nix-grpc-daemon[767]: ts=2026-09-23T04:44:32Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:571385175worker1 # [ 103.352334] nix-grpc-daemon[767]: ts=2026-09-23T04:44:32Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:57138 duration_s=0 paths=1 nar_bytes_out=1205176client # [ 103.472980] nix-daemon[1149]: reaped child process 1279, status = succeeded5177client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.86 seconds)5178(finished: subtest: an input only one worker has still reaches the builder, in 9.21 seconds)5179subtest: upload whose reference the worker lacks is completed from the cache5180client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache5181client # this derivation will be built:5182client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv5183client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'...5184client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache, in 0.37 seconds)5185client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache5186client # this derivation will be built:5187client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv5188client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'...5189client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache, in 0.32 seconds)5190client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure5191client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.14 seconds)5192worker1: must succeed: nix-store --import < /tmp/shared/ref.closure5193worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.19 seconds)5194worker2 # [ 104.651233] systemd[1]: Stopping Nix worker-protocol over gRPC...5195worker2 # [ 104.858640] nix-grpc-daemon[770]: ts=2026-09-23T04:44:34Z level=info event=signal_exit5196lb # [ 105.305375] niks3-server[973]: 2026/09/23 04:44:34 INFO lead: released remote=192.168.1.4:380185197worker2 # [ 106.363779] nix-grpc-daemon[770]: ts=2026-09-23T04:44:35Z level=info event=scheduler_restarting5198worker1 # [ 106.262733] nix-grpc-daemon[767]: ts=2026-09-23T04:44:35Z level=info event=scheduler_restarting addr=lb:500515199worker2 # [ 106.375571] nix-grpc-daemon[770]: ts=2026-09-23T04:44:35Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:516445200worker2 # [ 106.382315] nix-grpc-daemon[770]: ts=2026-09-23T04:44:35Z level=info event=worker_gone addr=192.168.1.3:500515201worker2 # [ 106.386740] nix-grpc-daemon[770]: ts=2026-09-23T04:44:35Z level=info event=scheduler_restarting addr=lb:500515202worker2 # [ 106.391775] nix-grpc-daemon[770]: ts=2026-09-23T04:44:35Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:516445203worker2 # [ 106.396581] nix-grpc-daemon[770]: ts=2026-09-23T04:44:35Z level=info event=worker_gone addr=192.168.1.4:500515204worker2 # [ 106.426917] nix-daemon[856]: reaped child process 858, status = succeeded5205worker1 # [ 106.463100] nix-grpc-daemon[767]: ts=2026-09-23T04:44:35Z level=info event=scheduler_connected addr=lb:500515206worker1 # [ 106.468648] nix-grpc-daemon[767]: ts=2026-09-23T04:44:35Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5207worker2 # [ 106.598701] nix-grpc-daemon[770]: ts=2026-09-23T04:44:36Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5208worker1 # [ 106.669410] nix-grpc-daemon[767]: ts=2026-09-23T04:44:36Z level=info event=scheduler_connected addr=lb:500515209worker1 # [ 106.674618] nix-grpc-daemon[767]: ts=2026-09-23T04:44:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5210worker1 # [ 106.875290] nix-grpc-daemon[767]: ts=2026-09-23T04:44:36Z level=info event=scheduler_connected addr=lb:500515211worker1 # [ 106.880515] nix-grpc-daemon[767]: ts=2026-09-23T04:44:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5212worker2 # [ 106.988358] nix-grpc-daemon[770]: E0000 00:00:1790138676.510024 816 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 19, started true, shutdown true5213worker2 # [ 106.995179] nix-grpc-daemon[770]: E0000 00:00:1790138676.510381 816 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5214worker1 # [ 107.081250] nix-grpc-daemon[767]: ts=2026-09-23T04:44:36Z level=info event=scheduler_connected addr=lb:500515215worker1 # [ 107.086387] nix-grpc-daemon[767]: ts=2026-09-23T04:44:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5216worker1 # [ 107.288848] nix-grpc-daemon[767]: ts=2026-09-23T04:44:36Z level=info event=scheduler_connected addr=lb:500515217worker1 # [ 107.292923] nix-grpc-daemon[767]: ts=2026-09-23T04:44:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5218worker1 # [ 107.489408] nix-grpc-daemon[767]: ts=2026-09-23T04:44:36Z level=info event=scheduler_connected addr=lb:500515219worker1 # [ 107.494474] nix-grpc-daemon[767]: ts=2026-09-23T04:44:36Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5220worker1 # [ 107.695205] nix-grpc-daemon[767]: ts=2026-09-23T04:44:37Z level=info event=scheduler_connected addr=lb:500515221worker1 # [ 107.700440] nix-grpc-daemon[767]: ts=2026-09-23T04:44:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5222worker2 # [ 107.968952] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5223worker2 # [ 107.976491] systemd[1]: Stopped Nix worker-protocol over gRPC.5224worker2 # [ 107.979108] systemd[1]: nix-grpc-daemon.service: Consumed 3.904s CPU time over 1min 28.356s wall clock time, 70.2M memory peak, 113.2K incoming IP traffic, 275.2K outgoing IP traffic.5225worker2 # [ 107.996465] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5226worker2 # [ 108.005290] systemd[1]: Closed Nix worker-protocol over gRPC.5227worker1 # [ 107.901275] nix-grpc-daemon[767]: ts=2026-09-23T04:44:37Z level=info event=scheduler_connected addr=lb:500515228lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy5229lb: must succeed: curl -sf localhost:9901/clusters5230worker2 # [ 108.010161] systemd[1]: nix-grpc-daemon.socket: Consumed 704us CPU time over 1min 37.667s wall clock time, 256K memory peak, 126.2K incoming IP traffic, 105.9K outgoing IP traffic.5231worker1 # [ 107.906562] nix-grpc-daemon[767]: ts=2026-09-23T04:44:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5232lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5233lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.08 seconds)5234client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&25235worker1 # [ 108.107264] nix-grpc-daemon[767]: ts=2026-09-23T04:44:37Z level=info event=scheduler_connected addr=lb:500515236worker1 # [ 108.112621] nix-grpc-daemon[767]: ts=2026-09-23T04:44:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5237worker1 # [ 108.200516] nix-grpc-daemon[767]: ts=2026-09-23T04:44:37Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:571385238worker1 # [ 108.209427] nix-grpc-daemon[767]: ts=2026-09-23T04:44:37Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:571385239client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache5240client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.27 seconds)5241worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache5242worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.03 seconds)5243worker1 # [ 108.315086] nix-grpc-daemon[767]: ts=2026-09-23T04:44:37Z level=info event=scheduler_connected addr=lb:500515244worker1 # [ 108.319246] nix-grpc-daemon[767]: ts=2026-09-23T04:44:37Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5245worker2 # [ 108.453756] systemd[1]: Listening on Nix worker-protocol over gRPC.5246worker1 # [ 108.423928] systemd[1]: Stopping Nix worker-protocol over gRPC...5247worker1 # [ 108.515756] nix-grpc-daemon[767]: ts=2026-09-23T04:44:38Z level=info event=scheduler_connected addr=lb:500515248worker1 # [ 108.521140] nix-grpc-daemon[767]: ts=2026-09-23T04:44:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5249worker1 # [ 108.637467] nix-grpc-daemon[767]: ts=2026-09-23T04:44:38Z level=info event=signal_exit5250worker1 # [ 108.721753] nix-grpc-daemon[767]: ts=2026-09-23T04:44:38Z level=info event=scheduler_connected addr=lb:500515251worker1 # [ 108.726968] nix-grpc-daemon[767]: ts=2026-09-23T04:44:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5252worker2 # [ 108.963249] systemd[1]: Starting scheduler-token.service...5253worker1 # [ 108.927905] nix-grpc-daemon[767]: ts=2026-09-23T04:44:38Z level=info event=scheduler_connected addr=lb:500515254worker1 # [ 108.933387] nix-grpc-daemon[767]: ts=2026-09-23T04:44:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5255worker2 # [ 109.049392] systemd[1]: scheduler-token.service: Deactivated successfully.5256worker2 # [ 109.056982] systemd[1]: Finished scheduler-token.service.5257worker2 # [ 109.058545] systemd[1]: scheduler-token.service: Consumed 58ms CPU time over 88ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5258worker2 # [ 109.068474] systemd[1]: Starting Nix worker-protocol over gRPC...5259worker2 # [ 109.193252] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:38Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5260worker2 # [ 109.195483] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:38Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5261worker2 # [ 109.199383] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:38Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15262worker2 # [ 109.204944] nix-grpc-daemon[1323]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5263worker2 # [ 109.206380] nix-grpc-daemon[1323]: E0000 00:00:1790138678.726591 1323 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5264lb # [ 109.166073] niks3-server[973]: 2026/09/23 04:44:38 INFO lead: acquired remote=192.168.1.4:408625265worker1 # [ 109.134123] nix-grpc-daemon[767]: ts=2026-09-23T04:44:38Z level=info event=scheduler_connected addr=lb:500515266worker2 # [ 109.248194] nix-grpc-daemon[1323]: E0000 00:00:1790138678.732712 1323 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5267worker2 # [ 109.250461] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:38Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5268worker2 # [ 109.261212] nix-grpc-daemon[1323]: E0000 00:00:1790138678.744347 1366 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5269worker2 # [ 109.264065] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:38Z level=info event=scheduler_take_over5270worker2 # [ 109.265353] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:38Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:492165271worker2 # [ 109.266872] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:38Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05272worker2 # [ 109.271258] nix-grpc-daemon[1323]: E0000 00:00:1790138678.763735 1366 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5273worker2 # [ 109.276164] nix-grpc-daemon[1323]: E0000 00:00:1790138678.763851 1366 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5274worker2 # [ 109.278884] systemd[1]: Started Nix worker-protocol over gRPC.5275worker2 # [ 109.279882] nix-grpc-daemon[1323]: E0000 00:00:1790138678.782487 1366 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5276worker2 # [ 109.282929] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:38Z level=info event=scheduler_connected addr=lb:500515277worker2 # [ 109.286052] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:38Z level=info event=oidc_jwks provider=mock keys=15278worker2 # [ 109.287325] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:38Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:492165279worker2 # [ 109.289112] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:38Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05280lb # [ 109.421331] envoy[900]: conn peer=[2001:db8:1::1]:36434 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3765 tx=2123 ms=26215281lb # [ 109.426372] envoy[900]: conn peer=[2001:db8:1::1]:36442 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=685282lb # [ 109.431088] envoy[900]: conn peer=[2001:db8:1::1]:34412 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=325283lb # [ 109.435772] envoy[900]: conn peer=[2001:db8:1::1]:34400 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3753 tx=2100 ms=24265284lb # [ 109.441371] envoy[900]: conn peer=[2001:db8:1::4]:36966 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=58704 tx=3607 ms=866845285lb # [ 109.446358] envoy[900]: conn peer=[2001:db8:1::1]:34420 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=305286worker1 # [ 110.261257] nix-grpc-daemon[767]: ts=2026-09-23T04:44:39Z level=info event=scheduler_restarting5287worker2 # [ 110.377538] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:39Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:492165288worker2 # [ 110.382360] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:39Z level=info event=worker_gone addr=192.168.1.3:500515289worker1 # [ 110.305608] nix-grpc-daemon[767]: ts=2026-09-23T04:44:39Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5290worker1 # [ 110.322415] nix-daemon[834]: reaped child process 836, status = succeeded5291worker1 # [ 110.801565] nix-grpc-daemon[767]: E0000 00:00:1790138680.300044 813 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true5292worker1 # [ 110.810425] nix-grpc-daemon[767]: E0000 00:00:1790138680.308836 813 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5293worker1 # [ 111.811024] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5294worker1 # [ 111.818357] systemd[1]: Stopped Nix worker-protocol over gRPC.5295worker1 # [ 111.825099] systemd[1]: nix-grpc-daemon.service: Consumed 3.016s CPU time over 1min 32.433s wall clock time, 60.6M memory peak, 88.3K incoming IP traffic, 157.5K outgoing IP traffic.5296worker1 # [ 111.840978] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5297worker1 # [ 111.843204] systemd[1]: Closed Nix worker-protocol over gRPC.5298worker1 # [ 111.849607] systemd[1]: nix-grpc-daemon.socket: Consumed 617us CPU time over 1min 42.157s wall clock time, 256K memory peak, 95.2K incoming IP traffic, 82.5K outgoing IP traffic.5299lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy5300lb: must succeed: curl -sf localhost:9901/clusters5301lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5302lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.07 seconds)5303client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&25304worker2 # [ 112.261186] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:341365305worker2 # [ 112.276436] nix-daemon[856]: accepted connection from pid 1323, user nix-grpc-daemon (trusted)5306worker2 # [ 112.301603] nix-grpc-daemon[1323]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 21 ms (attempt 1/5)5307worker2 # [ 112.331633] nix-grpc-daemon[1323]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 66 ms (attempt 2/5)5308worker2 # [ 112.399543] nix-grpc-daemon[1323]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 88 ms (attempt 3/5)5309worker2 # [ 112.489588] nix-grpc-daemon[1323]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 45 ms (attempt 4/5)5310worker2 # [ 112.548098] nix-grpc-daemon[1323]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5311worker2 # [ 112.551832] nix-grpc-daemon[1323]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5312client # copying 2 paths...5313client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'...5314worker2 # [ 112.577742] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:42Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:341365315client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'...5316worker2 # [ 112.595375] nix-daemon[856]: accepted connection from pid 1323, user nix-grpc-daemon (trusted)5317worker2 # [ 112.626439] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:42Z level=info event=niks3_push_started pid=13845318worker2 # [ 112.777472] nix-grpc-daemon[1384]: time=2026-09-23T04:44:42.299Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""5319worker2 # [ 112.887513] nix-daemon[856]: accepted connection from pid 1396, user nix-grpc-daemon (trusted)5320worker2 # [ 112.908109] nix-daemon[856]: reaped child process 1399, status = succeeded5321lb # [ 112.853937] niks3-server[973]: 2026/09/23 04:44:42 INFO Received uploads request method=POST path=/api/pending_closures5322lb # [ 112.863741] niks3-server[973]: 2026/09/23 04:44:42 INFO Received uploads request method=POST path=/api/pending_closures5323worker2 # [ 112.936102] nix-grpc-daemon[1384]: time=2026-09-23T04:44:42.456Z level=INFO msg="Uploading 2 paths to lb (0 already cached)"5324worker2 # [ 112.937702] nix-grpc-daemon[1384]: time=2026-09-23T04:44:42.456Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)"5325worker2 # [ 112.939557] nix-grpc-daemon[1384]: time=2026-09-23T04:44:42.456Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)"5326lb # [ 112.917822] niks3-server[973]: 2026/09/23 04:44:42 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst5327lb # [ 112.934318] niks3-server[973]: 2026/09/23 04:44:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign5328lb # [ 112.946302] niks3-server[973]: 2026/09/23 04:44:42 INFO Signed narinfos id=35 count=15329lb # [ 112.951596] niks3-server[973]: 2026/09/23 04:44:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign5330lb # [ 112.960225] niks3-server[973]: 2026/09/23 04:44:42 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls5331lb # [ 112.962706] niks3-server[973]: 2026/09/23 04:44:42 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls5332lb # [ 112.968070] niks3-server[973]: 2026/09/23 04:44:42 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst5333worker2 # [ 113.028135] nix-grpc-daemon[1384]: time=2026-09-23T04:44:42.549Z level=INFO msg="Uploading 3 narinfos"5334lb # [ 112.971241] niks3-server[973]: 2026/09/23 04:44:42 INFO Signed narinfos id=36 count=25335lb # [ 112.995653] niks3-server[973]: 2026/09/23 04:44:42 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete5336lb # [ 113.005920] niks3-server[973]: 2026/09/23 04:44:42 INFO Completed upload id=355337lb # [ 113.007277] niks3-server[973]: 2026/09/23 04:44:42 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete5338lb # [ 113.013795] niks3-server[973]: 2026/09/23 04:44:42 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo5339worker2 # [ 113.071391] nix-grpc-daemon[1384]: time=2026-09-23T04:44:42.593Z level=INFO msg="Upload complete. (290ms)"5340lb # [ 113.017069] niks3-server[973]: 2026/09/23 04:44:42 INFO Completed upload id=365341lb # [ 113.018423] niks3-server[973]: 2026/09/23 04:44:42 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5342lb # [ 113.021645] niks3-server[973]: 2026/09/23 04:44:42 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo5343worker2 # [ 113.084438] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34136 duration_s=0 paths=2 nar_bytes_in=3125344worker2 # [ 113.101243] nix-daemon[856]: reaped child process 1382, status = succeeded5345client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.08 seconds)5346worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache5347worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.03 seconds)5348worker1 # [ 113.113275] systemd[1]: Listening on Nix worker-protocol over gRPC.5349worker1 # [ 113.121880] systemd[1]: Starting Nix worker-protocol over gRPC...5350worker1 # [ 113.268429] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:42Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5351worker1 # [ 113.270863] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:42Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5352worker1 # [ 113.274876] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:42Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15353worker1 # [ 113.280068] nix-grpc-daemon[1153]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5354worker1 # [ 113.281519] nix-grpc-daemon[1153]: E0000 00:00:1790138682.778446 1153 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5355lb: waiting until exactly no endpoints of aarch64-linux are unhealthy5356lb: must succeed: curl -sf localhost:9901/clusters5357worker1 # [ 113.326274] nix-grpc-daemon[1153]: E0000 00:00:1790138682.784943 1153 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5358worker1 # [ 113.330165] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:42Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5359worker1 # [ 113.332402] nix-grpc-daemon[1153]: E0000 00:00:1790138682.792663 1193 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5360worker1 # [ 113.334921] nix-grpc-daemon[1153]: E0000 00:00:1790138682.809860 1193 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5361worker1 # [ 113.337298] nix-grpc-daemon[1153]: E0000 00:00:1790138682.810029 1193 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5362worker1 # [ 113.339608] systemd[1]: Started Nix worker-protocol over gRPC.5363worker1 # [ 113.340474] nix-grpc-daemon[1153]: E0000 00:00:1790138682.827998 1193 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5364worker1 # [ 113.348119] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:42Z level=info event=scheduler_connected addr=lb:500515365worker2 # [ 113.454608] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:42Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:492165366worker2 # [ 113.460284] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:42Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05367lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5368lb: must succeed: curl -sf localhost:9901/clusters5369lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5370lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.15 seconds)5371(finished: subtest: upload whose reference the worker lacks is completed from the cache, in 11.00 seconds)5372subtest: a niks3 restart does not cost the leader its role5373lb: must succeed: curl -sf localhost:9901/clusters5374lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5375lb: must succeed: curl -sf localhost:9901/clusters5376lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5377worker1 # [ 114.580126] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:44Z level=info event=scheduler_lock_lost status=2005378lb # [ 114.622546] systemd[1]: Stopping niks3 server...5379lb # [ 114.623447] niks3-server[973]: 2026/09/23 04:44:44 INFO Shutdown signal received, draining in-flight requests timeout=10s5380lb # [ 114.633122] niks3-server[973]: 2026/09/23 04:44:44 INFO lead: released remote=192.168.1.4:408625381worker2 # [ 114.693791] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:44Z level=info event=scheduler_lock_lost status=2005382lb # [ 114.650172] systemd[1]: niks3.service: Deactivated successfully.5383lb # [ 114.651933] systemd[1]: Stopped niks3 server.5384lb # [ 114.652992] systemd[1]: niks3.service: Consumed 683ms CPU time over 1min 30.589s wall clock time, 27.6M memory peak, 90.7K incoming IP traffic, 134.2K outgoing IP traffic.5385lb # [ 114.663267] systemd[1]: Starting niks3 server...5386lb # [ 114.725161] niks3-server[1114]: 2026/09/23 04:44:44 goose: no migrations to run. current version: 202609200000005387lb # [ 114.731498] niks3-server[1114]: 2026/09/23 04:44:44 OK 1_commit_pending_closure.sql (4.54ms)5388lb # [ 114.735199] niks3-server[1114]: 2026/09/23 04:44:44 OK 2_object_stats_trigger.sql (3.36ms)5389lb # [ 114.737784] niks3-server[1114]: 2026/09/23 04:44:44 goose: up to current file version: 25390lb # [ 114.739130] niks3-server[1114]: 2026/09/23 04:44:44 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key5391lb # [ 114.749344] niks3-server[1114]: 2026/09/23 04:44:44 INFO Read proxy enabled — serving cache objects from S35392lb # [ 114.750677] niks3-server[1114]: 2026/09/23 04:44:44 INFO Using socket-activated listener address=0.0.0.0:57515393lb # [ 114.755780] niks3-server[1114]: 2026/09/23 04:44:44 INFO systemd watchdog enabled interval=15s5394lb # [ 114.757741] niks3-server[1114]: 2026/09/23 04:44:44 INFO Starting HTTP server address=0.0.0.0:57515395lb # [ 114.760229] systemd[1]: Started niks3 server.5396lb: waiting for unit niks3.service5397lb: (finished: waiting for unit niks3.service, in 0.03 seconds)5398lb: must succeed: curl -sf localhost:9901/clusters5399lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.02 seconds)5400worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5401worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5402lb: must succeed: curl -sf localhost:9901/clusters5403lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5404lb: must succeed: curl -sf localhost:9901/clusters5405lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5406worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield5407worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.07 seconds)5408client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&25409worker1 # [ 115.202478] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:44Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:419045410worker1 # [ 115.216301] nix-daemon[834]: accepted connection from pid 1153, user nix-grpc-daemon (trusted)5411client # don't know how to build these paths:5412client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5413worker2 # [ 115.344306] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:341365414worker2 # [ 115.375220] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:44Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=05415worker2 # [ 115.382752] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:44Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.4:50051 clients=15416worker2 # [ 115.423703] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:341365417worker2 # [ 115.439088] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:341365418client # copying 1 paths to 'lb:50051'...5419client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'...5420worker2 # [ 115.454697] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:341365421worker2 # [ 115.469540] nix-daemon[856]: accepted connection from pid 1323, user nix-grpc-daemon (trusted)5422worker2 # [ 115.627844] nix-daemon[856]: accepted connection from pid 1416, user nix-grpc-daemon (trusted)5423worker2 # [ 115.649958] nix-daemon[856]: reaped child process 1419, status = succeeded5424lb # [ 115.588700] niks3-server[1114]: 2026/09/23 04:44:45 INFO Received uploads request method=POST path=/api/pending_closures5425worker2 # [ 115.661874] nix-grpc-daemon[1384]: time=2026-09-23T04:44:45.183Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5426worker2 # [ 115.663317] nix-grpc-daemon[1384]: time=2026-09-23T04:44:45.183Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)"5427lb # [ 115.632685] niks3-server[1114]: 2026/09/23 04:44:45 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst5428lb # [ 115.656363] niks3-server[1114]: 2026/09/23 04:44:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign5429worker2 # [ 115.734759] nix-grpc-daemon[1384]: time=2026-09-23T04:44:45.255Z level=INFO msg="Uploading 1 narinfos"5430lb # [ 115.684343] niks3-server[1114]: 2026/09/23 04:44:45 INFO Signed narinfos id=37 count=15431lb # [ 115.692214] niks3-server[1114]: 2026/09/23 04:44:45 INFO lead: acquired remote=192.168.1.4:339565432lb # [ 115.703278] niks3-server[1114]: 2026/09/23 04:44:45 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete5433lb # [ 115.717543] niks3-server[1114]: 2026/09/23 04:44:45 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls5434lb # [ 115.731075] niks3-server[1114]: 2026/09/23 04:44:45 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.narinfo5435lb # [ 115.741546] niks3-server[1114]: 2026/09/23 04:44:45 INFO Completed upload id=375436worker2 # [ 115.801163] nix-grpc-daemon[1384]: time=2026-09-23T04:44:45.323Z level=INFO msg="Upload complete. (314ms)"5437worker2 # [ 115.814158] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34136 duration_s=0 paths=1 nar_bytes_in=5445438worker2 # [ 115.828411] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:341365439worker2 # [ 115.843014] nix-daemon[856]: accepted connection from pid 1323, user nix-grpc-daemon (trusted)5440worker2 # [ 115.846698] nix-daemon[856]: reaped child process 1414, status = succeeded5441client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051...5442worker2 # [ 115.858602] nix-daemon[856]: accepted connection from pid 1323, user nix-grpc-daemon (trusted)5443client # farm-a> worker2: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5444client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5445client # farm-a> LOG-farm-a-after-niks3-restart5446worker2 # [ 117.319750] nix-daemon[856]: reaped child process 1425, status = succeeded5447worker2 # [ 117.454263] nix-daemon[856]: accepted connection from pid 1441, user nix-grpc-daemon (trusted)5448worker2 # [ 117.473740] nix-daemon[856]: reaped child process 1444, status = succeeded5449lb # [ 117.420642] niks3-server[1114]: 2026/09/23 04:44:46 INFO Received uploads request method=POST path=/api/pending_closures5450worker2 # [ 117.493602] nix-grpc-daemon[1384]: time=2026-09-23T04:44:47.013Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5451worker2 # [ 117.495045] nix-grpc-daemon[1384]: time=2026-09-23T04:44:47.013Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)"5452lb # [ 117.466591] niks3-server[1114]: 2026/09/23 04:44:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign5453worker2 # [ 117.537549] nix-grpc-daemon[1384]: time=2026-09-23T04:44:47.059Z level=INFO msg="Uploading 1 narinfos"5454lb # [ 117.481524] niks3-server[1114]: 2026/09/23 04:44:46 INFO Signed narinfos id=38 count=15455lb # [ 117.498732] niks3-server[1114]: 2026/09/23 04:44:47 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls5456lb # [ 117.504509] niks3-server[1114]: 2026/09/23 04:44:47 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst5457lb # [ 117.524946] niks3-server[1114]: 2026/09/23 04:44:47 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete5458worker2 # [ 117.594754] nix-grpc-daemon[1384]: time=2026-09-23T04:44:47.116Z level=INFO msg="Upload complete. (322ms)"5459lb # [ 117.534557] niks3-server[1114]: 2026/09/23 04:44:47 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv5460lb # [ 117.544620] niks3-server[1114]: 2026/09/23 04:44:47 INFO Completed upload id=385461worker2 # [ 117.617439] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34136 duration_s=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=15462worker2 # [ 117.635220] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:47Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=05463worker2 # [ 117.639933] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:47Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.4:50051 clients=15464worker2 # [ 117.644817] nix-daemon[856]: reaped child process 1423, status = succeeded5465worker2 # [ 117.659549] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:341365466worker2 # [ 117.673324] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:341365467worker2 # [ 117.687369] nix-grpc-daemon[1384]: time=2026-09-23T04:44:47.209Z level=INFO msg="All 1 paths already cached"5468client # copying 1 paths to 'lb:50051'...5469client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'...5470worker2 # [ 117.696855] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:341365471worker2 # [ 117.708929] nix-daemon[856]: accepted connection from pid 1323, user nix-grpc-daemon (trusted)5472worker2 # [ 117.850861] nix-daemon[856]: accepted connection from pid 1450, user nix-grpc-daemon (trusted)5473worker2 # [ 117.872874] nix-daemon[856]: reaped child process 1453, status = succeeded5474lb # [ 117.814530] niks3-server[1114]: 2026/09/23 04:44:47 INFO Received uploads request method=POST path=/api/pending_closures5475worker2 # [ 117.887501] nix-grpc-daemon[1384]: time=2026-09-23T04:44:47.409Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5476worker2 # [ 117.889218] nix-grpc-daemon[1384]: time=2026-09-23T04:44:47.409Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)"5477worker2 # [ 117.925535] nix-grpc-daemon[1384]: time=2026-09-23T04:44:47.447Z level=INFO msg="Uploading 1 narinfos"5478lb # [ 117.860506] niks3-server[1114]: 2026/09/23 04:44:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign5479lb # [ 117.873297] niks3-server[1114]: 2026/09/23 04:44:47 INFO Signed narinfos id=39 count=15480lb # [ 117.875435] niks3-server[1114]: 2026/09/23 04:44:47 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls5481lb # [ 117.887546] niks3-server[1114]: 2026/09/23 04:44:47 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst5482worker2 # [ 117.952173] nix-grpc-daemon[1384]: time=2026-09-23T04:44:47.474Z level=INFO msg="Upload complete. (228ms)"5483lb # [ 117.894452] niks3-server[1114]: 2026/09/23 04:44:47 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete5484lb # [ 117.897866] niks3-server[1114]: 2026/09/23 04:44:47 INFO Completed upload id=395485lb # [ 117.900548] niks3-server[1114]: 2026/09/23 04:44:47 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo5486worker2 # [ 117.960170] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34136 duration_s=0 paths=1 nar_bytes_in=6565487worker2 # [ 117.976696] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:341365488worker2 # [ 117.983213] nix-daemon[856]: accepted connection from pid 1323, user nix-grpc-daemon (trusted)5489worker2 # [ 117.989440] nix-daemon[856]: reaped child process 1448, status = succeeded5490client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051...5491worker2 # [ 118.000208] nix-daemon[856]: accepted connection from pid 1323, user nix-grpc-daemon (trusted)5492client # farm-b> worker2: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5493client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5494client # farm-b> LOG-farm-b-after-niks3-restart5495worker2 # [ 119.265737] nix-daemon[856]: reaped child process 1457, status = succeeded5496worker2 # [ 119.387199] nix-daemon[856]: accepted connection from pid 1474, user nix-grpc-daemon (trusted)5497worker2 # [ 119.403768] nix-daemon[856]: reaped child process 1477, status = succeeded5498lb # [ 119.351492] niks3-server[1114]: 2026/09/23 04:44:48 INFO Received uploads request method=POST path=/api/pending_closures5499worker2 # [ 119.421416] nix-grpc-daemon[1384]: time=2026-09-23T04:44:48.942Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5500worker2 # [ 119.422998] nix-grpc-daemon[1384]: time=2026-09-23T04:44:48.942Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)"5501lb # [ 119.389483] niks3-server[1114]: 2026/09/23 04:44:48 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv5502lb # [ 119.402603] niks3-server[1114]: 2026/09/23 04:44:48 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst5503worker2 # [ 119.472686] nix-grpc-daemon[1384]: time=2026-09-23T04:44:48.994Z level=INFO msg="Uploading 1 narinfos"5504lb # [ 119.416974] niks3-server[1114]: 2026/09/23 04:44:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign5505lb # [ 119.429069] niks3-server[1114]: 2026/09/23 04:44:48 INFO Signed narinfos id=40 count=15506lb # [ 119.431733] niks3-server[1114]: 2026/09/23 04:44:48 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls5507lb # [ 119.448489] envoy[900]: conn peer=[2001:db8:1::3]:42958 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=13887 tx=3853 ms=906115508lb # [ 119.451504] envoy[900]: conn peer=[2001:db8:1::1]:60082 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="TLS_error:|33554464:system_library:OPENSSL_internal:Broken_pipe:TLS_error_end" rx=1005 tx=305 ms=8395509lb # [ 119.459334] niks3-server[1114]: 2026/09/23 04:44:48 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete5510worker2 # [ 119.519393] nix-grpc-daemon[1384]: time=2026-09-23T04:44:49.041Z level=INFO msg="Upload complete. (307ms)"5511lb # [ 119.462630] niks3-server[1114]: 2026/09/23 04:44:48 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.narinfo5512lb # [ 119.466252] niks3-server[1114]: 2026/09/23 04:44:48 INFO Completed upload id=405513worker2 # [ 119.537309] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34136 duration_s=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=15514worker2 # [ 119.556484] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:49Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=05515worker2 # [ 119.563219] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:49Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.4:50051 clients=15516worker2 # [ 119.568962] nix-daemon[856]: reaped child process 1455, status = succeeded5517worker2 # [ 119.579536] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:341365518worker2 # [ 119.594939] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:341365519worker2 # [ 119.609379] nix-grpc-daemon[1384]: time=2026-09-23T04:44:49.131Z level=INFO msg="All 2 paths already cached"5520client # copying 1 paths to 'lb:50051'...5521client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'...5522worker2 # [ 119.620446] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:341365523worker2 # [ 119.633935] nix-daemon[856]: accepted connection from pid 1323, user nix-grpc-daemon (trusted)5524worker2 # [ 119.783939] nix-daemon[856]: accepted connection from pid 1483, user nix-grpc-daemon (trusted)5525worker2 # [ 119.804644] nix-daemon[856]: reaped child process 1486, status = succeeded5526lb # [ 119.744303] niks3-server[1114]: 2026/09/23 04:44:49 INFO Received uploads request method=POST path=/api/pending_closures5527worker2 # [ 119.812798] nix-grpc-daemon[1384]: time=2026-09-23T04:44:49.334Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5528worker2 # [ 119.814274] nix-grpc-daemon[1384]: time=2026-09-23T04:44:49.334Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)"5529lb # [ 119.772712] niks3-server[1114]: 2026/09/23 04:44:49 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls5530lb # [ 119.787840] niks3-server[1114]: 2026/09/23 04:44:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign5531worker2 # [ 119.854951] nix-grpc-daemon[1384]: time=2026-09-23T04:44:49.376Z level=INFO msg="Uploading 1 narinfos"5532lb # [ 119.799103] niks3-server[1114]: 2026/09/23 04:44:49 INFO Signed narinfos id=41 count=15533lb # [ 119.807397] niks3-server[1114]: 2026/09/23 04:44:49 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst5534lb # [ 119.820114] niks3-server[1114]: 2026/09/23 04:44:49 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete5535lb # [ 119.831395] niks3-server[1114]: 2026/09/23 04:44:49 INFO Completed upload id=415536worker2 # [ 119.890808] nix-grpc-daemon[1384]: time=2026-09-23T04:44:49.412Z level=INFO msg="Upload complete. (241ms)"5537lb # [ 119.834757] niks3-server[1114]: 2026/09/23 04:44:49 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo5538worker2 # [ 119.900437] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:34136 duration_s=0 paths=1 nar_bytes_in=7845539worker2 # [ 119.919197] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:341365540worker2 # [ 119.926237] nix-daemon[856]: accepted connection from pid 1323, user nix-grpc-daemon (trusted)5541worker2 # [ 119.931436] nix-daemon[856]: reaped child process 1481, status = succeeded5542client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051...5543worker2 # [ 119.943622] nix-daemon[856]: accepted connection from pid 1323, user nix-grpc-daemon (trusted)5544client # farm-top> worker2: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5545client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5546client # farm-top> LOG-farm-top-after-niks3-restart5547worker2 # [ 120.778121] nix-daemon[856]: reaped child process 1490, status = succeeded5548worker2 # [ 120.887022] nix-daemon[856]: accepted connection from pid 1506, user nix-grpc-daemon (trusted)5549worker2 # [ 120.907613] nix-daemon[856]: reaped child process 1509, status = succeeded5550lb # [ 120.853019] niks3-server[1114]: 2026/09/23 04:44:50 INFO Received uploads request method=POST path=/api/pending_closures5551worker2 # [ 120.922610] nix-grpc-daemon[1384]: time=2026-09-23T04:44:50.443Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5552worker2 # [ 120.924135] nix-grpc-daemon[1384]: time=2026-09-23T04:44:50.443Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)"5553lb # [ 120.896168] niks3-server[1114]: 2026/09/23 04:44:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign5554worker2 # [ 120.965937] nix-grpc-daemon[1384]: time=2026-09-23T04:44:50.487Z level=INFO msg="Uploading 1 narinfos"5555lb # [ 120.909850] niks3-server[1114]: 2026/09/23 04:44:50 INFO Signed narinfos id=42 count=15556lb # [ 120.916494] niks3-server[1114]: 2026/09/23 04:44:50 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv5557lb # [ 120.925376] niks3-server[1114]: 2026/09/23 04:44:50 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls5558worker2 # [ 120.995972] nix-grpc-daemon[1384]: time=2026-09-23T04:44:50.518Z level=INFO msg="Upload complete. (239ms)"5559lb # [ 120.936759] niks3-server[1114]: 2026/09/23 04:44:50 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst5560lb # [ 120.947064] niks3-server[1114]: 2026/09/23 04:44:50 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete5561lb # [ 120.952500] niks3-server[1114]: 2026/09/23 04:44:50 INFO Completed upload id=425562worker2 # [ 121.008916] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:34136 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=15563lb # [ 120.954818] niks3-server[1114]: 2026/09/23 04:44:50 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo5564worker2 # [ 121.028196] nix-daemon[856]: reaped child process 1488, status = succeeded5565client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2, in 5.97 seconds)5566client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath5567client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath, in 0.18 seconds)5568(finished: subtest: a niks3 restart does not cost the leader its role, in 6.67 seconds)5569subtest: scheduler down: the other node takes the lock and keeps it5570lb: must succeed: curl -sf localhost:9901/clusters5571lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5572lb: must succeed: curl -sf localhost:9901/clusters5573lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5574lb: must succeed: curl -sf localhost:9901/clusters5575lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5576lb: must succeed: curl -sf localhost:9901/clusters5577lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5578worker2 # [ 121.479731] systemd[1]: Stopping Nix worker-protocol over gRPC...5579worker2 # [ 122.220317] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:51Z level=info event=signal_exit5580lb # [ 122.648938] niks3-server[1114]: 2026/09/23 04:44:52 INFO lead: released remote=192.168.1.4:339565581worker1 # [ 123.606762] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:53Z level=info event=scheduler_restarting addr=lb:500515582worker2 # [ 123.713380] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:53Z level=info event=scheduler_restarting5583worker2 # [ 123.717721] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:53Z level=info event=scheduler_restarting addr=lb:500515584worker2 # [ 123.723117] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:53Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:492165585worker2 # [ 123.728570] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:53Z level=info event=worker_gone addr=192.168.1.4:500515586worker2 # [ 123.732858] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:53Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:492165587worker2 # [ 123.737828] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:53Z level=info event=worker_gone addr=192.168.1.3:500515588worker2 # [ 123.763046] nix-daemon[856]: reaped child process 1376, status = succeeded5589worker1 # [ 123.806421] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:53Z level=info event=scheduler_connected addr=lb:500515590worker1 # [ 123.812127] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5591worker2 # [ 123.929903] nix-grpc-daemon[1323]: ts=2026-09-23T04:44:53Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5592worker1 # [ 124.012748] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:53Z level=info event=scheduler_connected addr=lb:500515593worker1 # [ 124.017924] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5594worker2 # [ 124.264888] nix-grpc-daemon[1323]: E0000 00:00:1790138693.786740 1369 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true5595worker2 # [ 124.271521] nix-grpc-daemon[1323]: E0000 00:00:1790138693.786945 1369 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5596worker1 # [ 124.218713] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:53Z level=info event=scheduler_connected addr=lb:500515597worker1 # [ 124.223868] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5598worker1 # [ 124.424808] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:53Z level=info event=scheduler_connected addr=lb:500515599worker1 # [ 124.429954] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5600worker1 # [ 124.632437] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:54Z level=info event=scheduler_connected addr=lb:500515601worker1 # [ 124.636605] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5602worker1 # [ 124.833176] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:54Z level=info event=scheduler_connected addr=lb:500515603worker1 # [ 124.838069] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5604worker2 # [ 124.970615] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5605worker2 # [ 124.975720] systemd[1]: Stopped Nix worker-protocol over gRPC.5606worker2 # [ 124.984639] systemd[1]: nix-grpc-daemon.service: Consumed 1.367s CPU time over 15.910s wall clock time, 39.1M memory peak, 46K incoming IP traffic, 78K outgoing IP traffic.5607worker2 # [ 124.996808] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully.5608worker2 # [ 125.002269] systemd[1]: Closed Nix worker-protocol over gRPC.5609worker2 # [ 125.004550] systemd[1]: nix-grpc-daemon.socket: Consumed 765us CPU time over 16.558s wall clock time, 256K memory peak, 38.9K incoming IP traffic, 29K outgoing IP traffic.5610worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over5611worker1 # [ 125.038970] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:54Z level=info event=scheduler_connected addr=lb:500515612worker1 # [ 125.044331] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5613worker1 # [ 125.245041] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:54Z level=info event=scheduler_connected addr=lb:500515614worker1 # [ 125.250320] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5615worker1 # [ 125.451145] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:54Z level=info event=scheduler_connected addr=lb:500515616worker1 # [ 125.456409] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5617worker1 # [ 125.585946] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:55Z level=info event=scheduler_take_over5618lb # [ 125.630400] niks3-server[1114]: 2026/09/23 04:44:55 INFO lead: acquired remote=192.168.1.3:568285619worker1 # [ 125.658932] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:55Z level=info event=scheduler_connected addr=lb:500515620worker1 # [ 125.663152] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5621worker1 # [ 125.859459] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:55Z level=info event=scheduler_connected addr=lb:500515622worker1 # [ 125.868389] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:55Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:548265623worker1 # [ 125.874395] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:55Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05624worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 1.15 seconds)5625worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5626worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds)5627client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&25628worker1 # [ 126.318857] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:55Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:419045629client # don't know how to build these paths:5630client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5631worker1 # [ 126.329085] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419045632worker1 # [ 126.344148] nix-grpc-daemon[1153]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 95 ms (attempt 1/5)5633worker1 # [ 126.448372] nix-grpc-daemon[1153]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 143 ms (attempt 2/5)5634worker1 # [ 126.593376] nix-grpc-daemon[1153]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 55 ms (attempt 3/5)5635worker1 # [ 126.650395] nix-grpc-daemon[1153]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 473 ms (attempt 4/5)5636worker1 # [ 127.128226] nix-grpc-daemon[1153]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5637worker1 # [ 127.131904] nix-grpc-daemon[1153]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5638worker1 # [ 127.163387] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:56Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=05639worker1 # [ 127.170600] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:56Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.3:50051 clients=15640worker1 # [ 127.208973] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419045641worker1 # [ 127.224746] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419045642client # copying 1 paths to 'lb:50051'...5643client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'...5644worker1 # [ 127.239119] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:419045645worker1 # [ 127.254234] nix-daemon[834]: accepted connection from pid 1153, user nix-grpc-daemon (trusted)5646worker1 # [ 127.293684] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:56Z level=info event=niks3_push_started pid=12335647worker1 # [ 127.430001] nix-grpc-daemon[1233]: time=2026-09-23T04:44:56.928Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""5648worker1 # [ 127.534643] nix-daemon[834]: accepted connection from pid 1246, user nix-grpc-daemon (trusted)5649worker1 # [ 127.553164] nix-daemon[834]: reaped child process 1249, status = succeeded5650lb # [ 127.603082] niks3-server[1114]: 2026/09/23 04:44:57 INFO Received uploads request method=POST path=/api/pending_closures5651worker1 # [ 127.569518] nix-grpc-daemon[1233]: time=2026-09-23T04:44:57.067Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5652worker1 # [ 127.570967] nix-grpc-daemon[1233]: time=2026-09-23T04:44:57.067Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)"5653lb # [ 127.649048] niks3-server[1114]: 2026/09/23 04:44:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign5654worker1 # [ 127.612107] nix-grpc-daemon[1233]: time=2026-09-23T04:44:57.110Z level=INFO msg="Uploading 1 narinfos"5655lb # [ 127.658640] niks3-server[1114]: 2026/09/23 04:44:57 INFO Signed narinfos id=43 count=15656lb # [ 127.669730] niks3-server[1114]: 2026/09/23 04:44:57 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls5657lb # [ 127.679682] niks3-server[1114]: 2026/09/23 04:44:57 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst5658lb # [ 127.697555] niks3-server[1114]: 2026/09/23 04:44:57 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete5659lb # [ 127.706252] niks3-server[1114]: 2026/09/23 04:44:57 INFO Completed upload id=435660worker1 # [ 127.661964] nix-grpc-daemon[1233]: time=2026-09-23T04:44:57.160Z level=INFO msg="Upload complete. (228ms)"5661lb # [ 127.709398] niks3-server[1114]: 2026/09/23 04:44:57 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo5662worker1 # [ 127.672360] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41904 duration_s=0 paths=1 nar_bytes_in=5205663worker1 # [ 127.689934] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419045664worker1 # [ 127.696923] nix-daemon[834]: accepted connection from pid 1153, user nix-grpc-daemon (trusted)5665worker1 # [ 127.702346] nix-daemon[834]: reaped child process 1231, status = succeeded5666client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051...5667worker1 # [ 127.715067] nix-daemon[834]: accepted connection from pid 1153, user nix-grpc-daemon (trusted)5668client # farm-a> node-a: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5669client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds5670client # farm-a> LOG-farm-a-on-standby5671worker1 # [ 128.565827] nix-daemon[834]: reaped child process 1253, status = succeeded5672worker1 # [ 128.691675] nix-daemon[834]: accepted connection from pid 1269, user nix-grpc-daemon (trusted)5673worker1 # [ 128.710255] nix-daemon[834]: reaped child process 1272, status = succeeded5674lb # [ 128.759842] niks3-server[1114]: 2026/09/23 04:44:58 INFO Received uploads request method=POST path=/api/pending_closures5675worker1 # [ 128.727963] nix-grpc-daemon[1233]: time=2026-09-23T04:44:58.224Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5676worker1 # [ 128.730685] nix-grpc-daemon[1233]: time=2026-09-23T04:44:58.224Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)"5677lb # [ 128.804138] niks3-server[1114]: 2026/09/23 04:44:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign5678worker1 # [ 128.772340] nix-grpc-daemon[1233]: time=2026-09-23T04:44:58.270Z level=INFO msg="Uploading 1 narinfos"5679lb # [ 128.820644] niks3-server[1114]: 2026/09/23 04:44:58 INFO Signed narinfos id=44 count=15680lb # [ 128.835546] niks3-server[1114]: 2026/09/23 04:44:58 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv5681lb # [ 128.845964] niks3-server[1114]: 2026/09/23 04:44:58 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst5682lb # [ 128.862277] niks3-server[1114]: 2026/09/23 04:44:58 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete5683lb # [ 128.866102] niks3-server[1114]: 2026/09/23 04:44:58 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo5684lb # [ 128.871988] niks3-server[1114]: 2026/09/23 04:44:58 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls5685worker1 # [ 128.828150] nix-grpc-daemon[1233]: time=2026-09-23T04:44:58.326Z level=INFO msg="Upload complete. (299ms)"5686lb # [ 128.875096] niks3-server[1114]: 2026/09/23 04:44:58 INFO Completed upload id=445687worker1 # [ 128.853125] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41904 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=15688worker1 # [ 128.868341] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:58Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=05689worker1 # [ 128.879154] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:58Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.3:50051 clients=15690worker1 # [ 128.885337] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419045691client # copying 1 paths to 'lb:50051'...5692worker1 # [ 128.890776] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419045693worker1 # [ 128.894967] nix-daemon[834]: reaped child process 1251, status = succeeded5694client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'...5695worker1 # [ 128.897995] nix-grpc-daemon[1233]: time=2026-09-23T04:44:58.388Z level=INFO msg="All 1 paths already cached"5696worker1 # [ 128.902916] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:419045697worker1 # [ 128.916422] nix-daemon[834]: accepted connection from pid 1153, user nix-grpc-daemon (trusted)5698worker1 # [ 129.044086] nix-daemon[834]: accepted connection from pid 1278, user nix-grpc-daemon (trusted)5699worker1 # [ 129.066375] nix-daemon[834]: reaped child process 1281, status = succeeded5700lb # [ 129.108071] niks3-server[1114]: 2026/09/23 04:44:58 INFO Received uploads request method=POST path=/api/pending_closures5701worker1 # [ 129.073299] nix-grpc-daemon[1233]: time=2026-09-23T04:44:58.571Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5702worker1 # [ 129.074761] nix-grpc-daemon[1233]: time=2026-09-23T04:44:58.571Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)"5703worker1 # [ 129.099403] nix-grpc-daemon[1233]: time=2026-09-23T04:44:58.597Z level=INFO msg="Uploading 1 narinfos"5704lb # [ 129.136810] niks3-server[1114]: 2026/09/23 04:44:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign5705lb # [ 129.148953] niks3-server[1114]: 2026/09/23 04:44:58 INFO Signed narinfos id=45 count=15706lb # [ 129.159618] niks3-server[1114]: 2026/09/23 04:44:58 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls5707lb # [ 129.168896] niks3-server[1114]: 2026/09/23 04:44:58 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst5708lb # [ 129.185510] niks3-server[1114]: 2026/09/23 04:44:58 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete5709worker1 # [ 129.151227] nix-grpc-daemon[1233]: time=2026-09-23T04:44:58.649Z level=INFO msg="Upload complete. (228ms)"5710lb # [ 129.195056] niks3-server[1114]: 2026/09/23 04:44:58 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo5711lb # [ 129.202013] niks3-server[1114]: 2026/09/23 04:44:58 INFO Completed upload id=455712worker1 # [ 129.160301] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41904 duration_s=0 paths=1 nar_bytes_in=6405713worker1 # [ 129.176379] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419045714worker1 # [ 129.188072] nix-daemon[834]: accepted connection from pid 1153, user nix-grpc-daemon (trusted)5715worker1 # [ 129.196328] nix-daemon[834]: reaped child process 1276, status = succeeded5716client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051...5717worker1 # [ 129.203587] nix-daemon[834]: accepted connection from pid 1153, user nix-grpc-daemon (trusted)5718client # farm-b> node-a: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5719lb # [ 129.423616] envoy[900]: conn peer=[2001:db8:1::1]:60098 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9736 tx=4448 ms=57235720lb # [ 129.429066] envoy[900]: conn peer=[2001:db8:1::4]:36438 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=1708 tx=442 ms=146465721client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds5722client # farm-b> LOG-farm-b-on-standby5723worker1 # [ 130.270675] nix-daemon[834]: reaped child process 1285, status = succeeded5724worker1 # [ 130.385131] nix-daemon[834]: accepted connection from pid 1301, user nix-grpc-daemon (trusted)5725worker1 # [ 130.405875] nix-daemon[834]: reaped child process 1304, status = succeeded5726lb # [ 130.449540] niks3-server[1114]: 2026/09/23 04:44:59 INFO Received uploads request method=POST path=/api/pending_closures5727worker1 # [ 130.418585] nix-grpc-daemon[1233]: time=2026-09-23T04:44:59.916Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"5728worker1 # [ 130.420404] nix-grpc-daemon[1233]: time=2026-09-23T04:44:59.916Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)"5729worker1 # [ 130.460035] nix-grpc-daemon[1233]: time=2026-09-23T04:44:59.958Z level=INFO msg="Uploading 1 narinfos"5730lb # [ 130.491582] niks3-server[1114]: 2026/09/23 04:45:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign5731lb # [ 130.507518] niks3-server[1114]: 2026/09/23 04:45:00 INFO Signed narinfos id=46 count=15732lb # [ 130.515235] niks3-server[1114]: 2026/09/23 04:45:00 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls5733lb # [ 130.525668] niks3-server[1114]: 2026/09/23 04:45:00 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv5734worker1 # [ 130.487325] nix-grpc-daemon[1233]: time=2026-09-23T04:44:59.986Z level=INFO msg="Upload complete. (274ms)"5735lb # [ 130.532217] niks3-server[1114]: 2026/09/23 04:45:00 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst5736lb # [ 130.536731] niks3-server[1114]: 2026/09/23 04:45:00 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete5737lb # [ 130.541785] niks3-server[1114]: 2026/09/23 04:45:00 INFO Completed upload id=465738lb # [ 130.544342] niks3-server[1114]: 2026/09/23 04:45:00 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo5739worker1 # [ 130.504686] nix-grpc-daemon[1153]: ts=2026-09-23T04:44:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41904 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=15740worker1 # [ 130.524527] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:00Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=05741worker1 # [ 130.529606] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:00Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.3:50051 clients=15742worker1 # [ 130.534612] nix-daemon[834]: reaped child process 1283, status = succeeded5743worker1 # [ 130.565456] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419045744worker1 # [ 130.580942] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419045745worker1 # [ 130.596619] nix-grpc-daemon[1233]: time=2026-09-23T04:45:00.095Z level=INFO msg="All 2 paths already cached"5746client # copying 1 paths to 'lb:50051'...5747client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'...5748worker1 # [ 130.608956] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:419045749worker1 # [ 130.622990] nix-daemon[834]: accepted connection from pid 1153, user nix-grpc-daemon (trusted)5750worker1 # [ 130.770095] nix-daemon[834]: accepted connection from pid 1310, user nix-grpc-daemon (trusted)5751worker1 # [ 130.790714] nix-daemon[834]: reaped child process 1313, status = succeeded5752lb # [ 130.838790] niks3-server[1114]: 2026/09/23 04:45:00 INFO Received uploads request method=POST path=/api/pending_closures5753worker1 # [ 130.804092] nix-grpc-daemon[1233]: time=2026-09-23T04:45:00.302Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5754worker1 # [ 130.805715] nix-grpc-daemon[1233]: time=2026-09-23T04:45:00.302Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)"5755lb # [ 130.872530] niks3-server[1114]: 2026/09/23 04:45:00 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst5756worker1 # [ 130.839174] nix-grpc-daemon[1233]: time=2026-09-23T04:45:00.337Z level=INFO msg="Uploading 1 narinfos"5757lb # [ 130.887733] niks3-server[1114]: 2026/09/23 04:45:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign5758lb # [ 130.899236] niks3-server[1114]: 2026/09/23 04:45:00 INFO Signed narinfos id=47 count=15759lb # [ 130.908445] niks3-server[1114]: 2026/09/23 04:45:00 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls5760worker1 # [ 130.872313] nix-grpc-daemon[1233]: time=2026-09-23T04:45:00.370Z level=INFO msg="Upload complete. (233ms)"5761lb # [ 130.918847] niks3-server[1114]: 2026/09/23 04:45:00 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete5762lb # [ 130.924500] niks3-server[1114]: 2026/09/23 04:45:00 INFO Completed upload id=475763lb # [ 130.927962] niks3-server[1114]: 2026/09/23 04:45:00 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo5764worker1 # [ 130.884825] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41904 duration_s=0 paths=1 nar_bytes_in=7685765worker1 # [ 130.898864] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419045766worker1 # [ 130.905421] nix-daemon[834]: accepted connection from pid 1153, user nix-grpc-daemon (trusted)5767worker1 # [ 130.910429] nix-daemon[834]: reaped child process 1308, status = succeeded5768client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051...5769worker1 # [ 130.921184] nix-daemon[834]: accepted connection from pid 1153, user nix-grpc-daemon (trusted)5770client # farm-top> node-a: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5771client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds5772client # farm-top> LOG-farm-top-on-standby5773worker1 # [ 132.218270] nix-daemon[834]: reaped child process 1317, status = succeeded5774worker1 # [ 132.306648] nix-daemon[834]: accepted connection from pid 1333, user nix-grpc-daemon (trusted)5775worker1 # [ 132.327027] nix-daemon[834]: reaped child process 1336, status = succeeded5776lb # [ 132.374885] niks3-server[1114]: 2026/09/23 04:45:01 INFO Received uploads request method=POST path=/api/pending_closures5777worker1 # [ 132.342556] nix-grpc-daemon[1233]: time=2026-09-23T04:45:01.839Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"5778worker1 # [ 132.344357] nix-grpc-daemon[1233]: time=2026-09-23T04:45:01.839Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)"5779lb # [ 132.411358] niks3-server[1114]: 2026/09/23 04:45:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign5780worker1 # [ 132.377854] nix-grpc-daemon[1233]: time=2026-09-23T04:45:01.876Z level=INFO msg="Uploading 1 narinfos"5781lb # [ 132.426956] niks3-server[1114]: 2026/09/23 04:45:01 INFO Signed narinfos id=48 count=15782lb # [ 132.435535] niks3-server[1114]: 2026/09/23 04:45:01 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst5783lb # [ 132.447790] niks3-server[1114]: 2026/09/23 04:45:01 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv5784worker1 # [ 132.409051] nix-grpc-daemon[1233]: time=2026-09-23T04:45:01.907Z level=INFO msg="Upload complete. (275ms)"5785lb # [ 132.456803] niks3-server[1114]: 2026/09/23 04:45:01 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls5786lb # [ 132.463785] niks3-server[1114]: 2026/09/23 04:45:01 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete5787lb # [ 132.468895] niks3-server[1114]: 2026/09/23 04:45:01 INFO Completed upload id=485788worker1 # [ 132.421063] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41904 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=15789lb # [ 132.471201] niks3-server[1114]: 2026/09/23 04:45:01 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo5790worker1 # [ 132.438940] nix-daemon[834]: reaped child process 1315, status = succeeded5791client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2, in 6.37 seconds)5792client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath5793client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath, in 0.18 seconds)5794worker2 # [ 132.828695] systemd[1]: Listening on Nix worker-protocol over gRPC.5795worker2 # [ 132.836181] systemd[1]: Starting scheduler-token.service...5796worker2 # [ 132.912611] systemd[1]: scheduler-token.service: Deactivated successfully.5797worker2 # [ 132.920582] systemd[1]: Finished scheduler-token.service.5798worker2 # [ 132.922206] systemd[1]: scheduler-token.service: Consumed 44ms CPU time over 80ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.5799worker2 # [ 132.928608] systemd[1]: Starting Nix worker-protocol over gRPC...5800worker2 # [ 133.055758] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:02Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5801worker2 # [ 133.058150] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:02Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5802worker2 # [ 133.061924] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:02Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=15803worker2 # [ 133.067297] nix-grpc-daemon[1530]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5804worker2 # [ 133.091754] nix-grpc-daemon[1530]: E0000 00:00:1790138702.588552 1530 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5805lb: must succeed: curl -sf localhost:9901/clusters5806worker2 # [ 133.119153] nix-grpc-daemon[1530]: E0000 00:00:1790138702.592221 1530 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5807worker2 # [ 133.122287] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:02Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5808worker2 # [ 133.124479] nix-grpc-daemon[1530]: E0000 00:00:1790138702.604139 1571 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5809worker2 # [ 133.126992] nix-grpc-daemon[1530]: E0000 00:00:1790138702.625574 1571 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5810worker2 # [ 133.129343] nix-grpc-daemon[1530]: E0000 00:00:1790138702.625729 1571 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5811worker2 # [ 133.131720] nix-grpc-daemon[1530]: E0000 00:00:1790138702.641084 1571 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5812worker2 # [ 133.136937] systemd[1]: Started Nix worker-protocol over gRPC.5813worker2 # [ 133.139609] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:02Z level=info event=scheduler_connected addr=lb:500515814worker1 # [ 133.050042] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:02Z level=info event=oidc_jwks provider=mock keys=15815worker1 # [ 133.054780] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:02Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:548265816worker1 # [ 133.060291] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:02Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05817lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5818worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics5819worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5820lb: must succeed: curl -sf localhost:9901/clusters5821lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)5822lb: must succeed: curl -sf localhost:9901/clusters5823lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5824worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5825worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)5826(finished: subtest: scheduler down: the other node takes the lock and keeps it, in 12.19 seconds)5827subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise5828client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs5829client # [ 133.407644] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs.5830client # Running as unit: rs.service; invocation ID: 8a3f2718cdf6436fb8f5de11ea8808895831client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs, in 0.09 seconds)5832client # [ 133.617798] nix[1402]: don't know how to build these paths:5833client # [ 133.618804] nix[1402]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5834worker1 # [ 133.591769] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:419045835worker1 # [ 133.605364] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:03Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=05836worker1 # [ 133.612095] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:03Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.3:50051 clients=15837worker1 # [ 133.653153] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419045838worker1 # [ 133.668374] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:419045839client # [ 133.708241] nix[1402]: copying 1 paths to 'lb:50051'...5840worker1 # [ 133.683184] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:419045841client # [ 133.710649] nix[1402]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'...5842worker1 # [ 133.698395] nix-daemon[834]: accepted connection from pid 1153, user nix-grpc-daemon (trusted)5843worker1 # [ 133.852159] nix-daemon[834]: accepted connection from pid 1349, user nix-grpc-daemon (trusted)5844worker1 # [ 133.873528] nix-daemon[834]: reaped child process 1352, status = succeeded5845lb # [ 133.917191] niks3-server[1114]: 2026/09/23 04:45:03 INFO Received uploads request method=POST path=/api/pending_closures5846worker1 # [ 133.886709] nix-grpc-daemon[1233]: time=2026-09-23T04:45:03.384Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5847worker1 # [ 133.888650] nix-grpc-daemon[1233]: time=2026-09-23T04:45:03.384Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)"5848lb # [ 133.953206] niks3-server[1114]: 2026/09/23 04:45:03 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst5849worker1 # [ 133.921743] nix-grpc-daemon[1233]: time=2026-09-23T04:45:03.420Z level=INFO msg="Uploading 1 narinfos"5850lb # [ 133.968342] niks3-server[1114]: 2026/09/23 04:45:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign5851lb # [ 133.976457] niks3-server[1114]: 2026/09/23 04:45:03 INFO Signed narinfos id=49 count=15852lb # [ 133.981056] niks3-server[1114]: 2026/09/23 04:45:03 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls5853worker1 # [ 133.946091] nix-grpc-daemon[1233]: time=2026-09-23T04:45:03.444Z level=INFO msg="Upload complete. (231ms)"5854lb # [ 133.991045] niks3-server[1114]: 2026/09/23 04:45:03 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete5855lb # [ 133.995312] niks3-server[1114]: 2026/09/23 04:45:03 INFO Completed upload id=495856lb # [ 133.998236] niks3-server[1114]: 2026/09/23 04:45:03 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo5857worker1 # [ 133.954807] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41904 duration_s=0 paths=1 nar_bytes_in=4165858worker1 # [ 133.969364] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:419045859worker1 # [ 133.977243] nix-daemon[834]: accepted connection from pid 1153, user nix-grpc-daemon (trusted)5860worker1 # [ 133.979966] nix-daemon[834]: reaped child process 1347, status = succeeded5861worker1 # [ 133.989901] nix-daemon[834]: accepted connection from pid 1153, user nix-grpc-daemon (trusted)5862client # [ 134.014054] nix[1402]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051...5863lb: must succeed: curl -sf localhost:9901/clusters5864lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5865lb: must succeed: curl -sf localhost:9901/clusters5866lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)5867lb: must succeed: curl -sf localhost:9901/clusters5868lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds)5869lb: must succeed: curl -sf localhost:9901/clusters5870lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds)5871worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true5872worker2 # [ 136.269496] systemd-journald[271]: Received client request to rotate journal, rotating.5873worker2 # [ 136.294016] systemd-journald[271]: Vacuuming done, freed 0B of archived journals from /var/log/journal/12727907fe3b47708e2dec830f45bb7c.5874worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.09 seconds)5875worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service5876worker1 # [ 136.233868] systemd[1]: Stopping Nix worker-protocol over gRPC...5877worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.04 seconds)5878client: must succeed: sleep 5.05879worker1 # [ 136.307279] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:05Z level=info event=draining5880lb # [ 139.424007] envoy[900]: conn peer=[2001:db8:1::1]:44662 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9632 tx=4420 ms=61215881client: (finished: must succeed: sleep 5.0, in 5.05 seconds)5882worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr='5883worker1 # [ 141.698853] nix-daemon[834]: reaped child process 1356, status = succeeded5884worker1 # [ 141.769067] nix-daemon[834]: accepted connection from pid 1397, user nix-grpc-daemon (trusted)5885worker1 # [ 141.787129] nix-daemon[834]: reaped child process 1400, status = succeeded5886lb # [ 141.838380] niks3-server[1114]: 2026/09/23 04:45:11 INFO Received uploads request method=POST path=/api/pending_closures5887worker1 # [ 141.808435] nix-grpc-daemon[1233]: time=2026-09-23T04:45:11.306Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5888worker1 # [ 141.810211] nix-grpc-daemon[1233]: time=2026-09-23T04:45:11.306Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)"5889worker1 # [ 141.853995] nix-grpc-daemon[1233]: time=2026-09-23T04:45:11.352Z level=INFO msg="Uploading 1 narinfos"5890lb # [ 141.888607] niks3-server[1114]: 2026/09/23 04:45:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign5891lb # [ 141.904408] niks3-server[1114]: 2026/09/23 04:45:11 INFO Signed narinfos id=50 count=15892lb # [ 141.906923] niks3-server[1114]: 2026/09/23 04:45:11 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst5893lb # [ 141.924475] niks3-server[1114]: 2026/09/23 04:45:11 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls5894worker1 # [ 141.886095] nix-grpc-daemon[1233]: time=2026-09-23T04:45:11.384Z level=INFO msg="Upload complete. (279ms)"5895lb # [ 141.928002] niks3-server[1114]: 2026/09/23 04:45:11 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv5896lb # [ 141.939180] niks3-server[1114]: 2026/09/23 04:45:11 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete5897lb # [ 141.944605] niks3-server[1114]: 2026/09/23 04:45:11 INFO Completed upload id=505898lb # [ 141.947703] niks3-server[1114]: 2026/09/23 04:45:11 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo5899worker1 # [ 141.901287] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41904 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=15900worker1 # [ 141.919395] nix-daemon[834]: reaped child process 1354, status = succeeded5901client # [ 141.960214] systemd[1]: rs.service: Deactivated successfully.5902client # [ 141.964793] systemd[1]: rs.service: Consumed 197ms CPU time over 8.556s wall clock time, 43.2M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic.5903worker1 # [ 142.309135] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:11Z level=info event=signal_exit5904lb # [ 142.632803] niks3-server[1114]: 2026/09/23 04:45:12 INFO lead: released remote=192.168.1.3:568285905lb # [ 143.026593] niks3-server[1114]: 2026/09/23 04:45:12 INFO lead: acquired remote=192.168.1.4:552585906worker2 # [ 143.091152] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:12Z level=info event=scheduler_take_over5907worker2 # [ 143.692872] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:13Z level=info event=scheduler_restarting addr=lb:500515908worker1 # [ 143.594841] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:13Z level=info event=scheduler_restarting5909worker1 # [ 143.595245] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:13Z level=info event=scheduler_restarting addr=lb:500515910worker1 # [ 143.595447] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:13Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:548265911worker1 # [ 143.595578] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:13Z level=info event=worker_gone addr=192.168.1.4:500515912worker1 # [ 143.595707] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:13Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:548265913worker1 # [ 143.595820] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:13Z level=info event=worker_gone addr=192.168.1.3:500515914worker1 # [ 143.643733] nix-daemon[834]: reaped child process 1206, status = succeeded5915worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 2.18 seconds)5916client: waiting for success: ! systemctl is-active rs5917client: (finished: waiting for success: ! systemctl is-active rs, in 0.06 seconds)5918client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }5919worker2 # [ 143.894574] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:13Z level=info event=scheduler_connected addr=lb:500515920worker2 # [ 143.899992] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5921worker1 # [ 143.811637] nix-grpc-daemon[1153]: ts=2026-09-23T04:45:13Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"5922client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds)5923client: must fail: journalctl -u rs | grep -q 'reconnecting'5924client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds)5925lb: must succeed: curl -sf localhost:9901/clusters5926worker1 # [ 143.870498] nix-grpc-daemon[1153]: E0000 00:00:1790138713.369019 1199 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true5927worker1 # [ 143.895964] nix-grpc-daemon[1153]: E0000 00:00:1790138713.394492 1199 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true5928lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)5929worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5930worker2 # [ 144.103379] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:13Z level=info event=scheduler_connected addr=lb:500515931worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5932worker2 # [ 144.107637] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:13Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"5933worker2 # [ 144.303564] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:13Z level=info event=scheduler_connected addr=lb:500515934worker2 # [ 144.320125] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:13Z level=info event=oidc_jwks provider=mock keys=15935worker2 # [ 144.324774] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:13Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:482785936worker2 # [ 144.330028] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:13Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=05937lb: must succeed: curl -sf localhost:9901/clusters5938lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5939worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5940worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5941worker1 # [ 145.186789] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.5942worker1 # [ 145.192474] systemd[1]: Stopped Nix worker-protocol over gRPC.5943worker1 # [ 145.195041] systemd[1]: nix-grpc-daemon.service: Consumed 1.598s CPU time over 32.070s wall clock time, 39.1M memory peak, 47.5K incoming IP traffic, 88.7K outgoing IP traffic.5944worker1 # [ 145.215486] systemd[1]: Starting Nix worker-protocol over gRPC...5945worker1 # [ 145.357192] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:14Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"5946worker1 # [ 145.359452] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:14Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc5947worker1 # [ 145.363300] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:14Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=15948worker1 # [ 145.368073] nix-grpc-daemon[1407]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR5949worker1 # [ 145.384131] nix-grpc-daemon[1407]: E0000 00:00:1790138714.866813 1407 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5950worker1 # [ 145.410620] nix-grpc-daemon[1407]: E0000 00:00:1790138714.870342 1407 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5951worker1 # [ 145.413545] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:14Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket5952worker1 # [ 145.415355] nix-grpc-daemon[1407]: E0000 00:00:1790138714.876032 1450 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5953worker1 # [ 145.417857] systemd[1]: Started Nix worker-protocol over gRPC.5954worker1 # [ 145.421576] nix-grpc-daemon[1407]: E0000 00:00:1790138714.894039 1450 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5955worker1 # [ 145.425598] nix-grpc-daemon[1407]: E0000 00:00:1790138714.894180 1450 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.5956worker1 # [ 145.428450] nix-grpc-daemon[1407]: E0000 00:00:1790138714.908563 1450 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.5957worker1 # [ 145.430770] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:14Z level=info event=scheduler_connected addr=lb:500515958worker2 # [ 145.530883] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:15Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:482785959worker2 # [ 145.537241] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:15Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=05960lb: must succeed: curl -sf localhost:9901/clusters5961lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)5962worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics5963worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)5964(finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.98 seconds)5965subtest: scheduler restart mid-build: build finishes, no double build5966client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid5967client # [ 146.386829] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid.5968client # Running as unit: mid.service; invocation ID: 302ac9fae2d44edbadbf01c0411b52095969client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid, in 0.09 seconds)5970worker2 # [ 146.685681] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:16Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:482865971worker2 # [ 146.698664] nix-daemon[856]: accepted connection from pid 1530, user nix-grpc-daemon (trusted)5972client # [ 146.638592] nix[1438]: don't know how to build these paths:5973client # [ 146.640607] nix[1438]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv5974worker2 # [ 146.731258] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:16Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=05975worker2 # [ 146.738152] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:16Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=15976worker2 # [ 146.783857] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482865977worker2 # [ 146.799501] nix-grpc-daemon[1530]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 16 ms (attempt 1/5)5978worker2 # [ 146.824664] nix-grpc-daemon[1530]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 160 ms (attempt 2/5)5979worker2 # [ 146.986629] nix-grpc-daemon[1530]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 300 ms (attempt 3/5)5980worker2 # [ 147.288661] nix-grpc-daemon[1530]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 110 ms (attempt 4/5)5981worker2 # [ 147.408208] nix-grpc-daemon[1530]: disabling binary cache 'https://cache.nixos.org' for 60 seconds5982worker2 # [ 147.414572] nix-grpc-daemon[1530]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org5983worker2 # [ 147.432322] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:482865984client # [ 147.368225] nix[1438]: copying 1 paths to 'lb:50051'...5985worker2 # [ 147.447246] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:482865986client # [ 147.368553] nix[1438]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'...5987worker2 # [ 147.463371] nix-daemon[856]: accepted connection from pid 1530, user nix-grpc-daemon (trusted)5988worker2 # [ 147.488717] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:17Z level=info event=niks3_push_started pid=16565989worker2 # [ 147.638698] nix-grpc-daemon[1656]: time=2026-09-23T04:45:17.160Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""5990worker2 # [ 147.752787] nix-daemon[856]: accepted connection from pid 1668, user nix-grpc-daemon (trusted)5991worker2 # [ 147.788125] nix-daemon[856]: reaped child process 1674, status = succeeded5992lb # [ 147.725621] niks3-server[1114]: 2026/09/23 04:45:17 INFO Received uploads request method=POST path=/api/pending_closures5993worker2 # [ 147.795948] nix-grpc-daemon[1656]: time=2026-09-23T04:45:17.315Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"5994worker2 # [ 147.800110] nix-grpc-daemon[1656]: time=2026-09-23T04:45:17.315Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)"5995lb # [ 147.771385] niks3-server[1114]: 2026/09/23 04:45:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign5996worker2 # [ 147.843091] nix-grpc-daemon[1656]: time=2026-09-23T04:45:17.364Z level=INFO msg="Uploading 1 narinfos"5997lb # [ 147.784655] niks3-server[1114]: 2026/09/23 04:45:17 INFO Signed narinfos id=51 count=15998lb # [ 147.787205] niks3-server[1114]: 2026/09/23 04:45:17 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst5999lb # [ 147.800251] niks3-server[1114]: 2026/09/23 04:45:17 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls6000lb # [ 147.808199] niks3-server[1114]: 2026/09/23 04:45:17 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete6001worker2 # [ 147.872406] nix-grpc-daemon[1656]: time=2026-09-23T04:45:17.394Z level=INFO msg="Upload complete. (230ms)"6002lb # [ 147.813821] niks3-server[1114]: 2026/09/23 04:45:17 INFO Completed upload id=516003lb # [ 147.816134] niks3-server[1114]: 2026/09/23 04:45:17 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo6004worker2 # [ 147.880587] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48286 duration_s=0 paths=1 nar_bytes_in=4246005worker2 # [ 147.890620] nix-grpc-daemon[1530]: ts=2026-09-23T04:45:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:482866006worker2 # [ 147.895496] nix-daemon[856]: accepted connection from pid 1530, user nix-grpc-daemon (trusted)6007worker2 # [ 147.899851] nix-daemon[856]: reaped child process 1654, status = succeeded6008worker2 # [ 147.908986] nix-daemon[856]: accepted connection from pid 1530, user nix-grpc-daemon (trusted)6009client # [ 147.831291] nix[1438]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...6010lb # [ 149.429220] envoy[900]: conn peer=[2001:db8:1::1]:60534 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=1516 ms=83246011lb # [ 149.434068] envoy[900]: conn peer=[2001:db8:1::3]:57824 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2839 tx=977 ms=304586012lb: must succeed: curl -sf localhost:9901/clusters6013lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6014lb: must succeed: curl -sf localhost:9901/clusters6015lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6016worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service6017worker2 # [ 150.353073] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1530 (nix-grpc-daemon) on client request.6018worker2 # [ 150.354667] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request.6019lb # [ 150.309481] niks3-server[1114]: 2026/09/23 04:45:19 INFO lead: released remote=192.168.1.4:552586020worker2 # [ 150.372101] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL6021worker2 # [ 150.374084] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'.6022worker2 # [ 150.375326] systemd[1]: nix-grpc-daemon.service: Consumed 496ms CPU time over 17.438s wall clock time, 34.8M memory peak, 15.9K incoming IP traffic, 28.4K outgoing IP traffic.6023client # [ 150.298019] nix[1438]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again6024worker1 # [ 150.272803] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:19Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination"6025client # [ 150.304437] nix[1438]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting6026worker2 # [ 150.393591] systemd[1]: Starting scheduler-token.service...6027worker2 # [ 150.419226] nix-daemon[856]: reaped child process 1677, status = succeeded6028worker2 # [ 150.423045] nix-daemon[856]: reaped child process 1646, status = succeeded6029worker2 # [ 150.428225] nix-daemon[1679]: unexpected Nix daemon error: error: interrupted by the user6030worker2 # [ 150.434631] nix-daemon[856]: reaped child process 1679, status = failed with exit code 16031worker2 # [ 150.464139] systemd[1]: scheduler-token.service: Deactivated successfully.6032worker2 # [ 150.465100] systemd[1]: Finished scheduler-token.service.6033worker2 # [ 150.465852] systemd[1]: scheduler-token.service: Consumed 23ms CPU time over 70ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6034worker2 # [ 150.467698] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1.6035worker2 # [ 150.471511] systemd[1]: Starting scheduler-token.service...6036lb # [ 150.423427] niks3-server[1114]: 2026/09/23 04:45:19 INFO lead: acquired remote=192.168.1.3:382246037worker1 # [ 150.383898] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:19Z level=info event=scheduler_take_over6038worker2 # [ 150.517482] systemd[1]: scheduler-token.service: Deactivated successfully.6039worker2 # [ 150.520418] systemd[1]: Finished scheduler-token.service.6040worker2 # [ 150.521209] systemd[1]: scheduler-token.service: Consumed 21ms CPU time over 46ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6041worker2 # [ 150.524738] systemd[1]: Starting Nix worker-protocol over gRPC...6042worker1 # [ 150.473052] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:19Z level=info event=scheduler_connected addr=lb:500516043worker2 # [ 150.625685] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:20Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6044worker2 # [ 150.627931] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:20Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6045worker2 # [ 150.631666] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:20Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16046worker2 # [ 150.637256] nix-grpc-daemon[1722]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6047worker2 # [ 150.638709] nix-grpc-daemon[1722]: E0000 00:00:1790138720.158514 1722 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6048worker1 # [ 150.553744] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:20Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6049worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.37 seconds)6050client: must succeed: sleep 5.06051worker2 # [ 150.684139] nix-grpc-daemon[1722]: E0000 00:00:1790138720.164513 1722 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6052worker2 # [ 150.686400] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:20Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6053worker2 # [ 150.690588] nix-grpc-daemon[1722]: E0000 00:00:1790138720.174193 1763 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6054worker2 # [ 150.693499] nix-grpc-daemon[1722]: E0000 00:00:1790138720.192222 1763 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6055worker2 # [ 150.695913] nix-grpc-daemon[1722]: E0000 00:00:1790138720.192406 1763 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6056worker2 # [ 150.701260] systemd[1]: Started Nix worker-protocol over gRPC.6057worker2 # [ 150.704402] nix-grpc-daemon[1722]: E0000 00:00:1790138720.211912 1763 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6058worker2 # [ 150.707213] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:20Z level=info event=scheduler_connected addr=lb:500516059worker2 # [ 150.708571] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:20Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6060worker1 # [ 150.754675] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:20Z level=info event=scheduler_connected addr=lb:500516061worker1 # [ 150.760256] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:20Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6062client # [ 150.803320] nix[1438]: scheduler at lb:50051: scheduler passive, another node serves, reconnecting6063worker2 # [ 150.908557] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:20Z level=info event=scheduler_connected addr=lb:500516064worker2 # [ 150.915272] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:20Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6065worker1 # [ 150.961378] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:20Z level=info event=scheduler_connected addr=lb:500516066worker1 # [ 150.966841] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:20Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6067worker2 # [ 151.116598] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:20Z level=info event=scheduler_connected addr=lb:500516068worker2 # [ 151.123560] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:20Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves"6069worker1 # [ 151.167675] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:20Z level=info event=scheduler_connected addr=lb:500516070worker1 # [ 151.173245] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6071worker2 # [ 151.324960] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:20Z level=info event=scheduler_connected addr=lb:500516072worker2 # [ 151.330071] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6073worker1 # [ 151.376579] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:20Z level=info event=scheduler_connected addr=lb:500516074worker1 # [ 151.380737] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6075worker2 # [ 151.531090] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:21Z level=info event=scheduler_connected addr=lb:500516076worker2 # [ 151.536320] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6077worker1 # [ 151.576399] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:21Z level=info event=scheduler_connected addr=lb:500516078worker1 # [ 151.581631] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6079worker2 # [ 151.739023] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:21Z level=info event=scheduler_connected addr=lb:500516080worker2 # [ 151.743241] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6081client # [ 151.807151] nix[1438]: scheduler at lb:50051: no healthy upstream, reconnecting6082worker1 # [ 151.782339] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:21Z level=info event=scheduler_connected addr=lb:500516083worker1 # [ 151.787523] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6084worker2 # [ 151.939871] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:21Z level=info event=scheduler_connected addr=lb:500516085worker2 # [ 151.945060] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6086worker1 # [ 151.988458] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:21Z level=info event=scheduler_connected addr=lb:500516087worker1 # [ 151.993691] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6088worker2 # [ 152.145796] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:21Z level=info event=scheduler_connected addr=lb:500516089worker2 # [ 152.150907] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6090worker1 # [ 152.194420] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:21Z level=info event=scheduler_connected addr=lb:500516091worker1 # [ 152.199655] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream"6092worker2 # [ 152.351991] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:21Z level=info event=scheduler_connected addr=lb:500516093worker1 # [ 152.271076] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:21Z level=info event=oidc_jwks provider=mock keys=16094worker1 # [ 152.275645] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:21Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:608806095worker1 # [ 152.281154] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:21Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=06096worker1 # [ 152.403041] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:21Z level=info event=scheduler_connected addr=lb:500516097worker1 # [ 152.407231] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:21Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:608806098worker1 # [ 152.412165] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:21Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=06099worker1 # [ 153.794275] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:23Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=06100worker1 # [ 153.800536] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:23Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=16101worker2 # [ 153.951757] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:453546102worker2 # [ 153.966797] nix-daemon[856]: accepted connection from pid 1722, user nix-grpc-daemon (trusted)6103worker2 # [ 153.985978] nix-daemon[856]: accepted connection from pid 1722, user nix-grpc-daemon (trusted)6104client # [ 153.929032] nix[1438]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051...6105worker2 # [ 154.011980] nix-daemon[856]: accepted connection from pid 1722, user nix-grpc-daemon (trusted)6106client: (finished: must succeed: sleep 5.0, in 5.05 seconds)6107client: waiting for success: ! systemctl is-active mid6108worker2 # [ 156.030329] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:25Z level=info event=niks3_push_started pid=18046109worker2 # [ 156.051230] nix-daemon[856]: reaped child process 1779, status = succeeded6110worker2 # [ 156.158138] nix-grpc-daemon[1804]: time=2026-09-23T04:45:25.679Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""6111worker2 # [ 156.266326] nix-daemon[856]: accepted connection from pid 1817, user nix-grpc-daemon (trusted)6112worker2 # [ 156.281754] nix-daemon[856]: reaped child process 1820, status = succeeded6113lb # [ 156.231387] niks3-server[1114]: 2026/09/23 04:45:25 INFO Received uploads request method=POST path=/api/pending_closures6114worker2 # [ 156.303404] nix-grpc-daemon[1804]: time=2026-09-23T04:45:25.823Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6115worker2 # [ 156.307118] nix-grpc-daemon[1804]: time=2026-09-23T04:45:25.823Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)"6116lb # [ 156.285304] niks3-server[1114]: 2026/09/23 04:45:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign6117lb # [ 156.303138] niks3-server[1114]: 2026/09/23 04:45:25 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls6118lb # [ 156.318611] niks3-server[1114]: 2026/09/23 04:45:25 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv6119worker2 # [ 156.392093] nix-grpc-daemon[1804]: time=2026-09-23T04:45:25.913Z level=INFO msg="Uploading 1 narinfos"6120lb # [ 156.334465] niks3-server[1114]: 2026/09/23 04:45:25 INFO Signed narinfos id=52 count=16121lb # [ 156.348638] niks3-server[1114]: 2026/09/23 04:45:25 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst6122lb # [ 156.356999] niks3-server[1114]: 2026/09/23 04:45:25 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete6123lb # [ 156.364511] niks3-server[1114]: 2026/09/23 04:45:25 INFO Completed upload id=526124worker2 # [ 156.424096] nix-grpc-daemon[1804]: time=2026-09-23T04:45:25.946Z level=INFO msg="Upload complete. (262ms)"6125lb # [ 156.369359] niks3-server[1114]: 2026/09/23 04:45:25 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo6126worker2 # [ 156.436186] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45354 duration_s=2 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=16127worker2 # [ 156.455452] nix-daemon[856]: reaped child process 1777, status = succeeded6128client # [ 156.394128] systemd[1]: mid.service: Deactivated successfully.6129client # [ 156.399273] systemd[1]: mid.service: Consumed 210ms CPU time over 10.008s wall clock time, 41.4M memory peak, 9.2K incoming IP traffic, 11.3K outgoing IP traffic.6130client: (finished: waiting for success: ! systemctl is-active mid, in 1.12 seconds)6131client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }6132client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.05 seconds)6133(finished: subtest: scheduler restart mid-build: build finishes, no double build, in 10.75 seconds)6134subtest: interrupting the client stops the build on the worker6135client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr6136client # [ 157.109529] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr.6137client # Running as unit: intr.service; invocation ID: 30bf8d0b50184215bcfac1390ce8e94b6138client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr, in 0.05 seconds)6139worker2 # [ 157.357011] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:453546140client # [ 157.285168] nix[1474]: don't know how to build these paths:6141client # [ 157.286337] nix[1474]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv6142worker1 # [ 157.276822] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:26Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=06143worker1 # [ 157.282280] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:26Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.3:50051 clients=16144worker1 # [ 157.335997] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966145worker1 # [ 157.350175] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6146worker1 # [ 157.373814] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 91 ms (attempt 1/5)6147worker1 # [ 157.473432] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 101 ms (attempt 2/5)6148worker1 # [ 157.576497] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 180 ms (attempt 3/5)6149worker1 # [ 157.765849] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 307 ms (attempt 4/5)6150worker1 # [ 158.075089] nix-grpc-daemon[1407]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6151worker1 # [ 158.082127] nix-grpc-daemon[1407]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6152worker1 # [ 158.096463] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608966153client # [ 158.136137] nix[1474]: copying 1 paths to 'lb:50051'...6154worker1 # [ 158.112105] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608966155client # [ 158.140098] nix[1474]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'...6156worker1 # [ 158.126415] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6157worker1 # [ 158.159718] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:27Z level=info event=niks3_push_started pid=15046158worker1 # [ 158.315642] nix-grpc-daemon[1504]: time=2026-09-23T04:45:27.812Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca=""6159worker1 # [ 158.440112] nix-daemon[834]: accepted connection from pid 1521, user nix-grpc-daemon (trusted)6160worker1 # [ 158.460172] nix-daemon[834]: reaped child process 1524, status = succeeded6161lb # [ 158.506007] niks3-server[1114]: 2026/09/23 04:45:28 INFO Received uploads request method=POST path=/api/pending_closures6162worker1 # [ 158.475534] nix-grpc-daemon[1504]: time=2026-09-23T04:45:27.973Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6163worker1 # [ 158.479404] nix-grpc-daemon[1504]: time=2026-09-23T04:45:27.973Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)"6164lb # [ 158.549306] niks3-server[1114]: 2026/09/23 04:45:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign6165worker1 # [ 158.517858] nix-grpc-daemon[1504]: time=2026-09-23T04:45:28.015Z level=INFO msg="Uploading 1 narinfos"6166lb # [ 158.565779] niks3-server[1114]: 2026/09/23 04:45:28 INFO Signed narinfos id=53 count=16167lb # [ 158.576900] niks3-server[1114]: 2026/09/23 04:45:28 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst6168lb # [ 158.589527] niks3-server[1114]: 2026/09/23 04:45:28 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls6169lb # [ 158.594474] niks3-server[1114]: 2026/09/23 04:45:28 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete6170lb # [ 158.600083] niks3-server[1114]: 2026/09/23 04:45:28 INFO Completed upload id=536171worker1 # [ 158.556554] nix-grpc-daemon[1504]: time=2026-09-23T04:45:28.055Z level=INFO msg="Upload complete. (242ms)"6172lb # [ 158.603020] niks3-server[1114]: 2026/09/23 04:45:28 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo6173worker1 # [ 158.567018] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 paths=1 nar_bytes_in=4246174worker1 # [ 158.581998] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966175worker1 # [ 158.587945] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6176worker1 # [ 158.593305] nix-daemon[834]: reaped child process 1502, status = succeeded6177client # [ 158.629230] nix[1474]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051...6178worker1 # [ 158.606844] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6179lb # [ 159.427718] envoy[900]: conn peer=[2001:db8:1::4]:54940 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=3457 tx=408 ms=172486180lb # [ 159.432931] envoy[900]: conn peer=[2001:db8:1::1]:49014 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4314 tx=2093 ms=97826181client: must succeed: systemctl kill -s INT intr6182worker1 # [ 160.792647] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:30Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv6183client # [ 160.815254] systemd[1]: intr.service: Sent signal SIGINT to main process 1474 (nix) on client request.6184client # [ 160.825350] nix[1474]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED)6185client # [ 160.836779] nix[1474]: error: interrupted by the user6186client: (finished: must succeed: systemctl kill -s INT intr, in 0.07 seconds)6187client # [ 160.859546] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE6188client # [ 160.863676] systemd[1]: intr.service: Failed with result 'exit-code'.6189client # [ 160.866250] systemd[1]: intr.service: Consumed 163ms CPU time over 3.752s wall clock time, 41.4M memory peak, 6.5K incoming IP traffic, 8.3K outgoing IP traffic.6190worker1 # [ 160.849234] nix-daemon[834]: reaped child process 1526, status = succeeded6191worker1 # [ 160.854645] nix-daemon[1528]: unexpected Nix daemon error: error: interrupted by the user6192worker1 # [ 160.868189] nix-daemon[834]: reaped child process 1528, status = failed with exit code 16193(finished: subtest: interrupting the client stops the build on the worker, in 3.89 seconds)6194subtest: a deploy mid-build drains: the build finishes, the new generation takes over6195client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw6196client # [ 161.028763] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw.6197client # Running as unit: sw.service; invocation ID: 2ea878511b454325ad00a41b4f0b2cda6198client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw, in 0.08 seconds)6199worker1 # [ 161.208049] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:30Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:608966200client # [ 161.241343] nix[1495]: don't know how to build these paths:6201client # [ 161.242341] nix[1495]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv6202worker1 # [ 161.225847] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:30Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=06203worker1 # [ 161.232813] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:30Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.4:50051 clients=16204worker2 # [ 161.377129] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:453546205worker2 # [ 161.393368] nix-grpc-daemon[1722]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 30 ms (attempt 1/5)6206worker2 # [ 161.432690] nix-grpc-daemon[1722]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 161 ms (attempt 2/5)6207worker2 # [ 161.596290] nix-grpc-daemon[1722]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 50 ms (attempt 3/5)6208worker2 # [ 161.656673] nix-grpc-daemon[1722]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 99 ms (attempt 4/5)6209worker2 # [ 161.766161] nix-grpc-daemon[1722]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6210worker2 # [ 161.773086] nix-grpc-daemon[1722]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6211worker2 # [ 161.791580] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:453546212client # [ 161.728169] nix[1495]: copying 1 paths to 'lb:50051'...6213worker2 # [ 161.807180] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:453546214client # [ 161.730624] nix[1495]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'...6215worker2 # [ 161.825559] nix-daemon[856]: accepted connection from pid 1722, user nix-grpc-daemon (trusted)6216worker2 # [ 161.995215] nix-daemon[856]: accepted connection from pid 1852, user nix-grpc-daemon (trusted)6217worker2 # [ 162.017921] nix-daemon[856]: reaped child process 1855, status = succeeded6218lb # [ 161.957367] niks3-server[1114]: 2026/09/23 04:45:31 INFO Received uploads request method=POST path=/api/pending_closures6219worker2 # [ 162.028397] nix-grpc-daemon[1804]: time=2026-09-23T04:45:31.549Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6220worker2 # [ 162.029911] nix-grpc-daemon[1804]: time=2026-09-23T04:45:31.549Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)"6221lb # [ 161.990384] niks3-server[1114]: 2026/09/23 04:45:31 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls6222worker2 # [ 162.061142] nix-grpc-daemon[1804]: time=2026-09-23T04:45:31.582Z level=INFO msg="Uploading 1 narinfos"6223lb # [ 162.003885] niks3-server[1114]: 2026/09/23 04:45:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign6224lb # [ 162.014058] niks3-server[1114]: 2026/09/23 04:45:31 INFO Signed narinfos id=54 count=16225worker2 # [ 162.084445] nix-grpc-daemon[1804]: time=2026-09-23T04:45:31.606Z level=INFO msg="Upload complete. (240ms)"6226lb # [ 162.025023] niks3-server[1114]: 2026/09/23 04:45:31 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst6227lb # [ 162.029866] niks3-server[1114]: 2026/09/23 04:45:31 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete6228lb # [ 162.034174] niks3-server[1114]: 2026/09/23 04:45:31 INFO Completed upload id=546229worker2 # [ 162.095438] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:45354 duration_s=0 paths=1 nar_bytes_in=4166230lb # [ 162.038018] niks3-server[1114]: 2026/09/23 04:45:31 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo6231worker2 # [ 162.108561] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:453546232worker2 # [ 162.114280] nix-daemon[856]: accepted connection from pid 1722, user nix-grpc-daemon (trusted)6233worker2 # [ 162.120195] nix-daemon[856]: reaped child process 1850, status = succeeded6234client # [ 162.051247] nix[1495]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051...6235worker2 # [ 162.132382] nix-daemon[856]: accepted connection from pid 1722, user nix-grpc-daemon (trusted)6236worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&26237worker2 # Checking switch inhibitors... done6238worker2 # [ 165.273350] nixos[1895]: switching to system configuration /nix/store/wbg62602schcdgcrfazqip0hk2w5m5n6-nixos-system-worker2-test6239worker2 # activating the configuration...6240worker2 # [ 165.280941] systemd[1]: Stopped target Remote File Systems.6241worker2 # [ 165.281746] systemd[1]: Stopped target Local File Systems.6242worker2 # setting up /etc...6243worker2 # [ 165.820563] systemd[1]: Reload requested from client PID 1895 ('.switch-to-conf') (unit backdoor.service)...6244worker2 # [ 165.822381] systemd[1]: Reloading...6245lb # [ 169.424359] envoy[900]: conn peer=[2001:db8:1::1]:49546 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3013 tx=850 ms=35676246worker2 # [ 170.715176] systemd[1]: Reloading finished in 4886 ms.6247worker2 # restarting sysinit-reactivation.target6248worker2 # [ 170.829084] systemd[1]: Starting Re-setup tmpfiles on a system that is already running....6249worker2 # [ 170.950081] systemd[1]: Finished Re-setup tmpfiles on a system that is already running..6250worker2 # [ 170.951273] systemd[1]: Reached target Reactivate sysinit units.6251worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service6252worker2 # [ 170.959790] systemd[1]: Reloading D-Bus System Message Bus...6253worker2 # [ 170.967134] dbus-broker-launch[493]: Caught SIGHUP, trigger reload.6254worker2 # [ 170.971149] systemd[1]: Reloading Nix worker-protocol over gRPC...6255worker2 # [ 170.978649] dbus-broker-launch[493]: Looking up NSS user entry for 'systemd-timesync'...6256worker2 # [ 170.983480] dbus-broker-launch[493]: NSS returned no entry for 'systemd-timesync'6257worker2 # [ 170.992143] dbus-broker-launch[493]: Invalid user-name in /nix/store/ng2ydfkv03gvmdws4f4mjf9wj6qj3kq6-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync"6258worker2 # [ 171.008325] systemd[1]: Reloaded Nix worker-protocol over gRPC.6259worker2 # [ 171.020658] systemd[1]: Reloaded D-Bus System Message Bus.6260worker2 # [ 171.024641] systemd[1]: Reached target Remote File Systems.6261worker2 # [ 171.032996] systemd[1]: Starting linger-users.service...6262worker2 # [ 171.042880] systemd[1]: Reached target Local File Systems.6263worker2 # [ 171.043744] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore6264worker2 # [ 171.057452] systemd[1]: Starting Create SUID/SGID Wrappers...6265worker2 # [ 171.058284] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met.6266worker2 # [ 171.059476] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc676267worker2 # [ 171.073225] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore6268worker2 # [ 171.074807] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os6269worker2 # [ 171.086992] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os6270worker2 # [ 171.092733] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os6271worker2 # [ 171.094016] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki6272worker2 # [ 171.106952] systemd[1]: linger-users.service: Deactivated successfully.6273worker2 # [ 171.107896] systemd[1]: Finished linger-users.service.6274worker2 # [ 171.409791] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully.6275worker2 # [ 171.410837] systemd[1]: Finished Create SUID/SGID Wrappers.6276worker2 # [ 171.652967] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:41Z level=info event=draining6277worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service6278worker2 # [ 171.668630] nixos[1895]: finished switching to system configuration /nix/store/wbg62602schcdgcrfazqip0hk2w5m5n6-nixos-system-worker2-test6279worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 6.83 seconds)6280worker2: must succeed: pgrep -f 'read -t [9]0 x'6281worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds)6282client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&26283client # don't know how to build these paths:6284client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6285worker2 # [ 171.957199] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:453546286worker1 # [ 171.851621] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:608966287worker1 # [ 171.877733] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:41Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=06288worker1 # [ 171.884718] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:41Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.3:50051 clients=16289worker1 # [ 171.924794] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966290worker1 # [ 171.940145] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608966291client # copying 1 paths to 'lb:50051'...6292worker1 # [ 171.956111] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608966293client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'...6294worker1 # [ 171.986618] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6295worker1 # [ 172.159432] nix-daemon[834]: accepted connection from pid 1583, user nix-grpc-daemon (trusted)6296worker1 # [ 172.182525] nix-daemon[834]: reaped child process 1586, status = succeeded6297lb # [ 172.232340] niks3-server[1114]: 2026/09/23 04:45:41 INFO Received uploads request method=POST path=/api/pending_closures6298worker1 # [ 172.201591] nix-grpc-daemon[1504]: time=2026-09-23T04:45:41.699Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6299worker1 # [ 172.203317] nix-grpc-daemon[1504]: time=2026-09-23T04:45:41.699Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)"6300lb # [ 172.263860] niks3-server[1114]: 2026/09/23 04:45:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign6301worker1 # [ 172.227136] nix-grpc-daemon[1504]: time=2026-09-23T04:45:41.725Z level=INFO msg="Uploading 1 narinfos"6302lb # [ 172.276434] niks3-server[1114]: 2026/09/23 04:45:41 INFO Signed narinfos id=55 count=16303lb # [ 172.286560] niks3-server[1114]: 2026/09/23 04:45:41 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls6304lb # [ 172.301916] niks3-server[1114]: 2026/09/23 04:45:41 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete6305lb # [ 172.310515] niks3-server[1114]: 2026/09/23 04:45:41 INFO Completed upload id=556306worker1 # [ 172.265514] nix-grpc-daemon[1504]: time=2026-09-23T04:45:41.764Z level=INFO msg="Upload complete. (261ms)"6307lb # [ 172.320694] niks3-server[1114]: 2026/09/23 04:45:41 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst6308worker1 # [ 172.275690] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 paths=1 nar_bytes_in=5286309lb # [ 172.329577] niks3-server[1114]: 2026/09/23 04:45:41 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo6310worker1 # [ 172.291487] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966311worker1 # [ 172.305690] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6312worker1 # [ 172.315648] nix-daemon[834]: reaped child process 1581, status = succeeded6313client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051...6314client # farm-a> node-a: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv6315worker1 # [ 172.325362] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6316client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds6317client # farm-a> LOG-farm-a-during-drain6318worker1 # [ 173.233804] nix-daemon[834]: reaped child process 1590, status = succeeded6319worker1 # [ 173.306849] nix-daemon[834]: accepted connection from pid 1606, user nix-grpc-daemon (trusted)6320worker1 # [ 173.330657] nix-daemon[834]: reaped child process 1609, status = succeeded6321lb # [ 173.373618] niks3-server[1114]: 2026/09/23 04:45:42 INFO Received uploads request method=POST path=/api/pending_closures6322worker1 # [ 173.348497] nix-grpc-daemon[1504]: time=2026-09-23T04:45:42.845Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6323worker1 # [ 173.350432] nix-grpc-daemon[1504]: time=2026-09-23T04:45:42.845Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)"6324lb # [ 173.455467] niks3-server[1114]: 2026/09/23 04:45:42 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst6325lb # [ 173.464460] niks3-server[1114]: 2026/09/23 04:45:42 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv6326lb # [ 173.472419] niks3-server[1114]: 2026/09/23 04:45:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign6327worker1 # [ 173.434427] nix-grpc-daemon[1504]: time=2026-09-23T04:45:42.932Z level=INFO msg="Uploading 1 narinfos"6328lb # [ 173.482134] niks3-server[1114]: 2026/09/23 04:45:42 INFO Signed narinfos id=56 count=16329lb # [ 173.489284] niks3-server[1114]: 2026/09/23 04:45:43 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls6330lb # [ 173.504159] niks3-server[1114]: 2026/09/23 04:45:43 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete6331lb # [ 173.510759] niks3-server[1114]: 2026/09/23 04:45:43 INFO Completed upload id=566332lb # [ 173.516849] niks3-server[1114]: 2026/09/23 04:45:43 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo6333worker1 # [ 173.467293] nix-grpc-daemon[1504]: time=2026-09-23T04:45:42.965Z level=INFO msg="Upload complete. (330ms)"6334worker1 # [ 173.492862] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=16335worker1 # [ 173.516595] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:42Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=06336worker1 # [ 173.521242] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:42Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.3:50051 clients=16337worker1 # [ 173.526551] nix-daemon[834]: reaped child process 1588, status = succeeded6338worker1 # [ 173.544712] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966339worker1 # [ 173.565363] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608966340worker1 # [ 173.589070] nix-grpc-daemon[1504]: time=2026-09-23T04:45:43.087Z level=INFO msg="All 1 paths already cached"6341client # copying 1 paths to 'lb:50051'...6342worker1 # [ 173.602747] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608966343client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'...6344worker1 # [ 173.626815] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6345worker1 # [ 173.836185] nix-daemon[834]: accepted connection from pid 1615, user nix-grpc-daemon (trusted)6346worker1 # [ 173.855693] nix-daemon[834]: reaped child process 1618, status = succeeded6347lb # [ 173.908311] niks3-server[1114]: 2026/09/23 04:45:43 INFO Received uploads request method=POST path=/api/pending_closures6348worker1 # [ 173.876081] nix-grpc-daemon[1504]: time=2026-09-23T04:45:43.374Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6349worker1 # [ 173.877767] nix-grpc-daemon[1504]: time=2026-09-23T04:45:43.374Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)"6350worker1 # [ 173.922453] nix-grpc-daemon[1504]: time=2026-09-23T04:45:43.420Z level=INFO msg="Uploading 1 narinfos"6351lb # [ 173.956580] niks3-server[1114]: 2026/09/23 04:45:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign6352lb # [ 173.971705] niks3-server[1114]: 2026/09/23 04:45:43 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls6353lb # [ 173.981866] niks3-server[1114]: 2026/09/23 04:45:43 INFO Signed narinfos id=57 count=16354lb # [ 173.989346] niks3-server[1114]: 2026/09/23 04:45:43 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst6355lb # [ 174.003955] niks3-server[1114]: 2026/09/23 04:45:43 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete6356lb # [ 174.014974] niks3-server[1114]: 2026/09/23 04:45:43 INFO Completed upload id=576357worker1 # [ 173.970021] nix-grpc-daemon[1504]: time=2026-09-23T04:45:43.468Z level=INFO msg="Upload complete. (320ms)"6358lb # [ 174.021536] niks3-server[1114]: 2026/09/23 04:45:43 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo6359worker1 # [ 173.980409] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:43Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 paths=1 nar_bytes_in=6406360worker1 # [ 173.997452] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966361worker1 # [ 174.012902] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6362worker1 # [ 174.023837] nix-daemon[834]: reaped child process 1613, status = succeeded6363client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051...6364client # farm-b> node-a: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv6365worker1 # [ 174.043132] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6366client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds6367client # farm-b> LOG-farm-b-during-drain6368worker1 # [ 175.149621] nix-daemon[834]: reaped child process 1622, status = succeeded6369worker1 # [ 175.227631] nix-daemon[834]: accepted connection from pid 1638, user nix-grpc-daemon (trusted)6370worker1 # [ 175.250271] nix-daemon[834]: reaped child process 1641, status = succeeded6371lb # [ 175.301990] niks3-server[1114]: 2026/09/23 04:45:44 INFO Received uploads request method=POST path=/api/pending_closures6372worker1 # [ 175.274998] nix-grpc-daemon[1504]: time=2026-09-23T04:45:44.772Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6373worker1 # [ 175.277045] nix-grpc-daemon[1504]: time=2026-09-23T04:45:44.772Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)"6374lb # [ 175.358405] niks3-server[1114]: 2026/09/23 04:45:44 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst6375worker1 # [ 175.331522] nix-grpc-daemon[1504]: time=2026-09-23T04:45:44.829Z level=INFO msg="Uploading 1 narinfos"6376lb # [ 175.378329] niks3-server[1114]: 2026/09/23 04:45:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign6377lb # [ 175.388342] niks3-server[1114]: 2026/09/23 04:45:44 INFO Signed narinfos id=58 count=16378lb # [ 175.391008] niks3-server[1114]: 2026/09/23 04:45:44 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv6379lb # [ 175.404330] niks3-server[1114]: 2026/09/23 04:45:44 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls6380lb # [ 175.414635] niks3-server[1114]: 2026/09/23 04:45:44 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete6381worker1 # [ 175.370361] nix-grpc-daemon[1504]: time=2026-09-23T04:45:44.868Z level=INFO msg="Upload complete. (312ms)"6382lb # [ 175.418533] niks3-server[1114]: 2026/09/23 04:45:44 INFO Completed upload id=586383lb # [ 175.420774] niks3-server[1114]: 2026/09/23 04:45:44 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo6384worker1 # [ 175.392861] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=16385worker1 # [ 175.412973] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:44Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=06386worker1 # [ 175.424475] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:44Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.3:50051 clients=16387worker1 # [ 175.430691] nix-daemon[834]: reaped child process 1620, status = succeeded6388worker1 # [ 175.449927] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966389worker1 # [ 175.469910] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608966390worker1 # [ 175.487613] nix-grpc-daemon[1504]: time=2026-09-23T04:45:44.986Z level=INFO msg="All 2 paths already cached"6391client # copying 1 paths to 'lb:50051'...6392worker1 # [ 175.502109] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608966393client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'...6394worker1 # [ 175.527894] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6395worker1 # [ 175.710455] nix-daemon[834]: accepted connection from pid 1647, user nix-grpc-daemon (trusted)6396worker1 # [ 175.732250] nix-daemon[834]: reaped child process 1650, status = succeeded6397lb # [ 175.783158] niks3-server[1114]: 2026/09/23 04:45:45 INFO Received uploads request method=POST path=/api/pending_closures6398worker1 # [ 175.752697] nix-grpc-daemon[1504]: time=2026-09-23T04:45:45.251Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6399worker1 # [ 175.754583] nix-grpc-daemon[1504]: time=2026-09-23T04:45:45.251Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)"6400lb # [ 175.824855] niks3-server[1114]: 2026/09/23 04:45:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign6401worker1 # [ 175.791096] nix-grpc-daemon[1504]: time=2026-09-23T04:45:45.289Z level=INFO msg="Uploading 1 narinfos"6402lb # [ 175.841309] niks3-server[1114]: 2026/09/23 04:45:45 INFO Signed narinfos id=59 count=16403lb # [ 175.843889] niks3-server[1114]: 2026/09/23 04:45:45 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls6404lb # [ 175.854802] niks3-server[1114]: 2026/09/23 04:45:45 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst6405lb # [ 175.869421] niks3-server[1114]: 2026/09/23 04:45:45 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete6406worker1 # [ 175.831085] nix-grpc-daemon[1504]: time=2026-09-23T04:45:45.329Z level=INFO msg="Upload complete. (283ms)"6407lb # [ 175.880949] niks3-server[1114]: 2026/09/23 04:45:45 INFO Completed upload id=596408lb # [ 175.887736] niks3-server[1114]: 2026/09/23 04:45:45 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo6409worker1 # [ 175.845312] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 paths=1 nar_bytes_in=7686410worker1 # [ 175.856400] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966411worker1 # [ 175.874566] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6412worker1 # [ 175.881711] nix-daemon[834]: reaped child process 1645, status = succeeded6413client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051...6414client # farm-top> node-a: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6415worker1 # [ 175.900117] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6416client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds6417client # farm-top> LOG-farm-top-during-drain6418worker1 # [ 177.371178] nix-daemon[834]: reaped child process 1654, status = succeeded6419worker1 # [ 177.486914] nix-daemon[834]: accepted connection from pid 1670, user nix-grpc-daemon (trusted)6420worker1 # [ 177.506512] nix-daemon[834]: reaped child process 1673, status = succeeded6421lb # [ 177.558109] niks3-server[1114]: 2026/09/23 04:45:47 INFO Received uploads request method=POST path=/api/pending_closures6422worker1 # [ 177.524395] nix-grpc-daemon[1504]: time=2026-09-23T04:45:47.020Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"6423worker1 # [ 177.526188] nix-grpc-daemon[1504]: time=2026-09-23T04:45:47.020Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)"6424lb # [ 177.597682] niks3-server[1114]: 2026/09/23 04:45:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign6425worker1 # [ 177.564766] nix-grpc-daemon[1504]: time=2026-09-23T04:45:47.063Z level=INFO msg="Uploading 1 narinfos"6426lb # [ 177.617829] niks3-server[1114]: 2026/09/23 04:45:47 INFO Signed narinfos id=60 count=16427lb # [ 177.626032] niks3-server[1114]: 2026/09/23 04:45:47 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv6428lb # [ 177.635885] niks3-server[1114]: 2026/09/23 04:45:47 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst6429lb # [ 177.641628] niks3-server[1114]: 2026/09/23 04:45:47 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls6430lb # [ 177.650742] niks3-server[1114]: 2026/09/23 04:45:47 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete6431lb # [ 177.654965] niks3-server[1114]: 2026/09/23 04:45:47 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo6432worker1 # [ 177.611421] nix-grpc-daemon[1504]: time=2026-09-23T04:45:47.110Z level=INFO msg="Upload complete. (297ms)"6433lb # [ 177.658409] niks3-server[1114]: 2026/09/23 04:45:47 INFO Completed upload id=606434worker1 # [ 177.625252] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=16435worker1 # [ 177.645770] nix-daemon[834]: reaped child process 1652, status = succeeded6436client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2, in 6.05 seconds)6437client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath6438client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath, in 0.19 seconds)6439worker2: must succeed: pkill -f 'read -t [9]0 x'6440worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.06 seconds)6441client: waiting for success: ! systemctl is-active sw6442worker2 # [ 178.256797] nix-daemon[856]: reaped child process 1859, status = succeeded6443worker2 # [ 178.323867] nix-daemon[856]: accepted connection from pid 2036, user nix-grpc-daemon (trusted)6444worker2 # [ 178.340980] nix-daemon[856]: reaped child process 2039, status = succeeded6445lb # [ 178.289519] niks3-server[1114]: 2026/09/23 04:45:47 INFO Received uploads request method=POST path=/api/pending_closures6446worker2 # [ 178.359373] nix-grpc-daemon[1804]: time=2026-09-23T04:45:47.880Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6447worker2 # [ 178.361123] nix-grpc-daemon[1804]: time=2026-09-23T04:45:47.880Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)"6448lb # [ 178.331704] niks3-server[1114]: 2026/09/23 04:45:47 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv6449worker2 # [ 178.400492] nix-grpc-daemon[1804]: time=2026-09-23T04:45:47.922Z level=INFO msg="Uploading 1 narinfos"6450lb # [ 178.346713] niks3-server[1114]: 2026/09/23 04:45:47 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst6451lb # [ 178.360828] niks3-server[1114]: 2026/09/23 04:45:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign6452lb # [ 178.366450] niks3-server[1114]: 2026/09/23 04:45:47 INFO Signed narinfos id=61 count=16453worker2 # [ 178.424985] nix-grpc-daemon[1804]: time=2026-09-23T04:45:47.947Z level=INFO msg="Upload complete. (263ms)"6454lb # [ 178.370811] niks3-server[1114]: 2026/09/23 04:45:47 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls6455lb # [ 178.375436] niks3-server[1114]: 2026/09/23 04:45:47 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete6456lb # [ 178.379678] niks3-server[1114]: 2026/09/23 04:45:47 INFO Completed upload id=616457lb # [ 178.382531] niks3-server[1114]: 2026/09/23 04:45:47 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo6458worker2 # [ 178.441061] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:47Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:45354 duration_s=16 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=16459worker2 # [ 178.455815] nix-daemon[856]: reaped child process 1857, status = succeeded6460client # [ 178.399373] systemd[1]: sw.service: Deactivated successfully.6461client # [ 178.404110] systemd[1]: sw.service: Consumed 202ms CPU time over 17.371s wall clock time, 39.2M memory peak, 7.7K incoming IP traffic, 8.7K outgoing IP traffic.6462worker2 # [ 178.655814] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:48Z level=info event=signal_exit6463client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds)6464client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }6465client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.05 seconds)6466worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service6467worker2 # [ 179.256778] systemd[1]: Starting scheduler-token.service...6468worker2 # [ 179.324079] systemd[1]: scheduler-token.service: Deactivated successfully.6469worker2 # [ 179.333924] systemd[1]: Finished scheduler-token.service.6470worker2 # [ 179.335521] systemd[1]: scheduler-token.service: Consumed 38ms CPU time over 70ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic.6471lb # [ 179.425946] envoy[900]: conn peer=[2001:db8:1::1]:44226 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9625 tx=4417 ms=57906472lb # [ 179.426508] envoy[900]: conn peer=[2001:db8:1::1]:59952 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3038 tx=1578 ms=171446473worker1 # [ 180.565777] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:50Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:608806474worker1 # [ 180.566068] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:50Z level=info event=worker_gone addr=192.168.1.4:500516475worker2 # [ 180.686025] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:50Z level=info event=scheduler_restarting6476worker2 # [ 180.703177] nix-grpc-daemon[1722]: ts=2026-09-23T04:45:50Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited"6477worker2 # [ 180.729522] nix-daemon[856]: reaped child process 1775, status = succeeded6478worker2 # [ 181.158004] nix-grpc-daemon[1722]: E0000 00:00:1790138750.679906 1768 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true6479worker2 # [ 181.211304] nix-grpc-daemon[1722]: E0000 00:00:1790138750.733260 1768 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true6480worker2 # [ 182.697082] systemd[1]: nix-grpc-daemon.service: Deactivated successfully.6481worker2 # [ 182.703180] systemd[1]: nix-grpc-daemon.service: Consumed 814ms CPU time over 32.332s wall clock time, 36M memory peak, 22.9K incoming IP traffic, 36.8K outgoing IP traffic.6482worker2 # [ 182.724448] systemd[1]: Starting Nix worker-protocol over gRPC...6483worker2 # [ 182.871802] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:52Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear"6484worker2 # [ 182.874333] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:52Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc6485worker2 # [ 182.878326] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:52Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=16486worker2 # [ 182.884053] nix-grpc-daemon[2056]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR6487worker2 # [ 182.885505] nix-grpc-daemon[2056]: E0000 00:00:1790138752.404782 2056 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6488worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 3.75 seconds)6489lb: waiting until exactly no endpoints of aarch64-linux are unhealthy6490lb: must succeed: curl -sf localhost:9901/clusters6491worker2 # [ 182.929087] nix-grpc-daemon[2056]: E0000 00:00:1790138752.411173 2056 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6492worker2 # [ 182.931385] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:52Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket6493worker2 # [ 182.935777] nix-grpc-daemon[2056]: E0000 00:00:1790138752.419416 2096 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6494worker2 # [ 182.938857] nix-grpc-daemon[2056]: E0000 00:00:1790138752.436013 2096 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6495worker2 # [ 182.941412] nix-grpc-daemon[2056]: E0000 00:00:1790138752.436211 2096 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates.6496worker2 # [ 182.943747] systemd[1]: Started Nix worker-protocol over gRPC.6497worker2 # [ 182.944648] nix-grpc-daemon[2056]: E0000 00:00:1790138752.456894 2096 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates.6498worker2 # [ 182.957597] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:52Z level=info event=scheduler_connected addr=lb:500516499worker1 # [ 182.858254] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:52Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:608806500worker1 # [ 182.864485] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:52Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=06501lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds)6502lb: must succeed: curl -sf localhost:9901/clusters6503lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6504lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.14 seconds)6505lb: must succeed: curl -sf localhost:9901/clusters6506lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6507worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6508worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6509(finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 23.14 seconds)6510subtest: queue gauges show a burst while every build blocks6511lb: must succeed: curl -sf localhost:9901/clusters6512lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6513lb: must succeed: curl -sf localhost:9901/clusters6514lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6515worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6516worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)6517client: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]'6518client # [ 184.348922] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto --impure --expr "map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ \"q0\" \"q1\" \"q2\" \"q3\" ]".6519client # Running as unit: burst.service; invocation ID: f9b29435ece74a928ceff68043aea9616520client: (finished: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]', in 0.09 seconds)6521lb: must succeed: curl -sf localhost:9901/clusters6522lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds)6523lb: must succeed: curl -sf localhost:9901/clusters6524lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6525worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6526worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6527worker2 # [ 184.629280] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:446706528worker2 # [ 184.644892] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)6529client # [ 184.585083] nix[1551]: don't know how to build these paths:6530client # [ 184.586638] nix[1551]: /nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv6531client # [ 184.590167] nix[1551]: /nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv6532client # [ 184.591982] nix[1551]: /nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv6533client # [ 184.593835] nix[1551]: /nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv6534worker1 # [ 184.577761] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:54Z level=debug event=want client=5 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv cached=06535worker1 # [ 184.583247] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:54Z level=debug event=assigned drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv worker=192.168.1.3:50051 clients=16536worker1 # [ 184.589285] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:54Z level=debug event=want client=5 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv cached=06537worker1 # [ 184.594608] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:54Z level=debug event=assigned drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv worker=192.168.1.4:50051 clients=16538worker1 # [ 184.600780] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:54Z level=debug event=want client=5 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv cached=06539worker1 # [ 184.605530] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:54Z level=debug event=want client=5 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv cached=06540worker1 # [ 184.621075] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966541worker2 # [ 184.724538] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:446706542worker1 # [ 184.637601] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608966543worker2 # [ 184.741485] nix-grpc-daemon[2056]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 5 ms (attempt 1/5)6544client # [ 184.676238] nix[1551]: copying 1 paths to 'lb:50051'...6545worker1 # [ 184.652454] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608966546client # [ 184.680133] nix[1551]: copying path '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' to 'lb:50051'...6547worker2 # [ 184.755787] nix-grpc-daemon[2056]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 95 ms (attempt 2/5)6548worker1 # [ 184.670117] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6549worker2 # [ 184.860852] nix-grpc-daemon[2056]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 178 ms (attempt 3/5)6550worker1 # [ 184.819983] nix-daemon[834]: accepted connection from pid 1688, user nix-grpc-daemon (trusted)6551worker1 # [ 184.843648] nix-daemon[834]: reaped child process 1691, status = succeeded6552lb # [ 184.885721] niks3-server[1114]: 2026/09/23 04:45:54 INFO Received uploads request method=POST path=/api/pending_closures6553worker1 # [ 184.853208] nix-grpc-daemon[1504]: time=2026-09-23T04:45:54.351Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6554worker1 # [ 184.854651] nix-grpc-daemon[1504]: time=2026-09-23T04:45:54.351Z level=INFO msg="Uploading d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv (416B)"6555lb # [ 184.915130] niks3-server[1114]: 2026/09/23 04:45:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign6556worker1 # [ 184.879901] nix-grpc-daemon[1504]: time=2026-09-23T04:45:54.378Z level=INFO msg="Uploading 1 narinfos"6557lb # [ 184.927782] niks3-server[1114]: 2026/09/23 04:45:54 INFO Signed narinfos id=62 count=16558lb # [ 184.935360] niks3-server[1114]: 2026/09/23 04:45:54 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.ls6559lb # [ 184.948161] niks3-server[1114]: 2026/09/23 04:45:54 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete6560worker1 # [ 184.913047] nix-grpc-daemon[1504]: time=2026-09-23T04:45:54.411Z level=INFO msg="Upload complete. (226ms)"6561lb # [ 184.959374] niks3-server[1114]: 2026/09/23 04:45:54 INFO Completed upload id=626562lb # [ 184.968226] niks3-server[1114]: 2026/09/23 04:45:54 INFO Registered completed upload object_key=nar/06ismykzsix0w29371dm1hbzzf3ljf9jn7dyh8skp5wqfkkw0vlk.nar.zst6563worker1 # [ 184.924960] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:54Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 paths=1 nar_bytes_in=4166564worker1 # [ 184.932444] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966565lb # [ 184.974816] niks3-server[1114]: 2026/09/23 04:45:54 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.narinfo6566worker2 # [ 185.040988] nix-grpc-daemon[2056]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 448 ms (attempt 4/5)6567worker1 # [ 184.944514] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6568worker1 # [ 184.949870] nix-daemon[834]: reaped child process 1686, status = succeeded6569client # [ 184.984856] nix[1551]: building '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' on lb:50051...6570worker1 # [ 184.963473] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6571worker2 # [ 185.491755] nix-grpc-daemon[2056]: disabling binary cache 'https://cache.nixos.org' for 60 seconds6572worker2 # [ 185.498564] nix-grpc-daemon[2056]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org6573worker2 # [ 185.516598] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:446706574client # [ 185.452412] nix[1551]: copying 1 paths to 'lb:50051'...6575worker2 # [ 185.531989] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:446706576client # [ 185.456142] nix[1551]: copying path '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' to 'lb:50051'...6577worker2 # [ 185.548721] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)6578worker2 # [ 185.577786] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:55Z level=info event=niks3_push_started pid=21176579lb: must succeed: curl -sf localhost:9901/clusters6580lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6581lb: must succeed: curl -sf localhost:9901/clusters6582worker2 # [ 185.733973] nix-grpc-daemon[2117]: time=2026-09-23T04:45:55.255Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca=""6583lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6584worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6585worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds)6586worker2 # [ 185.845958] nix-daemon[856]: accepted connection from pid 2130, user nix-grpc-daemon (trusted)6587worker2 # [ 185.865911] nix-daemon[856]: reaped child process 2133, status = succeeded6588lb # [ 185.805516] niks3-server[1114]: 2026/09/23 04:45:55 INFO Received uploads request method=POST path=/api/pending_closures6589worker2 # [ 185.875321] nix-grpc-daemon[2117]: time=2026-09-23T04:45:55.396Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6590worker2 # [ 185.878687] nix-grpc-daemon[2117]: time=2026-09-23T04:45:55.396Z level=INFO msg="Uploading dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv (416B)"6591lb # [ 185.842771] niks3-server[1114]: 2026/09/23 04:45:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign6592worker2 # [ 185.913327] nix-grpc-daemon[2117]: time=2026-09-23T04:45:55.434Z level=INFO msg="Uploading 1 narinfos"6593lb # [ 185.857364] niks3-server[1114]: 2026/09/23 04:45:55 INFO Signed narinfos id=63 count=16594lb # [ 185.859864] niks3-server[1114]: 2026/09/23 04:45:55 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.ls6595worker2 # [ 185.932296] nix-grpc-daemon[2117]: time=2026-09-23T04:45:55.454Z level=INFO msg="Upload complete. (194ms)"6596lb # [ 185.872648] niks3-server[1114]: 2026/09/23 04:45:55 INFO Registered completed upload object_key=nar/06mdwsnf16qmfi86y8r33asi4vcrp5z7kdxqki87psgfpgvvpdki.nar.zst6597worker2 # [ 185.938290] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44670 duration_s=0 paths=1 nar_bytes_in=4166598lb # [ 185.880968] niks3-server[1114]: 2026/09/23 04:45:55 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete6599lb # [ 185.886543] niks3-server[1114]: 2026/09/23 04:45:55 INFO Completed upload id=636600lb # [ 185.888958] niks3-server[1114]: 2026/09/23 04:45:55 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.narinfo6601worker2 # [ 185.952189] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:446706602worker2 # [ 185.962628] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)6603worker2 # [ 185.963818] nix-daemon[856]: reaped child process 2115, status = succeeded6604worker2 # [ 185.972719] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)6605client # [ 185.897043] nix[1551]: building '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' on lb:50051...6606worker2 # [ 187.196086] nix-daemon[856]: reaped child process 2141, status = succeeded6607worker2 # [ 187.289356] nix-daemon[856]: accepted connection from pid 2162, user nix-grpc-daemon (trusted)6608worker2 # [ 187.309007] nix-daemon[856]: reaped child process 2165, status = succeeded6609lb # [ 187.255121] niks3-server[1114]: 2026/09/23 04:45:56 INFO Received uploads request method=POST path=/api/pending_closures6610worker2 # [ 187.324374] nix-grpc-daemon[2117]: time=2026-09-23T04:45:56.845Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6611worker2 # [ 187.325869] nix-grpc-daemon[2117]: time=2026-09-23T04:45:56.845Z level=INFO msg="Uploading y3qkgrgq2zzpgvi2rf5chq94i627wkdb-slow-q3 (120B)"6612lb # [ 187.297747] niks3-server[1114]: 2026/09/23 04:45:56 INFO Registered completed upload object_key=nar/1v1grzkvw3yj09x0dkdm2avihvqvnri156qzp768hk85zbw7lkn9.nar.zst6613worker1 # [ 187.274412] nix-daemon[834]: reaped child process 1695, status = succeeded6614worker2 # [ 187.379999] nix-grpc-daemon[2117]: time=2026-09-23T04:45:56.901Z level=INFO msg="Uploading 1 narinfos"6615lb # [ 187.310637] niks3-server[1114]: 2026/09/23 04:45:56 INFO Registered completed upload object_key=log/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv6616lb # [ 187.323779] niks3-server[1114]: 2026/09/23 04:45:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign6617lb # [ 187.333123] niks3-server[1114]: 2026/09/23 04:45:56 INFO Signed narinfos id=64 count=16618lb # [ 187.335847] niks3-server[1114]: 2026/09/23 04:45:56 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.ls6619lb # [ 187.347813] niks3-server[1114]: 2026/09/23 04:45:56 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete6620lb # [ 187.354011] niks3-server[1114]: 2026/09/23 04:45:56 INFO Completed upload id=646621lb # [ 187.357148] niks3-server[1114]: 2026/09/23 04:45:56 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.narinfo6622worker2 # [ 187.412477] nix-grpc-daemon[2117]: time=2026-09-23T04:45:56.934Z level=INFO msg="Upload complete. (230ms)"6623worker1 # [ 187.318679] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:56Z level=debug event=assigned drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv worker=192.168.1.4:50051 clients=16624worker2 # [ 187.425373] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44670 duration_s=1 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv assign_id=8 outputs=16625worker2 # [ 187.444626] nix-daemon[856]: reaped child process 2139, status = succeeded6626worker1 # [ 187.354473] nix-daemon[834]: accepted connection from pid 1723, user nix-grpc-daemon (trusted)6627worker1 # [ 187.375847] nix-daemon[834]: reaped child process 1726, status = succeeded6628worker2 # [ 187.475692] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:446706629lb # [ 187.417963] niks3-server[1114]: 2026/09/23 04:45:56 INFO Received uploads request method=POST path=/api/pending_closures6630worker1 # [ 187.387124] nix-grpc-daemon[1504]: time=2026-09-23T04:45:56.883Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6631worker1 # [ 187.388874] nix-grpc-daemon[1504]: time=2026-09-23T04:45:56.883Z level=INFO msg="Uploading k6zivhvzgmlyxfmy64fzd759bxlhf9b5-slow-q0 (120B)"6632worker2 # [ 187.492887] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:446706633client # [ 187.437375] nix[1551]: copying 1 paths to 'lb:50051'...6634worker2 # [ 187.515934] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:446706635client # [ 187.437655] nix[1551]: copying path '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' to 'lb:50051'...6636lb # [ 187.465144] niks3-server[1114]: 2026/09/23 04:45:56 INFO Registered completed upload object_key=nar/0xr809jj32p5n6mnc6i7lqsz48wrm5f9jp14m77k724041711iq7.nar.zst6637worker2 # [ 187.532190] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)6638worker1 # [ 187.432110] nix-grpc-daemon[1504]: time=2026-09-23T04:45:56.930Z level=INFO msg="Uploading 1 narinfos"6639lb # [ 187.477261] niks3-server[1114]: 2026/09/23 04:45:56 INFO Registered completed upload object_key=log/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv6640lb # [ 187.484660] niks3-server[1114]: 2026/09/23 04:45:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign6641lb # [ 187.494053] niks3-server[1114]: 2026/09/23 04:45:56 INFO Signed narinfos id=65 count=16642worker1 # [ 187.457553] nix-grpc-daemon[1504]: time=2026-09-23T04:45:56.956Z level=INFO msg="Upload complete. (282ms)"6643lb # [ 187.502111] niks3-server[1114]: 2026/09/23 04:45:56 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.ls6644lb # [ 187.506803] niks3-server[1114]: 2026/09/23 04:45:57 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete6645lb # [ 187.514214] niks3-server[1114]: 2026/09/23 04:45:57 INFO Completed upload id=656646worker1 # [ 187.469491] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=2 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv assign_id=7 outputs=16647lb # [ 187.518263] niks3-server[1114]: 2026/09/23 04:45:57 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.narinfo6648worker1 # [ 187.488322] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:56Z level=debug event=assigned drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker=192.168.1.3:50051 clients=16649worker1 # [ 187.499415] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966650worker1 # [ 187.505337] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608966651worker1 # [ 187.510917] nix-daemon[834]: reaped child process 1693, status = succeeded6652client # [ 187.536770] nix[1551]: copying 1 paths to 'lb:50051'...6653worker1 # [ 187.517085] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608966654client # [ 187.542356] nix[1551]: copying path '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' to 'lb:50051'...6655worker1 # [ 187.527853] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6656worker2 # [ 187.692114] nix-daemon[856]: accepted connection from pid 2169, user nix-grpc-daemon (trusted)6657worker2 # [ 187.712252] nix-daemon[856]: reaped child process 2172, status = succeeded6658lb # [ 187.656492] niks3-server[1114]: 2026/09/23 04:45:57 INFO Received uploads request method=POST path=/api/pending_closures6659worker2 # [ 187.725946] nix-grpc-daemon[2117]: time=2026-09-23T04:45:57.247Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6660worker2 # [ 187.727492] nix-grpc-daemon[2117]: time=2026-09-23T04:45:57.247Z level=INFO msg="Uploading jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv (416B)"6661worker2 # [ 187.746341] nix-grpc-daemon[2117]: time=2026-09-23T04:45:57.268Z level=INFO msg="Uploading 1 narinfos"6662lb # [ 187.686677] niks3-server[1114]: 2026/09/23 04:45:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign6663lb # [ 187.700773] niks3-server[1114]: 2026/09/23 04:45:57 INFO Signed narinfos id=66 count=16664worker1 # [ 187.660735] nix-daemon[834]: accepted connection from pid 1730, user nix-grpc-daemon (trusted)6665lb # [ 187.709404] niks3-server[1114]: 2026/09/23 04:45:57 INFO Registered completed upload object_key=nar/1fyfjzlx8vymsfmdv5cv8ymgjkr18fwm737qvys29plwwiwmyvwr.nar.zst6666worker1 # [ 187.681234] nix-daemon[834]: reaped child process 1733, status = succeeded6667lb # [ 187.717981] niks3-server[1114]: 2026/09/23 04:45:57 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.ls6668worker1 # [ 187.687314] nix-grpc-daemon[1504]: time=2026-09-23T04:45:57.185Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6669worker2 # [ 187.788660] nix-grpc-daemon[2117]: time=2026-09-23T04:45:57.310Z level=INFO msg="Upload complete. (239ms)"6670worker1 # [ 187.689070] nix-grpc-daemon[1504]: time=2026-09-23T04:45:57.185Z level=INFO msg="Uploading pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv (416B)"6671lb # [ 187.732569] niks3-server[1114]: 2026/09/23 04:45:57 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete6672worker2 # [ 187.798048] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44670 duration_s=0 paths=1 nar_bytes_in=4166673lb # [ 187.742065] niks3-server[1114]: 2026/09/23 04:45:57 INFO Received uploads request method=POST path=/api/pending_closures6674lb # [ 187.748831] niks3-server[1114]: 2026/09/23 04:45:57 INFO Completed upload id=666675lb # [ 187.750987] niks3-server[1114]: 2026/09/23 04:45:57 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.narinfo6676worker2 # [ 187.815448] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:446706677lb # [ 187.762578] niks3-server[1114]: 2026/09/23 04:45:57 INFO Registered completed upload object_key=nar/1k21jrz411xc9gkca2pnr9mrjq1s3bg3zhsam3nnv4j066dgzgm7.nar.zst6678worker2 # [ 187.822722] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)6679worker1 # [ 187.724566] nix-grpc-daemon[1504]: time=2026-09-23T04:45:57.223Z level=INFO msg="Uploading 1 narinfos"6680worker2 # [ 187.828741] nix-daemon[856]: reaped child process 2167, status = succeeded6681lb # [ 187.770130] niks3-server[1114]: 2026/09/23 04:45:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign6682lb # [ 187.776870] niks3-server[1114]: 2026/09/23 04:45:57 INFO Signed narinfos id=67 count=16683client # [ 187.761735] nix[1551]: building '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' on lb:50051...6684worker2 # [ 187.843233] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)6685lb # [ 187.783566] niks3-server[1114]: 2026/09/23 04:45:57 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.ls6686worker1 # [ 187.745800] nix-grpc-daemon[1504]: time=2026-09-23T04:45:57.244Z level=INFO msg="Upload complete. (206ms)"6687lb # [ 187.790776] niks3-server[1114]: 2026/09/23 04:45:57 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete6688lb # [ 187.794943] niks3-server[1114]: 2026/09/23 04:45:57 INFO Completed upload id=676689lb # [ 187.797629] niks3-server[1114]: 2026/09/23 04:45:57 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.narinfo6690worker1 # [ 187.754468] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 paths=1 nar_bytes_in=4166691worker1 # [ 187.767050] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966692worker1 # [ 187.772298] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6693worker1 # [ 187.776698] nix-daemon[834]: reaped child process 1728, status = succeeded6694client # [ 187.809675] nix[1551]: building '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' on lb:50051...6695worker1 # [ 187.787654] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6696lb # [ 189.426800] envoy[900]: conn peer=[2001:db8:1::4]:53804 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=7599 tx=679 ms=299746697worker1 # [ 189.414820] nix-daemon[834]: reaped child process 1737, status = succeeded6698worker1 # [ 189.508445] nix-daemon[834]: accepted connection from pid 1762, user nix-grpc-daemon (trusted)6699worker1 # [ 189.526163] nix-daemon[834]: reaped child process 1765, status = succeeded6700lb # [ 189.575176] niks3-server[1114]: 2026/09/23 04:45:59 INFO Received uploads request method=POST path=/api/pending_closures6701worker1 # [ 189.543926] nix-grpc-daemon[1504]: time=2026-09-23T04:45:59.039Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6702worker1 # [ 189.545601] nix-grpc-daemon[1504]: time=2026-09-23T04:45:59.039Z level=INFO msg="Uploading l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx-slow-q1 (120B)"6703lb # [ 189.614090] niks3-server[1114]: 2026/09/23 04:45:59 INFO Registered completed upload object_key=nar/0c1c367r3msa4fz6imwp70zxi4fxvrkb86v3hv921wsrf13yd8v3.nar.zst6704lb # [ 189.629321] niks3-server[1114]: 2026/09/23 04:45:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign6705worker1 # [ 189.592220] nix-grpc-daemon[1504]: time=2026-09-23T04:45:59.090Z level=INFO msg="Uploading 1 narinfos"6706lb # [ 189.641549] niks3-server[1114]: 2026/09/23 04:45:59 INFO Signed narinfos id=68 count=16707lb # [ 189.646582] niks3-server[1114]: 2026/09/23 04:45:59 INFO Registered completed upload object_key=log/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv6708lb # [ 189.655593] niks3-server[1114]: 2026/09/23 04:45:59 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.ls6709worker1 # [ 189.616985] nix-grpc-daemon[1504]: time=2026-09-23T04:45:59.115Z level=INFO msg="Upload complete. (220ms)"6710lb # [ 189.661335] niks3-server[1114]: 2026/09/23 04:45:59 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete6711lb # [ 189.664701] niks3-server[1114]: 2026/09/23 04:45:59 INFO Completed upload id=686712lb # [ 189.668366] niks3-server[1114]: 2026/09/23 04:45:59 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.narinfo6713worker1 # [ 189.628935] nix-grpc-daemon[1407]: ts=2026-09-23T04:45:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=1 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv assign_id=10 outputs=16714worker1 # [ 189.641800] nix-daemon[834]: reaped child process 1735, status = succeeded6715worker2 # [ 189.748592] nix-daemon[856]: reaped child process 2176, status = succeeded6716worker2 # [ 189.820009] nix-daemon[856]: accepted connection from pid 2201, user nix-grpc-daemon (trusted)6717worker2 # [ 189.838091] nix-daemon[856]: reaped child process 2204, status = succeeded6718lb # [ 189.783292] niks3-server[1114]: 2026/09/23 04:45:59 INFO Received uploads request method=POST path=/api/pending_closures6719worker2 # [ 189.852331] nix-grpc-daemon[2117]: time=2026-09-23T04:45:59.373Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6720worker2 # [ 189.853859] nix-grpc-daemon[2117]: time=2026-09-23T04:45:59.373Z level=INFO msg="Uploading gjyi8j90pk5flrsqqdhj6h132pcb7n33-slow-q2 (120B)"6721lb # [ 189.822306] niks3-server[1114]: 2026/09/23 04:45:59 INFO Registered completed upload object_key=log/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv6722worker2 # [ 189.894361] nix-grpc-daemon[2117]: time=2026-09-23T04:45:59.416Z level=INFO msg="Uploading 1 narinfos"6723lb # [ 189.838196] niks3-server[1114]: 2026/09/23 04:45:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign6724lb # [ 189.846842] niks3-server[1114]: 2026/09/23 04:45:59 INFO Registered completed upload object_key=nar/15imdx7zyy6mzjjv6qvdzywh0d9w1n6ikc0rq4rq95aw9x6swfzj.nar.zst6725worker2 # [ 189.915532] nix-grpc-daemon[2117]: time=2026-09-23T04:45:59.437Z level=INFO msg="Upload complete. (261ms)"6726lb # [ 189.857720] niks3-server[1114]: 2026/09/23 04:45:59 INFO Signed narinfos id=69 count=16727lb # [ 189.859871] niks3-server[1114]: 2026/09/23 04:45:59 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.ls6728lb # [ 189.866291] niks3-server[1114]: 2026/09/23 04:45:59 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete6729worker2 # [ 189.924769] nix-grpc-daemon[2056]: ts=2026-09-23T04:45:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44670 duration_s=2 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv assign_id=9 outputs=16730lb # [ 189.870508] niks3-server[1114]: 2026/09/23 04:45:59 INFO Completed upload id=696731lb # [ 189.873798] niks3-server[1114]: 2026/09/23 04:45:59 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.narinfo6732worker2 # [ 189.940973] nix-daemon[856]: reaped child process 2174, status = succeeded6733client # [ 189.888681] systemd[1]: burst.service: Deactivated successfully.6734client # [ 189.893776] systemd[1]: burst.service: Consumed 226ms CPU time over 5.541s wall clock time, 57.6M memory peak, 14K incoming IP traffic, 18.5K outgoing IP traffic.6735client: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }6736client: (finished: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }, in 0.07 seconds)6737lb: must succeed: curl -sf localhost:9901/clusters6738lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)6739lb: must succeed: curl -sf localhost:9901/clusters6740lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)6741worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics6742worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)6743(finished: subtest: queue gauges show a burst while every build blocks, in 6.85 seconds)6744subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does6745client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&16746worker1 # [ 191.156995] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:00Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:608966747worker2 # [ 191.273105] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:446706748worker1 # [ 191.215726] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:00Z level=debug event=want client=6 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=06749worker1 # [ 191.222712] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:00Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=16750worker1 # [ 191.228862] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:00Z level=debug event=want client=6 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=06751worker1 # [ 191.234092] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:00Z level=debug event=want client=6 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=06752worker1 # [ 191.260735] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966753worker1 # [ 191.275804] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608966754worker1 # [ 191.291389] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608966755worker1 # [ 191.313284] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6756worker1 # [ 191.464613] nix-daemon[834]: accepted connection from pid 1778, user nix-grpc-daemon (trusted)6757worker1 # [ 191.483648] nix-daemon[834]: reaped child process 1781, status = succeeded6758lb # [ 191.531621] niks3-server[1114]: 2026/09/23 04:46:01 INFO Received uploads request method=POST path=/api/pending_closures6759worker1 # [ 191.500170] nix-grpc-daemon[1504]: time=2026-09-23T04:46:00.998Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6760worker1 # [ 191.501806] nix-grpc-daemon[1504]: time=2026-09-23T04:46:00.998Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)"6761lb # [ 191.568579] niks3-server[1114]: 2026/09/23 04:46:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign6762worker1 # [ 191.533373] nix-grpc-daemon[1504]: time=2026-09-23T04:46:01.031Z level=INFO msg="Uploading 1 narinfos"6763lb # [ 191.583474] niks3-server[1114]: 2026/09/23 04:46:01 INFO Signed narinfos id=70 count=16764lb # [ 191.590744] niks3-server[1114]: 2026/09/23 04:46:01 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst6765lb # [ 191.600772] niks3-server[1114]: 2026/09/23 04:46:01 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls6766lb # [ 191.609323] niks3-server[1114]: 2026/09/23 04:46:01 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete6767worker1 # [ 191.565016] nix-grpc-daemon[1504]: time=2026-09-23T04:46:01.063Z level=INFO msg="Upload complete. (242ms)"6768lb # [ 191.619657] niks3-server[1114]: 2026/09/23 04:46:01 INFO Completed upload id=706769worker1 # [ 191.576276] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 paths=1 nar_bytes_in=5126770lb # [ 191.622789] niks3-server[1114]: 2026/09/23 04:46:01 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo6771worker1 # [ 191.591918] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966772worker1 # [ 191.602412] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6773worker1 # [ 191.606177] nix-daemon[834]: reaped child process 1776, status = succeeded6774worker1 # [ 191.619683] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6775worker1 # [ 193.135196] nix-daemon[834]: reaped child process 1785, status = succeeded6776worker1 # [ 193.208251] nix-daemon[834]: accepted connection from pid 1801, user nix-grpc-daemon (trusted)6777worker1 # [ 193.229412] nix-daemon[834]: reaped child process 1804, status = succeeded6778lb # [ 193.278918] niks3-server[1114]: 2026/09/23 04:46:02 INFO Received uploads request method=POST path=/api/pending_closures6779worker1 # [ 193.244611] nix-grpc-daemon[1504]: time=2026-09-23T04:46:02.742Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6780worker1 # [ 193.247599] nix-grpc-daemon[1504]: time=2026-09-23T04:46:02.743Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)"6781lb # [ 193.320796] niks3-server[1114]: 2026/09/23 04:46:02 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst6782worker1 # [ 193.290302] nix-grpc-daemon[1504]: time=2026-09-23T04:46:02.788Z level=INFO msg="Uploading 1 narinfos"6783lb # [ 193.337091] niks3-server[1114]: 2026/09/23 04:46:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign6784lb # [ 193.347499] niks3-server[1114]: 2026/09/23 04:46:02 INFO Signed narinfos id=71 count=16785lb # [ 193.353667] niks3-server[1114]: 2026/09/23 04:46:02 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls6786lb # [ 193.366679] niks3-server[1114]: 2026/09/23 04:46:02 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv6787lb # [ 193.372340] niks3-server[1114]: 2026/09/23 04:46:02 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete6788lb # [ 193.376937] niks3-server[1114]: 2026/09/23 04:46:02 INFO Completed upload id=716789lb # [ 193.380214] niks3-server[1114]: 2026/09/23 04:46:02 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo6790worker1 # [ 193.328716] nix-grpc-daemon[1504]: time=2026-09-23T04:46:02.827Z level=INFO msg="Upload complete. (292ms)"6791worker1 # [ 193.346426] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=11 outputs=16792worker1 # [ 193.368531] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:02Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=16793worker1 # [ 193.377037] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966794worker1 # [ 193.383616] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:02Z level=debug event=want client=6 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=06795worker1 # [ 193.389377] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608966796worker1 # [ 193.401544] nix-daemon[834]: reaped child process 1783, status = succeeded6797worker1 # [ 193.403749] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6798worker1 # [ 193.412712] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608966799worker1 # [ 193.545632] nix-daemon[834]: accepted connection from pid 1810, user nix-grpc-daemon (trusted)6800worker1 # [ 193.562655] nix-daemon[834]: reaped child process 1813, status = succeeded6801lb # [ 193.612641] niks3-server[1114]: 2026/09/23 04:46:03 INFO Received uploads request method=POST path=/api/pending_closures6802worker1 # [ 193.577321] nix-grpc-daemon[1504]: time=2026-09-23T04:46:03.075Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6803worker1 # [ 193.578794] nix-grpc-daemon[1504]: time=2026-09-23T04:46:03.075Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)"6804lb # [ 193.640657] niks3-server[1114]: 2026/09/23 04:46:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign6805worker1 # [ 193.603249] nix-grpc-daemon[1504]: time=2026-09-23T04:46:03.101Z level=INFO msg="Uploading 1 narinfos"6806lb # [ 193.654049] niks3-server[1114]: 2026/09/23 04:46:03 INFO Signed narinfos id=72 count=16807lb # [ 193.663510] niks3-server[1114]: 2026/09/23 04:46:03 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls6808lb # [ 193.678165] niks3-server[1114]: 2026/09/23 04:46:03 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst6809worker1 # [ 193.642601] nix-grpc-daemon[1504]: time=2026-09-23T04:46:03.141Z level=INFO msg="Upload complete. (224ms)"6810lb # [ 193.687374] niks3-server[1114]: 2026/09/23 04:46:03 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete6811worker1 # [ 193.652591] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 paths=1 nar_bytes_in=5126812lb # [ 193.699526] niks3-server[1114]: 2026/09/23 04:46:03 INFO Completed upload id=726813lb # [ 193.702784] niks3-server[1114]: 2026/09/23 04:46:03 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo6814worker1 # [ 193.669024] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966815worker1 # [ 193.672835] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6816worker1 # [ 193.680200] nix-daemon[834]: reaped child process 1808, status = succeeded6817worker1 # [ 193.691405] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6818worker1 # [ 194.988240] nix-daemon[834]: reaped child process 1817, status = succeeded6819worker1 # [ 195.060442] nix-daemon[834]: accepted connection from pid 1833, user nix-grpc-daemon (trusted)6820worker1 # [ 195.080867] nix-daemon[834]: reaped child process 1836, status = succeeded6821lb # [ 195.124719] niks3-server[1114]: 2026/09/23 04:46:04 INFO Received uploads request method=POST path=/api/pending_closures6822worker1 # [ 195.092450] nix-grpc-daemon[1504]: time=2026-09-23T04:46:04.588Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6823worker1 # [ 195.094012] nix-grpc-daemon[1504]: time=2026-09-23T04:46:04.588Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)"6824lb # [ 195.172323] niks3-server[1114]: 2026/09/23 04:46:04 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls6825worker1 # [ 195.139880] nix-grpc-daemon[1504]: time=2026-09-23T04:46:04.638Z level=INFO msg="Uploading 1 narinfos"6826lb # [ 195.186217] niks3-server[1114]: 2026/09/23 04:46:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign6827lb # [ 195.194559] niks3-server[1114]: 2026/09/23 04:46:04 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst6828lb # [ 195.206033] niks3-server[1114]: 2026/09/23 04:46:04 INFO Signed narinfos id=73 count=16829lb # [ 195.209350] niks3-server[1114]: 2026/09/23 04:46:04 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv6830worker1 # [ 195.165486] nix-grpc-daemon[1504]: time=2026-09-23T04:46:04.664Z level=INFO msg="Upload complete. (263ms)"6831lb # [ 195.212664] niks3-server[1114]: 2026/09/23 04:46:04 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete6832lb # [ 195.216978] niks3-server[1114]: 2026/09/23 04:46:04 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo6833lb # [ 195.221704] niks3-server[1114]: 2026/09/23 04:46:04 INFO Completed upload id=736834worker1 # [ 195.183681] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=12 outputs=16835worker1 # [ 195.199235] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:04Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=16836worker1 # [ 195.204070] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966837worker1 # [ 195.208973] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:04Z level=debug event=want client=6 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=06838worker1 # [ 195.213737] nix-daemon[834]: reaped child process 1815, status = succeeded6839worker1 # [ 195.215461] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608966840worker1 # [ 195.223699] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608966841worker1 # [ 195.234399] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6842worker1 # [ 195.367108] nix-daemon[834]: accepted connection from pid 1842, user nix-grpc-daemon (trusted)6843worker1 # [ 195.387813] nix-daemon[834]: reaped child process 1845, status = succeeded6844lb # [ 195.434253] niks3-server[1114]: 2026/09/23 04:46:04 INFO Received uploads request method=POST path=/api/pending_closures6845worker1 # [ 195.398759] nix-grpc-daemon[1504]: time=2026-09-23T04:46:04.897Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6846worker1 # [ 195.400331] nix-grpc-daemon[1504]: time=2026-09-23T04:46:04.897Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)"6847lb # [ 195.463351] niks3-server[1114]: 2026/09/23 04:46:04 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls6848lb # [ 195.474263] niks3-server[1114]: 2026/09/23 04:46:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign6849worker1 # [ 195.436136] nix-grpc-daemon[1504]: time=2026-09-23T04:46:04.934Z level=INFO msg="Uploading 1 narinfos"6850lb # [ 195.487347] niks3-server[1114]: 2026/09/23 04:46:04 INFO Signed narinfos id=74 count=16851lb # [ 195.492258] niks3-server[1114]: 2026/09/23 04:46:05 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst6852worker1 # [ 195.459389] nix-grpc-daemon[1504]: time=2026-09-23T04:46:04.958Z level=INFO msg="Upload complete. (213ms)"6853lb # [ 195.503658] niks3-server[1114]: 2026/09/23 04:46:05 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete6854lb # [ 195.509633] niks3-server[1114]: 2026/09/23 04:46:05 INFO Completed upload id=746855lb # [ 195.514117] niks3-server[1114]: 2026/09/23 04:46:05 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo6856worker1 # [ 195.468239] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 paths=1 nar_bytes_in=5126857worker1 # [ 195.481305] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966858worker1 # [ 195.489559] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6859worker1 # [ 195.492402] nix-daemon[834]: reaped child process 1840, status = succeeded6860worker1 # [ 195.504519] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6861worker1 # [ 196.934920] nix-daemon[834]: reaped child process 1849, status = succeeded6862worker1 # [ 197.040106] nix-daemon[834]: accepted connection from pid 1865, user nix-grpc-daemon (trusted)6863worker1 # [ 197.056461] nix-daemon[834]: reaped child process 1868, status = succeeded6864lb # [ 197.106419] niks3-server[1114]: 2026/09/23 04:46:06 INFO Received uploads request method=POST path=/api/pending_closures6865worker1 # [ 197.072327] nix-grpc-daemon[1504]: time=2026-09-23T04:46:06.570Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"6866worker1 # [ 197.074945] nix-grpc-daemon[1504]: time=2026-09-23T04:46:06.572Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)"6867lb # [ 197.148635] niks3-server[1114]: 2026/09/23 04:46:06 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst6868worker1 # [ 197.117736] nix-grpc-daemon[1504]: time=2026-09-23T04:46:06.616Z level=INFO msg="Uploading 1 narinfos"6869lb # [ 197.163147] niks3-server[1114]: 2026/09/23 04:46:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign6870lb # [ 197.173450] niks3-server[1114]: 2026/09/23 04:46:06 INFO Signed narinfos id=75 count=16871lb # [ 197.175842] niks3-server[1114]: 2026/09/23 04:46:06 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv6872lb # [ 197.186966] niks3-server[1114]: 2026/09/23 04:46:06 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls6873lb # [ 197.191730] niks3-server[1114]: 2026/09/23 04:46:06 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete6874worker1 # [ 197.148167] nix-grpc-daemon[1504]: time=2026-09-23T04:46:06.646Z level=INFO msg="Upload complete. (233ms)"6875lb # [ 197.194770] niks3-server[1114]: 2026/09/23 04:46:06 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo6876lb # [ 197.199045] niks3-server[1114]: 2026/09/23 04:46:06 INFO Completed upload id=756877worker1 # [ 197.168220] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=13 outputs=16878worker1 # [ 197.176785] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:06Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=16879worker1 # [ 197.186316] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966880worker1 # [ 197.191714] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:06Z level=debug event=want client=6 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=06881worker1 # [ 197.196669] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608966882worker1 # [ 197.199858] nix-daemon[834]: reaped child process 1847, status = succeeded6883worker1 # [ 197.203099] nix-grpc-daemon[1504]: time=2026-09-23T04:46:06.700Z level=INFO msg="All 1 paths already cached"6884worker1 # [ 197.207040] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608966885worker1 # [ 197.220237] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6886worker1 # [ 197.344716] nix-daemon[834]: accepted connection from pid 1874, user nix-grpc-daemon (trusted)6887worker1 # [ 197.362889] nix-daemon[834]: reaped child process 1877, status = succeeded6888lb # [ 197.412290] niks3-server[1114]: 2026/09/23 04:46:06 INFO Received uploads request method=POST path=/api/pending_closures6889worker1 # [ 197.377458] nix-grpc-daemon[1504]: time=2026-09-23T04:46:06.875Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6890worker1 # [ 197.378932] nix-grpc-daemon[1504]: time=2026-09-23T04:46:06.875Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)"6891lb # [ 197.441747] niks3-server[1114]: 2026/09/23 04:46:06 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls6892worker1 # [ 197.414338] nix-grpc-daemon[1504]: time=2026-09-23T04:46:06.912Z level=INFO msg="Uploading 1 narinfos"6893lb # [ 197.452100] niks3-server[1114]: 2026/09/23 04:46:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign6894lb # [ 197.461770] niks3-server[1114]: 2026/09/23 04:46:06 INFO Signed narinfos id=76 count=16895lb # [ 197.470789] niks3-server[1114]: 2026/09/23 04:46:06 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst6896lb # [ 197.484562] niks3-server[1114]: 2026/09/23 04:46:06 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete6897worker1 # [ 197.448311] nix-grpc-daemon[1504]: time=2026-09-23T04:46:06.946Z level=INFO msg="Upload complete. (221ms)"6898lb # [ 197.495160] niks3-server[1114]: 2026/09/23 04:46:07 INFO Completed upload id=766899lb # [ 197.500870] niks3-server[1114]: 2026/09/23 04:46:07 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo6900worker1 # [ 197.457255] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 paths=1 nar_bytes_in=6246901worker1 # [ 197.468524] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966902worker1 # [ 197.479328] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6903worker1 # [ 197.488400] nix-daemon[834]: reaped child process 1872, status = succeeded6904worker1 # [ 197.496840] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6905worker1 # [ 198.141293] nix-daemon[834]: reaped child process 1881, status = succeeded6906worker1 # [ 198.256137] nix-daemon[834]: accepted connection from pid 1897, user nix-grpc-daemon (trusted)6907worker1 # [ 198.272648] nix-daemon[834]: reaped child process 1900, status = succeeded6908lb # [ 198.323242] niks3-server[1114]: 2026/09/23 04:46:07 INFO Received uploads request method=POST path=/api/pending_closures6909worker1 # [ 198.290281] nix-grpc-daemon[1504]: time=2026-09-23T04:46:07.787Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6910worker1 # [ 198.291832] nix-grpc-daemon[1504]: time=2026-09-23T04:46:07.787Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)"6911lb # [ 198.362652] niks3-server[1114]: 2026/09/23 04:46:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign6912worker1 # [ 198.326236] nix-grpc-daemon[1504]: time=2026-09-23T04:46:07.824Z level=INFO msg="Uploading 1 narinfos"6913lb # [ 198.376086] niks3-server[1114]: 2026/09/23 04:46:07 INFO Signed narinfos id=77 count=16914lb # [ 198.378780] niks3-server[1114]: 2026/09/23 04:46:07 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv6915lb # [ 198.393205] niks3-server[1114]: 2026/09/23 04:46:07 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls6916worker1 # [ 198.354913] nix-grpc-daemon[1504]: time=2026-09-23T04:46:07.853Z level=INFO msg="Upload complete. (245ms)"6917lb # [ 198.398629] niks3-server[1114]: 2026/09/23 04:46:07 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst6918lb # [ 198.403200] niks3-server[1114]: 2026/09/23 04:46:07 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete6919lb # [ 198.407513] niks3-server[1114]: 2026/09/23 04:46:07 INFO Completed upload id=776920lb # [ 198.410900] niks3-server[1114]: 2026/09/23 04:46:07 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo6921worker1 # [ 198.369324] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=14 outputs=16922worker1 # [ 198.384652] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:07Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=16923worker1 # [ 198.391450] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966924worker1 # [ 198.396754] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:07Z level=debug event=want client=6 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=06925worker1 # [ 198.401982] nix-daemon[834]: reaped child process 1879, status = succeeded6926worker1 # [ 198.405464] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608966927worker1 # [ 198.410245] nix-grpc-daemon[1504]: time=2026-09-23T04:46:07.908Z level=INFO msg="All 1 paths already cached"6928worker1 # [ 198.413785] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608966929worker1 # [ 198.427980] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6930worker1 # [ 198.548115] nix-daemon[834]: accepted connection from pid 1906, user nix-grpc-daemon (trusted)6931worker1 # [ 198.564302] nix-daemon[834]: reaped child process 1909, status = succeeded6932lb # [ 198.613940] niks3-server[1114]: 2026/09/23 04:46:08 INFO Received uploads request method=POST path=/api/pending_closures6933worker1 # [ 198.578682] nix-grpc-daemon[1504]: time=2026-09-23T04:46:08.077Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6934worker1 # [ 198.580183] nix-grpc-daemon[1504]: time=2026-09-23T04:46:08.077Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)"6935worker1 # [ 198.598196] nix-grpc-daemon[1504]: time=2026-09-23T04:46:08.096Z level=INFO msg="Uploading 1 narinfos"6936lb # [ 198.642484] niks3-server[1114]: 2026/09/23 04:46:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign6937lb # [ 198.655477] niks3-server[1114]: 2026/09/23 04:46:08 INFO Signed narinfos id=78 count=16938lb # [ 198.664642] niks3-server[1114]: 2026/09/23 04:46:08 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls6939lb # [ 198.673292] niks3-server[1114]: 2026/09/23 04:46:08 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst6940worker1 # [ 198.643534] nix-grpc-daemon[1504]: time=2026-09-23T04:46:08.142Z level=INFO msg="Upload complete. (210ms)"6941lb # [ 198.687978] niks3-server[1114]: 2026/09/23 04:46:08 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete6942lb # [ 198.694396] niks3-server[1114]: 2026/09/23 04:46:08 INFO Completed upload id=786943lb # [ 198.697789] niks3-server[1114]: 2026/09/23 04:46:08 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo6944worker1 # [ 198.652729] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 paths=1 nar_bytes_in=6246945worker1 # [ 198.666502] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966946worker1 # [ 198.672594] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6947worker1 # [ 198.677226] nix-daemon[834]: reaped child process 1904, status = succeeded6948worker1 # [ 198.688769] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6949lb # [ 199.427401] envoy[900]: conn peer=[2001:db8:1::1]:42710 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9723 tx=5202 ms=53246950worker1 # [ 200.160423] nix-daemon[834]: reaped child process 1913, status = succeeded6951worker1 # [ 200.267711] nix-daemon[834]: accepted connection from pid 1929, user nix-grpc-daemon (trusted)6952worker1 # [ 200.286460] nix-daemon[834]: reaped child process 1932, status = succeeded6953lb # [ 200.336655] niks3-server[1114]: 2026/09/23 04:46:09 INFO Received uploads request method=POST path=/api/pending_closures6954worker1 # [ 200.302845] nix-grpc-daemon[1504]: time=2026-09-23T04:46:09.800Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6955worker1 # [ 200.304726] nix-grpc-daemon[1504]: time=2026-09-23T04:46:09.800Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)"6956lb # [ 200.371782] niks3-server[1114]: 2026/09/23 04:46:09 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv6957lb # [ 200.386303] niks3-server[1114]: 2026/09/23 04:46:09 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls6958worker1 # [ 200.350534] nix-grpc-daemon[1504]: time=2026-09-23T04:46:09.848Z level=INFO msg="Uploading 1 narinfos"6959lb # [ 200.399116] niks3-server[1114]: 2026/09/23 04:46:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign6960lb # [ 200.408232] niks3-server[1114]: 2026/09/23 04:46:09 INFO Signed narinfos id=79 count=16961lb # [ 200.410946] niks3-server[1114]: 2026/09/23 04:46:09 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst6962lb # [ 200.425565] niks3-server[1114]: 2026/09/23 04:46:09 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete6963worker1 # [ 200.381360] nix-grpc-daemon[1504]: time=2026-09-23T04:46:09.880Z level=INFO msg="Upload complete. (282ms)"6964lb # [ 200.429225] niks3-server[1114]: 2026/09/23 04:46:09 INFO Completed upload id=796965lb # [ 200.432658] niks3-server[1114]: 2026/09/23 04:46:09 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo6966worker1 # [ 200.404270] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=15 outputs=16967worker1 # [ 200.416469] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:09Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=16968worker1 # [ 200.425787] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966969worker1 # [ 200.430398] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:09Z level=debug event=want client=6 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=06970worker1 # [ 200.434960] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608966971worker1 # [ 200.439552] nix-daemon[834]: reaped child process 1911, status = succeeded6972worker1 # [ 200.441462] nix-grpc-daemon[1504]: time=2026-09-23T04:46:09.937Z level=INFO msg="All 1 paths already cached"6973worker1 # [ 200.444714] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608966974worker1 # [ 200.454669] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6975worker1 # [ 200.586463] nix-daemon[834]: accepted connection from pid 1938, user nix-grpc-daemon (trusted)6976worker1 # [ 200.604650] nix-daemon[834]: reaped child process 1941, status = succeeded6977lb # [ 200.656045] niks3-server[1114]: 2026/09/23 04:46:10 INFO Received uploads request method=POST path=/api/pending_closures6978worker1 # [ 200.621380] nix-grpc-daemon[1504]: time=2026-09-23T04:46:10.119Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6979worker1 # [ 200.622843] nix-grpc-daemon[1504]: time=2026-09-23T04:46:10.119Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)"6980worker1 # [ 200.641855] nix-grpc-daemon[1504]: time=2026-09-23T04:46:10.140Z level=INFO msg="Uploading 1 narinfos"6981lb # [ 200.685796] niks3-server[1114]: 2026/09/23 04:46:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/80/sign6982lb # [ 200.697709] niks3-server[1114]: 2026/09/23 04:46:10 INFO Signed narinfos id=80 count=16983lb # [ 200.707397] niks3-server[1114]: 2026/09/23 04:46:10 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls6984lb # [ 200.720475] niks3-server[1114]: 2026/09/23 04:46:10 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst6985worker1 # [ 200.684919] nix-grpc-daemon[1504]: time=2026-09-23T04:46:10.183Z level=INFO msg="Upload complete. (220ms)"6986lb # [ 200.728910] niks3-server[1114]: 2026/09/23 04:46:10 INFO Received complete upload request method=POST path=/api/pending_closures/80/complete6987lb # [ 200.735114] niks3-server[1114]: 2026/09/23 04:46:10 INFO Completed upload id=806988worker1 # [ 200.695570] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 paths=1 nar_bytes_in=6246989lb # [ 200.739952] niks3-server[1114]: 2026/09/23 04:46:10 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo6990worker1 # [ 200.710019] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608966991worker1 # [ 200.716186] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6992worker1 # [ 200.720760] nix-daemon[834]: reaped child process 1936, status = succeeded6993worker1 # [ 200.732818] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)6994worker1 # [ 202.212313] nix-daemon[834]: reaped child process 1945, status = succeeded6995worker1 # [ 202.309484] nix-daemon[834]: accepted connection from pid 1961, user nix-grpc-daemon (trusted)6996worker1 # [ 202.328232] nix-daemon[834]: reaped child process 1964, status = succeeded6997lb # [ 202.376907] niks3-server[1114]: 2026/09/23 04:46:11 INFO Received uploads request method=POST path=/api/pending_closures6998worker1 # [ 202.344451] nix-grpc-daemon[1504]: time=2026-09-23T04:46:11.841Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"6999worker1 # [ 202.346184] nix-grpc-daemon[1504]: time=2026-09-23T04:46:11.841Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)"7000lb # [ 202.423582] niks3-server[1114]: 2026/09/23 04:46:11 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv7001worker1 # [ 202.390521] nix-grpc-daemon[1504]: time=2026-09-23T04:46:11.888Z level=INFO msg="Uploading 1 narinfos"7002lb # [ 202.436747] niks3-server[1114]: 2026/09/23 04:46:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/81/sign7003lb # [ 202.450780] niks3-server[1114]: 2026/09/23 04:46:11 INFO Signed narinfos id=81 count=17004lb # [ 202.457005] niks3-server[1114]: 2026/09/23 04:46:11 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls7005lb # [ 202.466333] niks3-server[1114]: 2026/09/23 04:46:11 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst7006worker1 # [ 202.432596] nix-grpc-daemon[1504]: time=2026-09-23T04:46:11.931Z level=INFO msg="Upload complete. (290ms)"7007lb # [ 202.477502] niks3-server[1114]: 2026/09/23 04:46:11 INFO Received complete upload request method=POST path=/api/pending_closures/81/complete7008lb # [ 202.483758] niks3-server[1114]: 2026/09/23 04:46:11 INFO Completed upload id=817009lb # [ 202.488424] niks3-server[1114]: 2026/09/23 04:46:11 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo7010worker1 # [ 202.447353] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=16 outputs=17011worker1 # [ 202.472401] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:11Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=17012worker1 # [ 202.478493] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608967013worker1 # [ 202.483704] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:11Z level=debug event=want client=6 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=07014worker1 # [ 202.490217] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608967015worker1 # [ 202.496327] nix-daemon[834]: reaped child process 1943, status = succeeded7016worker1 # [ 202.500595] nix-grpc-daemon[1504]: time=2026-09-23T04:46:11.997Z level=INFO msg="All 2 paths already cached"7017worker1 # [ 202.504738] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608967018worker1 # [ 202.520320] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)7019worker1 # [ 202.653445] nix-daemon[834]: accepted connection from pid 1970, user nix-grpc-daemon (trusted)7020worker1 # [ 202.669635] nix-daemon[834]: reaped child process 1973, status = succeeded7021lb # [ 202.719834] niks3-server[1114]: 2026/09/23 04:46:12 INFO Received uploads request method=POST path=/api/pending_closures7022worker1 # [ 202.684824] nix-grpc-daemon[1504]: time=2026-09-23T04:46:12.183Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7023worker1 # [ 202.686274] nix-grpc-daemon[1504]: time=2026-09-23T04:46:12.183Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)"7024worker1 # [ 202.705331] nix-grpc-daemon[1504]: time=2026-09-23T04:46:12.203Z level=INFO msg="Uploading 1 narinfos"7025lb # [ 202.749144] niks3-server[1114]: 2026/09/23 04:46:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/82/sign7026lb # [ 202.762810] niks3-server[1114]: 2026/09/23 04:46:12 INFO Signed narinfos id=82 count=17027lb # [ 202.770299] niks3-server[1114]: 2026/09/23 04:46:12 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst7028worker1 # [ 202.740993] nix-grpc-daemon[1504]: time=2026-09-23T04:46:12.239Z level=INFO msg="Upload complete. (214ms)"7029lb # [ 202.785292] niks3-server[1114]: 2026/09/23 04:46:12 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls7030lb # [ 202.791974] niks3-server[1114]: 2026/09/23 04:46:12 INFO Received complete upload request method=POST path=/api/pending_closures/82/complete7031worker1 # [ 202.748835] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:12Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 paths=1 nar_bytes_in=7527032lb # [ 202.796003] niks3-server[1114]: 2026/09/23 04:46:12 INFO Completed upload id=827033lb # [ 202.800155] niks3-server[1114]: 2026/09/23 04:46:12 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo7034worker1 # [ 202.761308] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608967035worker1 # [ 202.766754] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)7036worker1 # [ 202.770873] nix-daemon[834]: reaped child process 1968, status = succeeded7037worker1 # [ 202.780148] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)7038worker1 # [ 203.656500] nix-daemon[834]: reaped child process 1977, status = succeeded7039worker1 # [ 203.763620] nix-daemon[834]: accepted connection from pid 1993, user nix-grpc-daemon (trusted)7040worker1 # [ 203.780888] nix-daemon[834]: reaped child process 1996, status = succeeded7041lb # [ 203.831474] niks3-server[1114]: 2026/09/23 04:46:13 INFO Received uploads request method=POST path=/api/pending_closures7042worker1 # [ 203.803083] nix-grpc-daemon[1504]: time=2026-09-23T04:46:13.300Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7043worker1 # [ 203.805174] nix-grpc-daemon[1504]: time=2026-09-23T04:46:13.300Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)"7044lb # [ 203.878531] niks3-server[1114]: 2026/09/23 04:46:13 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv7045worker1 # [ 203.845782] nix-grpc-daemon[1504]: time=2026-09-23T04:46:13.344Z level=INFO msg="Uploading 1 narinfos"7046lb # [ 203.892356] niks3-server[1114]: 2026/09/23 04:46:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/83/sign7047lb # [ 203.895533] niks3-server[1114]: 2026/09/23 04:46:13 INFO Signed narinfos id=83 count=17048lb # [ 203.904316] niks3-server[1114]: 2026/09/23 04:46:13 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst7049lb # [ 203.911002] niks3-server[1114]: 2026/09/23 04:46:13 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls7050lb # [ 203.928855] niks3-server[1114]: 2026/09/23 04:46:13 INFO Received complete upload request method=POST path=/api/pending_closures/83/complete7051worker1 # [ 203.888598] nix-grpc-daemon[1504]: time=2026-09-23T04:46:13.387Z level=INFO msg="Upload complete. (298ms)"7052lb # [ 203.935068] niks3-server[1114]: 2026/09/23 04:46:13 INFO Completed upload id=837053lb # [ 203.938407] niks3-server[1114]: 2026/09/23 04:46:13 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo7054worker1 # [ 203.905198] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:13Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=17 outputs=17055worker1 # [ 203.921393] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:13Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=17056worker1 # [ 203.926726] nix-daemon[834]: reaped child process 1975, status = succeeded7057worker1 # [ 203.944174] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608967058worker1 # [ 203.957405] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608967059worker1 # [ 203.969554] nix-grpc-daemon[1504]: time=2026-09-23T04:46:13.468Z level=INFO msg="All 2 paths already cached"7060worker1 # [ 203.978803] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:13Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608967061worker1 # [ 203.992980] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)7062worker1 # [ 204.138707] nix-daemon[834]: accepted connection from pid 2000, user nix-grpc-daemon (trusted)7063worker1 # [ 204.156265] nix-daemon[834]: reaped child process 2003, status = succeeded7064lb # [ 204.205196] niks3-server[1114]: 2026/09/23 04:46:13 INFO Received uploads request method=POST path=/api/pending_closures7065worker1 # [ 204.169962] nix-grpc-daemon[1504]: time=2026-09-23T04:46:13.668Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7066worker1 # [ 204.171623] nix-grpc-daemon[1504]: time=2026-09-23T04:46:13.668Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)"7067lb # [ 204.236835] niks3-server[1114]: 2026/09/23 04:46:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/84/sign7068worker1 # [ 204.198873] nix-grpc-daemon[1504]: time=2026-09-23T04:46:13.697Z level=INFO msg="Uploading 1 narinfos"7069lb # [ 204.250948] niks3-server[1114]: 2026/09/23 04:46:13 INFO Signed narinfos id=84 count=17070lb # [ 204.258657] niks3-server[1114]: 2026/09/23 04:46:13 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls7071lb # [ 204.268957] niks3-server[1114]: 2026/09/23 04:46:13 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst7072lb # [ 204.277748] niks3-server[1114]: 2026/09/23 04:46:13 INFO Received complete upload request method=POST path=/api/pending_closures/84/complete7073worker1 # [ 204.230489] nix-grpc-daemon[1504]: time=2026-09-23T04:46:13.729Z level=INFO msg="Upload complete. (221ms)"7074lb # [ 204.282783] niks3-server[1114]: 2026/09/23 04:46:13 INFO Completed upload id=847075worker1 # [ 204.240495] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 paths=1 nar_bytes_in=7527076lb # [ 204.287400] niks3-server[1114]: 2026/09/23 04:46:13 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo7077worker1 # [ 204.254378] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608967078worker1 # [ 204.260747] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)7079worker1 # [ 204.266183] nix-daemon[834]: reaped child process 1998, status = succeeded7080worker1 # [ 204.279212] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)7081worker1 # [ 205.752548] nix-daemon[834]: reaped child process 2007, status = succeeded7082worker1 # [ 205.856329] nix-daemon[834]: accepted connection from pid 2023, user nix-grpc-daemon (trusted)7083worker1 # [ 205.874273] nix-daemon[834]: reaped child process 2026, status = succeeded7084lb # [ 205.929363] niks3-server[1114]: 2026/09/23 04:46:15 INFO Received uploads request method=POST path=/api/pending_closures7085worker1 # [ 205.896507] nix-grpc-daemon[1504]: time=2026-09-23T04:46:15.391Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7086worker1 # [ 205.898423] nix-grpc-daemon[1504]: time=2026-09-23T04:46:15.391Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)"7087lb # [ 205.972050] niks3-server[1114]: 2026/09/23 04:46:15 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls7088worker1 # [ 205.939266] nix-grpc-daemon[1504]: time=2026-09-23T04:46:15.437Z level=INFO msg="Uploading 1 narinfos"7089lb # [ 205.985228] niks3-server[1114]: 2026/09/23 04:46:15 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv7090lb # [ 205.994734] niks3-server[1114]: 2026/09/23 04:46:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/85/sign7091lb # [ 206.005507] niks3-server[1114]: 2026/09/23 04:46:15 INFO Signed narinfos id=85 count=17092worker1 # [ 205.961164] nix-grpc-daemon[1504]: time=2026-09-23T04:46:15.459Z level=INFO msg="Upload complete. (271ms)"7093lb # [ 206.012408] niks3-server[1114]: 2026/09/23 04:46:15 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst7094lb # [ 206.019812] niks3-server[1114]: 2026/09/23 04:46:15 INFO Received complete upload request method=POST path=/api/pending_closures/85/complete7095worker1 # [ 205.973131] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:15Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=18 outputs=17096lb # [ 206.024680] niks3-server[1114]: 2026/09/23 04:46:15 INFO Completed upload id=857097lb # [ 206.027354] niks3-server[1114]: 2026/09/23 04:46:15 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo7098worker1 # [ 205.992111] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:15Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=17099worker1 # [ 205.997148] nix-daemon[834]: reaped child process 2005, status = succeeded7100worker1 # [ 206.016607] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608967101worker1 # [ 206.031443] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:15Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608967102worker1 # [ 206.045495] nix-grpc-daemon[1504]: time=2026-09-23T04:46:15.544Z level=INFO msg="All 2 paths already cached"7103worker1 # [ 206.055216] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:15Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:608967104worker1 # [ 206.069379] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)7105worker1 # [ 206.214739] nix-daemon[834]: accepted connection from pid 2030, user nix-grpc-daemon (trusted)7106worker1 # [ 206.234029] nix-daemon[834]: reaped child process 2033, status = succeeded7107lb # [ 206.283130] niks3-server[1114]: 2026/09/23 04:46:15 INFO Received uploads request method=POST path=/api/pending_closures7108worker1 # [ 206.248113] nix-grpc-daemon[1504]: time=2026-09-23T04:46:15.746Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7109worker1 # [ 206.249656] nix-grpc-daemon[1504]: time=2026-09-23T04:46:15.746Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)"7110lb # [ 206.314675] niks3-server[1114]: 2026/09/23 04:46:15 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst7111worker1 # [ 206.281630] nix-grpc-daemon[1504]: time=2026-09-23T04:46:15.780Z level=INFO msg="Uploading 1 narinfos"7112lb # [ 206.330201] niks3-server[1114]: 2026/09/23 04:46:15 INFO Received sign narinfos request method=POST path=/api/pending_closures/86/sign7113lb # [ 206.341140] niks3-server[1114]: 2026/09/23 04:46:15 INFO Signed narinfos id=86 count=17114worker1 # [ 206.304823] nix-grpc-daemon[1504]: time=2026-09-23T04:46:15.803Z level=INFO msg="Upload complete. (222ms)"7115lb # [ 206.349442] niks3-server[1114]: 2026/09/23 04:46:15 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls7116worker1 # [ 206.314100] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=0 paths=1 nar_bytes_in=7527117lb # [ 206.359018] niks3-server[1114]: 2026/09/23 04:46:15 INFO Received complete upload request method=POST path=/api/pending_closures/86/complete7118lb # [ 206.364396] niks3-server[1114]: 2026/09/23 04:46:15 INFO Completed upload id=867119lb # [ 206.368064] niks3-server[1114]: 2026/09/23 04:46:15 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo7120worker1 # [ 206.330016] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:608967121worker1 # [ 206.335532] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)7122worker1 # [ 206.339866] nix-daemon[834]: reaped child process 2028, status = succeeded7123worker1 # [ 206.349620] nix-daemon[834]: accepted connection from pid 1407, user nix-grpc-daemon (trusted)7124worker1 # [ 207.594127] nix-daemon[834]: reaped child process 2037, status = succeeded7125worker1 # [ 207.697408] nix-daemon[834]: accepted connection from pid 2053, user nix-grpc-daemon (trusted)7126worker1 # [ 207.717793] nix-daemon[834]: reaped child process 2056, status = succeeded7127lb # [ 207.760795] niks3-server[1114]: 2026/09/23 04:46:17 INFO Received uploads request method=POST path=/api/pending_closures7128worker1 # [ 207.728345] nix-grpc-daemon[1504]: time=2026-09-23T04:46:17.225Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7129worker1 # [ 207.729809] nix-grpc-daemon[1504]: time=2026-09-23T04:46:17.225Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)"7130lb # [ 207.802631] niks3-server[1114]: 2026/09/23 04:46:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/87/sign7131worker1 # [ 207.767405] nix-grpc-daemon[1504]: time=2026-09-23T04:46:17.265Z level=INFO msg="Uploading 1 narinfos"7132lb # [ 207.817420] niks3-server[1114]: 2026/09/23 04:46:17 INFO Signed narinfos id=87 count=17133lb # [ 207.823246] niks3-server[1114]: 2026/09/23 04:46:17 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv7134worker1 # [ 207.793227] nix-grpc-daemon[1504]: time=2026-09-23T04:46:17.291Z level=INFO msg="Upload complete. (233ms)"7135lb # [ 207.837366] niks3-server[1114]: 2026/09/23 04:46:17 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls7136lb # [ 207.846627] niks3-server[1114]: 2026/09/23 04:46:17 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst7137worker1 # [ 207.803916] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:17Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60896 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=19 outputs=17138lb # [ 207.851788] niks3-server[1114]: 2026/09/23 04:46:17 INFO Received complete upload request method=POST path=/api/pending_closures/87/complete7139lb # [ 207.856606] niks3-server[1114]: 2026/09/23 04:46:17 INFO Completed upload id=877140lb # [ 207.858769] niks3-server[1114]: 2026/09/23 04:46:17 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo7141worker1 # [ 207.820465] nix-daemon[834]: reaped child process 2035, status = succeeded7142client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 16.94 seconds)7143worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7144worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds)7145client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]'7146client # [ 208.024785] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features "[\"gpu\"]".7147client # Running as unit: gpu.service; invocation ID: 93fe9b85190b4360bdc35b6d7a2b4bf97148client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds)7149lb: must succeed: curl -sf localhost:9901/clusters7150lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)7151lb: must succeed: curl -sf localhost:9901/clusters7152lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7153worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7154client # [ 208.219282] nix[1620]: don't know how to build these paths:7155client # [ 208.220379] nix[1620]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv7156worker2 # [ 208.297233] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:17Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:446707157worker1 # [ 208.201795] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608967158worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7159worker1 # [ 208.227659] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:17Z level=debug event=want client=7 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=07160client # [ 208.258041] nix[1620]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip}, waiting up to 30s for a worker7161lb: must succeed: curl -sf localhost:9901/clusters7162lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)7163lb: must succeed: curl -sf localhost:9901/clusters7164lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7165worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7166lb # [ 209.428495] envoy[900]: conn peer=[2001:db8:1::1]:36024 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27170 tx=12416 ms=166707167worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7168client: waiting for success: systemctl show -p Result gpu | grep -q exit-code7169client # [ 238.265708] nix[1620]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip}7170client # [ 238.280136] nix[1620]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed7171client # [ 238.318275] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE7172client # [ 238.326468] systemd[1]: gpu.service: Failed with result 'exit-code'.7173client # [ 238.330227] systemd[1]: gpu.service: Consumed 188ms CPU time over 30.299s wall clock time, 37.6M memory peak, 5.6K incoming IP traffic, 5.2K outgoing IP traffic.7174client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.07 seconds)7175client: must succeed: journalctl -u gpu -o cat7176client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds)7177lb: must succeed: curl -sf localhost:9901/clusters7178lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds)7179lb: must succeed: curl -sf localhost:9901/clusters7180lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds)7181worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics7182worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds)7183(finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 47.75 seconds)7184subtest: low disk drains a worker and builds go to the other7185worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill7186lb # [ 239.434207] envoy[900]: conn peer=[2001:db8:1::1]:56770 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=931 tx=613 ms=300727187worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.19 seconds)7188worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free'7189worker1 # [ 240.435176] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:49Z level=info event=unhealthy reason=min_free7190worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.10 seconds)7191client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&27192client # don't know how to build these paths:7193client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv7194worker1 # [ 241.194392] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:50Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:608967195worker1 # [ 241.202486] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:608967196worker1 # [ 241.229070] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:50Z level=debug event=want client=8 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=07197worker1 # [ 241.235117] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:50Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=17198worker2 # [ 241.381878] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:446707199worker2 # [ 241.398551] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:446707200client # copying 1 paths to 'lb:50051'...7201client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'...7202worker2 # [ 241.415167] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:446707203worker2 # [ 241.430352] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)7204worker2 # [ 241.594344] nix-daemon[856]: accepted connection from pid 2212, user nix-grpc-daemon (trusted)7205worker2 # [ 241.616925] nix-daemon[856]: reaped child process 2215, status = succeeded7206lb # [ 241.561877] niks3-server[1114]: 2026/09/23 04:46:51 INFO Received uploads request method=POST path=/api/pending_closures7207worker2 # [ 241.633709] nix-grpc-daemon[2117]: time=2026-09-23T04:46:51.154Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7208worker2 # [ 241.635761] nix-grpc-daemon[2117]: time=2026-09-23T04:46:51.154Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)"7209lb # [ 241.598177] niks3-server[1114]: 2026/09/23 04:46:51 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst7210lb # [ 241.620041] niks3-server[1114]: 2026/09/23 04:46:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/88/sign7211worker2 # [ 241.686872] nix-grpc-daemon[2117]: time=2026-09-23T04:46:51.208Z level=INFO msg="Uploading 1 narinfos"7212lb # [ 241.632419] niks3-server[1114]: 2026/09/23 04:46:51 INFO Signed narinfos id=88 count=17213lb # [ 241.638950] niks3-server[1114]: 2026/09/23 04:46:51 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls7214lb # [ 241.662368] niks3-server[1114]: 2026/09/23 04:46:51 INFO Received complete upload request method=POST path=/api/pending_closures/88/complete7215lb # [ 241.669287] niks3-server[1114]: 2026/09/23 04:46:51 INFO Completed upload id=887216worker2 # [ 241.728657] nix-grpc-daemon[2117]: time=2026-09-23T04:46:51.250Z level=INFO msg="Upload complete. (280ms)"7217lb # [ 241.672349] niks3-server[1114]: 2026/09/23 04:46:51 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo7218worker2 # [ 241.739561] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44670 duration_s=0 paths=1 nar_bytes_in=5127219worker2 # [ 241.756626] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:446707220worker2 # [ 241.767073] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)7221worker2 # [ 241.779776] nix-daemon[856]: reaped child process 2210, status = succeeded7222client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051...7223client # farm-a> worker2-next: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv7224worker2 # [ 241.788642] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)7225client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds7226client # farm-a> LOG-farm-a-drain7227worker2 # [ 242.860984] nix-daemon[856]: reaped child process 2219, status = succeeded7228worker2 # [ 242.972473] nix-daemon[856]: accepted connection from pid 2235, user nix-grpc-daemon (trusted)7229worker2 # [ 242.990313] nix-daemon[856]: reaped child process 2238, status = succeeded7230lb # [ 242.936850] niks3-server[1114]: 2026/09/23 04:46:52 INFO Received uploads request method=POST path=/api/pending_closures7231worker2 # [ 243.008486] nix-grpc-daemon[2117]: time=2026-09-23T04:46:52.528Z level=INFO msg="Uploading 1 paths to lb (0 already cached)"7232worker2 # [ 243.011097] nix-grpc-daemon[2117]: time=2026-09-23T04:46:52.528Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)"7233lb # [ 242.984398] niks3-server[1114]: 2026/09/23 04:46:52 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst7234lb # [ 242.998008] niks3-server[1114]: 2026/09/23 04:46:52 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv7235worker2 # [ 243.057727] nix-grpc-daemon[2117]: time=2026-09-23T04:46:52.579Z level=INFO msg="Uploading 1 narinfos"7236lb # [ 243.009054] niks3-server[1114]: 2026/09/23 04:46:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/89/sign7237lb # [ 243.015693] niks3-server[1114]: 2026/09/23 04:46:52 INFO Signed narinfos id=89 count=17238lb # [ 243.023563] niks3-server[1114]: 2026/09/23 04:46:52 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls7239lb # [ 243.028482] niks3-server[1114]: 2026/09/23 04:46:52 INFO Received complete upload request method=POST path=/api/pending_closures/89/complete7240lb # [ 243.034881] niks3-server[1114]: 2026/09/23 04:46:52 INFO Completed upload id=897241worker2 # [ 243.094411] nix-grpc-daemon[2117]: time=2026-09-23T04:46:52.616Z level=INFO msg="Upload complete. (294ms)"7242lb # [ 243.039347] niks3-server[1114]: 2026/09/23 04:46:52 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo7243worker1 # [ 243.010374] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:52Z level=debug event=want client=8 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=07244worker2 # [ 243.109143] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44670 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=20 outputs=17245worker1 # [ 243.016524] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:52Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=17246worker2 # [ 243.127767] nix-daemon[856]: reaped child process 2217, status = succeeded7247worker2 # [ 243.159911] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:446707248worker2 # [ 243.175690] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:446707249worker2 # [ 243.189448] nix-grpc-daemon[2117]: time=2026-09-23T04:46:52.711Z level=INFO msg="All 1 paths already cached"7250client # copying 1 paths to 'lb:50051'...7251client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'...7252worker2 # [ 243.200136] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:446707253worker2 # [ 243.214466] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)7254worker2 # [ 243.367487] nix-daemon[856]: accepted connection from pid 2242, user nix-grpc-daemon (trusted)7255worker2 # [ 243.388139] nix-daemon[856]: reaped child process 2245, status = succeeded7256lb # [ 243.331486] niks3-server[1114]: 2026/09/23 04:46:52 INFO Received uploads request method=POST path=/api/pending_closures7257worker2 # [ 243.401769] nix-grpc-daemon[2117]: time=2026-09-23T04:46:52.923Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7258worker2 # [ 243.403287] nix-grpc-daemon[2117]: time=2026-09-23T04:46:52.923Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)"7259lb # [ 243.363228] niks3-server[1114]: 2026/09/23 04:46:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/90/sign7260worker2 # [ 243.428974] nix-grpc-daemon[2117]: time=2026-09-23T04:46:52.950Z level=INFO msg="Uploading 1 narinfos"7261lb # [ 243.376814] niks3-server[1114]: 2026/09/23 04:46:52 INFO Signed narinfos id=90 count=17262lb # [ 243.379491] niks3-server[1114]: 2026/09/23 04:46:52 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst7263lb # [ 243.397761] niks3-server[1114]: 2026/09/23 04:46:52 INFO Received complete upload request method=POST path=/api/pending_closures/90/complete7264lb # [ 243.405317] niks3-server[1114]: 2026/09/23 04:46:52 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls7265worker2 # [ 243.464158] nix-grpc-daemon[2117]: time=2026-09-23T04:46:52.986Z level=INFO msg="Upload complete. (234ms)"7266lb # [ 243.408702] niks3-server[1114]: 2026/09/23 04:46:52 INFO Completed upload id=907267lb # [ 243.411447] niks3-server[1114]: 2026/09/23 04:46:52 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo7268worker2 # [ 243.475979] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44670 duration_s=0 paths=1 nar_bytes_in=6327269worker2 # [ 243.489274] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:446707270worker2 # [ 243.497802] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)7271worker2 # [ 243.500972] nix-daemon[856]: reaped child process 2240, status = succeeded7272client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051...7273client # farm-b> worker2-next: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv7274worker2 # [ 243.515449] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)7275client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds7276client # farm-b> LOG-farm-b-drain7277worker2 # [ 244.954111] nix-daemon[856]: reaped child process 2249, status = succeeded7278worker2 # [ 245.054146] nix-daemon[856]: accepted connection from pid 2265, user nix-grpc-daemon (trusted)7279worker2 # [ 245.071148] nix-daemon[856]: reaped child process 2268, status = succeeded7280lb # [ 245.019650] niks3-server[1114]: 2026/09/23 04:46:54 INFO Received uploads request method=POST path=/api/pending_closures7281worker2 # [ 245.088397] nix-grpc-daemon[2117]: time=2026-09-23T04:46:54.610Z level=INFO msg="Uploading 1 paths to lb (1 already cached)"7282worker2 # [ 245.089857] nix-grpc-daemon[2117]: time=2026-09-23T04:46:54.610Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)"7283lb # [ 245.059433] niks3-server[1114]: 2026/09/23 04:46:54 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst7284worker2 # [ 245.133718] nix-grpc-daemon[2117]: time=2026-09-23T04:46:54.655Z level=INFO msg="Uploading 1 narinfos"7285lb # [ 245.074042] niks3-server[1114]: 2026/09/23 04:46:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/91/sign7286lb # [ 245.088234] niks3-server[1114]: 2026/09/23 04:46:54 INFO Signed narinfos id=91 count=17287lb # [ 245.091154] niks3-server[1114]: 2026/09/23 04:46:54 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls7288lb # [ 245.107268] niks3-server[1114]: 2026/09/23 04:46:54 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv7289worker2 # [ 245.171466] nix-grpc-daemon[2117]: time=2026-09-23T04:46:54.693Z level=INFO msg="Upload complete. (244ms)"7290lb # [ 245.114421] niks3-server[1114]: 2026/09/23 04:46:54 INFO Received complete upload request method=POST path=/api/pending_closures/91/complete7291lb # [ 245.117955] niks3-server[1114]: 2026/09/23 04:46:54 INFO Completed upload id=917292lb # [ 245.121421] niks3-server[1114]: 2026/09/23 04:46:54 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo7293worker2 # [ 245.185135] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44670 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=21 outputs=17294worker1 # [ 245.089279] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:54Z level=debug event=want client=8 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=07295worker1 # [ 245.095678] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:54Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=17296worker2 # [ 245.203511] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:54Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:446707297worker2 # [ 245.209015] nix-daemon[856]: reaped child process 2247, status = succeeded7298worker2 # [ 245.220705] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:446707299worker2 # [ 245.234136] nix-grpc-daemon[2117]: time=2026-09-23T04:46:54.756Z level=INFO msg="All 2 paths already cached"7300client # copying 1 paths to 'lb:50051'...7301client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'...7302worker2 # [ 245.245726] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:54Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:446707303worker2 # [ 245.260382] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)7304worker2 # [ 245.396321] nix-daemon[856]: accepted connection from pid 2272, user nix-grpc-daemon (trusted)7305worker2 # [ 245.413189] nix-daemon[856]: reaped child process 2275, status = succeeded7306lb # [ 245.358943] niks3-server[1114]: 2026/09/23 04:46:54 INFO Received uploads request method=POST path=/api/pending_closures7307worker2 # [ 245.428098] nix-grpc-daemon[2117]: time=2026-09-23T04:46:54.949Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7308worker2 # [ 245.429581] nix-grpc-daemon[2117]: time=2026-09-23T04:46:54.949Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)"7309lb # [ 245.387814] niks3-server[1114]: 2026/09/23 04:46:54 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst7310worker2 # [ 245.461815] nix-grpc-daemon[2117]: time=2026-09-23T04:46:54.983Z level=INFO msg="Uploading 1 narinfos"7311lb # [ 245.406896] niks3-server[1114]: 2026/09/23 04:46:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/92/sign7312lb # [ 245.416441] niks3-server[1114]: 2026/09/23 04:46:54 INFO Signed narinfos id=92 count=17313lb # [ 245.422259] niks3-server[1114]: 2026/09/23 04:46:54 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls7314worker2 # [ 245.492364] nix-grpc-daemon[2117]: time=2026-09-23T04:46:55.014Z level=INFO msg="Upload complete. (221ms)"7315lb # [ 245.434890] niks3-server[1114]: 2026/09/23 04:46:54 INFO Received complete upload request method=POST path=/api/pending_closures/92/complete7316lb # [ 245.442223] niks3-server[1114]: 2026/09/23 04:46:54 INFO Completed upload id=927317lb # [ 245.446894] niks3-server[1114]: 2026/09/23 04:46:54 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo7318worker2 # [ 245.502078] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:44670 duration_s=0 paths=1 nar_bytes_in=7607319worker2 # [ 245.513056] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:446707320worker2 # [ 245.525532] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)7321worker2 # [ 245.530321] nix-daemon[856]: reaped child process 2270, status = succeeded7322client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051...7323worker2 # [ 245.543105] nix-daemon[856]: accepted connection from pid 2056, user nix-grpc-daemon (trusted)7324client # farm-top> worker2-next: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv7325client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds7326client # farm-top> LOG-farm-top-drain7327worker2 # [ 246.596007] nix-daemon[856]: reaped child process 2279, status = succeeded7328worker2 # [ 246.708865] nix-daemon[856]: accepted connection from pid 2295, user nix-grpc-daemon (trusted)7329worker2 # [ 246.725993] nix-daemon[856]: reaped child process 2298, status = succeeded7330lb # [ 246.677722] niks3-server[1114]: 2026/09/23 04:46:56 INFO Received uploads request method=POST path=/api/pending_closures7331worker2 # [ 246.748458] nix-grpc-daemon[2117]: time=2026-09-23T04:46:56.267Z level=INFO msg="Uploading 1 paths to lb (2 already cached)"7332worker2 # [ 246.750171] nix-grpc-daemon[2117]: time=2026-09-23T04:46:56.267Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)"7333lb # [ 246.721014] niks3-server[1114]: 2026/09/23 04:46:56 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv7334lb # [ 246.733211] niks3-server[1114]: 2026/09/23 04:46:56 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst7335worker2 # [ 246.801862] nix-grpc-daemon[2117]: time=2026-09-23T04:46:56.323Z level=INFO msg="Uploading 1 narinfos"7336lb # [ 246.748980] niks3-server[1114]: 2026/09/23 04:46:56 INFO Received sign narinfos request method=POST path=/api/pending_closures/93/sign7337lb # [ 246.759208] niks3-server[1114]: 2026/09/23 04:46:56 INFO Signed narinfos id=93 count=17338lb # [ 246.765309] niks3-server[1114]: 2026/09/23 04:46:56 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls7339worker2 # [ 246.833077] nix-grpc-daemon[2117]: time=2026-09-23T04:46:56.355Z level=INFO msg="Upload complete. (280ms)"7340lb # [ 246.775557] niks3-server[1114]: 2026/09/23 04:46:56 INFO Received complete upload request method=POST path=/api/pending_closures/93/complete7341lb # [ 246.781497] niks3-server[1114]: 2026/09/23 04:46:56 INFO Completed upload id=937342lb # [ 246.785525] niks3-server[1114]: 2026/09/23 04:46:56 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo7343worker2 # [ 246.845266] nix-grpc-daemon[2056]: ts=2026-09-23T04:46:56Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:44670 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=22 outputs=17344worker2 # [ 246.866507] nix-daemon[856]: reaped child process 2277, status = succeeded7345client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2, in 5.82 seconds)7346client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath7347client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath, in 0.18 seconds)7348worker1: must succeed: rm /nix/.rw-store/fill7349worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.18 seconds)7350worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free'7351worker1 # [ 247.439849] nix-grpc-daemon[1407]: ts=2026-09-23T04:46:56Z level=info event=healthy reason=min_free7352worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.14 seconds)7353(finished: subtest: low disk drains a worker and builds go to the other, in 9.70 seconds)7354(finished: run the VM test script, in 249.41 seconds)7355test script finished in 249.79s7356cleanup7357kill QemuMachine (pid 49)7358client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)7359client # [2026-09-23T04:46:58Z INFO virtiofsd] Client disconnected, shutting down7360client # [2026-09-23T04:46:58Z INFO virtiofsd] Client disconnected, shutting down7361client # [2026-09-23T04:46:58Z INFO virtiofsd] Client disconnected, shutting down7362kill QemuMachine (pid 50)7363lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)7364lb # [2026-09-23T04:46:58Z INFO virtiofsd] Client disconnected, shutting down7365lb # [2026-09-23T04:46:58Z INFO virtiofsd] Client disconnected, shutting down7366lb # [2026-09-23T04:46:58Z INFO virtiofsd] Client disconnected, shutting down7367kill QemuMachine (pid 52)7368worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)7369worker1 # [2026-09-23T04:46:59Z INFO virtiofsd] Client disconnected, shutting down7370worker1 # [2026-09-23T04:46:59Z INFO virtiofsd] Client disconnected, shutting down7371worker1 # [2026-09-23T04:46:59Z INFO virtiofsd] Client disconnected, shutting down7372kill QemuMachine (pid 51)7373worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14)7374worker2 # [2026-09-23T04:46:59Z INFO virtiofsd] Client disconnected, shutting down7375worker2 # [2026-09-23T04:46:59Z INFO virtiofsd] Client disconnected, shutting down7376worker2 # [2026-09-23T04:46:59Z INFO virtiofsd] Client disconnected, shutting down7377(finished: cleanup, in 1.23 seconds)7378additionally exposed symbols:7379 client, lb, worker1, worker2,7380 vlan1,7381 start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh