tribuchet: building on eliza Machine state will be reset. To keep it, pass --keep-machine-state start all VLans (finished: start all VLans, in 0.00 seconds) Test will time out and terminate in 900.0 seconds run the VM test script start all VMs client: starting vm lb: starting vm worker1: starting vm worker2: starting vm client # Disk image does not exist, creating the virtualisation disk image... worker2 # Disk image does not exist, creating the virtualisation disk image... client: QEMU running (pid 51) client # Formatting '/build/vm-state-client/tmp.Zy9QhGmZxx', fmt=raw size=1073741824 worker2: QEMU running (pid 52) lb: QEMU running (pid 49) client # mke2fs 1.47.4 (6-Mar-2025) worker2 # Formatting '/build/vm-state-worker2/tmp.pXNuHuyN3Q', fmt=raw size=1073741824 client # Discarding device blocks: 0/262144 done worker2 # mke2fs 1.47.4 (6-Mar-2025) client # Creating filesystem with 262144 4k blocks and 65536 inodes worker2 # Discarding device blocks: 0/262144 done client # Filesystem UUID: a62f6ea9-c1fc-4c8f-81b0-c7a81bf4e5dd worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes client # Superblock backups stored on blocks: worker2 # Filesystem UUID: 66c3ca7b-51d7-4602-8e39-346aa8580ad6 client # 32768, 98304, 163840, 229376 worker2 # Superblock backups stored on blocks: client # worker2 # 32768, 98304, 163840, 229376 client # Allocating group tables: 0/8 done worker2 # client # Writing inode tables: 0/8 done worker2 # Allocating group tables: 0/8 done lb # Disk image does not exist, creating the virtualisation disk image... worker1: QEMU running (pid 50) worker1 # Disk image does not exist, creating the virtualisation disk image... worker1 # Formatting '/build/vm-state-worker1/tmp.PkMhTaEi8c', fmt=raw size=1073741824 worker2 # Writing inode tables: 0/8 done client # Creating journal (8192 blocks): done worker2 # Creating journal (8192 blocks): done (finished: start all VMs, in 0.55 seconds) lb # Formatting '/build/vm-state-lb/tmp.1HpU8q3NUG', fmt=raw size=1073741824 worker2 # Writing superblocks and filesystem accounting information: 0/8 done client: waiting for unit network-addresses-eth1.service lb # mke2fs 1.47.4 (6-Mar-2025) worker2 # client: waiting for the VM to finish booting lb # Discarding device blocks: 0/262144 done worker2 # Virtualisation disk image created. worker1 # mke2fs 1.47.4 (6-Mar-2025) client # Writing superblocks and filesystem accounting information: 0/8 done lb # Creating filesystem with 262144 4k blocks and 65536 inodes worker2 # Starting virtiofs daemons... worker1 # Discarding device blocks: 0/262144 done client # lb # Filesystem UUID: 1cd19ffa-09ce-40ed-92b0-dba9e25172a2 worker2 # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes client # Virtualisation disk image created. lb # Superblock backups stored on blocks: worker2 # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # Filesystem UUID: 6e4ea723-c219-4b7b-8934-6eb51079b2c2 client # Starting virtiofs daemons... lb # 32768, 98304, 163840, 229376 worker2 # [2026-09-23T05:11:53Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # Superblock backups stored on blocks: client # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # worker2 # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # 32768, 98304, 163840, 229376 client # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # Allocating group tables: 0/8 done lb # Writing inode tables: 0/8 done worker1 # worker2 # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-23T05:11:53Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # Creating journal (8192 blocks): done worker1 # Allocating group tables: 0/8 done worker2 # [2026-09-23T05:11:53Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # Writing superblocks and filesystem accounting information: 0/8 done worker1 # Writing inode tables: 0/8 done worker2 # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # worker1 # Creating journal (8192 blocks): done worker2 # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether client # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # Virtualisation disk image created. worker1 # Writing superblocks and filesystem accounting information: 0/8 done worker2 # [2026-09-23T05:11:53Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # Starting virtiofs daemons... worker1 # worker2 # [2026-09-23T05:11:53Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-23T05:11:53Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # Virtualisation disk image created. worker2 # [2026-09-23T05:11:53Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-23T05:11:53Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # Starting virtiofs daemons... worker2 # [2026-09-23T05:11:53Z INFO virtiofsd] Client connected, servicing requests client # [2026-09-23T05:11:53Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-23T05:11:53Z INFO virtiofsd] Waiting for vhost-user socket connection... client # [2026-09-23T05:11:53Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) client # [2026-09-23T05:11:53Z INFO virtiofsd] Client connected, servicing requests lb # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-23T05:11:53Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) worker1 # [2026-09-23T05:11:53Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-23T05:11:53Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether lb # [2026-09-23T05:11:53Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-23T05:11:53Z INFO virtiofsd] Waiting for vhost-user socket connection... lb # [2026-09-23T05:11:53Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] Failed to open file handle for the root node: Operation not permitted (os error 1) lb # [2026-09-23T05:11:53Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-23T05:11:53Z WARN virtiofsd::passthrough] File handles do not appear safe to use, disabling file handles altogether worker1 # [2026-09-23T05:11:53Z INFO virtiofsd] Waiting for vhost-user socket connection... worker1 # [2026-09-23T05:11:53Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-23T05:11:53Z INFO virtiofsd] Client connected, servicing requests worker1 # [2026-09-23T05:11:53Z INFO virtiofsd] Client connected, servicing requests client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] client # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 client # [ 0.000000] KASLR enabled client # [ 0.000000] random: crng init done client # [ 0.000000] Machine model: linux,dummy-virt client # [ 0.000000] efi: UEFI not found. client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] client # [ 0.000000] Zone ranges: client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] DMA32 empty client # [ 0.000000] Normal empty client # [ 0.000000] Device empty client # [ 0.000000] Movable zone start for each node client # [ 0.000000] Early memory node ranges client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 client # [ 0.000000] psci: probing for conduit method from DT. client # [ 0.000000] psci: PSCIv1.3 detected in firmware. client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs client # [ 0.000000] psci: Trusted OS migration not required client # [ 0.000000] psci: SMC Calling Convention v1.1 client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) client # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 client # [ 0.000000] Detected PIPT I-cache on CPU0 client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) client # [ 0.000000] CPU features: detected: GICv3 CPU interface client # [ 0.000000] CPU features: detected: Spectre-v4 client # [ 0.000000] CPU features: detected: Spectre-BHB client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 client # [ 0.000000] alternatives: applying boot alternatives client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/5fx3y1hbxhkdng69ksrxcn3js8649kk7-nixos-system-client-test/init regInfo=/nix/store/nsqdzrzkl3zyhgxccqczms6gsa6s08c3-closure-info/registration console=ttyAMA0,115200n8 console=tty0 client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/nsqdzrzkl3zyhgxccqczms6gsa6s08c3-closure-info/registration", will be passed to user space. client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB client # [ 0.000000] software IO TLB: area num 1. client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) client # [ 0.000000] Fallback order for Node 0: 0 client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 client # [ 0.000000] Policy zone: DMA client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 client # [ 0.000000] allocated 3145728 bytes of page_ext client # [ 0.000000] ftrace: allocating 74950 entries in 294 pages client # [ 0.000000] ftrace: allocated 294 pages with 4 groups client # [ 0.000000] rcu: Hierarchical RCU implementation. client # [ 0.000000] rcu: RCU event tracing is enabled. lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. client # [ 0.000000] Trampoline variant of Tasks RCU enabled. lb # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 client # [ 0.000000] Rude variant of Tasks RCU enabled. lb # [ 0.000000] KASLR enabled lb # [ 0.000000] random: crng init done client # [ 0.000000] Tracing variant of Tasks RCU enabled. lb # [ 0.000000] Machine model: linux,dummy-virt lb # [ 0.000000] efi: UEFI not found. client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] lb # [ 0.000000] Zone ranges: client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] DMA32 empty client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] Normal empty lb # [ 0.000000] Device empty client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 lb # [ 0.000000] Movable zone start for each node client # [ 0.000000] GICv3: 256 SPIs implemented lb # [ 0.000000] Early memory node ranges client # [ 0.000000] GICv3: 0 Extended SPIs implemented lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] Root IRQ handler: gic_handle_irq client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 lb # [ 0.000000] psci: probing for conduit method from DT. client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] lb # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker2 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 worker2 # [ 0.000000] KASLR enabled lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker2 # [ 0.000000] random: crng init done lb # [ 0.000000] psci: Trusted OS migration not required worker2 # [ 0.000000] Machine model: linux,dummy-virt lb # [ 0.000000] psci: SMC Calling Convention v1.1 worker2 # [ 0.000000] efi: UEFI not found. client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 lb # [ 0.000000] Detected PIPT I-cache on CPU0 worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. worker2 # [ 0.000000] Zone ranges: lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker2 # [ 0.000000] DMA32 empty lb # [ 0.000000] CPU features: detected: GICv3 CPU interface worker2 # [ 0.000000] Normal empty client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). lb # [ 0.000000] CPU features: detected: Spectre-v4 worker2 # [ 0.000000] Device empty lb # [ 0.000000] CPU features: detected: Spectre-BHB worker2 # [ 0.000000] Movable zone start for each node worker2 # [ 0.000000] Early memory node ranges client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] alternatives: applying boot alternatives client # [ 0.000037] arm-pv: using stolen time PV worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker2 # [ 0.000000] psci: probing for conduit method from DT. client # [ 0.000847] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware. client # [ 0.001015] Console: colour dummy device 80x25 worker1 # [ 0.000000] Linux version 6.18.52 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 14 11:36:19 UTC 2026 client # [ 0.001023] printk: legacy console [tty0] enabled worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker1 # [ 0.000000] KASLR enabled worker1 # [ 0.000000] random: crng init done worker2 # [ 0.000000] psci: Trusted OS migration not required worker1 # [ 0.000000] Machine model: linux,dummy-virt worker2 # [ 0.000000] psci: SMC Calling Convention v1.1 client # [ 0.001216] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker1 # [ 0.000000] efi: UEFI not found. lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/kgjzg78npi7z07am2szqxkr41f06jp5w-nixos-system-lb-test/init regInfo=/nix/store/lcw0xq38k7svc2ifmp8wza1s56mbyq61-closure-info/registration console=ttyAMA0,115200n8 console=tty0 client # [ 0.001222] pid_max: default: 32768 minimum: 301 worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT client # [ 0.001300] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.001485] landlock: Up and running. lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/lcw0xq38k7svc2ifmp8wza1s56mbyq61-closure-info/registration", will be passed to user space. worker2 # [ 0.000000] Detected PIPT I-cache on CPU0 client # [ 0.001488] Yama: becoming mindful. worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] client # [ 0.002204] LSM support for eBPF active lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker1 # [ 0.000000] Zone ranges: worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.002388] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker1 # [ 0.000000] DMA32 empty worker2 # [ 0.000000] CPU features: detected: Spectre-v4 worker1 # [ 0.000000] Normal empty client # [ 0.002445] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker1 # [ 0.000000] Device empty worker2 # [ 0.000000] CPU features: detected: Spectre-BHB client # [ 0.004501] rcu: Hierarchical SRCU implementation. worker1 # [ 0.000000] Movable zone start for each node lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker1 # [ 0.000000] Early memory node ranges client # [ 0.004508] rcu: Max phase no-delay instances is 1000. lb # [ 0.000000] software IO TLB: area num 1. client # [ 0.005770] fsl-mc MSI: its@8080000 domain created worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB client # [ 0.005868] EFI services will not be available. worker2 # [ 0.000000] alternatives: applying boot alternatives worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.005937] smp: Bringing up secondary CPUs ... lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) client # [ 0.005946] smp: Brought up 1 node, 1 CPU worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 lb # [ 0.000000] Fallback order for Node 0: 0 worker1 # [ 0.000000] psci: probing for conduit method from DT. client # [ 0.005949] SMP: Total of 1 processors activated. client # [ 0.005955] CPU: All CPU(s) started at EL1 lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware. lb # [ 0.000000] Policy zone: DMA worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs client # [ 0.005965] CPU features: detected: Branch Target Identification lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off worker1 # [ 0.000000] psci: Trusted OS migration not required client # [ 0.005970] CPU features: detected: ARMv8.4 Translation Table Level worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/8rxzyh0zqd97xa57p8wqswwb0drnznwf-nixos-system-worker2-test/init regInfo=/nix/store/lrx87j4jd62l04f6j8b9gzxii6h2jx5g-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker1 # [ 0.000000] psci: SMC Calling Convention v1.1 lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 client # [ 0.005973] CPU features: detected: Instruction cache invalidation not required for I/D coherence lb # [ 0.000000] allocated 3145728 bytes of page_ext worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) lb # [ 0.000000] ftrace: allocating 74950 entries in 294 pages worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/lrx87j4jd62l04f6j8b9gzxii6h2jx5g-closure-info/registration", will be passed to user space. client # [ 0.005977] CPU features: detected: Data cache clean to the PoU not required for I/D coherence worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186712 r8192 d116392 u311296 lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker1 # [ 0.000000] Detected PIPT I-cache on CPU0 client # [ 0.005981] CPU features: detected: Common not Private translations worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes lb # [ 0.000000] rcu: Hierarchical RCU implementation. client # [ 0.005984] CPU features: detected: CRC32 instructions lb # [ 0.000000] rcu: RCU event tracing is enabled. worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) client # [ 0.005987] CPU features: detected: Data cache clean to Point of Deep Persistence worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker1 # [ 0.000000] CPU features: detected: Spectre-v4 lb # [ 0.000000] Trampoline variant of Tasks RCU enabled. client # [ 0.005990] CPU features: detected: Data cache clean to Point of Persistence worker1 # [ 0.000000] CPU features: detected: Spectre-BHB worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB lb # [ 0.000000] Rude variant of Tasks RCU enabled. client # [ 0.005993] CPU features: detected: Data independent timing control (DIT) worker2 # [ 0.000000] software IO TLB: area num 1. lb # [ 0.000000] Tracing variant of Tasks RCU enabled. worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 client # [ 0.005997] CPU features: detected: E0PD worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. client # [ 0.005999] CPU features: detected: Enhanced Counter Virtualization worker1 # [ 0.000000] alternatives: applying boot alternatives worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 client # [ 0.006002] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) worker2 # [ 0.000000] Fallback order for Node 0: 0 client # [ 0.006006] CPU features: detected: Enhanced Virtualization Traps worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] Policy zone: DMA client # [ 0.006009] CPU features: detected: Fine Grained Traps worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.006012] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 client # [ 0.006017] CPU features: detected: RCpc load-acquire (LDAPR) worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/3j2yqs8i8ynzphhqwmkrm3aah3k3l7j5-nixos-system-worker1-test/init regInfo=/nix/store/rjgpcx9an49sbd2sna23nqm8paywvxlm-closure-info/registration console=ttyAMA0,115200n8 console=tty0 client # [ 0.006020] CPU features: detected: LSE atomic instructions worker2 # [ 0.000000] allocated 3145728 bytes of page_ext lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 client # [ 0.006023] CPU features: detected: Privileged Access Never lb # [ 0.000000] GICv3: 256 SPIs implemented worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups client # [ 0.006026] CPU features: detected: PMUv3 worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/rjgpcx9an49sbd2sna23nqm8paywvxlm-closure-info/registration", will be passed to user space. lb # [ 0.000000] GICv3: 0 Extended SPIs implemented worker2 # [ 0.000000] rcu: Hierarchical RCU implementation. client # [ 0.006029] CPU features: detected: RAS Extension Support lb # [ 0.000000] Root IRQ handler: gic_handle_irq worker2 # [ 0.000000] rcu: RCU event tracing is enabled. worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes client # [ 0.006032] CPU features: detected: RASv1p1 Extension Support lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 client # [ 0.006034] CPU features: detected: Random Number Generator worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled. client # [ 0.006037] CPU features: detected: Speculation barrier (SB) lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker2 # [ 0.000000] Rude variant of Tasks RCU enabled. lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] client # [ 0.006039] CPU features: detected: Stage-2 Force Write-Back worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled. worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB client # [ 0.006042] CPU features: detected: TLB range maintenance instructions worker1 # [ 0.000000] software IO TLB: area num 1. worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB client # [ 0.006047] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 client # [ 0.006089] alternatives: applying system-wide alternatives lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.000000] Fallback order for Node 0: 0 lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 client # [ 0.009142] CPU features: detected: BBM Level 2 without TLB conflict abort worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.000000] Policy zone: DMA lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. client # [ 0.009323] Memory: 1405648K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124660K reserved, 32768K cma-reserved) worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.009794] devtmpfs: initialized worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker2 # [ 0.000000] GICv3: 256 SPIs implemented worker1 # [ 0.000000] allocated 3145728 bytes of page_ext client # [ 0.011693] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented worker1 # [ 0.000000] ftrace: allocating 74950 entries in 294 pages client # [ 0.011717] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI client # [ 0.011930] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker1 # [ 0.000000] rcu: Hierarchical RCU implementation. worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 client # [ 0.011934] 0 pages in range for non-PLT usage worker1 # [ 0.000000] rcu: RCU event tracing is enabled. lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns client # [ 0.011935] 508272 pages in range for PLT usage worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. lb # [ 0.000041] arm-pv: using stolen time PV client # [ 0.012067] pinctrl core: initialized pinctrl subsystem worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled. client # [ 0.012846] DMI not present or invalid. worker1 # [ 0.000000] Rude variant of Tasks RCU enabled. lb # [ 0.000675] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) client # [ 0.017005] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled. lb # [ 0.000851] Console: colour dummy device 80x25 client # [ 0.019334] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations lb # [ 0.000860] printk: legacy console [tty0] enabled worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. client # [ 0.019538] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 lb # [ 0.001068] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) client # [ 0.019747] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 lb # [ 0.001075] pid_max: default: 32768 minimum: 301 worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.019777] audit: initializing netlink subsys (disabled) lb # [ 0.001176] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. lb # [ 0.001344] landlock: Up and running. client # [ 0.020409] thermal_sys: Registered thermal governor 'fair_share' worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.001347] Yama: becoming mindful. worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns client # [ 0.020412] thermal_sys: Registered thermal governor 'bang_bang' lb # [ 0.001965] LSM support for eBPF active worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). client # [ 0.020415] thermal_sys: Registered thermal governor 'step_wise' lb # [ 0.002145] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 client # [ 0.020418] thermal_sys: Registered thermal governor 'user_space' worker1 # [ 0.000000] GICv3: 256 SPIs implemented lb # [ 0.002178] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented client # [ 0.020424] thermal_sys: Registered thermal governor 'power_allocator' lb # [ 0.004192] rcu: Hierarchical SRCU implementation. worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns client # [ 0.020450] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI lb # [ 0.004199] rcu: Max phase no-delay instances is 1000. worker2 # [ 0.000032] arm-pv: using stolen time PV client # [ 0.020461] cpuidle: using governor ladder lb # [ 0.005505] fsl-mc MSI: its@8080000 domain created worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 client # [ 0.020466] cpuidle: using governor menu lb # [ 0.005606] EFI services will not be available. worker2 # [ 0.000503] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 client # [ 0.020685] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. lb # [ 0.005720] smp: Bringing up secondary CPUs ... worker2 # [ 0.000690] Console: colour dummy device 80x25 worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] lb # [ 0.005729] smp: Brought up 1 node, 1 CPU client # [ 0.020703] ASID allocator initialised with 65536 entries worker2 # [ 0.000698] printk: legacy console [tty0] enabled lb # [ 0.005732] SMP: Total of 1 processors activated. client # [ 0.021948] Serial: AMBA PL011 UART driver worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) lb # [ 0.005739] CPU: All CPU(s) started at EL1 worker2 # [ 0.000906] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) client # [ 0.027465] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 lb # [ 0.005749] CPU features: detected: Branch Target Identification worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) worker2 # [ 0.000915] pid_max: default: 32768 minimum: 301 client # [ 0.027603] printk: console [ttyAMA0] enabled lb # [ 0.005754] CPU features: detected: ARMv8.4 Translation Table Level worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker2 # [ 0.001003] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker2 # [ 0.001196] landlock: Up and running. worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker2 # [ 0.001200] Yama: becoming mindful. lb # [ 0.005757] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker2 # [ 0.001690] LSM support for eBPF active worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. lb # [ 0.005761] CPU features: detected: Data cache clean to the PoU not required for I/D coherence worker2 # [ 0.001852] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) lb # [ 0.005765] CPU features: detected: Common not Private translations worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns worker2 # [ 0.001888] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) lb # [ 0.005768] CPU features: detected: CRC32 instructions worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker2 # [ 0.003919] rcu: Hierarchical SRCU implementation. lb # [ 0.005771] CPU features: detected: Data cache clean to Point of Deep Persistence worker2 # [ 0.003925] rcu: Max phase no-delay instances is 1000. worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker2 # [ 0.005255] fsl-mc MSI: its@8080000 domain created lb # [ 0.005774] CPU features: detected: Data cache clean to Point of Persistence worker2 # [ 0.005350] EFI services will not be available. lb # [ 0.005777] CPU features: detected: Data independent timing control (DIT) worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker2 # [ 0.005422] smp: Bringing up secondary CPUs ... lb # [ 0.005780] CPU features: detected: E0PD worker1 # [ 0.000036] arm-pv: using stolen time PV worker2 # [ 0.005431] smp: Brought up 1 node, 1 CPU lb # [ 0.005783] CPU features: detected: Enhanced Counter Virtualization worker2 # [ 0.005434] SMP: Total of 1 processors activated. worker1 # [ 0.000423] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker2 # [ 0.005439] CPU: All CPU(s) started at EL1 lb # [ 0.005786] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) worker1 # [ 0.000625] Console: colour dummy device 80x25 worker2 # [ 0.005450] CPU features: detected: Branch Target Identification lb # [ 0.005789] CPU features: detected: Enhanced Virtualization Traps worker1 # [ 0.000633] printk: legacy console [tty0] enabled client # [ 0.149809] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages worker2 # [ 0.005455] CPU features: detected: ARMv8.4 Translation Table Level lb # [ 0.005792] CPU features: detected: Fine Grained Traps client # [ 0.149830] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker1 # [ 0.000825] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker2 # [ 0.005458] CPU features: detected: Instruction cache invalidation not required for I/D coherence lb # [ 0.005795] CPU features: detected: Generic authentication (architected QARMA5 algorithm) client # [ 0.149836] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages worker1 # [ 0.000832] pid_max: default: 32768 minimum: 301 lb # [ 0.005800] CPU features: detected: RCpc load-acquire (LDAPR) worker2 # [ 0.005462] CPU features: detected: Data cache clean to the PoU not required for I/D coherence client # [ 0.149840] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page worker1 # [ 0.000915] LSM: initializing lsm=capability,landlock,yama,bpf,ima lb # [ 0.005803] CPU features: detected: LSE atomic instructions worker1 # [ 0.001053] landlock: Up and running. worker2 # [ 0.005466] CPU features: detected: Common not Private translations client # [ 0.149844] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages worker1 # [ 0.001056] Yama: becoming mindful. lb # [ 0.005806] CPU features: detected: Privileged Access Never worker2 # [ 0.005469] CPU features: detected: CRC32 instructions worker1 # [ 0.001529] LSM support for eBPF active lb # [ 0.005830] CPU features: detected: PMUv3 client # [ 0.149848] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page worker2 # [ 0.005472] CPU features: detected: Data cache clean to Point of Deep Persistence lb # [ 0.005833] CPU features: detected: RAS Extension Support worker1 # [ 0.001675] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.149853] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages lb # [ 0.005836] CPU features: detected: RASv1p1 Extension Support worker2 # [ 0.005475] CPU features: detected: Data cache clean to Point of Persistence worker1 # [ 0.001709] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.149857] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page lb # [ 0.005838] CPU features: detected: Random Number Generator worker1 # [ 0.003640] rcu: Hierarchical SRCU implementation. worker2 # [ 0.005479] CPU features: detected: Data independent timing control (DIT) worker2 # [ 0.005482] CPU features: detected: E0PD lb # [ 0.005841] CPU features: detected: Speculation barrier (SB) worker1 # [ 0.003646] rcu: Max phase no-delay instances is 1000. worker1 # [ 0.005006] fsl-mc MSI: its@8080000 domain created lb # [ 0.005844] CPU features: detected: Stage-2 Force Write-Back worker2 # [ 0.005484] CPU features: detected: Enhanced Counter Virtualization client # [ 0.157255] fbcon: Taking over console worker1 # [ 0.005097] EFI services will not be available. lb # [ 0.005847] CPU features: detected: TLB range maintenance instructions worker2 # [ 0.005487] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) client # [ 0.157272] ACPI: Interpreter disabled. worker1 # [ 0.005182] smp: Bringing up secondary CPUs ... worker1 # [ 0.005191] smp: Brought up 1 node, 1 CPU worker2 # [ 0.005491] CPU features: detected: Enhanced Virtualization Traps lb # [ 0.005852] CPU features: detected: Speculative Store Bypassing Safe (SSBS) client # [ 0.159175] iommu: Default domain type: Translated worker1 # [ 0.005194] SMP: Total of 1 processors activated. worker2 # [ 0.005494] CPU features: detected: Fine Grained Traps lb # [ 0.005890] alternatives: applying system-wide alternatives worker1 # [ 0.005200] CPU: All CPU(s) started at EL1 client # [ 0.159186] iommu: DMA domain TLB invalidation policy: strict mode lb # [ 0.008913] CPU features: detected: BBM Level 2 without TLB conflict abort worker2 # [ 0.005497] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker1 # [ 0.005210] CPU features: detected: Branch Target Identification client # [ 0.160873] SCSI subsystem initialized worker2 # [ 0.005502] CPU features: detected: RCpc load-acquire (LDAPR) worker1 # [ 0.005214] CPU features: detected: ARMv8.4 Translation Table Level worker2 # [ 0.005505] CPU features: detected: LSE atomic instructions lb # [ 0.009114] Memory: 1405672K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124636K reserved, 32768K cma-reserved) worker2 # [ 0.005508] CPU features: detected: Privileged Access Never worker1 # [ 0.005217] CPU features: detected: Instruction cache invalidation not required for I/D coherence lb # [ 0.009589] devtmpfs: initialized worker2 # [ 0.005510] CPU features: detected: PMUv3 lb # [ 0.011538] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) worker1 # [ 0.005221] CPU features: detected: Data cache clean to the PoU not required for I/D coherence worker2 # [ 0.005513] CPU features: detected: RAS Extension Support worker2 # [ 0.005516] CPU features: detected: RASv1p1 Extension Support worker1 # [ 0.005225] CPU features: detected: Common not Private translations worker1 # [ 0.005229] CPU features: detected: CRC32 instructions worker2 # [ 0.005519] CPU features: detected: Random Number Generator lb # [ 0.011563] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker1 # [ 0.005232] CPU features: detected: Data cache clean to Point of Deep Persistence lb # [ 0.011767] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL worker2 # [ 0.005521] CPU features: detected: Speculation barrier (SB) worker1 # [ 0.005235] CPU features: detected: Data cache clean to Point of Persistence lb # [ 0.011771] 0 pages in range for non-PLT usage client # [ 0.166163] usbcore: registered new interface driver usbfs worker2 # [ 0.005524] CPU features: detected: Stage-2 Force Write-Back lb # [ 0.011772] 508272 pages in range for PLT usage client # [ 0.166200] usbcore: registered new interface driver hub worker1 # [ 0.005238] CPU features: detected: Data independent timing control (DIT) worker2 # [ 0.005527] CPU features: detected: TLB range maintenance instructions worker1 # [ 0.005241] CPU features: detected: E0PD client # [ 0.166218] usbcore: registered new device driver usb lb # [ 0.011899] pinctrl core: initialized pinctrl subsystem lb # [ 0.012764] DMI not present or invalid. client # [ 0.166528] pps_core: LinuxPPS API ver. 1 registered worker2 # [ 0.005531] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker1 # [ 0.005244] CPU features: detected: Enhanced Counter Virtualization lb # [ 0.016177] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker2 # [ 0.005568] alternatives: applying system-wide alternatives worker1 # [ 0.005247] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) client # [ 0.166534] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti lb # [ 0.018622] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations worker1 # [ 0.005250] CPU features: detected: Enhanced Virtualization Traps client # [ 0.166545] PTP clock support registered client # [ 0.166594] EDAC MC: Ver: 3.0.0 worker2 # [ 0.008750] CPU features: detected: BBM Level 2 without TLB conflict abort worker1 # [ 0.005254] CPU features: detected: Fine Grained Traps lb # [ 0.018828] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations client # [ 0.171127] scmi_core: SCMI protocol bus registered worker1 # [ 0.005257] CPU features: detected: Generic authentication (architected QARMA5 algorithm) lb # [ 0.019031] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker2 # [ 0.008936] Memory: 1405648K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124632K reserved, 32768K cma-reserved) worker2 # [ 0.009447] devtmpfs: initialized worker1 # [ 0.005261] CPU features: detected: RCpc load-acquire (LDAPR) lb # [ 0.019056] audit: initializing netlink subsys (disabled) client # [ 0.172126] FPGA manager framework client # [ 0.173157] vgaarb: loaded worker2 # [ 0.011431] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) worker1 # [ 0.005264] CPU features: detected: LSE atomic instructions lb # [ 0.019716] thermal_sys: Registered thermal governor 'fair_share' worker1 # [ 0.005267] CPU features: detected: Privileged Access Never lb # [ 0.019719] thermal_sys: Registered thermal governor 'bang_bang' worker2 # [ 0.011453] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). client # [ 0.173881] clocksource: Switched to clocksource arch_sys_counter worker1 # [ 0.005269] CPU features: detected: PMUv3 lb # [ 0.019722] thermal_sys: Registered thermal governor 'step_wise' worker2 # [ 0.011650] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL worker1 # [ 0.005272] CPU features: detected: RAS Extension Support lb # [ 0.019725] thermal_sys: Registered thermal governor 'user_space' worker2 # [ 0.011654] 0 pages in range for non-PLT usage worker1 # [ 0.005274] CPU features: detected: RASv1p1 Extension Support worker2 # [ 0.011656] 508272 pages in range for PLT usage lb # [ 0.019731] thermal_sys: Registered thermal governor 'power_allocator' worker1 # [ 0.005277] CPU features: detected: Random Number Generator worker2 # [ 0.011778] pinctrl core: initialized pinctrl subsystem worker1 # [ 0.005279] CPU features: detected: Speculation barrier (SB) worker2 # [ 0.012557] DMI not present or invalid. lb # [ 0.019764] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 worker1 # [ 0.005282] CPU features: detected: Stage-2 Force Write-Back lb # [ 0.019773] cpuidle: using governor ladder worker2 # [ 0.015795] NET: Registered PF_NETLINK/PF_ROUTE protocol family lb # [ 0.019778] cpuidle: using governor menu worker1 # [ 0.005284] CPU features: detected: TLB range maintenance instructions worker2 # [ 0.018415] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations lb # [ 0.019971] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. worker1 # [ 0.005289] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker2 # [ 0.018621] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations lb # [ 0.019991] ASID allocator initialised with 65536 entries worker1 # [ 0.005326] alternatives: applying system-wide alternatives lb # [ 0.021275] Serial: AMBA PL011 UART driver client # [ 0.180284] VFS: Disk quotas dquot_6.6.0 worker2 # [ 0.018888] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker1 # [ 0.008403] CPU features: detected: BBM Level 2 without TLB conflict abort worker2 # [ 0.018916] audit: initializing netlink subsys (disabled) lb # [ 0.026743] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 client # [ 0.180338] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) lb # [ 0.026884] printk: console [ttyAMA0] enabled worker2 # [ 0.019553] thermal_sys: Registered thermal governor 'fair_share' client # [ 0.181977] netfs: FS-Cache loaded worker1 # [ 0.008659] Memory: 1405616K/1572864K available (24448K kernel code, 7094K rwdata, 26372K rodata, 4736K init, 1107K bss, 124664K reserved, 32768K cma-reserved) worker2 # [ 0.019556] thermal_sys: Registered thermal governor 'bang_bang' client # [ 0.182112] pnp: PnP ACPI: disabled worker1 # [ 0.009135] devtmpfs: initialized worker2 # [ 0.019560] thermal_sys: Registered thermal governor 'step_wise' worker1 # [ 0.011138] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) worker2 # [ 0.019563] thermal_sys: Registered thermal governor 'user_space' worker1 # [ 0.011163] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker2 # [ 0.019568] thermal_sys: Registered thermal governor 'power_allocator' worker1 # [ 0.011346] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL worker2 # [ 0.019597] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 worker1 # [ 0.011351] 0 pages in range for non-PLT usage worker2 # [ 0.019606] cpuidle: using governor ladder worker1 # [ 0.011352] 508272 pages in range for PLT usage worker2 # [ 0.019612] cpuidle: using governor menu worker1 # [ 0.011462] pinctrl core: initialized pinctrl subsystem worker1 # [ 0.012352] DMI not present or invalid. worker2 # [ 0.019809] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. worker2 # [ 0.019826] ASID allocator initialised with 65536 entries worker1 # [ 0.015667] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker2 # [ 0.021068] Serial: AMBA PL011 UART driver worker1 # [ 0.018220] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations worker2 # [ 0.026416] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 worker1 # [ 0.018430] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations client # [ 0.186015] NET: Registered PF_INET protocol family worker2 # [ 0.026558] printk: console [ttyAMA0] enabled client # [ 0.186297] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker1 # [ 0.018643] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations lb # [ 0.149999] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages worker1 # [ 0.018674] audit: initializing netlink subsys (disabled) lb # [ 0.150020] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker1 # [ 0.019305] thermal_sys: Registered thermal governor 'fair_share' lb # [ 0.150026] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages worker1 # [ 0.019307] thermal_sys: Registered thermal governor 'bang_bang' lb # [ 0.150031] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page worker1 # [ 0.019310] thermal_sys: Registered thermal governor 'step_wise' lb # [ 0.150035] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages worker1 # [ 0.019313] thermal_sys: Registered thermal governor 'user_space' lb # [ 0.150039] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page worker1 # [ 0.019318] thermal_sys: Registered thermal governor 'power_allocator' lb # [ 0.150044] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages worker1 # [ 0.019346] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 lb # [ 0.150048] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker1 # [ 0.019355] cpuidle: using governor ladder worker2 # [ 0.144373] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages worker1 # [ 0.019361] cpuidle: using governor menu worker2 # [ 0.144394] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker1 # [ 0.019565] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. worker2 # [ 0.144400] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages worker1 # [ 0.019585] ASID allocator initialised with 65536 entries lb # [ 0.157735] fbcon: Taking over console worker1 # [ 0.020982] Serial: AMBA PL011 UART driver worker2 # [ 0.144404] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page lb # [ 0.157751] ACPI: Interpreter disabled. worker1 # [ 0.026357] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 worker2 # [ 0.144409] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages worker1 # [ 0.026507] printk: console [ttyAMA0] enabled worker2 # [ 0.144413] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page worker2 # [ 0.144417] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages worker2 # [ 0.144421] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker2 # [ 0.152118] fbcon: Taking over console worker2 # [ 0.152134] ACPI: Interpreter disabled. lb # [ 0.166387] iommu: Default domain type: Translated lb # [ 0.166399] iommu: DMA domain TLB invalidation policy: strict mode lb # [ 0.166763] SCSI subsystem initialized lb # [ 0.168787] usbcore: registered new interface driver usbfs worker1 # [ 0.146973] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages lb # [ 0.168819] usbcore: registered new interface driver hub worker1 # [ 0.146993] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page lb # [ 0.168835] usbcore: registered new device driver usb worker1 # [ 0.146999] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages lb # [ 0.169127] pps_core: LinuxPPS API ver. 1 registered worker1 # [ 0.147004] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page lb # [ 0.169133] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti worker1 # [ 0.147009] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages lb # [ 0.169143] PTP clock support registered worker2 # [ 0.161199] iommu: Default domain type: Translated lb # [ 0.169193] EDAC MC: Ver: 3.0.0 worker1 # [ 0.147013] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page worker2 # [ 0.161212] iommu: DMA domain TLB invalidation policy: strict mode lb # [ 0.174011] scmi_core: SCMI protocol bus registered worker1 # [ 0.147018] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages worker2 # [ 0.161648] SCSI subsystem initialized worker1 # [ 0.147022] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page lb # [ 0.175002] FPGA manager framework worker2 # [ 0.163660] usbcore: registered new interface driver usbfs lb # [ 0.175970] vgaarb: loaded worker2 # [ 0.163693] usbcore: registered new interface driver hub lb # [ 0.176682] clocksource: Switched to clocksource arch_sys_counter worker2 # [ 0.163710] usbcore: registered new device driver usb worker1 # [ 0.154708] fbcon: Taking over console worker2 # [ 0.164007] pps_core: LinuxPPS API ver. 1 registered worker1 # [ 0.154729] ACPI: Interpreter disabled. client # [ 0.217261] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) client # [ 0.217331] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) worker2 # [ 0.164013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti worker2 # [ 0.164024] PTP clock support registered client # [ 0.217367] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) worker2 # [ 0.164074] EDAC MC: Ver: 3.0.0 client # [ 0.217454] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) worker2 # [ 0.168816] scmi_core: SCMI protocol bus registered client # [ 0.217596] TCP: Hash tables configured (established 16384 bind 16384) lb # [ 0.181301] VFS: Disk quotas dquot_6.6.0 worker2 # [ 0.169835] FPGA manager framework worker2 # [ 0.170821] vgaarb: loaded client # [ 0.217690] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) lb # [ 0.181354] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) client # [ 0.217736] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.183077] netfs: FS-Cache loaded worker2 # [ 0.171479] clocksource: Switched to clocksource arch_sys_counter lb # [ 0.183194] pnp: PnP ACPI: disabled client # [ 0.217773] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.217884] NET: Registered PF_UNIX/PF_LOCAL protocol family worker1 # [ 0.163329] iommu: Default domain type: Translated worker1 # [ 0.163341] iommu: DMA domain TLB invalidation policy: strict mode worker1 # [ 0.163805] SCSI subsystem initialized worker2 # [ 0.176240] VFS: Disk quotas dquot_6.6.0 worker1 # [ 0.165842] usbcore: registered new interface driver usbfs worker2 # [ 0.176284] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) lb # [ 0.187100] NET: Registered PF_INET protocol family worker1 # [ 0.165876] usbcore: registered new interface driver hub lb # [ 0.187363] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker1 # [ 0.165893] usbcore: registered new device driver usb worker1 # [ 0.166192] pps_core: LinuxPPS API ver. 1 registered worker1 # [ 0.166197] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti worker2 # [ 0.180279] netfs: FS-Cache loaded worker1 # [ 0.166207] PTP clock support registered worker2 # [ 0.180425] pnp: PnP ACPI: disabled worker1 # [ 0.166258] EDAC MC: Ver: 3.0.0 client # [ 0.225806] NET: Registered PF_XDP protocol family worker1 # [ 0.170999] scmi_core: SCMI protocol bus registered client # [ 0.225828] PCI: CLS 0 bytes, default 64 client # [ 0.225982] Trying to unpack rootfs image as initramfs... worker1 # [ 0.172033] FPGA manager framework worker1 # [ 0.173043] vgaarb: loaded worker1 # [ 0.173694] clocksource: Switched to clocksource arch_sys_counter client # [ 0.235866] kvm [1]: HYP mode not available worker2 # [ 0.184494] NET: Registered PF_INET protocol family worker1 # [ 0.176990] VFS: Disk quotas dquot_6.6.0 worker2 # [ 0.184780] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker1 # [ 0.177028] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) worker1 # [ 0.181122] netfs: FS-Cache loaded worker1 # [ 0.181268] pnp: PnP ACPI: disabled worker1 # [ 0.185428] NET: Registered PF_INET protocol family worker1 # [ 0.186361] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) lb # [ 0.219612] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) lb # [ 0.219671] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) lb # [ 0.219704] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) lb # [ 0.219792] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) lb # [ 0.219938] TCP: Hash tables configured (established 16384 bind 16384) lb # [ 0.220033] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) lb # [ 0.220088] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) worker2 # [ 0.216215] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) lb # [ 0.220130] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.220228] NET: Registered PF_UNIX/PF_LOCAL protocol family worker2 # [ 0.216275] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) lb # [ 0.220249] NET: Registered PF_XDP protocol family worker2 # [ 0.216309] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) lb # [ 0.220271] PCI: CLS 0 bytes, default 64 lb # [ 0.220524] Trying to unpack rootfs image as initramfs... worker2 # [ 0.216389] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) worker2 # [ 0.216534] TCP: Hash tables configured (established 16384 bind 16384) worker2 # [ 0.216647] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) worker2 # [ 0.216684] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) worker2 # [ 0.216721] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) worker2 # [ 0.216860] NET: Registered PF_UNIX/PF_LOCAL protocol family worker2 # [ 0.216886] NET: Registered PF_XDP protocol family worker2 # [ 0.216908] PCI: CLS 0 bytes, default 64 worker2 # [ 0.217158] Trying to unpack rootfs image as initramfs... lb # [ 0.238625] kvm [1]: HYP mode not available worker1 # [ 0.216809] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) worker1 # [ 0.216865] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) worker1 # [ 0.216898] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) worker2 # [ 0.233325] kvm [1]: HYP mode not available worker1 # [ 0.216985] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) worker1 # [ 0.217129] TCP: Hash tables configured (established 16384 bind 16384) worker1 # [ 0.217222] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) worker1 # [ 0.217261] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) worker1 # [ 0.217297] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) worker1 # [ 0.217400] NET: Registered PF_UNIX/PF_LOCAL protocol family worker1 # [ 0.217446] NET: Registered PF_XDP protocol family worker1 # [ 0.217464] PCI: CLS 0 bytes, default 64 worker1 # [ 0.217754] Trying to unpack rootfs image as initramfs... worker1 # [ 0.235639] kvm [1]: HYP mode not available client # [ 0.358566] Initialise system trusted keyrings client # [ 0.359327] workingset: timestamp_bits=42 max_order=19 bucket_order=0 client # [ 0.360663] squashfs: version 4.0 (2009/01/31) Phillip Lougher client # [ 0.361428] 9p: Installing v9fs 9p2000 file system support client # [ 0.390369] Key type asymmetric registered client # [ 0.390405] Asymmetric key parser 'x509' registered client # [ 0.390506] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) client # [ 0.392636] io scheduler mq-deadline registered client # [ 0.392649] io scheduler kyber registered lb # [ 0.361311] Initialise system trusted keyrings client # [ 0.402057] pl061_gpio 9030000.pl061: PL061 GPIO chip registered lb # [ 0.362148] workingset: timestamp_bits=42 max_order=19 bucket_order=0 lb # [ 0.363437] squashfs: version 4.0 (2009/01/31) Phillip Lougher lb # [ 0.364268] 9p: Installing v9fs 9p2000 file system support worker2 # [ 0.356149] Initialise system trusted keyrings worker2 # [ 0.356917] workingset: timestamp_bits=42 max_order=19 bucket_order=0 client # [ 0.403599] ledtrig-cpu: registered to indicate activity on CPUs worker2 # [ 0.358229] squashfs: version 4.0 (2009/01/31) Phillip Lougher client # [ 0.404004] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker2 # [ 0.359031] 9p: Installing v9fs 9p2000 file system support client # [ 0.404022] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 client # [ 0.404034] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 client # [ 0.404043] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 client # [ 0.404063] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker1 # [ 0.352959] Initialise system trusted keyrings client # [ 0.404090] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] client # [ 0.404168] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 client # [ 0.404178] pci_bus 0000:00: root bus resource [bus 00-ff] client # [ 0.404184] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] client # [ 0.404189] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] client # [ 0.404194] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] client # [ 0.404291] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint client # [ 0.404737] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.404925] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] client # [ 0.404945] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.404974] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.361789] workingset: timestamp_bits=42 max_order=19 bucket_order=0 client # [ 0.404990] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.363233] squashfs: version 4.0 (2009/01/31) Phillip Lougher client # [ 0.405443] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint client # [ 0.405628] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] client # [ 0.405644] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.405673] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.406158] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint client # [ 0.406339] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] client # [ 0.406355] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.406386] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.364043] 9p: Installing v9fs 9p2000 file system support client # [ 0.406863] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.407048] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] client # [ 0.407064] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.393159] Key type asymmetric registered lb # [ 0.393190] Asymmetric key parser 'x509' registered client # [ 0.407093] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.393268] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) client # [ 0.407112] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] lb # [ 0.395463] io scheduler mq-deadline registered client # [ 0.407564] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint lb # [ 0.395475] io scheduler kyber registered client # [ 0.407751] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.407780] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.408230] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker1 # [ 0.376816] Key type asymmetric registered client # [ 0.408416] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.387843] Key type asymmetric registered worker1 # [ 0.376828] Asymmetric key parser 'x509' registered client # [ 0.408445] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.387873] Asymmetric key parser 'x509' registered worker1 # [ 0.376886] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) client # [ 0.408845] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker2 # [ 0.387949] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) client # [ 0.409025] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] worker2 # [ 0.390104] io scheduler mq-deadline registered worker2 # [ 0.390116] io scheduler kyber registered client # [ 0.409301] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.409491] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.409521] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.404851] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker1 # [ 0.385780] io scheduler mq-deadline registered worker1 # [ 0.385803] io scheduler kyber registered worker2 # [ 0.399679] pl061_gpio 9030000.pl061: PL061 GPIO chip registered lb # [ 0.406423] ledtrig-cpu: registered to indicate activity on CPUs lb # [ 0.406851] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: lb # [ 0.406869] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker2 # [ 0.401209] ledtrig-cpu: registered to indicate activity on CPUs lb # [ 0.406883] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker2 # [ 0.401667] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: lb # [ 0.406892] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker2 # [ 0.401689] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 lb # [ 0.406913] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker2 # [ 0.401703] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 lb # [ 0.406941] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker2 # [ 0.401712] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 lb # [ 0.407038] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker2 # [ 0.401733] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker1 # [ 0.397895] pl061_gpio 9030000.pl061: PL061 GPIO chip registered lb # [ 0.407048] pci_bus 0000:00: root bus resource [bus 00-ff] worker2 # [ 0.401761] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker2 # [ 0.401841] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 lb # [ 0.407054] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] lb # [ 0.407060] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker2 # [ 0.401850] pci_bus 0000:00: root bus resource [bus 00-ff] lb # [ 0.407065] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.399370] ledtrig-cpu: registered to indicate activity on CPUs worker2 # [ 0.401857] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] lb # [ 0.407123] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker2 # [ 0.401863] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.399833] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: client # [ 0.458089] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.407569] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.458312] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.401868] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.399850] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 lb # [ 0.407752] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] client # [ 0.458344] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.401927] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint lb # [ 0.407768] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.399863] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 lb # [ 0.407798] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.458909] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.402391] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.399871] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 client # [ 0.459112] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.407814] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.402579] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.402596] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.459145] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.399896] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits lb # [ 0.408271] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker2 # [ 0.402635] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.408455] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] client # [ 0.459618] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker1 # [ 0.399930] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker2 # [ 0.402651] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] lb # [ 0.408471] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.459922] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] worker1 # [ 0.400005] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 client # [ 0.459940] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.408500] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.403115] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.400014] pci_bus 0000:00: root bus resource [bus 00-ff] client # [ 0.459970] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.403296] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.400020] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] lb # [ 0.408992] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker2 # [ 0.403312] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.409175] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] client # [ 0.460458] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.400025] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker2 # [ 0.403341] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.409191] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.460649] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.400031] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] client # [ 0.460665] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.409220] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.400094] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint client # [ 0.460695] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.409665] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.400543] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.461282] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned lb # [ 0.409866] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.400734] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.409882] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.461294] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker1 # [ 0.400750] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.409911] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.461300] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker1 # [ 0.400780] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.409929] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.400797] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.461347] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned lb # [ 0.410409] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint client # [ 0.461395] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker1 # [ 0.401257] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.410597] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.401440] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] client # [ 0.461443] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned lb # [ 0.410626] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.401455] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.461493] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned worker1 # [ 0.401485] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.411071] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint client # [ 0.461543] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned lb # [ 0.411255] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.411285] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.461593] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker1 # [ 0.420490] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint lb # [ 0.411676] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint client # [ 0.461640] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker1 # [ 0.420676] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] lb # [ 0.411856] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] client # [ 0.461687] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned worker1 # [ 0.420693] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.412144] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.461733] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned worker1 # [ 0.420723] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.412332] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.421183] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.461810] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned lb # [ 0.412364] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.427940] pci 0000:00:03.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker1 # [ 0.421365] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.428154] pci 0000:00:03.0: BAR 0 [io 0x0000-0x007f] client # [ 0.461857] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned worker1 # [ 0.421380] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.428172] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.461879] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker1 # [ 0.421410] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.428202] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.461946] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned worker1 # [ 0.421429] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.461968] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned worker2 # [ 0.428666] pci 0000:00:04.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.428856] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] client # [ 0.461991] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned worker2 # [ 0.428873] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.462013] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned worker2 # [ 0.428902] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.462034] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned worker2 # [ 0.428923] pci 0000:00:04.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.462063] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned worker2 # [ 0.429379] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint client # [ 0.462085] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned worker2 # [ 0.429568] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.462106] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned worker2 # [ 0.429598] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.462129] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned worker2 # [ 0.430058] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint lb # [ 0.452955] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.462151] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned worker2 # [ 0.430251] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.453176] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.462172] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned worker2 # [ 0.430280] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.462193] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned lb # [ 0.453206] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.462214] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned worker2 # [ 0.430725] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint lb # [ 0.453696] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.462234] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned worker2 # [ 0.430910] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] lb # [ 0.453936] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.462255] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned worker2 # [ 0.431160] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.453970] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.462276] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned worker2 # [ 0.431348] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.462303] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] lb # [ 0.454459] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker2 # [ 0.431378] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.454797] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] client # [ 0.462312] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] lb # [ 0.454815] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.431847] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint client # [ 0.462317] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] worker2 # [ 0.432037] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.454848] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.463168] pci 0000:00:07.0: enabling device (0000 -> 0002) worker2 # [ 0.432068] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.433991] pci 0000:00:05.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint lb # [ 0.455318] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.434198] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.434229] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.455502] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.432526] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint lb # [ 0.455518] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.432713] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.434698] pci 0000:00:06.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint lb # [ 0.455548] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.434888] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.432742] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.456182] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned worker1 # [ 0.434917] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.433199] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint lb # [ 0.456193] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker2 # [ 0.433516] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] worker1 # [ 0.435313] pci 0000:00:07.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker2 # [ 0.433534] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.456198] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker1 # [ 0.435493] pci 0000:00:07.0: BAR 0 [mem 0x00000000-0x00000fff] worker2 # [ 0.433564] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.456245] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker1 # [ 0.435789] pci 0000:00:08.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.434028] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.435976] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.456293] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker2 # [ 0.434213] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.436006] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.456342] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned worker2 # [ 0.434229] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.436461] pci 0000:00:09.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.434259] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.456391] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned worker1 # [ 0.436647] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.434860] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned lb # [ 0.456439] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned worker1 # [ 0.436676] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.434872] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned lb # [ 0.456487] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker1 # [ 0.437134] pci 0000:00:0a.0: [1af4:105a] type 00 class 0x018000 conventional PCI endpoint worker2 # [ 0.434877] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker1 # [ 0.437321] pci 0000:00:0a.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.456536] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker2 # [ 0.434922] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker1 # [ 0.437351] pci 0000:00:0a.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.456582] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned worker2 # [ 0.434970] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker1 # [ 0.437836] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint lb # [ 0.456629] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned worker1 # [ 0.438114] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] worker2 # [ 0.435017] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned worker1 # [ 0.438131] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.435064] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned worker1 # [ 0.438160] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.435110] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned worker1 # [ 0.438630] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint client # [ 0.520198] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) worker2 # [ 0.435157] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker1 # [ 0.438826] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.438842] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.435205] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker1 # [ 0.438873] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.435251] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned worker1 # [ 0.439462] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned worker2 # [ 0.435298] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned worker1 # [ 0.439474] pci 0000:00:04.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker2 # [ 0.435380] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned worker1 # [ 0.439480] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker2 # [ 0.435433] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned worker1 # [ 0.439530] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker2 # [ 0.435455] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned lb # [ 0.484815] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned worker2 # [ 0.435476] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned worker1 # [ 0.439576] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned lb # [ 0.484883] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned worker1 # [ 0.439623] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned lb # [ 0.484906] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker1 # [ 0.439671] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned lb # [ 0.484929] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned worker1 # [ 0.439732] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned lb # [ 0.484951] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned worker1 # [ 0.439779] pci 0000:00:08.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned lb # [ 0.484974] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned worker1 # [ 0.439824] pci 0000:00:09.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned lb # [ 0.484997] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned lb # [ 0.485020] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned worker1 # [ 0.439869] pci 0000:00:0a.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned lb # [ 0.485048] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned worker1 # [ 0.439919] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned lb # [ 0.485071] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned worker1 # [ 0.440010] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned lb # [ 0.485094] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned worker1 # [ 0.440056] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned lb # [ 0.485118] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned worker1 # [ 0.440077] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker2 # [ 0.483604] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned lb # [ 0.485141] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned worker1 # [ 0.440098] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned worker2 # [ 0.483650] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned lb # [ 0.485162] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned worker1 # [ 0.440122] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned lb # [ 0.485184] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned worker2 # [ 0.483673] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned worker1 # [ 0.440143] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned lb # [ 0.485205] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned worker2 # [ 0.483698] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned worker1 # [ 0.440165] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned lb # [ 0.485227] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned worker2 # [ 0.483725] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned lb # [ 0.485248] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned worker1 # [ 0.440186] pci 0000:00:07.0: BAR 0 [mem 0x10086000-0x10086fff]: assigned worker2 # [ 0.483748] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned lb # [ 0.485270] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned worker1 # [ 0.440211] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned worker2 # [ 0.483772] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned lb # [ 0.485301] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] worker1 # [ 0.440233] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned worker2 # [ 0.483796] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned lb # [ 0.485310] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] worker1 # [ 0.440254] pci 0000:00:0a.0: BAR 1 [mem 0x10089000-0x10089fff]: assigned worker2 # [ 0.483821] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned lb # [ 0.485316] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] client # [ 0.538184] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker1 # [ 0.440276] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned worker2 # [ 0.483845] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned lb # [ 0.486176] pci 0000:00:07.0: enabling device (0000 -> 0002) client # [ 0.540303] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker1 # [ 0.440298] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned worker2 # [ 0.483866] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned client # [ 0.542521] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) worker1 # [ 0.440319] pci 0000:00:03.0: BAR 0 [io 0x1000-0x107f]: assigned worker2 # [ 0.483887] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned worker1 # [ 0.440340] pci 0000:00:0b.0: BAR 0 [io 0x1080-0x10bf]: assigned worker2 # [ 0.483909] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned client # [ 0.545251] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) worker1 # [ 0.440360] pci 0000:00:01.0: BAR 0 [io 0x10c0-0x10df]: assigned worker2 # [ 0.483932] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned worker1 # [ 0.440381] pci 0000:00:02.0: BAR 0 [io 0x10e0-0x10ff]: assigned worker2 # [ 0.483954] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned worker1 # [ 0.440401] pci 0000:00:04.0: BAR 0 [io 0x1100-0x111f]: assigned worker2 # [ 0.483985] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] worker1 # [ 0.440422] pci 0000:00:0c.0: BAR 0 [io 0x1120-0x113f]: assigned worker2 # [ 0.483995] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] worker1 # [ 0.440449] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] worker2 # [ 0.484000] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] worker1 # [ 0.440458] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] worker2 # [ 0.484838] pci 0000:00:07.0: enabling device (0000 -> 0002) worker1 # [ 0.440463] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] worker1 # [ 0.441292] pci 0000:00:07.0: enabling device (0000 -> 0002) client # [ 0.556130] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) worker1 # [ 0.497886] pci 0000:00:07.0: quirk_usb_early_handoff+0x0/0xa60 took 55270 usecs client # [ 0.562020] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) client # [ 0.564244] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) lb # [ 0.531761] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) lb # [ 0.534941] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) client # [ 0.575085] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) client # [ 0.577226] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) worker2 # [ 0.526423] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) worker2 # [ 0.529562] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker1 # [ 0.524400] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) lb # [ 0.544963] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) client # [ 0.589867] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) lb # [ 0.547218] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) worker1 # [ 0.527804] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) lb # [ 0.551563] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) worker2 # [ 0.540242] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker2 # [ 0.542787] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) worker2 # [ 0.546699] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) lb # [ 0.559147] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) worker1 # [ 0.538041] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker1 # [ 0.540319] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) client # [ 0.604611] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled worker1 # [ 0.544493] virtio-pci 0000:00:05.0: enabling device (0000 -> 0002) client # [ 0.607668] msm_serial: driver initialized client # [ 0.607825] SuperH (H)SCI(F) driver initialized client # [ 0.607882] STM32 USART driver initialized worker2 # [ 0.555748] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) worker2 # [ 0.557749] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker1 # [ 0.552566] virtio-pci 0000:00:06.0: enabling device (0000 -> 0002) lb # [ 0.568975] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) lb # [ 0.571503] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) lb # [ 0.573613] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) lb # [ 0.575633] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) worker1 # [ 0.562413] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker1 # [ 0.566702] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker1 # [ 0.568729] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) worker2 # [ 0.575855] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker2 # [ 0.578050] virtio-pci 0000:00:0a.0: enabling device (0000 -> 0002) lb # [ 0.594372] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) worker2 # [ 0.580121] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) worker2 # [ 0.582663] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) lb # [ 0.600474] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled worker1 # [ 0.580320] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) client # [ 0.645348] loop: module loaded client # [ 0.645575] virtio_blk virtio2: 1/0/0 default/read/poll queues client # [ 0.647750] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) lb # [ 0.606064] msm_serial: driver initialized lb # [ 0.606224] SuperH (H)SCI(F) driver initialized lb # [ 0.606281] STM32 USART driver initialized worker1 # [ 0.587155] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) worker2 # [ 0.600825] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled worker2 # [ 0.603699] msm_serial: driver initialized worker2 # [ 0.603917] SuperH (H)SCI(F) driver initialized worker2 # [ 0.603976] STM32 USART driver initialized client # [ 0.658791] megasas: 07.734.00.00-rc1 client # [ 0.659547] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] worker1 # [ 0.604627] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled client # [ 0.661991] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 client # [ 0.662017] Intel/Sharp Extended Query Table at 0x0031 client # [ 0.663860] Using buffer write method client # [ 0.663948] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] worker1 # [ 0.607278] msm_serial: driver initialized worker1 # [ 0.607423] SuperH (H)SCI(F) driver initialized client # [ 0.665771] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.607479] STM32 USART driver initialized client # [ 0.665796] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.643803] loop: module loaded lb # [ 0.644003] virtio_blk virtio2: 1/0/0 default/read/poll queues client # [ 0.686195] Using buffer write method client # [ 0.686237] Concatenating MTD devices: client # [ 0.686241] (0): "0.flash" client # [ 0.686246] (1): "0.flash" lb # [ 0.646060] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) client # [ 0.686256] into device "0.flash" worker2 # [ 0.638625] loop: module loaded worker2 # [ 0.638830] virtio_blk virtio2: 1/0/0 default/read/poll queues worker2 # [ 0.640856] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) lb # [ 0.652822] megasas: 07.734.00.00-rc1 lb # [ 0.653776] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] worker1 # [ 0.638841] loop: module loaded worker1 # [ 0.639041] virtio_blk virtio2: 1/0/0 default/read/poll queues worker1 # [ 0.640041] virtio_blk virtio2: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) lb # [ 0.665928] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.666004] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.667782] Using buffer write method worker2 # [ 0.652269] megasas: 07.734.00.00-rc1 lb # [ 0.667867] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] worker2 # [ 0.653061] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] worker2 # [ 0.655270] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.655304] Intel/Sharp Extended Query Table at 0x0031 worker2 # [ 0.657150] Using buffer write method worker2 # [ 0.657242] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] worker2 # [ 0.659116] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.659146] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.654520] megasas: 07.734.00.00-rc1 worker1 # [ 0.655296] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] lb # [ 0.683704] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.657892] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.683748] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.657922] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.659739] Using buffer write method lb # [ 0.687792] Using buffer write method worker1 # [ 0.659833] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] lb # [ 0.687835] Concatenating MTD devices: lb # [ 0.687840] (0): "0.flash" lb # [ 0.687844] (1): "0.flash" lb # [ 0.687856] into device "0.flash" worker2 # [ 0.679974] Using buffer write method worker2 # [ 0.680019] Concatenating MTD devices: worker2 # [ 0.680025] (0): "0.flash" worker2 # [ 0.680030] (1): "0.flash" worker2 # [ 0.680043] into device "0.flash" worker1 # [ 0.669747] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.669785] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.681755] Using buffer write method worker1 # [ 0.681794] Concatenating MTD devices: worker1 # [ 0.681798] (0): "0.flash" worker1 # [ 0.681807] (1): "0.flash" worker1 # [ 0.681810] into device "0.flash" client # [ 0.978660] Freeing initrd memory: 26972K client # [ 0.985005] tun: Universal TUN/TAP device driver, 1.6 client # [ 0.988870] thunder_xcv, ver 1.0 client # [ 0.988918] thunder_bgx, ver 1.0 client # [ 0.988940] nicpf, ver 1.0 client # [ 0.989499] e1000: Intel(R) PRO/1000 Network Driver client # [ 0.989507] e1000: Copyright (c) 1999-2006 Intel Corporation. client # [ 0.989530] e1000e: Intel(R) PRO/1000 Network Driver client # [ 0.989540] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. client # [ 0.989569] igb: Intel(R) Gigabit Ethernet Network Driver client # [ 0.989575] igb: Copyright (c) 2007-2014 Intel Corporation. client # [ 0.989618] igbvf: Intel(R) Gigabit Virtual Function Network Driver client # [ 0.989625] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. client # [ 0.989757] sky2: driver version 1.30 client # [ 0.998643] usbcore: registered new interface driver usb-storage client # [ 0.998774] usbcore: registered new interface driver usbserial_generic client # [ 0.998792] usbserial: USB Serial support registered for generic client # [ 0.999407] hv_vmbus: registering driver hyperv_keyboard client # [ 1.000802] ehci-pci 0000:00:07.0: EHCI Host Controller client # [ 1.000838] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 client # [ 1.001079] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 client # [ 1.006574] rtc-pl031 9010000.pl031: registered as rtc0 client # [ 1.006608] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T05:11:55 UTC (1790140315) client # [ 1.007011] i2c_dev: i2c /dev entries driver client # [ 1.010093] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 client # [ 1.011105] hub 1-0:1.0: USB hub found client # [ 1.011589] hub 1-0:1.0: 6 ports detected client # [ 1.014940] sdhci: Secure Digital Host Controller Interface driver client # [ 1.014951] sdhci: Copyright(c) Pierre Ossman client # [ 1.015247] Synopsys Designware Multimedia Card Interface Driver lb # [ 0.979210] Freeing initrd memory: 26948K client # [ 1.015645] sdhci-pltfm: SDHCI platform and OF driver helper client # [ 1.017212] hid: raw HID events driver (C) Jiri Kosina client # [ 1.017499] usbcore: registered new interface driver usbhid client # [ 1.017509] usbhid: USB HID core driver client # [ 1.023232] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available client # [ 1.024810] drop_monitor: Initializing network drop monitor service lb # [ 0.985750] tun: Universal TUN/TAP device driver, 1.6 client # [ 1.025013] NET: Registered PF_INET6 protocol family client # [ 1.028222] Segment Routing with IPv6 client # [ 1.028241] In-situ OAM (IOAM) with IPv6 client # [ 1.028273] NET: Registered PF_PACKET protocol family lb # [ 0.989857] thunder_xcv, ver 1.0 lb # [ 0.989904] thunder_bgx, ver 1.0 lb # [ 0.989927] nicpf, ver 1.0 client # [ 1.029964] 9pnet: Installing 9P2000 support client # [ 1.030013] Key type dns_resolver registered lb # [ 0.990519] e1000: Intel(R) PRO/1000 Network Driver lb # [ 0.990529] e1000: Copyright (c) 1999-2006 Intel Corporation. lb # [ 0.990552] e1000e: Intel(R) PRO/1000 Network Driver lb # [ 0.990559] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. lb # [ 0.990588] igb: Intel(R) Gigabit Ethernet Network Driver worker2 # [ 0.985243] Freeing initrd memory: 26944K client # [ 1.037051] registered taskstats version 1 lb # [ 0.990593] igb: Copyright (c) 2007-2014 Intel Corporation. client # [ 1.037215] Loading compiled-in X.509 certificates lb # [ 0.990615] igbvf: Intel(R) Gigabit Virtual Function Network Driver lb # [ 0.990620] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. lb # [ 0.990752] sky2: driver version 1.30 lb # [ 0.992324] usbcore: registered new interface driver usb-storage lb # [ 0.992450] usbcore: registered new interface driver usbserial_generic lb # [ 0.992463] usbserial: USB Serial support registered for generic worker2 # [ 0.991546] tun: Universal TUN/TAP device driver, 1.6 lb # [ 1.003132] ehci-pci 0000:00:07.0: EHCI Host Controller lb # [ 1.003166] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 lb # [ 1.003409] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 worker1 # [ 0.981175] Freeing initrd memory: 26976K client # [ 1.046165] Demotion targets for Node 0: null client # [ 1.046298] Key type .fscrypt registered lb # [ 1.006388] hv_vmbus: registering driver hyperv_keyboard client # [ 1.046305] Key type fscrypt-provisioning registered client # [ 1.046425] ima: No TPM chip found, activating TPM-bypass! client # [ 1.046445] ima: Allocated hash algorithm: sha1 lb # [ 1.008040] rtc-pl031 9010000.pl031: registered as rtc0 client # [ 1.046469] ima: No architecture policies found lb # [ 1.008072] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T05:11:55 UTC (1790140315) lb # [ 1.008375] i2c_dev: i2c /dev entries driver client # [ 1.050828] input: gpio-keys as /devices/platform/gpio-keys/input/input0 worker1 # [ 0.988357] tun: Universal TUN/TAP device driver, 1.6 worker2 # [ 0.995713] thunder_xcv, ver 1.0 worker2 # [ 0.995756] thunder_bgx, ver 1.0 worker2 # [ 0.995782] nicpf, ver 1.0 worker2 # [ 0.996353] e1000: Intel(R) PRO/1000 Network Driver worker2 # [ 0.996364] e1000: Copyright (c) 1999-2006 Intel Corporation. worker2 # [ 0.996387] e1000e: Intel(R) PRO/1000 Network Driver worker2 # [ 0.996396] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker1 # [ 0.992308] thunder_xcv, ver 1.0 worker2 # [ 0.996428] igb: Intel(R) Gigabit Ethernet Network Driver worker1 # [ 0.992357] thunder_bgx, ver 1.0 worker1 # [ 0.992379] nicpf, ver 1.0 worker2 # [ 0.996434] igb: Copyright (c) 2007-2014 Intel Corporation. lb # [ 1.013879] sdhci: Secure Digital Host Controller Interface driver worker1 # [ 0.992961] e1000: Intel(R) PRO/1000 Network Driver lb # [ 1.013895] sdhci: Copyright(c) Pierre Ossman worker2 # [ 0.996457] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker1 # [ 0.992971] e1000: Copyright (c) 1999-2006 Intel Corporation. worker2 # [ 0.996465] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. lb # [ 1.014169] Synopsys Designware Multimedia Card Interface Driver worker1 # [ 0.992994] e1000e: Intel(R) PRO/1000 Network Driver worker2 # [ 0.996618] sky2: driver version 1.30 lb # [ 1.014543] sdhci-pltfm: SDHCI platform and OF driver helper worker1 # [ 0.993001] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker2 # [ 0.998258] usbcore: registered new interface driver usb-storage lb # [ 1.016134] hid: raw HID events driver (C) Jiri Kosina worker1 # [ 0.993030] igb: Intel(R) Gigabit Ethernet Network Driver lb # [ 1.016400] usbcore: registered new interface driver usbhid worker2 # [ 0.998438] usbcore: registered new interface driver usbserial_generic worker1 # [ 0.993036] igb: Copyright (c) 2007-2014 Intel Corporation. lb # [ 1.016406] usbhid: USB HID core driver worker2 # [ 0.998457] usbserial: USB Serial support registered for generic worker1 # [ 0.993080] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker2 # [ 0.999063] hv_vmbus: registering driver hyperv_keyboard worker1 # [ 0.993086] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. lb # [ 1.021188] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker1 # [ 0.993226] sky2: driver version 1.30 lb # [ 1.021525] hub 1-0:1.0: USB hub found worker2 # [ 1.000199] ehci-pci 0000:00:07.0: EHCI Host Controller lb # [ 1.021593] hub 1-0:1.0: 6 ports detected worker2 # [ 1.000225] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker2 # [ 1.000446] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 worker1 # [ 1.002411] ehci-pci 0000:00:07.0: EHCI Host Controller lb # [ 1.024464] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker2 # [ 1.012822] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker2 # [ 1.013202] hub 1-0:1.0: USB hub found worker1 # [ 1.002448] ehci-pci 0000:00:07.0: new USB bus registered, assigned bus number 1 worker2 # [ 1.013223] hub 1-0:1.0: 6 ports detected worker1 # [ 1.002695] ehci-pci 0000:00:07.0: irq 16, io mem 0x10086000 lb # [ 1.027214] drop_monitor: Initializing network drop monitor service worker1 # [ 1.005528] usbcore: registered new interface driver usb-storage lb # [ 1.027417] NET: Registered PF_INET6 protocol family worker2 # [ 1.015916] rtc-pl031 9010000.pl031: registered as rtc0 worker1 # [ 1.005665] usbcore: registered new interface driver usbserial_generic client # [ 1.069672] clk: Disabling unused clocks worker1 # [ 1.005680] usbserial: USB Serial support registered for generic lb # [ 1.029410] Segment Routing with IPv6 worker2 # [ 1.015943] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T05:11:55 UTC (1790140315) client # [ 1.069710] PM: genpd: Disabling unused power domains lb # [ 1.029429] In-situ OAM (IOAM) with IPv6 worker2 # [ 1.016334] i2c_dev: i2c /dev entries driver lb # [ 1.029460] NET: Registered PF_PACKET protocol family worker1 # [ 1.008595] hv_vmbus: registering driver hyperv_keyboard lb # [ 1.031086] 9pnet: Installing 9P2000 support lb # [ 1.031143] Key type dns_resolver registered worker1 # [ 1.010202] rtc-pl031 9010000.pl031: registered as rtc0 client # [ 1.073923] Freeing unused kernel memory: 4736K client # [ 1.074120] Run /init as init process worker1 # [ 1.010238] rtc-pl031 9010000.pl031: setting system clock to 2026-09-23T05:11:55 UTC (1790140315) worker1 # [ 1.010551] i2c_dev: i2c /dev entries driver worker2 # [ 1.021783] sdhci: Secure Digital Host Controller Interface driver worker2 # [ 1.021800] sdhci: Copyright(c) Pierre Ossman worker2 # [ 1.022064] Synopsys Designware Multimedia Card Interface Driver worker2 # [ 1.022466] sdhci-pltfm: SDHCI platform and OF driver helper worker2 # [ 1.026922] hid: raw HID events driver (C) Jiri Kosina worker2 # [ 1.027191] usbcore: registered new interface driver usbhid worker1 # [ 1.016074] sdhci: Secure Digital Host Controller Interface driver worker2 # [ 1.027199] usbhid: USB HID core driver worker1 # [ 1.016086] sdhci: Copyright(c) Pierre Ossman lb # [ 1.038116] registered taskstats version 1 worker1 # [ 1.016349] Synopsys Designware Multimedia Card Interface Driver lb # [ 1.038259] Loading compiled-in X.509 certificates worker1 # [ 1.016728] sdhci-pltfm: SDHCI platform and OF driver helper worker1 # [ 1.020465] ehci-pci 0000:00:07.0: USB 2.0 started, EHCI 1.00 worker1 # [ 1.021528] hub 1-0:1.0: USB hub found worker2 # [ 1.030217] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker1 # [ 1.022069] hub 1-0:1.0: 6 ports detected worker2 # [ 1.032917] drop_monitor: Initializing network drop monitor service worker2 # [ 1.033126] NET: Registered PF_INET6 protocol family worker2 # [ 1.035081] Segment Routing with IPv6 worker2 # [ 1.035113] In-situ OAM (IOAM) with IPv6 worker1 # [ 1.023684] hid: raw HID events driver (C) Jiri Kosina worker2 # [ 1.035145] NET: Registered PF_PACKET protocol family worker1 # [ 1.023963] usbcore: registered new interface driver usbhid lb # [ 1.047321] Demotion targets for Node 0: null worker1 # [ 1.023970] usbhid: USB HID core driver lb # [ 1.047457] Key type .fscrypt registered lb # [ 1.047463] Key type fscrypt-provisioning registered lb # [ 1.047568] ima: No TPM chip found, activating TPM-bypass! lb # [ 1.047590] ima: Allocated hash algorithm: sha1 lb # [ 1.047614] ima: No architecture policies found worker2 # [ 1.037016] 9pnet: Installing 9P2000 support worker1 # [ 1.026933] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker2 # [ 1.037069] Key type dns_resolver registered worker1 # [ 1.028569] drop_monitor: Initializing network drop monitor service lb # [ 1.052027] input: gpio-keys as /devices/platform/gpio-keys/input/input0 client # [ 1.092859] systemd[1]: Successfully made /usr/ read-only. worker1 # [ 1.028723] NET: Registered PF_INET6 protocol family worker1 # [ 1.031806] Segment Routing with IPv6 worker1 # [ 1.031825] In-situ OAM (IOAM) with IPv6 worker1 # [ 1.031851] NET: Registered PF_PACKET protocol family worker1 # [ 1.033469] 9pnet: Installing 9P2000 support worker1 # [ 1.033522] Key type dns_resolver registered worker2 # [ 1.043932] registered taskstats version 1 worker2 # [ 1.044101] Loading compiled-in X.509 certificates worker1 # [ 1.040525] registered taskstats version 1 worker1 # [ 1.040693] Loading compiled-in X.509 certificates worker2 # [ 1.053007] Demotion targets for Node 0: null worker2 # [ 1.053135] Key type .fscrypt registered worker2 # [ 1.053141] Key type fscrypt-provisioning registered worker2 # [ 1.053249] ima: No TPM chip found, activating TPM-bypass! worker2 # [ 1.053268] ima: Allocated hash algorithm: sha1 worker2 # [ 1.053292] ima: No architecture policies found worker2 # [ 1.057464] input: gpio-keys as /devices/platform/gpio-keys/input/input0 lb # [ 1.071006] clk: Disabling unused clocks worker1 # [ 1.050074] Demotion targets for Node 0: null worker1 # [ 1.050220] Key type .fscrypt registered lb # [ 1.071043] PM: genpd: Disabling unused power domains worker1 # [ 1.050226] Key type fscrypt-provisioning registered worker1 # [ 1.050339] ima: No TPM chip found, activating TPM-bypass! worker1 # [ 1.050361] ima: Allocated hash algorithm: sha1 worker1 # [ 1.050386] ima: No architecture policies found lb # [ 1.075365] Freeing unused kernel memory: 4736K lb # [ 1.075728] Run /init as init process worker1 # [ 1.054627] input: gpio-keys as /devices/platform/gpio-keys/input/input0 worker2 # [ 1.077375] clk: Disabling unused clocks lb # [ 1.091391] systemd[1]: Successfully made /usr/ read-only. worker2 # [ 1.077420] PM: genpd: Disabling unused power domains worker2 # [ 1.081798] Freeing unused kernel memory: 4736K worker2 # [ 1.082026] Run /init as init process worker1 # [ 1.074112] clk: Disabling unused clocks worker1 # [ 1.074145] PM: genpd: Disabling unused power domains worker1 # [ 1.078465] Freeing unused kernel memory: 4736K worker1 # [ 1.078674] Run /init as init process worker2 # [ 1.098093] systemd[1]: Successfully made /usr/ read-only. worker1 # [ 1.096181] systemd[1]: Successfully made /usr/ read-only. client # [ 1.258023] usb 1-1: new high-speed USB device number 2 using ehci-pci lb # [ 1.268778] usb 1-1: new high-speed USB device number 2 using ehci-pci worker2 # [ 1.259581] usb 1-1: new high-speed USB device number 2 using ehci-pci worker1 # [ 1.273789] usb 1-1: new high-speed USB device number 2 using ehci-pci client # [ 1.414875] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 client # [ 1.428328] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 1.440820] systemd[1]: Detected virtualization qemu. client # [ 1.443115] systemd[1]: Detected architecture arm64. client # [ 1.445081] systemd[1]: Running in initrd. client # [ 1.448032] systemd[1]: Initializing machine ID from random generator. client # [ 1.451202] systemd[1]: Hostname set to . lb # [ 1.421571] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker2 # [ 1.412295] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 lb # [ 1.428307] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 1.441303] systemd[1]: Detected virtualization qemu. lb # [ 1.441425] systemd[1]: Detected architecture arm64. lb # [ 1.441449] systemd[1]: Running in initrd. lb # [ 1.442662] systemd[1]: Initializing machine ID from random generator. lb # [ 1.442975] systemd[1]: Hostname set to . worker1 # [ 1.428456] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker2 # [ 1.433203] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 1.445768] systemd[1]: Detected virtualization qemu. worker2 # [ 1.448048] systemd[1]: Detected architecture arm64. worker2 # [ 1.450009] systemd[1]: Running in initrd. worker2 # [ 1.452858] systemd[1]: Initializing machine ID from random generator. worker2 # [ 1.456123] systemd[1]: Hostname set to . worker1 # [ 1.435019] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 1.510292] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 worker1 # [ 1.435080] systemd[1]: Detected virtualization qemu. worker1 # [ 1.435178] systemd[1]: Detected architecture arm64. worker1 # [ 1.435204] systemd[1]: Running in initrd. worker1 # [ 1.436415] systemd[1]: Initializing machine ID from random generator. worker1 # [ 1.436750] systemd[1]: Hostname set to . lb # [ 1.513076] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 worker2 # [ 1.507838] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 worker1 # [ 1.518067] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:07.0-1/input0 client # [ 1.629947] usb 1-2: new high-speed USB device number 3 using ehci-pci lb # [ 1.636749] usb 1-2: new high-speed USB device number 3 using ehci-pci worker2 # [ 1.627581] usb 1-2: new high-speed USB device number 3 using ehci-pci worker1 # [ 1.641739] usb 1-2: new high-speed USB device number 3 using ehci-pci client # [ 1.774521] systemd[1]: bpf-restrict-fs: LSM BPF program attached client # [ 1.786099] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 client # [ 1.794057] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 lb # [ 1.783938] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 1.799439] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.791635] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 1.805341] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker2 # [ 1.794892] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.795314] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker1 # [ 1.798238] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:07.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker1 # [ 1.804038] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:07.0-2/input0 worker1 # [ 1.817453] systemd[1]: bpf-restrict-fs: LSM BPF program attached client # [ 1.884776] systemd[1]: Queued start job for default target Initrd Default Target. client # [ 1.897460] systemd[1]: Created slice Slice /system/modprobe. client # [ 1.898908] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 1.900321] systemd[1]: Expecting device /dev/disk/by-label/nixos... client # [ 1.901454] systemd[1]: Reached target Path Units. client # [ 1.902379] systemd[1]: Reached target Slice Units. client # [ 1.903284] systemd[1]: Reached target Swaps. client # [ 1.904064] systemd[1]: Reached target Timer Units. client # [ 1.905169] systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 1.906562] systemd[1]: Listening on Journal Socket (/dev/log). client # [ 1.906726] systemd[1]: Listening on Journal Sockets. client # [ 1.906909] systemd[1]: Listening on udev Control Socket. client # [ 1.907065] systemd[1]: Listening on udev Kernel Socket. client # [ 1.907093] systemd[1]: Reached target Socket Units. client # [ 1.913795] systemd[1]: Starting Create List of Static Device Nodes... client # [ 1.915089] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 1.923645] systemd[1]: Mounting Kernel Configuration File System... client # [ 1.935202] systemd[1]: Starting Journal Service... lb # [ 1.907609] systemd[1]: Queued start job for default target Initrd Default Target. worker2 # [ 1.905979] systemd[1]: Queued start job for default target Initrd Default Target. lb # [ 1.919448] systemd[1]: Created slice Slice /system/modprobe. lb # [ 1.920805] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 1.920851] systemd[1]: Expecting device /dev/disk/by-label/nixos... lb # [ 1.920881] systemd[1]: Reached target Path Units. lb # [ 1.920902] systemd[1]: Reached target Slice Units. lb # [ 1.920923] systemd[1]: Reached target Swaps. lb # [ 1.920943] systemd[1]: Reached target Timer Units. lb # [ 1.921141] systemd[1]: Listening on D-Bus System Message Bus Socket. lb # [ 1.921305] systemd[1]: Listening on Journal Socket (/dev/log). lb # [ 1.921485] systemd[1]: Listening on Journal Sockets. worker2 # [ 1.918881] systemd[1]: Created slice Slice /system/modprobe. client # [ 1.970528] systemd[1]: Starting Load Kernel Modules... lb # [ 1.921680] systemd[1]: Listening on udev Control Socket. client # [ 1.970681] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 1.910034] systemd[1]: Queued start job for default target Initrd Default Target. lb # [ 1.921829] systemd[1]: Listening on udev Kernel Socket. worker2 # [ 1.920395] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 1.921855] systemd[1]: Reached target Socket Units. worker2 # [ 1.921929] systemd[1]: Expecting device /dev/disk/by-label/nixos... worker2 # [ 1.923189] systemd[1]: Reached target Path Units. lb # [ 1.934808] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 1.924196] systemd[1]: Reached target Slice Units. worker2 # [ 1.925144] systemd[1]: Reached target Swaps. lb # [ 1.936012] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 1.926036] systemd[1]: Reached target Timer Units. worker2 # [ 1.927244] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 1.928754] systemd[1]: Listening on Journal Socket (/dev/log). worker2 # [ 1.930054] systemd[1]: Listening on Journal Sockets. worker1 # [ 1.920242] systemd[1]: Created slice Slice /system/modprobe. worker2 # [ 1.931201] systemd[1]: Listening on udev Control Socket. worker1 # [ 1.921380] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 1.922672] systemd[1]: Expecting device /dev/disk/by-label/nixos... lb # [ 1.944944] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 1.923654] systemd[1]: Reached target Path Units. worker2 # [ 1.932524] systemd[1]: Listening on udev Kernel Socket. worker1 # [ 1.924423] systemd[1]: Reached target Slice Units. worker2 # [ 1.932575] systemd[1]: Reached target Socket Units. worker1 # [ 1.925188] systemd[1]: Reached target Swaps. worker1 # [ 1.925892] systemd[1]: Reached target Timer Units. worker1 # [ 1.926827] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 1.936474] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 1.927964] systemd[1]: Listening on Journal Socket (/dev/log). worker1 # [ 1.929058] systemd[1]: Listening on Journal Sockets. worker2 # [ 1.936579] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 1.930025] systemd[1]: Listening on udev Control Socket. worker1 # [ 1.930172] systemd[1]: Listening on udev Kernel Socket. worker1 # [ 1.930228] systemd[1]: Reached target Socket Units. worker1 # [ 1.934354] systemd[1]: Starting Create List of Static Device Nodes... lb # [ 1.957033] systemd[1]: Starting Journal Service... worker1 # [ 1.935399] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 2.000912] systemd-journald[72]: Collecting audit messages is disabled. worker2 # [ 1.947758] systemd[1]: Mounting Kernel Configuration File System... client # [ 2.002074] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 1.942921] systemd[1]: Mounting Kernel Configuration File System... client # [ 2.007412] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 1.955858] systemd[1]: Starting Journal Service... worker1 # [ 1.954170] systemd[1]: Starting Journal Service... lb # [ 1.978388] systemd[1]: Starting Load Kernel Modules... lb # [ 1.980086] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 2.019609] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. client # [ 2.022453] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 2.025252] systemd[1]: Mounted Kernel Configuration File System. lb # [ 1.985169] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 1.991838] systemd[1]: Starting Load Kernel Modules... client # [ 2.037994] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker2 # [ 1.991989] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 2.044679] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 lb # [ 2.007132] systemd[1]: Finished Create List of Static Device Nodes. client # [ 2.044984] [drm] features: -virgl +edid -resource_blob -host_visible client # [ 2.045000] [drm] features: -context_init worker1 # [ 1.986049] systemd[1]: Starting Load Kernel Modules... client # [ 2.045769] [drm] number of scanouts: 1 client # [ 2.045791] [drm] number of cap sets: 0 worker1 # [ 1.986967] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 2.011994] systemd-journald[72]: Collecting audit messages is disabled. lb # [ 2.013822] systemd[1]: Mounted Kernel Configuration File System. lb # [ 2.025213] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker2 # [ 2.015733] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 2.024688] systemd-journald[72]: Collecting audit messages is disabled. client # [ 2.070937] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic client # [ 2.070965] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 worker2 # [ 2.028705] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 2.029529] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 2.017882] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 2.019876] systemd-journald[72]: Collecting audit messages is disabled. worker1 # [ 2.024361] systemd[1]: Finished Create List of Static Device Nodes. worker1 # [ 2.026224] systemd[1]: Mounted Kernel Configuration File System. worker2 # [ 2.045002] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker2 # [ 2.047697] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker1 # [ 2.040755] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. client # [ 2.099413] Console: switching to colour frame buffer device 160x50 client # [ 2.110735] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 2.113502] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 2.055601] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker2 # [ 2.062071] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 worker2 # [ 2.062344] [drm] features: -virgl +edid -resource_blob -host_visible worker2 # [ 2.062354] [drm] features: -context_init worker2 # [ 2.063067] [drm] number of scanouts: 1 worker2 # [ 2.063086] [drm] number of cap sets: 0 worker1 # [ 2.054197] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker1 # [ 2.058773] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev client # [ 2.127270] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device lb # [ 2.085565] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker1 # [ 2.075158] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 worker1 # [ 2.075402] [drm] features: -virgl +edid -resource_blob -host_visible worker1 # [ 2.075412] [drm] features: -context_init worker1 # [ 2.076153] [drm] number of scanouts: 1 worker1 # [ 2.076171] [drm] number of cap sets: 0 lb # [ 2.101481] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 2.103952] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev client # [ 2.150637] systemd[1]: Finished Load Kernel Modules. lb # [ 2.111729] [drm] pci: virtio-gpu-pci detected at 0000:00:05.0 lb # [ 2.111973] [drm] features: -virgl +edid -resource_blob -host_visible worker2 # [ 2.095913] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic lb # [ 2.111984] [drm] features: -context_init worker2 # [ 2.095939] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 lb # [ 2.115274] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 2.095324] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic worker1 # [ 2.095351] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 client # [ 2.158477] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 2.112268] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 2.115934] systemd[1]: Starting Create Static Device Nodes in /dev... lb # [ 2.136805] [drm] number of scanouts: 1 lb # [ 2.136844] [drm] number of cap sets: 0 client # [ 2.176519] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 2.119914] Console: switching to colour frame buffer device 160x50 client # [ 2.176740] systemd[1]: Reached target Preparation for Local File Systems. client # [ 2.176804] systemd[1]: Reached target Local File Systems. worker1 # [ 2.122481] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 2.139336] virtio-pci 0000:00:05.0: [drm] Registered 1 planes with drm panic lb # [ 2.139361] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:05.0 on minor 0 client # [ 2.186273] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker1 # [ 2.126116] Console: switching to colour frame buffer device 160x50 worker1 # [ 2.138148] systemd[1]: Starting Create Static Device Nodes in /dev... client # [ 2.188433] systemd-modules-load[73]: Inserted module 'dm_mod' worker2 # [ 2.152273] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device client # [ 2.189637] systemd-modules-load[73]: Module 'virtio_balloon' is built in client # [ 2.190775] systemd-modules-load[73]: Module 'virtio_console' is built in client # [ 2.191874] systemd-modules-load[73]: Inserted module 'virtio_gpu' lb # [ 2.169076] Console: switching to colour frame buffer device 160x50 client # [ 2.214206] systemd[1]: Started Journal Service. worker2 # [ 2.165080] systemd[1]: Finished Load Kernel Modules. worker1 # [ 2.154487] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device lb # [ 2.180124] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 2.180348] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 2.180373] systemd[1]: Reached target Local File Systems. worker2 # [ 2.172046] systemd[1]: Starting Apply Kernel Variables... client # [ 2.209410] systemd-modules-load[73]: Module 'virtio_rng' is built in worker1 # [ 2.167364] systemd[1]: Finished Load Kernel Modules. lb # [ 2.189365] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker1 # [ 2.174288] systemd[1]: Starting Apply Kernel Variables... lb # [ 2.197440] virtio-pci 0000:00:05.0: [drm] fb0: virtio_gpudrmfb frame buffer device client # [ 2.226066] systemd[1]: Finished Apply Kernel Variables. worker2 # [ 2.192302] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 2.195082] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 2.211858] systemd[1]: Finished Load Kernel Modules. client # [ 2.240132] systemd[1]: Starting Create System Files and Directories... worker2 # [ 2.203869] systemd[1]: Reached target Local File Systems. lb # [ 2.217176] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 2.212022] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 2.216352] systemd-modules-load[73]: Inserted module 'dm_mod' worker1 # [ 2.210152] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 2.219033] systemd-modules-load[73]: Module 'virtio_balloon' is built in worker1 # [ 2.210646] systemd[1]: Reached target Preparation for Local File Systems. worker2 # [ 2.209524] systemd-modules-load[73]: Inserted module 'dm_mod' worker1 # [ 2.210709] systemd[1]: Reached target Local File Systems. worker2 # [ 2.210673] systemd-modules-load[73]: Module 'virtio_balloon' is built in worker1 # [ 2.202512] systemd-modules-load[73]: Inserted module 'dm_mod' worker1 # [ 2.203847] systemd-modules-load[73]: Module 'virtio_balloon' is built in lb # [ 2.224403] systemd-modules-load[73]: Module 'virtio_console' is built in lb # [ 2.225559] systemd-modules-load[73]: Inserted module 'virtio_gpu' worker1 # [ 2.219178] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 2.226543] systemd-modules-load[73]: Module 'virtio_rng' is built in worker1 # [ 2.221647] systemd[1]: Started Journal Service. worker2 # [ 2.219675] systemd-modules-load[73]: Module 'virtio_console' is built in lb # [ 2.244826] systemd[1]: Started Journal Service. worker2 # [ 2.236559] systemd[1]: Started Journal Service. client # [ 2.272962] systemd[1]: Finished Create System Files and Directories. worker2 # [ 2.224482] systemd-modules-load[73]: Inserted module 'virtio_gpu' worker1 # [ 2.216583] systemd-modules-load[73]: Module 'virtio_console' is built in worker2 # [ 2.226093] systemd-modules-load[73]: Module 'virtio_rng' is built in client # [ 2.275658] systemd-udevd[80]: Using default interface naming scheme 'v261'. worker1 # [ 2.217747] systemd-modules-load[73]: Inserted module 'virtio_gpu' worker1 # [ 2.218826] systemd-modules-load[73]: Module 'virtio_rng' is built in worker2 # [ 2.236443] systemd[1]: Starting Create System Files and Directories... lb # [ 2.254268] systemd[1]: Starting Create System Files and Directories... worker1 # [ 2.244204] systemd[1]: Starting Create System Files and Directories... worker2 # [ 2.253904] systemd[1]: Finished Apply Kernel Variables. client # [ 2.305699] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 2.248383] systemd[1]: Finished Apply Kernel Variables. lb # [ 2.290043] systemd-udevd[78]: Using default interface naming scheme 'v261'. lb # [ 2.292177] systemd[1]: Finished Apply Kernel Variables. worker2 # [ 2.285438] systemd[1]: Finished Create System Files and Directories. worker1 # [ 2.285692] systemd[1]: Finished Create System Files and Directories. worker2 # [ 2.298169] systemd-udevd[80]: Using default interface naming scheme 'v261'. lb # [ 2.313130] systemd[1]: Finished Create System Files and Directories. client # [ 2.370818] systemd[1]: Starting Virtual Console Setup... worker1 # [ 2.313845] systemd-udevd[80]: Using default interface naming scheme 'v261'. lb # [ 2.333411] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 2.330109] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 2.345049] systemd[1]: Started Rule-based Manager for Device Events and Files. client # [ 2.424687] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 2.428227] systemd[1]: Finished Virtual Console Setup. lb # [ 2.400188] systemd[1]: Starting Virtual Console Setup... worker2 # [ 2.396785] systemd[1]: Starting Virtual Console Setup... worker1 # [ 2.414684] systemd[1]: Starting Virtual Console Setup... lb # [ 2.460660] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 2.464563] systemd[1]: Finished Virtual Console Setup. worker2 # [ 2.460580] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 2.463461] systemd[1]: Finished Virtual Console Setup. worker1 # [ 2.476581] systemd-vconsole-setup[101]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 2.480298] systemd[1]: Finished Virtual Console Setup. client # [ 3.080458] systemd[1]: Finished Coldplug All udev Devices. client # [ 3.081530] systemd[1]: Reached target System Initialization. client # [ 3.084129] systemd[1]: Reached target Basic System. lb # [ 3.098496] systemd[1]: Finished Coldplug All udev Devices. lb # [ 3.099512] systemd[1]: Reached target System Initialization. lb # [ 3.100413] systemd[1]: Reached target Basic System. worker2 # [ 3.101240] systemd[1]: Finished Coldplug All udev Devices. worker2 # [ 3.103002] systemd[1]: Reached target System Initialization. worker2 # [ 3.103856] systemd[1]: Reached target Basic System. worker1 # [ 3.097211] systemd[1]: Finished Coldplug All udev Devices. worker1 # [ 3.098976] systemd[1]: Reached target System Initialization. worker1 # [ 3.099869] systemd[1]: Reached target Basic System. client # [ 3.253498] (udev-worker)[90]: Network interface NamePolicy= disabled on kernel command line. client # [ 3.279679] (udev-worker)[104]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 3.284212] (udev-worker)[104]: Network interface NamePolicy= disabled on kernel command line. lb # [ 3.256728] (udev-worker)[92]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. lb # [ 3.294698] (udev-worker)[91]: Network interface NamePolicy= disabled on kernel command line. lb # [ 3.300489] (udev-worker)[92]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 3.290446] (udev-worker)[96]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 3.285668] (udev-worker)[94]: Network interface NamePolicy= disabled on kernel command line. client # [ 3.345132] systemd[1]: Found device /dev/disk/by-label/nixos. client # [ 3.346487] systemd[1]: Reached target Initrd Root Device. client # [ 3.350744] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... worker2 # [ 3.307106] (udev-worker)[96]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 3.302698] (udev-worker)[93]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker2 # [ 3.316695] (udev-worker)[94]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 3.309469] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line. client # [ 3.411506] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks client # [ 3.420342] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. client # [ 3.424746] systemd[1]: Mounting /sysroot... lb # [ 3.388908] systemd[1]: Found device /dev/disk/by-label/nixos. lb # [ 3.390791] systemd[1]: Reached target Initrd Root Device. lb # [ 3.395504] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... worker1 # [ 3.382047] systemd[1]: Found device /dev/disk/by-label/nixos. worker1 # [ 3.385417] systemd[1]: Reached target Initrd Root Device. worker1 # [ 3.392477] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... client # [ 3.485485] EXT4-fs (vda): mounted filesystem a62f6ea9-c1fc-4c8f-81b0-c7a81bf4e5dd r/w with ordered data mode. Quota mode: none. worker2 # [ 3.421875] systemd[1]: Found device /dev/disk/by-label/nixos. client # [ 3.474060] systemd[1]: Mounted /sysroot. worker2 # [ 3.424676] systemd[1]: Reached target Initrd Root Device. client # [ 3.476557] systemd[1]: Reached target Initrd Root File System. worker2 # [ 3.428884] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... client # [ 3.480494] systemd[1]: Starting Mountpoints Configured in the Real Root... lb # [ 3.460638] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks worker1 # [ 3.447628] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks lb # [ 3.468156] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. client # [ 3.508447] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. lb # [ 3.472682] systemd[1]: Mounting /sysroot... worker1 # [ 3.453818] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. client # [ 3.515890] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... client # [ 3.520122] systemd[1]: Reloading... worker1 # [ 3.467000] systemd[1]: Mounting /sysroot... worker2 # [ 3.482417] systemd-fsck[110]: nixos: clean, 12/65536 files, 13019/262144 blocks worker2 # [ 3.491112] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker2 # [ 3.494543] systemd[1]: Mounting /sysroot... lb # [ 3.532629] EXT4-fs (vda): mounted filesystem 1cd19ffa-09ce-40ed-92b0-dba9e25172a2 r/w with ordered data mode. Quota mode: none. worker1 # [ 3.516201] EXT4-fs (vda): mounted filesystem 6e4ea723-c219-4b7b-8934-6eb51079b2c2 r/w with ordered data mode. Quota mode: none. lb # [ 3.528220] systemd[1]: Mounted /sysroot. worker1 # [ 3.509714] systemd[1]: Mounted /sysroot. lb # [ 3.529214] systemd[1]: Reached target Initrd Root File System. worker1 # [ 3.511900] systemd[1]: Reached target Initrd Root File System. lb # [ 3.532499] systemd[1]: Starting Mountpoints Configured in the Real Root... worker1 # [ 3.516722] systemd[1]: Starting Mountpoints Configured in the Real Root... worker2 # [ 3.553167] EXT4-fs (vda): mounted filesystem 66c3ca7b-51d7-4602-8e39-346aa8580ad6 r/w with ordered data mode. Quota mode: none. worker2 # [ 3.543452] systemd[1]: Mounted /sysroot. worker2 # [ 3.544760] systemd[1]: Reached target Initrd Root File System. worker2 # [ 3.548462] systemd[1]: Starting Mountpoints Configured in the Real Root... lb # [ 3.564383] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. lb # [ 3.569471] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... lb # [ 3.573190] systemd[1]: Reloading... worker1 # [ 3.553232] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. worker1 # [ 3.559638] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... worker1 # [ 3.566036] systemd[1]: Reloading... worker2 # [ 3.579148] systemd-sysroot-fstab-check[118]: /sysroot should be mounted in the initrd, will request daemon-reload. worker2 # [ 3.586917] systemd[1]: Reload requested from client PID 118 ('systemd-sysroot') (unit initrd-parse-etc.service)... worker2 # [ 3.592004] systemd[1]: Reloading... client # [ 3.754397] systemd[1]: Reloading finished in 236 ms. client # [ 3.789892] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... client # [ 3.794762] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... client # [ 3.803850] systemd[1]: initrd-parse-etc.service: Deactivated successfully. client # [ 3.806265] systemd[1]: Finished Mountpoints Configured in the Real Root. client # [ 3.808834] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. lb # [ 3.810771] systemd[1]: Reloading finished in 239 ms. worker1 # [ 3.803909] systemd[1]: Reloading finished in 241 ms. worker2 # [ 3.819559] systemd[1]: Reloading finished in 233 ms. lb # [ 3.844973] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... lb # [ 3.850503] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... lb # [ 3.855628] systemd[1]: initrd-parse-etc.service: Deactivated successfully. lb # [ 3.859367] systemd[1]: Finished Mountpoints Configured in the Real Root. worker1 # [ 3.842314] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... lb # [ 3.861612] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. worker1 # [ 3.848252] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... worker2 # [ 3.857358] systemd-sysroot-fstab-check[118]: Requesting initrd-fs.target/start/replace... worker2 # [ 3.862056] systemd-sysroot-fstab-check[118]: Requesting swap.target/start/replace... worker1 # [ 3.856269] systemd[1]: initrd-parse-etc.service: Deactivated successfully. worker1 # [ 3.858724] systemd[1]: Finished Mountpoints Configured in the Real Root. worker1 # [ 3.861389] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. worker2 # [ 3.871277] systemd[1]: initrd-parse-etc.service: Deactivated successfully. worker2 # [ 3.876362] systemd[1]: Finished Mountpoints Configured in the Real Root. worker2 # [ 3.877503] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. client # [ 3.994375] systemd[1]: Mounting /sysroot/nix/.ro-store... client # [ 4.012848] systemd[1]: Mounting /sysroot/nix/.rw-store... lb # [ 3.974922] systemd[1]: Mounting /sysroot/nix/.ro-store... client # [ 4.022727] systemd[1]: Mounting /sysroot/run... lb # [ 3.985500] systemd[1]: Mounting /sysroot/nix/.rw-store... lb # [ 4.000243] systemd[1]: Mounting /sysroot/run... client # [ 4.057419] systemd[1]: Mounting /sysroot/tmp/shared... worker2 # [ 4.011373] systemd[1]: Mounting /sysroot/nix/.ro-store... lb # [ 4.022122] systemd[1]: Mounting /sysroot/tmp/shared... client # [ 4.070205] systemd[1]: Mounting /sysroot/tmp/xchg... worker2 # [ 4.024101] systemd[1]: Mounting /sysroot/nix/.rw-store... worker1 # [ 4.016238] systemd[1]: Mounting /sysroot/nix/.ro-store... lb # [ 4.042511] systemd[1]: Mounting /sysroot/tmp/xchg... worker2 # [ 4.036227] systemd[1]: Mounting /sysroot/run... worker1 # [ 4.027624] systemd[1]: Mounting /sysroot/nix/.rw-store... worker2 # [ 4.049902] systemd[1]: Mounting /sysroot/tmp/shared... worker1 # [ 4.042803] systemd[1]: Mounting /sysroot/run... client # [ 4.130466] fuse: init (API version 7.45) worker1 # [ 4.059317] systemd[1]: Mounting /sysroot/tmp/shared... client # [ 4.141461] virtiofs virtio6: discovered new tag: nix-store client # [ 4.127421] systemd[1]: Mounted /sysroot/nix/.rw-store. worker2 # [ 4.078463] systemd[1]: Mounting /sysroot/tmp/xchg... client # [ 4.152672] virtiofs virtio6: virtio_fs_setup_dax: No cache capability lb # [ 4.099332] systemd[1]: Mounted /sysroot/nix/.rw-store. lb # [ 4.119299] fuse: init (API version 7.45) client # [ 4.167777] virtiofs virtio7: discovered new tag: shared client # [ 4.168601] virtiofs virtio7: virtio_fs_setup_dax: No cache capability lb # [ 4.129351] virtiofs virtio6: discovered new tag: nix-store lb # [ 4.130259] virtiofs virtio6: virtio_fs_setup_dax: No cache capability client # [ 4.176462] virtiofs virtio8: discovered new tag: xchg client # [ 4.177245] virtiofs virtio8: virtio_fs_setup_dax: No cache capability worker1 # [ 4.105433] systemd[1]: Mounting /sysroot/tmp/xchg... lb # [ 4.146640] virtiofs virtio7: discovered new tag: shared lb # [ 4.147497] virtiofs virtio7: virtio_fs_setup_dax: No cache capability client # [ 4.174463] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 4.156108] virtiofs virtio8: discovered new tag: xchg client # [ 4.188107] systemd[1]: Mounted /sysroot/nix/.ro-store. lb # [ 4.166376] virtiofs virtio8: virtio_fs_setup_dax: No cache capability client # [ 4.194466] systemd[1]: Mounted /sysroot/run. lb # [ 4.157568] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 4.197174] systemd[1]: Mounted /sysroot/tmp/shared. lb # [ 4.172668] systemd[1]: Mounted /sysroot/run. client # [ 4.211876] systemd[1]: Mounted /sysroot/tmp/xchg. worker2 # [ 4.177092] fuse: init (API version 7.45) worker1 # [ 4.155339] systemd[1]: Mounted /sysroot/nix/.rw-store. lb # [ 4.178475] systemd[1]: Mounted /sysroot/nix/.ro-store. lb # [ 4.182748] systemd[1]: Mounted /sysroot/tmp/shared. lb # [ 4.186789] systemd[1]: Mounted /sysroot/tmp/xchg. worker2 # [ 4.190742] virtiofs virtio6: discovered new tag: nix-store worker1 # [ 4.182756] fuse: init (API version 7.45) worker1 # [ 4.173626] systemd[1]: Mounted /sysroot/run. client # [ 4.234439] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 4.236569] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 4.187121] systemd[1]: Mounted /sysroot/nix/.rw-store. worker2 # [ 4.204182] virtiofs virtio6: virtio_fs_setup_dax: No cache capability worker2 # [ 4.191974] systemd[1]: Mounted /sysroot/run. worker1 # [ 4.183683] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 4.208820] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 4.209962] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 4.221268] virtiofs virtio7: discovered new tag: shared worker2 # [ 4.222093] virtiofs virtio7: virtio_fs_setup_dax: No cache capability worker2 # [ 4.210691] systemd[1]: Starting rw-sysroot-nix-store.service... worker2 # [ 4.233555] virtiofs virtio8: discovered new tag: xchg worker2 # [ 4.234454] virtiofs virtio8: virtio_fs_setup_dax: No cache capability worker2 # [ 4.242054] systemd[1]: Mounted /sysroot/nix/.ro-store. worker2 # [ 4.245541] systemd[1]: Mounted /sysroot/tmp/shared. worker2 # [ 4.247212] systemd[1]: Mounted /sysroot/tmp/xchg. worker1 # [ 4.247494] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 4.267472] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker1 # [ 4.261300] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 4.270455] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 4.273331] systemd[1]: Mounting /sysroot/nix/store... worker1 # [ 4.289591] virtiofs virtio6: discovered new tag: nix-store worker1 # [ 4.321260] virtiofs virtio6: virtio_fs_setup_dax: No cache capability worker2 # [ 4.325112] systemd[1]: Mounted /sysroot/nix/store. worker2 # [ 4.327373] systemd[1]: Reached target Initrd File Systems. worker2 # [ 4.329823] systemd[1]: Starting Find NixOS closure... worker1 # [ 4.335549] virtiofs virtio7: discovered new tag: shared worker1 # [ 4.336404] virtiofs virtio7: virtio_fs_setup_dax: No cache capability worker1 # [ 4.341670] virtiofs virtio8: discovered new tag: xchg worker2 # [ 4.341006] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker1 # [ 4.346793] virtiofs virtio8: virtio_fs_setup_dax: No cache capability worker1 # [ 4.346965] systemd[1]: Mounted /sysroot/nix/.ro-store. worker1 # [ 4.350622] systemd[1]: Mounted /sysroot/tmp/shared. worker1 # [ 4.353021] systemd[1]: Mounted /sysroot/tmp/xchg. worker2 # [ 4.390533] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker2 # [ 4.401018] systemd[1]: Finished Find NixOS closure. worker2 # [ 4.405338] systemd[1]: Reached target Initrd Default Target. worker2 # [ 4.415546] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker2 # [ 4.450694] systemd[1]: initrd-cleanup.service: Deactivated successfully. worker2 # [ 4.456253] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker2 # [ 4.458516] systemd[1]: Stopped target Initrd Default Target. worker2 # [ 4.460175] systemd[1]: Stopped target Basic System. worker2 # [ 4.464316] systemd[1]: Stopped target Initrd Root Device. worker2 # [ 4.466113] systemd[1]: Stopped target Path Units. worker2 # [ 4.468154] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker2 # [ 4.471449] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker2 # [ 4.474998] systemd[1]: Stopped target Slice Units. worker2 # [ 4.481413] systemd[1]: Stopped target Socket Units. worker2 # [ 4.483759] systemd[1]: Stopped target System Initialization. worker2 # [ 4.484921] systemd[1]: Stopped target Swaps. worker2 # [ 4.486654] systemd[1]: Stopped target Timer Units. worker2 # [ 4.489791] systemd[1]: dbus.socket: Deactivated successfully. worker2 # [ 4.491469] systemd[1]: Closed D-Bus System Message Bus Socket. worker2 # [ 4.494023] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker2 # [ 4.495191] systemd[1]: Stopped Find NixOS closure. worker2 # [ 4.495918] systemd[1]: Starting rw-sysroot-nix-store.service... worker2 # [ 4.504307] systemd[1]: systemd-sysctl.service: Deactivated successfully. worker2 # [ 4.505390] systemd[1]: Stopped Apply Kernel Variables. worker2 # [ 4.512364] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker2 # [ 4.516630] systemd[1]: Stopped Load Kernel Modules. worker2 # [ 4.517883] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker2 # [ 4.520119] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker2 # [ 4.524381] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker2 # [ 4.525460] systemd[1]: Stopped Create System Files and Directories. worker2 # [ 4.528344] systemd[1]: Stopped target Local File Systems. worker2 # [ 4.529237] systemd[1]: Stopped target Preparation for Local File Systems. worker2 # [ 4.531228] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. worker2 # [ 4.532423] systemd[1]: Stopped Coldplug All udev Devices. worker2 # [ 4.533194] systemd[1]: Stopping Rule-based Manager for Device Events and Files... worker2 # [ 4.534180] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 4.535191] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 4.535895] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 4.542107] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 4.543988] systemd[1]: systemd-udevd.service: Deactivated successfully. worker2 # [ 4.549396] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker2 # [ 4.550417] systemd[1]: systemd-udevd.service: Consumed 1.052s CPU time over 2.343s wall clock time, 16M memory peak. worker2 # [ 4.551834] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. worker2 # [ 4.553613] systemd[1]: Closed udev Control Socket. worker2 # [ 4.554361] systemd[1]: Starting Cleanup udev Database... worker2 # [ 4.555153] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. worker2 # [ 4.557579] systemd[1]: Stopped Create Static Device Nodes in /dev. worker2 # [ 4.558472] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker2 # [ 4.559589] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. worker2 # [ 4.565433] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker2 # [ 4.566646] systemd[1]: Stopped Create List of Static Device Nodes. worker2 # [ 4.591475] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. worker2 # [ 4.594852] systemd[1]: Finished Cleanup udev Database. worker2 # [ 4.595720] systemd[1]: Reached target Switch Root. worker2 # [ 4.597234] systemd[1]: Starting NixOS Activation... worker2 # [ 4.688354] initrd-nixos-activation-start[170]: booting system configuration /nix/store/8rxzyh0zqd97xa57p8wqswwb0drnznwf-nixos-system-worker2-test worker2 # [ 4.725226] initrd-nixos-activation-start[170]: running activation script... client # [ 4.820924] (udev-worker)[99]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory client # [ 4.828243] (udev-worker)[99]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory client # [ 4.868134] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 4.869384] systemd[1]: Stopped Virtual Console Setup. client # [ 4.872207] systemd[1]: Stopping Virtual Console Setup... client # [ 4.873429] systemd[1]: Starting Virtual Console Setup... lb # [ 4.832808] (udev-worker)[104]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory client # [ 4.888742] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 4.848414] (udev-worker)[104]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory client # [ 4.892241] systemd[1]: Stopped Virtual Console Setup. client # [ 4.894813] systemd[1]: Starting Virtual Console Setup... lb # [ 4.876096] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 4.879449] systemd[1]: Stopped Virtual Console Setup. lb # [ 4.885052] systemd[1]: Stopping Virtual Console Setup... lb # [ 4.885898] systemd[1]: Starting Virtual Console Setup... client # [ 4.927442] systemd-vconsole-setup[152]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 4.931442] systemd[1]: Finished Virtual Console Setup. lb # [ 4.898394] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 4.899549] systemd[1]: Stopped Virtual Console Setup. lb # [ 4.904325] systemd[1]: Starting Virtual Console Setup... lb # [ 4.924710] systemd-vconsole-setup[152]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 4.928490] systemd[1]: Finished Virtual Console Setup. worker1 # [ 4.918303] (udev-worker)[98]: mtd0ro: Failed to find and pin callout binary "/nix/store/gnadg5slifq2r638xvz4pv6vrsdl9xkj-systemd-261.2/lib/udev/mtd_probe": No such file or directory worker1 # [ 4.924811] (udev-worker)[98]: mtd0ro: /etc/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory client # [ 4.992590] systemd[1]: Mounting /sysroot/nix/store... lb # [ 4.974049] systemd[1]: Mounting /sysroot/nix/store... worker1 # [ 4.971838] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 4.973795] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 4.975864] systemd[1]: Stopping Virtual Console Setup... worker1 # [ 4.977253] systemd[1]: Starting Virtual Console Setup... client # [ 5.037983] systemd[1]: Mounted /sysroot/nix/store. client # [ 5.040218] systemd[1]: Reached target Initrd File Systems. client # [ 5.042892] systemd[1]: Starting Find NixOS closure... client # [ 5.052433] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... lb # [ 5.017450] systemd[1]: Mounted /sysroot/nix/store. lb # [ 5.019456] systemd[1]: Reached target Initrd File Systems. lb # [ 5.022712] systemd[1]: Starting Find NixOS closure... worker1 # [ 5.006506] systemd-vconsole-setup[156]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 5.029653] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker1 # [ 5.012315] systemd[1]: Finished Virtual Console Setup. worker2 # [ 5.021716] initrd-nixos-activation-start[193]: setting up /etc... worker1 # [ 5.013201] systemd[1]: sysroot-run-credentials-systemd\x2dvconsole\x2dsetup.service.mount: Deactivated successfully. worker1 # [ 5.016958] systemd[1]: Mounting /sysroot/nix/store... client # [ 5.087187] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. client # [ 5.088719] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. client # [ 5.099673] systemd[1]: Finished Find NixOS closure. client # [ 5.102271] systemd[1]: Reached target Initrd Default Target. client # [ 5.103564] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker1 # [ 5.052161] systemd[1]: Mounted /sysroot/nix/store. lb # [ 5.070747] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker1 # [ 5.053034] systemd[1]: Reached target Initrd File Systems. worker1 # [ 5.056265] systemd[1]: Starting Find NixOS closure... lb # [ 5.076768] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. worker1 # [ 5.063089] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... lb # [ 5.084547] systemd[1]: Finished Find NixOS closure. lb # [ 5.087076] systemd[1]: Reached target Initrd Default Target. lb # [ 5.088695] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... client # [ 5.130126] systemd[1]: Stopped target Initrd Default Target. client # [ 5.131874] systemd[1]: Stopped target Basic System. client # [ 5.133317] systemd[1]: Stopped target Initrd Root Device. client # [ 5.136415] systemd[1]: Stopped target Path Units. client # [ 5.137317] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. client # [ 5.139748] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. client # [ 5.142394] systemd[1]: Stopped target Slice Units. client # [ 5.144315] systemd[1]: Stopped target Socket Units. client # [ 5.148174] systemd[1]: Stopped target System Initialization. client # [ 5.149152] systemd[1]: Stopped target Swaps. client # [ 5.149802] systemd[1]: Stopped target Timer Units. client # [ 5.150502] systemd[1]: dbus.socket: Deactivated successfully. worker1 # [ 5.093336] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. client # [ 5.153262] systemd[1]: Closed D-Bus System Message Bus Socket. lb # [ 5.114981] systemd[1]: Stopped target Initrd Default Target. client # [ 5.156857] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. lb # [ 5.120241] systemd[1]: Stopped target Basic System. lb # [ 5.121144] systemd[1]: Stopped target Initrd Root Device. client # [ 5.161080] systemd[1]: Stopped Find NixOS closure. lb # [ 5.121979] systemd[1]: Stopped target Path Units. lb # [ 5.122704] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker1 # [ 5.103853] systemd[1]: Finished Find NixOS closure. client # [ 5.163367] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 5.125023] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker1 # [ 5.106796] systemd[1]: Reached target Initrd Default Target. worker1 # [ 5.108151] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... client # [ 5.167506] systemd[1]: systemd-sysctl.service: Deactivated successfully. lb # [ 5.128828] systemd[1]: Stopped target Slice Units. lb # [ 5.131543] systemd[1]: Stopped target Socket Units. lb # [ 5.136974] systemd[1]: Stopped target System Initialization. client # [ 5.177107] systemd[1]: Stopped Apply Kernel Variables. lb # [ 5.137937] systemd[1]: Stopped target Swaps. lb # [ 5.138589] systemd[1]: Stopped target Timer Units. client # [ 5.178056] systemd[1]: systemd-modules-load.service: Deactivated successfully. lb # [ 5.139310] systemd[1]: dbus.socket: Deactivated successfully. client # [ 5.179086] systemd[1]: Stopped Load Kernel Modules. client # [ 5.179809] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. lb # [ 5.148211] systemd[1]: Closed D-Bus System Message Bus Socket. lb # [ 5.149080] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. lb # [ 5.150128] systemd[1]: Stopped Find NixOS closure. client # [ 5.189366] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. lb # [ 5.150873] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 5.151726] systemd[1]: systemd-sysctl.service: Deactivated successfully. worker1 # [ 5.133354] systemd[1]: Stopped target Initrd Default Target. worker1 # [ 5.135082] systemd[1]: Stopped target Basic System. worker1 # [ 5.136210] systemd[1]: Stopped target Initrd Root Device. client # [ 5.194921] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. client # [ 5.196264] systemd[1]: Stopped Create System Files and Directories. client # [ 5.197201] systemd[1]: Stopped target Local File Systems. lb # [ 5.158892] systemd[1]: Stopped Apply Kernel Variables. worker1 # [ 5.140473] systemd[1]: Stopped target Path Units. lb # [ 5.160346] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker1 # [ 5.141471] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. client # [ 5.200476] systemd[1]: Stopped target Preparation for Local File Systems. worker1 # [ 5.142569] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. client # [ 5.201543] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. lb # [ 5.162625] systemd[1]: Stopped Load Kernel Modules. worker1 # [ 5.144710] systemd[1]: Stopped target Slice Units. lb # [ 5.164292] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. client # [ 5.203930] systemd[1]: Stopped Coldplug All udev Devices. worker1 # [ 5.148193] systemd[1]: Stopped target Socket Units. worker1 # [ 5.149023] systemd[1]: Stopped target System Initialization. lb # [ 5.168274] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker1 # [ 5.149846] systemd[1]: Stopped target Swaps. client # [ 5.208132] systemd[1]: Stopping Rule-based Manager for Device Events and Files... lb # [ 5.169469] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. client # [ 5.209251] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 5.151239] systemd[1]: Stopped target Timer Units. worker2 # [ 5.159604] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. client # [ 5.210253] systemd[1]: Stopped Virtual Console Setup. lb # [ 5.170821] systemd[1]: Stopped Create System Files and Directories. client # [ 5.210963] systemd[1]: initrd-cleanup.service: Deactivated successfully. lb # [ 5.171713] systemd[1]: Stopped target Local File Systems. worker1 # [ 5.153027] systemd[1]: dbus.socket: Deactivated successfully. client # [ 5.211893] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker2 # [ 5.162976] systemd[1]: Finished NixOS Activation. client # [ 5.213588] systemd[1]: systemd-udevd.service: Deactivated successfully. worker2 # [ 5.164320] systemd[1]: Starting Switch Root... client # [ 5.214561] systemd[1]: Stopped Rule-based Manager for Device Events and Files. client # [ 5.215560] systemd[1]: systemd-udevd.service: Consumed 1.579s CPU time over 3.020s wall clock time, 22.1M memory peak. worker1 # [ 5.158200] systemd[1]: Closed D-Bus System Message Bus Socket. client # [ 5.217438] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. client # [ 5.218452] systemd[1]: Closed udev Control Socket. client # [ 5.219160] systemd[1]: Starting Cleanup udev Database... client # [ 5.219926] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. worker1 # [ 5.163086] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker1 # [ 5.165339] systemd[1]: Stopped Find NixOS closure. lb # [ 5.186094] systemd[1]: Stopped target Preparation for Local File Systems. lb # [ 5.187142] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. lb # [ 5.188589] systemd[1]: Stopped Coldplug All udev Devices. client # [ 5.228268] systemd[1]: Stopped Create Static Device Nodes in /dev. client # [ 5.229273] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. lb # [ 5.190225] systemd[1]: Stopping Rule-based Manager for Device Events and Files... lb # [ 5.191308] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 5.192494] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 5.173887] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 5.193259] systemd[1]: initrd-cleanup.service: Deactivated successfully. client # [ 5.232649] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. lb # [ 5.194172] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. lb # [ 5.195076] systemd[1]: systemd-udevd.service: Deactivated successfully. worker1 # [ 5.176471] systemd[1]: systemd-sysctl.service: Deactivated successfully. client # [ 5.236250] systemd[1]: kmod-static-nodes.service: Deactivated successfully. client # [ 5.237329] systemd[1]: Stopped Create List of Static Device Nodes. worker1 # [ 5.180233] systemd[1]: Stopped Apply Kernel Variables. worker2 # [ 5.189795] systemd[1]: Switching root. client # [ 5.240177] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker1 # [ 5.182155] systemd[1]: systemd-modules-load.service: Deactivated successfully. client # [ 5.241229] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 5.195989] systemd[1]: Stopped Rule-based Manager for Device Events and Files. lb # [ 5.204145] systemd[1]: systemd-udevd.service: Consumed 1.568s CPU time over 2.989s wall clock time, 22.4M memory peak. worker1 # [ 5.188075] systemd[1]: Stopped Load Kernel Modules. lb # [ 5.208388] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. lb # [ 5.209802] systemd[1]: Closed udev Control Socket. worker1 # [ 5.191061] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. lb # [ 5.212717] systemd[1]: Starting Cleanup udev Database... worker1 # [ 5.194042] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker1 # [ 5.196195] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. lb # [ 5.216235] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. worker1 # [ 5.198408] systemd[1]: Stopped Create System Files and Directories. lb # [ 5.218339] systemd[1]: Stopped Create Static Device Nodes in /dev. worker1 # [ 5.199852] systemd[1]: Stopped target Local File Systems. lb # [ 5.220235] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker1 # [ 5.202609] systemd[1]: Stopped target Preparation for Local File Systems. worker1 # [ 5.203845] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. lb # [ 5.224310] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. lb # [ 5.225388] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker1 # [ 5.208365] systemd[1]: Stopped Coldplug All udev Devices. lb # [ 5.228234] systemd[1]: Stopped Create List of Static Device Nodes. worker1 # [ 5.209203] systemd[1]: Stopping Rule-based Manager for Device Events and Files... lb # [ 5.229190] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker1 # [ 5.210273] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 5.211304] systemd[1]: Stopped Virtual Console Setup. client # [ 5.270077] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. lb # [ 5.232381] systemd[1]: Finished rw-sysroot-nix-store.service. worker1 # [ 5.214379] systemd[1]: initrd-cleanup.service: Deactivated successfully. client # [ 5.273134] systemd[1]: Finished Cleanup udev Database. worker1 # [ 5.215510] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. client # [ 5.274935] systemd[1]: Reached target Switch Root. worker1 # [ 5.216630] systemd[1]: systemd-udevd.service: Deactivated successfully. worker1 # [ 5.217570] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker1 # [ 5.218584] systemd[1]: systemd-udevd.service: Consumed 1.624s CPU time over 2.988s wall clock time, 22.1M memory peak. client # [ 5.280197] systemd[1]: Starting NixOS Activation... worker1 # [ 5.219989] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker1 # [ 5.225245] systemd[1]: Finished rw-sysroot-nix-store.service. worker1 # [ 5.226151] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. worker1 # [ 5.232311] systemd[1]: Closed udev Control Socket. worker1 # [ 5.233124] systemd[1]: Starting Cleanup udev Database... worker1 # [ 5.233896] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. worker1 # [ 5.234989] systemd[1]: Stopped Create Static Device Nodes in /dev. lb # [ 5.254652] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. worker1 # [ 5.236132] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. lb # [ 5.258191] systemd[1]: Finished Cleanup udev Database. lb # [ 5.259032] systemd[1]: Reached target Switch Root. worker1 # [ 5.240170] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. lb # [ 5.260454] systemd[1]: Starting NixOS Activation... worker1 # [ 5.241257] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker1 # [ 5.244131] systemd[1]: Stopped Create List of Static Device Nodes. worker1 # [ 5.270452] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. worker1 # [ 5.274918] systemd[1]: Finished Cleanup udev Database. worker1 # [ 5.275759] systemd[1]: Reached target Switch Root. worker1 # [ 5.280309] systemd[1]: Starting NixOS Activation... client # [ 5.367714] initrd-nixos-activation-start[177]: booting system configuration /nix/store/5fx3y1hbxhkdng69ksrxcn3js8649kk7-nixos-system-client-test lb # [ 5.353970] initrd-nixos-activation-start[176]: booting system configuration /nix/store/kgjzg78npi7z07am2szqxkr41f06jp5w-nixos-system-lb-test client # [ 5.404593] initrd-nixos-activation-start[177]: running activation script... worker2 # [ 5.372010] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). worker1 # [ 5.367074] initrd-nixos-activation-start[179]: booting system configuration /nix/store/3j2yqs8i8ynzphhqwmkrm3aah3k3l7j5-nixos-system-worker1-test lb # [ 5.394875] initrd-nixos-activation-start[176]: running activation script... worker1 # [ 5.403849] initrd-nixos-activation-start[179]: running activation script... client # [ 5.681778] initrd-nixos-activation-start[200]: setting up /etc... lb # [ 5.650599] initrd-nixos-activation-start[199]: setting up /etc... worker1 # [ 5.689675] initrd-nixos-activation-start[202]: setting up /etc... client # [ 5.813864] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. client # [ 5.816250] systemd[1]: Finished NixOS Activation. client # [ 5.818335] systemd[1]: Starting Switch Root... lb # [ 5.781576] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. lb # [ 5.784233] systemd[1]: Finished NixOS Activation. lb # [ 5.785988] systemd[1]: Starting Switch Root... client # [ 5.841308] systemd[1]: Switching root. lb # [ 5.809073] systemd[1]: Switching root. worker1 # [ 5.828424] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. worker1 # [ 5.831696] systemd[1]: Finished NixOS Activation. worker1 # [ 5.833067] systemd[1]: Starting Switch Root... worker1 # [ 5.857257] systemd[1]: Switching root. worker2 # [ 5.934729] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 5.947218] systemd[1]: Detected virtualization qemu. worker2 # [ 5.950530] systemd[1]: Detected architecture arm64. worker2 # [ 5.954638] systemd[1]: Detected first boot. worker2 # [ 5.960914] systemd[1]: Initializing machine ID from random generator. client # [ 6.033597] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). lb # [ 6.000022] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). worker1 # [ 6.045647] systemd-journald[72]: Received SIGTERM from PID 1 (systemd). worker2 # [ 6.310448] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 6.514014] systemd[1]: Applying preset policy. lb # [ 6.558693] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 6.599023] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 6.612098] systemd[1]: Detected virtualization qemu. lb # [ 6.571260] systemd[1]: Detected virtualization qemu. lb # [ 6.575577] systemd[1]: Detected architecture arm64. client # [ 6.615418] systemd[1]: Detected architecture arm64. lb # [ 6.577802] systemd[1]: Detected first boot. client # [ 6.619581] systemd[1]: Detected first boot. lb # [ 6.583416] systemd[1]: Initializing machine ID from random generator. client # [ 6.626967] systemd[1]: Initializing machine ID from random generator. worker1 # [ 6.597175] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker1 # [ 6.610365] systemd[1]: Detected virtualization qemu. worker1 # [ 6.614430] systemd[1]: Detected architecture arm64. worker1 # [ 6.616594] systemd[1]: Detected first boot. worker1 # [ 6.621902] systemd[1]: Initializing machine ID from random generator. worker2 # [ 6.777401] systemd[1]: Populated /etc with preset unit settings. lb # [ 6.906467] systemd[1]: bpf-restrict-fs: LSM BPF program attached client # [ 6.967301] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker1 # [ 6.969363] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 7.028431] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker2 # [ 7.030169] systemd[1]: Stopped initrd-switch-root.service. worker2 # [ 7.033737] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. worker2 # [ 7.038482] systemd[1]: Created slice Slice /system/getty. worker2 # [ 7.040988] systemd[1]: Created slice User and Session Slice. worker2 # [ 7.042312] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker2 # [ 7.044733] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker2 # [ 7.046718] systemd[1]: Expecting device /dev/hvc0... worker2 # [ 7.048597] systemd[1]: Expecting device /dev/ttyAMA0... worker2 # [ 7.050583] systemd[1]: Reached target Local Encrypted Volumes. worker2 # [ 7.052542] systemd[1]: Stopped target initrd-fs.target. worker2 # [ 7.054203] systemd[1]: Stopped target initrd-root-fs.target. worker2 # [ 7.056211] systemd[1]: Stopped target initrd-switch-root.target. worker2 # [ 7.058349] systemd[1]: Reached target Virtual Machines and Containers. worker2 # [ 7.060664] systemd[1]: Reached target Path Units. worker2 # [ 7.062676] systemd[1]: Reached target Remote File Systems. worker2 # [ 7.064824] systemd[1]: Reached target Slice Units. worker2 # [ 7.066820] systemd[1]: Reached target Swaps. worker2 # [ 7.070457] systemd[1]: Listening on Query the User Interactively for a Password. worker2 # [ 7.073780] systemd[1]: Listening on Process Core Dump Socket. worker2 # [ 7.076224] systemd[1]: Listening on Credential Encryption/Decryption. worker2 # [ 7.078748] systemd[1]: Listening on Factory Reset Management. worker2 # [ 7.080039] systemd[1]: Listening on Hostname Service Socket. worker2 # [ 7.084443] systemd[1]: Starting Journal Log Access Socket... lb # [ 7.097984] systemd[1]: Applying preset policy. worker2 # [ 7.086863] systemd[1]: Listening on Journal Audit Socket. worker2 # [ 7.090939] systemd[1]: Listening on Console Output Muting Service Socket. worker2 # [ 7.092598] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. worker2 # [ 7.095650] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 7.098590] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker2 # [ 7.104815] systemd[1]: Listening on Disk Repartitioning Service Socket. worker2 # [ 7.106244] systemd[1]: Listening on udev Control Socket. worker2 # [ 7.108038] systemd[1]: Listening on udev Varlink Socket. worker2 # [ 7.112942] systemd[1]: Mounting Huge Pages File System... worker2 # [ 7.121247] systemd[1]: Mounting POSIX Message Queue File System... worker2 # [ 7.125898] systemd[1]: Mounting Kernel Debug File System... worker2 # [ 7.142139] systemd[1]: Mounting Kernel Trace File System... client # [ 7.197845] systemd[1]: Applying preset policy. worker2 # [ 7.155818] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 7.156251] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 7.176307] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 7.176732] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm worker2 # [ 7.177078] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker2 # [ 7.177436] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker1 # [ 7.192760] systemd[1]: Applying preset policy. worker2 # [ 7.208236] systemd[1]: Mounting FUSE Control File System... worker2 # [ 7.208728] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker2 # [ 7.248748] systemd[1]: Starting Journal Service... worker2 # [ 7.273866] systemd[1]: Starting Load Kernel Modules... worker2 # [ 7.291250] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker2 # [ 7.297226] systemd[1]: Starting Remount Root and Kernel File Systems... worker2 # [ 7.299139] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 7.308735] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 7.314178] systemd[1]: Listening on Journal Log Access Socket. worker2 # [ 7.319954] systemd[1]: Mounted Huge Pages File System. worker2 # [ 7.323173] systemd[1]: Mounted POSIX Message Queue File System. worker2 # [ 7.327364] systemd[1]: Mounted Kernel Debug File System. worker2 # [ 7.331774] systemd[1]: Mounted Kernel Trace File System. worker2 # [ 7.334594] systemd[1]: Mounted Kernel Configuration File System. worker2 # [ 7.335095] systemd[1]: Mounted FUSE Control File System. worker2 # [ 7.345579] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 7.359015] systemd[1]: Populated /etc with preset unit settings. worker2 # [ 7.354372] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 7.450844] systemd[1]: Populated /etc with preset unit settings. worker2 # [ 7.442537] systemd[1]: Finished Load Kernel Modules. worker2 # [ 7.449443] systemd[1]: Starting Firewall... worker1 # [ 7.441957] systemd[1]: Populated /etc with preset unit settings. worker2 # [ 7.464556] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 7.472137] systemd-journald[264]: Collecting audit messages is enabled. worker2 # [ 7.491899] EXT4-fs (vda): re-mounted 66c3ca7b-51d7-4602-8e39-346aa8580ad6. worker2 # [ 7.509886] systemd[1]: Finished Remount Root and Kernel File Systems. worker2 # [ 7.512882] systemd[1]: Listening on Disk Image Download Service Socket. worker2 # [ 7.515338] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 7.524914] systemd[1]: Starting Load/Save OS Random Seed... worker2 # [ 7.527595] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 7.533778] systemd[1]: Started Journal Service. worker2 # [ 7.548618] systemd[1]: Queued start job for default target Multi-User System. worker2 # [ 7.550054] systemd[1]: systemd-journald.service: Deactivated successfully. worker2 # [ 7.558264] systemd-modules-load[265]: Module 'atkbd' is built in worker2 # [ 7.573933] systemd-modules-load[265]: Module 'loop' is built in worker2 # [ 7.584934] systemd[1]: Starting Flush Journal to Persistent Storage... lb # [ 7.616564] systemd[1]: initrd-switch-root.service: Deactivated successfully. lb # [ 7.618492] systemd[1]: Stopped initrd-switch-root.service. lb # [ 7.623333] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. lb # [ 7.628397] systemd[1]: Created slice Slice /system/getty. lb # [ 7.630555] systemd[1]: Created slice User and Session Slice. lb # [ 7.631915] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 7.634757] systemd[1]: Started Forward Password Requests to Wall Directory Watch. lb # [ 7.637144] systemd[1]: Expecting device /dev/hvc0... lb # [ 7.639090] systemd[1]: Expecting device /dev/ttyAMA0... lb # [ 7.641215] systemd[1]: Reached target Local Encrypted Volumes. lb # [ 7.643248] systemd[1]: Stopped target initrd-fs.target. lb # [ 7.645264] systemd[1]: Stopped target initrd-root-fs.target. lb # [ 7.647319] systemd[1]: Stopped target initrd-switch-root.target. lb # [ 7.649448] systemd[1]: Reached target Virtual Machines and Containers. lb # [ 7.651578] systemd[1]: Reached target Path Units. lb # [ 7.653613] systemd[1]: Reached target Remote File Systems. lb # [ 7.655640] systemd[1]: Reached target Slice Units. client # [ 7.696820] systemd[1]: initrd-switch-root.service: Deactivated successfully. lb # [ 7.657613] systemd[1]: Reached target Swaps. client # [ 7.698716] systemd[1]: Stopped initrd-switch-root.service. lb # [ 7.661591] systemd[1]: Listening on Query the User Interactively for a Password. client # [ 7.703233] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. lb # [ 7.664965] systemd[1]: Listening on Process Core Dump Socket. client # [ 7.707390] systemd[1]: Created slice Slice /system/getty. worker2 # [ 7.640403] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 7.667406] systemd[1]: Listening on Credential Encryption/Decryption. client # [ 7.709453] systemd[1]: Created slice User and Session Slice. lb # [ 7.669987] systemd[1]: Listening on Factory Reset Management. client # [ 7.711029] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 7.671249] systemd[1]: Listening on Hostname Service Socket. client # [ 7.712882] systemd[1]: Started Forward Password Requests to Wall Directory Watch. client # [ 7.714790] systemd[1]: Expecting device /dev/hvc0... worker2 # [ 7.647787] systemd[1]: Starting Create Static Device Nodes in /dev... lb # [ 7.676020] systemd[1]: Starting Journal Log Access Socket... client # [ 7.716436] systemd[1]: Expecting device /dev/ttyAMA0... client # [ 7.718102] systemd[1]: Reached target Local Encrypted Volumes. lb # [ 7.678406] systemd[1]: Listening on Journal Audit Socket. client # [ 7.719795] systemd[1]: Stopped target initrd-fs.target. client # [ 7.721366] systemd[1]: Stopped target initrd-root-fs.target. lb # [ 7.682523] systemd[1]: Listening on Console Output Muting Service Socket. client # [ 7.723133] systemd[1]: Stopped target initrd-switch-root.target. lb # [ 7.684114] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. client # [ 7.723500] systemd[1]: Reached target Virtual Machines and Containers. client # [ 7.723857] systemd[1]: Reached target Path Units. client # [ 7.724201] systemd[1]: Reached target Remote File Systems. client # [ 7.724523] systemd[1]: Reached target Slice Units. client # [ 7.724843] systemd[1]: Reached target Swaps. lb # [ 7.686034] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os lb # [ 7.689693] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki client # [ 7.730953] systemd[1]: Listening on Query the User Interactively for a Password. client # [ 7.734209] systemd[1]: Listening on Process Core Dump Socket. worker2 # [ 7.667605] systemd-oomd[266]: No swap; memory pressure usage will be degraded client # [ 7.736627] systemd[1]: Listening on Credential Encryption/Decryption. lb # [ 7.696006] systemd[1]: Listening on Disk Repartitioning Service Socket. lb # [ 7.697528] systemd[1]: Listening on udev Control Socket. client # [ 7.739155] systemd[1]: Listening on Factory Reset Management. lb # [ 7.699299] systemd[1]: Listening on udev Varlink Socket. client # [ 7.740366] systemd[1]: Listening on Hostname Service Socket. lb # [ 7.703199] systemd[1]: Mounting Huge Pages File System... client # [ 7.744946] systemd[1]: Starting Journal Log Access Socket... lb # [ 7.706704] systemd[1]: Mounting POSIX Message Queue File System... client # [ 7.747870] systemd[1]: Listening on Journal Audit Socket. worker2 # [ 7.684092] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. client # [ 7.751876] systemd[1]: Listening on Console Output Muting Service Socket. client # [ 7.753505] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. client # [ 7.755231] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os lb # [ 7.717415] systemd[1]: Mounting Kernel Debug File System... worker1 # [ 7.695746] systemd[1]: initrd-switch-root.service: Deactivated successfully. client # [ 7.759044] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker2 # [ 7.693511] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 7.697478] systemd[1]: Stopped initrd-switch-root.service. client # [ 7.765627] systemd[1]: Listening on Disk Repartitioning Service Socket. worker1 # [ 7.702455] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. client # [ 7.767129] systemd[1]: Listening on udev Control Socket. client # [ 7.768774] systemd[1]: Listening on udev Varlink Socket. worker1 # [ 7.706901] systemd[1]: Created slice Slice /system/getty. worker2 # [ 7.702954] systemd[1]: Finished Load/Save OS Random Seed. worker2 # [ 7.703867] systemd[1]: Reached target First Boot Complete. worker1 # [ 7.709121] systemd[1]: Created slice User and Session Slice. client # [ 7.772733] systemd[1]: Mounting Huge Pages File System... lb # [ 7.731965] systemd[1]: Mounting Kernel Trace File System... worker1 # [ 7.711483] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 7.776810] systemd[1]: Mounting POSIX Message Queue File System... worker1 # [ 7.714019] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker1 # [ 7.716328] systemd[1]: Expecting device /dev/hvc0... worker1 # [ 7.718343] systemd[1]: Expecting device /dev/ttyAMA0... lb # [ 7.741534] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 7.720310] systemd[1]: Reached target Local Encrypted Volumes. worker1 # [ 7.722521] systemd[1]: Stopped target initrd-fs.target. worker1 # [ 7.723567] systemd[1]: Stopped target initrd-root-fs.target. client # [ 7.787190] systemd[1]: Mounting Kernel Debug File System... worker1 # [ 7.726205] systemd[1]: Stopped target initrd-switch-root.target. lb # [ 7.748952] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 7.728271] systemd[1]: Reached target Virtual Machines and Containers. worker1 # [ 7.730488] systemd[1]: Reached target Path Units. worker2 # [ 7.741962] systemd-journald[264]: Received client request to flush runtime journal. worker1 # [ 7.732420] systemd[1]: Reached target Remote File Systems. worker1 # [ 7.734515] systemd[1]: Reached target Slice Units. worker1 # [ 7.736467] systemd[1]: Reached target Swaps. worker1 # [ 7.740146] systemd[1]: Listening on Query the User Interactively for a Password. client # [ 7.803359] systemd[1]: Mounting Kernel Trace File System... worker1 # [ 7.743421] systemd[1]: Listening on Process Core Dump Socket. lb # [ 7.764920] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 7.745875] systemd[1]: Listening on Credential Encryption/Decryption. worker1 # [ 7.748414] systemd[1]: Listening on Factory Reset Management. lb # [ 7.769129] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm worker1 # [ 7.749671] systemd[1]: Listening on Hostname Service Socket. client # [ 7.815526] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 7.754295] systemd[1]: Starting Journal Log Access Socket... lb # [ 7.777793] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker1 # [ 7.756588] systemd[1]: Listening on Journal Audit Socket. client # [ 7.821324] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 7.760773] systemd[1]: Listening on Console Output Muting Service Socket. worker1 # [ 7.762665] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. worker1 # [ 7.764925] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 7.766719] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki lb # [ 7.789231] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker1 # [ 7.773443] systemd[1]: Listening on Disk Repartitioning Service Socket. worker1 # [ 7.774914] systemd[1]: Listening on udev Control Socket. worker1 # [ 7.776720] systemd[1]: Listening on udev Varlink Socket. client # [ 7.842447] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 7.780956] systemd[1]: Mounting Huge Pages File System... client # [ 7.844929] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm lb # [ 7.808275] systemd[1]: Mounting FUSE Control File System... lb # [ 7.809771] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 client # [ 7.850974] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore client # [ 7.854041] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker1 # [ 7.793832] systemd[1]: Mounting POSIX Message Queue File System... worker1 # [ 7.798573] systemd[1]: Mounting Kernel Debug File System... worker1 # [ 7.809879] systemd[1]: Mounting Kernel Trace File System... worker2 # [ 7.815988] systemd[1]: Finished Flush Journal to Persistent Storage. worker1 # [ 7.820895] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 7.822844] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 7.887849] systemd[1]: Mounting FUSE Control File System... client # [ 7.891834] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 lb # [ 7.852096] systemd[1]: Starting Journal Service... worker1 # [ 7.841503] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 7.842862] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm worker1 # [ 7.844514] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore lb # [ 7.866557] systemd[1]: Starting Load Kernel Modules... worker1 # [ 7.846724] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse client # [ 7.914488] systemd[1]: Starting Journal Service... worker2 # [ 7.855484] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 7.857488] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 7.886246] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker2 # [ 7.861242] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker1 # [ 7.866469] systemd[1]: Mounting FUSE Control File System... client # [ 7.931885] systemd[1]: Starting Load Kernel Modules... worker1 # [ 7.866914] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 lb # [ 7.905010] systemd[1]: Starting Remount Root and Kernel File Systems... lb # [ 7.906455] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 7.898095] systemd[1]: Starting Journal Service... client # [ 7.962660] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... lb # [ 7.930888] systemd[1]: Starting Coldplug All udev Devices... client # [ 7.980718] systemd[1]: Starting Remount Root and Kernel File Systems... lb # [ 7.943446] systemd[1]: Listening on Journal Log Access Socket. worker1 # [ 7.930898] systemd[1]: Starting Load Kernel Modules... lb # [ 7.955179] systemd[1]: Mounted Huge Pages File System. client # [ 7.995466] systemd-journald[270]: Collecting audit messages is enabled. lb # [ 7.962361] systemd[1]: Mounted POSIX Message Queue File System. client # [ 8.003351] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 7.991883] systemd[1]: Queued start job for default target Multi-User System. client # [ 7.994045] systemd[1]: systemd-journald.service: Deactivated successfully. worker1 # [ 7.948952] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... lb # [ 7.971675] systemd-journald[269]: Collecting audit messages is enabled. lb # [ 7.979462] systemd[1]: Mounted Kernel Debug File System. lb # [ 7.981015] systemd[1]: Mounted Kernel Trace File System. worker1 # [ 7.958072] systemd[1]: Starting Remount Root and Kernel File Systems... client # [ 8.006284] systemd-modules-load[271]: Module 'atkbd' is built in worker1 # [ 7.958540] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 7.985036] systemd[1]: Finished Create List of Static Device Nodes. client # [ 8.026490] systemd[1]: Starting Coldplug All udev Devices... lb # [ 7.986951] systemd[1]: Mounted Kernel Configuration File System. client # [ 8.033228] systemd[1]: Started Journal Service. lb # [ 7.975101] systemd[1]: Queued start job for default target Multi-User System. worker1 # [ 7.971053] systemd[1]: Starting Coldplug All udev Devices... client # [ 8.016697] systemd-modules-load[271]: Module 'loop' is built in lb # [ 7.994151] systemd[1]: Started Journal Service. worker1 # [ 7.976077] systemd[1]: Listening on Journal Log Access Socket. lb # [ 7.984705] systemd[1]: systemd-journald.service: Deactivated successfully. worker1 # [ 7.980619] systemd[1]: Mounted Huge Pages File System. lb # [ 7.989449] systemd[1]: Mounted FUSE Control File System. worker1 # [ 7.984155] systemd[1]: Mounted POSIX Message Queue File System. client # [ 8.031387] systemd[1]: Listening on Journal Log Access Socket. worker1 # [ 7.986602] systemd[1]: Mounted Kernel Debug File System. worker1 # [ 7.990121] systemd[1]: Mounted Kernel Trace File System. worker1 # [ 7.992871] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 7.995837] systemd[1]: Mounted FUSE Control File System. client # [ 8.042178] systemd[1]: Mounted Huge Pages File System. client # [ 8.043097] systemd[1]: Mounted POSIX Message Queue File System. client # [ 8.043946] systemd[1]: Mounted Kernel Debug File System. lb # [ 8.009180] systemd-modules-load[270]: Module 'atkbd' is built in client # [ 8.051752] systemd[1]: Mounted Kernel Trace File System. lb # [ 8.014780] systemd-modules-load[270]: Module 'loop' is built in client # [ 8.061760] systemd[1]: Finished Create List of Static Device Nodes. client # [ 8.062828] systemd[1]: Finished Load Kernel Modules. worker1 # [ 8.017240] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 8.025219] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker2 # [ 8.016634] systemd[1]: Mounting /run/wrappers... worker1 # [ 8.023189] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 8.031549] systemd[1]: Finished Load Kernel Modules. lb # [ 8.055554] EXT4-fs (vda): re-mounted 1cd19ffa-09ce-40ed-92b0-dba9e25172a2. client # [ 8.103272] EXT4-fs (vda): re-mounted a62f6ea9-c1fc-4c8f-81b0-c7a81bf4e5dd. lb # [ 8.047829] systemd[1]: Starting Firewall... lb # [ 8.060182] systemd[1]: Starting Apply Kernel Variables... client # [ 8.100170] systemd[1]: Mounted Kernel Configuration File System. lb # [ 8.064753] systemd-oomd[271]: No swap; memory pressure usage will be degraded client # [ 8.107192] systemd[1]: Finished Remount Root and Kernel File Systems. client # [ 8.112339] systemd[1]: Listening on Disk Image Download Service Socket. client # [ 8.116699] systemd[1]: Starting Flush Journal to Persistent Storage... client # [ 8.119421] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore lb # [ 8.085461] systemd[1]: Finished Remount Root and Kernel File Systems. lb # [ 8.086612] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. client # [ 8.126432] systemd[1]: Starting Load/Save OS Random Seed... client # [ 8.145200] systemd[1]: Starting Apply Kernel Variables... client # [ 8.146166] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker2 # [ 8.097295] systemd[1]: Mounted /run/wrappers. client # [ 8.147242] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 8.098429] systemd[1]: Reached target Local File Systems. client # [ 8.157755] systemd-oomd[273]: No swap; memory pressure usage will be degraded worker2 # [ 8.109364] systemd[1]: Listening on Boot Loader Control Service Socket. worker1 # [ 8.115221] systemd[1]: Finished Load Kernel Modules. worker2 # [ 8.115660] systemd[1]: Starting register-nix-paths.service... client # [ 8.166317] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. lb # [ 8.127940] systemd[1]: Listening on Disk Image Download Service Socket. worker1 # [ 8.121684] systemd-journald[272]: Collecting audit messages is enabled. worker1 # [ 8.124983] systemd[1]: Starting Firewall... lb # [ 8.138698] systemd[1]: Starting Flush Journal to Persistent Storage... worker2 # [ 8.128933] systemd[1]: Starting Create SUID/SGID Wrappers... lb # [ 8.139858] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker1 # [ 8.138401] systemd[1]: Starting Apply Kernel Variables... worker1 # [ 8.142254] EXT4-fs (vda): re-mounted 6e4ea723-c219-4b7b-8934-6eb51079b2c2. worker2 # [ 8.144411] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. lb # [ 8.159739] systemd[1]: Starting Load/Save OS Random Seed... worker1 # [ 8.156296] systemd[1]: Finished Remount Root and Kernel File Systems. lb # [ 8.165539] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 8.159110] systemd[1]: Listening on Disk Image Download Service Socket. worker2 # [ 8.156358] systemd[1]: Starting Save Transient machine-id to Disk... worker1 # [ 8.161513] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 8.161182] systemd[1]: Starting Create System Files and Directories... worker1 # [ 8.158118] systemd[1]: Queued start job for default target Multi-User System. worker1 # [ 8.173845] systemd[1]: Starting Load/Save OS Random Seed... worker1 # [ 8.177780] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 8.178346] systemd[1]: Started Journal Service. client # [ 8.229503] systemd[1]: Mounted FUSE Control File System. worker1 # [ 8.189559] systemd[1]: systemd-journald.service: Deactivated successfully. client # [ 8.268940] systemd-journald[270]: Received client request to flush runtime journal. worker1 # [ 8.203009] systemd-modules-load[273]: Module 'atkbd' is built in worker1 # [ 8.211615] systemd-modules-load[273]: Module 'loop' is built in worker1 # [ 8.226650] systemd[1]: Starting Flush Journal to Persistent Storage... lb # [ 8.269181] systemd-journald[269]: Received client request to flush runtime journal. client # [ 8.306957] systemd[1]: Finished Load/Save OS Random Seed. client # [ 8.307955] systemd[1]: Reached target First Boot Complete. client # [ 8.316902] systemd[1]: Finished Flush Journal to Persistent Storage. client # [ 8.364520] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 8.322468] systemd-oomd[275]: No swap; memory pressure usage will be degraded lb # [ 8.348853] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 8.350236] systemd[1]: Starting Create Static Device Nodes in /dev... lb # [ 8.351179] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 8.348100] systemd-journald[272]: Received client request to flush runtime journal. lb # [ 8.370111] systemd[1]: Finished Flush Journal to Persistent Storage. worker2 # [ 8.391881] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. lb # [ 8.406355] systemd[1]: Finished Load/Save OS Random Seed. lb # [ 8.407336] systemd[1]: Reached target First Boot Complete. worker2 # [ 8.399820] systemd[1]: Finished Save Transient machine-id to Disk. worker2 # [ 8.428524] systemd[1]: Finished Create System Files and Directories. worker2 # [ 8.436594] systemd[1]: Starting Rebuild Journal Catalog... worker1 # [ 8.430745] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker1 # [ 8.434353] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 8.452300] systemd[1]: Starting Record System Boot/Shutdown in UTMP... worker1 # [ 8.452128] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker1 # [ 8.453771] systemd[1]: Finished Load/Save OS Random Seed. worker1 # [ 8.454623] systemd[1]: Reached target First Boot Complete. worker1 # [ 8.455419] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 8.465709] systemd[1]: Finished Flush Journal to Persistent Storage. lb # [ 8.552463] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 8.556284] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 8.561559] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker1 # [ 8.547439] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 8.556680] systemd-udevd[302]: Using default interface naming scheme 'v261'. worker1 # [ 8.549252] systemd[1]: Reached target Preparation for Local File Systems. client # [ 8.610886] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 8.613231] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 8.556551] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker2 # [ 8.591779] systemd[1]: Finished Record System Boot/Shutdown in UTMP. lb # [ 8.612834] systemd[1]: Mounting /run/wrappers... worker2 # [ 8.668134] systemd[1]: Finished Rebuild Journal Catalog. worker2 # [ 8.676462] systemd[1]: Starting Update is Completed... worker1 # [ 8.679489] systemd-udevd[310]: Using default interface naming scheme 'v261'. lb # [ 8.709110] systemd[1]: Mounted /run/wrappers. worker1 # [ 8.690026] systemd[1]: Mounting /run/wrappers... lb # [ 8.709971] systemd[1]: Reached target Local File Systems. lb # [ 8.718599] systemd[1]: Listening on Boot Loader Control Service Socket. lb # [ 8.727018] systemd[1]: Starting register-nix-paths.service... lb # [ 8.735547] systemd[1]: Starting Create SUID/SGID Wrappers... lb # [ 8.736936] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. lb # [ 8.743636] systemd[1]: Starting Save Transient machine-id to Disk... lb # [ 8.757422] systemd[1]: Starting Create System Files and Directories... lb # [ 8.763785] systemd-udevd[307]: Using default interface naming scheme 'v261'. worker1 # [ 8.757318] systemd[1]: Mounted /run/wrappers. worker1 # [ 8.758162] systemd[1]: Reached target Local File Systems. worker1 # [ 8.770838] systemd[1]: Listening on Boot Loader Control Service Socket. worker1 # [ 8.782497] systemd[1]: Starting register-nix-paths.service... worker2 # [ 8.798654] systemd[1]: Finished Update is Completed. worker1 # [ 8.796871] systemd[1]: Starting Create SUID/SGID Wrappers... worker1 # [ 8.800412] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker1 # [ 8.808673] systemd[1]: Starting Save Transient machine-id to Disk... worker1 # [ 8.816379] systemd[1]: Starting Create System Files and Directories... worker2 # [ 8.857656] systemd[1]: Started Rule-based Manager for Device Events and Files. client # [ 8.959704] systemd[1]: Finished Coldplug All udev Devices. client # [ 8.969205] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 8.971755] systemd[1]: Reached target Preparation for Local File Systems. client # [ 8.976190] systemd[1]: Mounting /run/wrappers... client # [ 8.979547] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker1 # [ 8.943101] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 8.963550] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker1 # [ 8.956735] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker1 # [ 8.957912] systemd[1]: Finished Save Transient machine-id to Disk. lb # [ 8.977952] systemd[1]: Finished Save Transient machine-id to Disk. client # [ 9.027733] systemd[1]: Mounted /run/wrappers. client # [ 9.029285] systemd[1]: Reached target Local File Systems. client # [ 9.032271] systemd[1]: Listening on Boot Loader Control Service Socket. client # [ 9.036218] systemd[1]: Starting register-nix-paths.service... client # [ 9.047738] systemd[1]: Starting Create SUID/SGID Wrappers... client # [ 9.048948] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. client # [ 9.057255] systemd[1]: Starting Save Transient machine-id to Disk... client # [ 9.070177] systemd[1]: Starting Create System Files and Directories... client # [ 9.079730] systemd-udevd[301]: Using default interface naming scheme 'v261'. lb # [ 9.060464] systemd[1]: Started Rule-based Manager for Device Events and Files. lb # [ 9.108834] systemd[1]: Finished Create System Files and Directories. worker1 # [ 9.098684] systemd[1]: Finished Create System Files and Directories. worker1 # [ 9.105796] systemd[1]: Starting Rebuild Journal Catalog... lb # [ 9.129594] systemd[1]: Starting Rebuild Journal Catalog... client # [ 9.176816] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker1 # [ 9.120926] systemd[1]: Starting Record System Boot/Shutdown in UTMP... lb # [ 9.146590] systemd[1]: Starting Record System Boot/Shutdown in UTMP... client # [ 9.190355] systemd[1]: Finished Save Transient machine-id to Disk. client # [ 9.253587] systemd[1]: Started Rule-based Manager for Device Events and Files. client # [ 9.258140] systemd[1]: Finished Create System Files and Directories. client # [ 9.272276] systemd[1]: Starting Rebuild Journal Catalog... client # [ 9.275649] systemd[1]: Starting Record System Boot/Shutdown in UTMP... worker1 # [ 9.236524] systemd[1]: Finished Record System Boot/Shutdown in UTMP. lb # [ 9.302942] systemd[1]: Finished Record System Boot/Shutdown in UTMP. worker1 # [ 9.312135] systemd[1]: Finished Rebuild Journal Catalog. worker1 # [ 9.317008] systemd[1]: Starting Update is Completed... lb # [ 9.343236] systemd[1]: Finished Rebuild Journal Catalog. client # [ 9.388530] systemd[1]: Finished Record System Boot/Shutdown in UTMP. lb # [ 9.353112] systemd[1]: Starting Update is Completed... client # [ 9.426901] systemd[1]: Finished Rebuild Journal Catalog. worker2 # [ 9.382001] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker2 # [ 9.385413] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 9.437707] systemd[1]: Starting Update is Completed... lb # [ 9.444926] systemd[1]: Finished Update is Completed. worker1 # [ 9.432430] systemd[1]: Finished Update is Completed. client # [ 9.504720] systemd[1]: Finished Update is Completed. client # [ 9.794979] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 9.847300] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker2 # [ 9.812452] systemd[1]: Finished Firewall. worker1 # [ 9.892472] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker1 # [ 9.897191] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 10.001714] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. client # [ 10.039644] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. client # [ 10.043453] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 10.058089] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. lb # [ 10.168850] systemd[1]: Finished Coldplug All udev Devices. client # [ 10.282200] (udev-worker)[349]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 10.290151] (udev-worker)[349]: Network interface NamePolicy= disabled on kernel command line. client # [ 10.291828] (udev-worker)[354]: Network interface NamePolicy= disabled on kernel command line. lb # [ 10.270063] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 10.265760] systemd[1]: Finished Coldplug All udev Devices. lb # [ 10.285177] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse worker2 # [ 10.301981] systemd[1]: Finished register-nix-paths.service. worker2 # [ 10.304291] systemd[1]: Reached target System Initialization. worker2 # [ 10.308441] systemd[1]: Started Discard unused filesystem blocks once a week. worker2 # [ 10.309488] systemd[1]: Started Daily Cleanup of Temporary Directories. worker2 # [ 10.312227] systemd[1]: Reached target Timer Units. worker2 # [ 10.316516] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 10.320645] systemd[1]: Listening on Nix Daemon Socket. lb # [ 10.333253] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. lb # [ 10.339703] systemd[1]: Finished Create SUID/SGID Wrappers. worker2 # [ 10.332270] systemd[1]: Listening on Nix worker-protocol over gRPC. worker2 # [ 10.336522] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. worker1 # [ 10.328543] systemd[1]: Finished Firewall. worker2 # [ 10.348442] systemd[1]: Reached target Socket Units. worker2 # [ 10.360119] systemd[1]: Reached target Basic System. worker2 # [ 10.365454] systemd[1]: Starting Import lastlog data into lastlog2 database... lb # [ 10.375866] systemd[1]: Finished register-nix-paths.service. lb # [ 10.379783] systemd[1]: Reached target System Initialization. worker2 # [ 10.374404] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 10.385462] systemd[1]: Started Discard unused filesystem blocks once a week. lb # [ 10.386515] systemd[1]: Started niks3 garbage collection timer. lb # [ 10.387381] systemd[1]: Started Daily Cleanup of Temporary Directories. lb # [ 10.391764] systemd[1]: Reached target Timer Units. worker2 # [ 10.382924] systemd[1]: Starting Post-Boot Actions... lb # [ 10.394874] systemd[1]: Listening on D-Bus System Message Bus Socket. lb # [ 10.400710] systemd[1]: Listening on niks3 server socket. lb # [ 10.401687] systemd[1]: Listening on Nix Daemon Socket. worker2 # [ 10.392159] systemd[1]: Started Reset console on configuration changes. lb # [ 10.402479] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. lb # [ 10.403649] systemd[1]: Reached target Socket Units. lb # [ 10.412112] systemd[1]: Reached target Basic System. worker2 # [ 10.402584] systemd[1]: Starting resolvconf update... lb # [ 10.412902] systemd[1]: Starting Import lastlog data into lastlog2 database... lb # [ 10.419898] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker2 # [ 10.411306] systemd[1]: Starting D-Bus System Message Bus... lb # [ 10.432238] systemd[1]: Starting Post-Boot Actions... lb # [ 10.452614] systemd[1]: Started Reset console on configuration changes. worker2 # [ 10.465740] systemd[1]: Finished Post-Boot Actions. lb # [ 10.500157] systemd[1]: Starting resolvconf update... client # [ 10.549398] systemd[1]: Finished register-nix-paths.service. client # [ 10.552811] systemd[1]: Reached target System Initialization. lb # [ 10.516549] systemd[1]: Started rustfs.service. client # [ 10.557027] systemd[1]: Started Discard unused filesystem blocks once a week. client # [ 10.558131] systemd[1]: Started Daily Cleanup of Temporary Directories. client # [ 10.559050] systemd[1]: Reached target Timer Units. client # [ 10.559771] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 10.512384] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 10.568809] systemd[1]: Listening on Nix Daemon Socket. worker2 # [ 10.521194] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 10.513505] systemd[1]: Finished Coldplug All udev Devices. client # [ 10.573449] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. client # [ 10.575950] systemd[1]: Reached target Socket Units. client # [ 10.580314] systemd[1]: Reached target Basic System. client # [ 10.581113] systemd[1]: Started backdoor.service. worker2 # [ 10.530446] nsncd[482]: Sep 23 05:12:05.018 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 10.581831] systemd[1]: Starting Import lastlog data into lastlog2 database... client # [ 10.589912] systemd[1]: Starting nftables firewall... worker2 # [ 10.545668] systemd[1]: Reached target User and Group Name Lookups. worker2 # [ 10.552647] systemd[1]: Starting User Login Management... client # [ 10.606220] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 10.570472] systemd[1]: Starting rustfs-bucket.service... worker2 # [ 10.566001] systemd[1]: Finished Import lastlog data into lastlog2 database. client # [ 10.625724] systemd[1]: Starting Post-Boot Actions... worker2 # [ 10.590370] dbus-broker-launch[486]: Looking up NSS user entry for 'systemd-timesync'... lb # [ 10.609524] nsncd[445]: Sep 23 05:12:05.117 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 10.654368] systemd[1]: Started Reset console on configuration changes. worker2 # [ 10.603373] dbus-broker-launch[486]: NSS returned no entry for 'systemd-timesync' worker2 # [ 10.613108] dbus-broker-launch[486]: Invalid user-name in /nix/store/flsxa7d757b7zpdiv9wsbs42iawcisak-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker1 # [ 10.605612] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker2 # [ 10.630573] systemd[1]: Started D-Bus System Message Bus. lb # [ 10.641390] systemd[1]: Starting D-Bus System Message Bus... lb # [ 10.642417] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 10.693767] systemd[1]: Starting resolvconf update... worker1 # [ 10.635265] systemd[1]: Load Kernel Module fuse skipped, unmet condition check ConditionKernelModuleLoaded=!fuse lb # [ 10.675041] systemd[1]: Finished Post-Boot Actions. client # connecting to host... worker2 # [ 10.679488] dbus-broker-launch[486]: Ready client # [ 10.761706] systemd[1]: Condition check resulted in Virtio network device being skipped. lb # [ 10.737993] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 10.723206] systemd[1]: Finished register-nix-paths.service. worker1 # [ 10.725227] systemd[1]: Reached target System Initialization. worker1 # [ 10.727244] systemd[1]: Started Discard unused filesystem blocks once a week. worker1 # [ 10.730902] systemd[1]: Started Daily Cleanup of Temporary Directories. worker2 # [ 10.741027] systemd-logind[503]: New seat seat0. lb # [ 10.755412] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 10.739026] systemd[1]: Reached target Timer Units. worker2 # [ 10.748363] systemd[1]: Started User Login Management. worker2 # [ 10.754112] systemd[1]: Starting linger-users.service... worker1 # [ 10.746346] systemd[1]: Listening on D-Bus System Message Bus Socket. worker1 # [ 10.747408] systemd[1]: Listening on Nix Daemon Socket. client # [ 10.809689] nsncd[394]: Sep 23 05:12:05.319 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client: Guest shell says: b'Spawning backdoor root shell...\n' client # [ 10.820471] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 10.829488] systemd[1]: Finished Post-Boot Actions. lb # [ 10.782581] systemd[1]: Starting User Login Management... client # [ 10.836964] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 10.770098] systemd[1]: Listening on Nix worker-protocol over gRPC. lb # [ 10.792387] systemd[1]: Finished Import lastlog data into lastlog2 database. worker2 # [ 10.777783] systemd[1]: Stopped target Host and Network Name Lookups. client # [ 10.837983] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 10.779513] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. lb # [ 10.807327] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker1 # [ 10.784805] systemd[1]: Reached target Socket Units. worker2 # [ 10.785498] systemd[1]: Stopping Host and Network Name Lookups... worker1 # [ 10.791289] systemd[1]: Reached target Basic System. client # [ 10.865173] systemd[1]: Starting D-Bus System Message Bus... client: connected to guest root shell worker2 # [ 10.786420] systemd[1]: Stopped target User and Group Name Lookups. worker1 # [ 10.797252] systemd[1]: Starting Import lastlog data into lastlog2 database... client # [ 10.884006] mousedev: PS/2 mouse device common for all mice client: (connecting took 11.40 seconds) worker2 # [ 10.787264] systemd[1]: Stopping User and Group Name Lookups... worker1 # [ 10.809249] systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 10.890159] systemd[1]: Starting User Login Management... client: (finished: waiting for the VM to finish booting, in 11.40 seconds) worker2 # [ 10.805708] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker1 # [ 10.820329] systemd[1]: Starting Post-Boot Actions... worker2 # [ 10.815246] systemd[1]: nscd.service: Deactivated successfully. worker1 # [ 10.831753] systemd[1]: Started Reset console on configuration changes. worker2 # [ 10.819146] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker1 # [ 10.841611] systemd[1]: Starting resolvconf update... worker2 # [ 10.823593] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 10.850534] systemd[1]: Starting D-Bus System Message Bus... worker2 # [ 10.862780] systemd[1]: linger-users.service: Deactivated successfully. worker1 # [ 10.894122] systemd[1]: Finished Post-Boot Actions. worker2 # [ 10.867437] systemd[1]: Finished linger-users.service. client # [ 10.906216] systemd[1]: Finished Import lastlog data into lastlog2 database. worker2 # [ 10.903640] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 10.908348] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 10.912975] nsncd[551]: Sep 23 05:12:05.405 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 10.919380] systemd[1]: Reached target User and Group Name Lookups. worker2 # [ 10.940566] systemd[1]: Finished resolvconf update. worker1 # [ 10.932868] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 10.942635] systemd[1]: Reached target Preparation for Network. worker2 # [ 10.948542] systemd[1]: Starting DHCP Client... worker1 # [ 10.941288] nsncd[488]: Sep 23 05:12:05.437 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 10.951879] systemd[1]: Starting Extra networking commands.... worker1 # [ 10.956616] systemd[1]: Reached target Host and Network Name Lookups. lb # [ 10.981104] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker1 # [ 10.965788] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 10.970576] systemd[1]: Starting User Login Management... lb # [ 11.002499] systemd[1]: Started backdoor.service. worker1 # [ 10.997106] systemd[1]: Finished Import lastlog data into lastlog2 database. worker1 # [ 11.014293] dbus-broker-launch[492]: Looking up NSS user entry for 'systemd-timesync'... lb # [ 11.051430] dbus-broker-launch[459]: Looking up NSS user entry for 'systemd-timesync'... worker1 # [ 11.031317] dbus-broker-launch[492]: NSS returned no entry for 'systemd-timesync' worker1 # [ 11.036617] dbus-broker-launch[492]: Invalid user-name in /nix/store/flsxa7d757b7zpdiv9wsbs42iawcisak-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker1 # [ 11.054425] systemd[1]: Started D-Bus System Message Bus. lb # [ 11.083619] dbus-broker-launch[459]: NSS returned no entry for 'systemd-timesync' lb # [ 11.090772] dbus-broker-launch[459]: Invalid user-name in /nix/store/4yiprls1sll47jid3xzq0m9nmi1qz2ww-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" client # [ 11.145893] systemd-logind[423]: Watching system buttons on /dev/input/event0 (gpio-keys) client # [ 11.163026] systemd-logind[423]: New seat seat0. worker1 # [ 11.108777] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. client # [ 11.171060] systemd[1]: Started User Login Management. worker1 # [ 11.113570] dbus-broker-launch[492]: Ready client # [ 11.175850] dbus-broker-launch[421]: Looking up NSS user entry for 'systemd-timesync'... client # [ 11.185350] systemd[1]: Starting linger-users.service... client # [ 11.190241] dbus-broker-launch[421]: NSS returned no entry for 'systemd-timesync' client # [ 11.191324] dbus-broker-launch[421]: Invalid user-name in /nix/store/6wvgzhp3xj1v1xybf1ixpbqi5gif5fbv-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" lb # [ 11.174878] systemd[1]: Started D-Bus System Message Bus. lb # [ 11.187048] systemd-logind[480]: New seat seat0. worker1 # [ 11.168748] systemd-logind[500]: New seat seat0. client # [ 11.235521] systemd[1]: Stopped target Host and Network Name Lookups. lb # [ 11.196838] systemd[1]: Started User Login Management. worker1 # [ 11.178451] systemd[1]: Started User Login Management. client # [ 11.242601] systemd[1]: Stopping Host and Network Name Lookups... client # [ 11.243539] systemd[1]: Stopped target User and Group Name Lookups. worker1 # [ 11.185453] systemd[1]: Starting linger-users.service... lb # [ 11.205447] systemd[1]: Stopped target Host and Network Name Lookups. worker1 # [ 11.192503] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. lb # [ 11.212472] systemd[1]: Stopping Host and Network Name Lookups... lb # [ 11.213580] systemd[1]: Stopped target User and Group Name Lookups. worker2 # [ 11.203888] dhcpcd[583]: dhcpcd-10.3.2 starting lb # [ 11.214546] systemd[1]: Stopping User and Group Name Lookups... client # [ 11.257062] systemd[1]: Stopping User and Group Name Lookups... client # [ 11.258025] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker1 # [ 11.200185] systemd[1]: Started backdoor.service. client # [ 11.258916] systemd[1]: Started D-Bus System Message Bus. worker2 # [ 11.216556] dhcpcd[624]: dev: loaded udev worker2 # [ 11.221255] dhcpcd[624]: no valid interfaces found worker1 # [ 11.214808] systemd[1]: Stopped target Host and Network Name Lookups. worker2 # [ 11.225527] dhcpcd[624]: no valid interfaces found lb # [ 11.236240] systemd[1]: Starting linger-users.service... worker2 # [ 11.227676] dhcpcd[624]: libudev: received NULL device client # [ 11.279551] systemd[1]: nscd.service: Deactivated successfully. worker2 # [ 11.230576] dhcpcd[624]: libudev: received NULL device worker1 # [ 11.221985] systemd[1]: Stopping Host and Network Name Lookups... worker1 # [ 11.226665] systemd[1]: Stopped target User and Group Name Lookups. lb # [ 11.245884] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... client # [ 11.286502] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker1 # [ 11.230239] systemd[1]: Stopping User and Group Name Lookups... lb # [ 11.253850] systemd[1]: nscd.service: Deactivated successfully. worker1 # [ 11.238142] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker1 # [ 11.249356] systemd[1]: nscd.service: Deactivated successfully. lb # connecting to host... lb # [ 11.270121] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker2 # [ 11.259781] systemd[1]: Finished Extra networking commands.. client # [ 11.309956] systemd-logind[423]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) worker2 # [ 11.263983] systemd[1]: Reached target Network. lb # [ 11.278718] dbus-broker-launch[459]: Ready worker2 # [ 11.269180] systemd[1]: Starting Permit User Sessions... worker1 # [ 11.260559] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker1 # [ 11.268326] systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 11.328621] systemd[1]: Starting Name Service Cache Daemon (nsncd)... client # [ 11.337148] dbus-broker-launch[421]: Ready client # [ 11.366485] systemd[1]: linger-users.service: Deactivated successfully. worker2 # [ 11.316937] systemd[1]: Finished Permit User Sessions. worker2 # [ 11.320326] systemd[1]: Started Getty on tty1. client # [ 11.371530] systemd[1]: Finished linger-users.service. worker2 # [ 11.325289] systemd[1]: Reached target Login Prompts. worker1 # [ 11.322902] systemd[1]: linger-users.service: Deactivated successfully. worker1 # [ 11.326944] systemd[1]: Finished linger-users.service. worker1 # connecting to host... lb # [ 11.381892] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 11.373048] nsncd[560]: Sep 23 05:12:05.873 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 11.375365] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 11.441875] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 11.443220] nsncd[487]: Sep 23 05:12:05.951 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker1 # [ 11.388119] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 11.389097] systemd[1]: Reached target User and Group Name Lookups. client # [ 11.455327] systemd[1]: Reached target Host and Network Name Lookups. client # [ 11.457686] systemd[1]: Reached target User and Group Name Lookups. client # [ 11.463761] systemd[1]: Finished resolvconf update. worker2 # [ 11.437795] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 client # [ 11.475915] systemd[1]: Starting DHCP Client... worker1 # [ 11.418034] systemd[1]: Finished resolvconf update. worker1 # [ 11.418949] systemd[1]: Reached target Preparation for Network. worker1 # [ 11.428931] systemd[1]: Starting DHCP Client... worker1 # [ 11.436290] systemd[1]: Starting Extra networking commands.... lb # [ 11.490775] systemd[1]: linger-users.service: Deactivated successfully. lb # [ 11.499201] systemd[1]: Finished linger-users.service. lb # [ 11.505882] (udev-worker)[403]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. lb # [ 11.508000] (udev-worker)[403]: Network interface NamePolicy= disabled on kernel command line. lb # [ 11.523908] (udev-worker)[408]: Network interface NamePolicy= disabled on kernel command line. lb # [ 11.552107] systemd[1]: Finished resolvconf update. lb # [ 11.558657] nsncd[561]: Sep 23 05:12:06.062 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" lb # [ 11.569297] systemd[1]: Started Name Service Cache Daemon (nsncd). lb # [ 11.570247] systemd[1]: Reached target Host and Network Name Lookups. lb # [ 11.571117] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 11.557550] (udev-worker)[455]: Network interface NamePolicy= disabled on kernel command line. lb # [ 11.580304] systemd[1]: Starting DHCP Client... worker1 # [ 11.573275] (udev-worker)[356]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 11.575547] (udev-worker)[356]: Network interface NamePolicy= disabled on kernel command line. client # [ 11.707943] dhcpcd[521]: dhcpcd-10.3.2 starting client # [ 11.717213] dhcpcd[529]: dev: loaded udev worker1 # [ 11.684948] dhcpcd[595]: dhcpcd-10.3.2 starting worker1 # [ 11.695501] dhcpcd[635]: dev: loaded udev client # [ 11.777327] 8021q: 802.1Q VLAN Support v1.8 worker1 # [ 11.740687] 8021q: 802.1Q VLAN Support v1.8 worker1 # [ 11.790473] systemd[1]: Finished Extra networking commands.. worker1 # [ 11.794571] systemd[1]: Reached target Network. worker1 # [ 11.798720] systemd[1]: Starting Permit User Sessions... worker1 # [ 11.816715] systemd[1]: Condition check resulted in Virtio network device being skipped. client # [ 11.902181] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker1 # [ 11.832361] systemd[1]: Starting Address configuration of eth1... lb # [ 11.852280] systemd[1]: Condition check resulted in Virtio network device being skipped. worker2 # [ 11.856746] systemd[1]: Starting Virtual Console Setup... worker1 # [ 11.871170] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker2 # [ 11.880772] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 11.884242] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 11.887250] systemd[1]: Starting Virtual Console Setup... worker1 # [ 11.897339] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker1 # [ 11.898688] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' worker1 # [ 11.902027] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 worker1 # [ 11.902388] cfg80211: failed to load regulatory.db client # [ 11.970540] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' client # [ 11.971101] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' client # [ 11.975366] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 client # [ 11.975728] cfg80211: failed to load regulatory.db worker2 # [ 11.911498] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 11.932422] dhcpcd[605]: dhcpcd-10.3.2 starting worker1 # [ 11.916297] systemd[1]: Finished Permit User Sessions. worker1 # [ 11.922798] systemd[1]: Started Getty on tty1. worker1 # [ 11.923572] systemd[1]: Reached target Login Prompts. lb # [ 11.950880] dhcpcd[625]: dev: loaded udev client # [ 12.022740] systemd[1]: Finished nftables firewall. worker1 # [ 11.979432] 8021q: adding VLAN 0 to HW filter on device eth1 client # [ 12.028191] systemd[1]: Reached target Preparation for Network. client # [ 12.035446] systemd[1]: Starting Address configuration of eth1... client # [ 12.043423] systemd[1]: Starting Extra networking commands.... worker1 # [ 11.989697] network-addresses-eth1-start[652]: adding address 192.168.1.3/24... done lb # [ 12.037353] 8021q: 802.1Q VLAN Support v1.8 worker1 # [ 12.009754] network-addresses-eth1-start[652]: adding address 2001:db8:1::3/64... done worker1 # [ 12.027140] 8021q: adding VLAN 0 to HW filter on device eth0 worker1 # [ 12.020280] dhcpcd[635]: eth0: waiting for carrier client # [ 12.098854] 8021q: adding VLAN 0 to HW filter on device eth0 worker1 # [ 12.025128] dhcpcd[635]: eth0: waiting for carrier client # [ 12.086514] dhcpcd[529]: eth0: waiting for carrier worker1 # [ 12.028429] dhcpcd[635]: eth0: carrier acquired client # [ 12.089817] dhcpcd[529]: libudev: received NULL device client # [ 12.090604] dhcpcd[529]: libudev: received NULL device client # [ 12.091455] dhcpcd[529]: eth0: carrier acquired worker1 # [ 12.040550] systemd[1]: Finished Address configuration of eth1. worker1 # [ 12.046487] dhcpcd[635]: DUID 00:01:00:01:32:46:1c:26:52:54:00:12:34:56 worker1 # [ 12.047504] dhcpcd[635]: eth0: IAID 00:12:34:56 worker1 # [ 12.050587] dhcpcd[635]: eth0: adding address fe80::5054:ff:fe12:3456 client # [ 12.116157] dhcpcd[529]: DUID 00:01:00:01:32:46:1c:26:52:54:00:12:34:56 client # [ 12.117210] dhcpcd[529]: eth0: IAID 00:12:34:56 client # [ 12.117897] dhcpcd[529]: eth0: adding address fe80::5054:ff:fe12:3456 lb # [ 12.108567] systemd[1]: Finished Firewall. lb # [ 12.109407] systemd[1]: Reached target Preparation for Network. client # [ 12.183377] 8021q: adding VLAN 0 to HW filter on device eth1 lb # [ 12.130827] systemd[1]: Starting Address configuration of eth1... lb # [ 12.138118] systemd[1]: Starting Extra networking commands.... worker2 # [ 12.129649] systemd-vconsole-setup[653]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 12.138785] systemd[1]: Finished Virtual Console Setup. client # [ 12.189110] network-addresses-eth1-start[549]: adding address 192.168.1.1/24... done client # [ 12.208654] network-addresses-eth1-start[549]: adding address 2001:db8:1::1/64... done worker2 # [ 12.175869] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. client # [ 12.251761] systemd[1]: Finished Address configuration of eth1. worker2 # [ 12.217669] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker2 # [ 12.222229] systemd[1]: Started backdoor.service. worker1 # [ 12.215130] systemd-logind[500]: Watching system buttons on /dev/input/event0 (gpio-keys) lb # [ 12.266820] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker2 # connecting to host... worker2 # [ 12.337079] 8021q: 802.1Q VLAN Support v1.8 lb # [ 12.352263] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' client # [ 12.376739] systemd[1]: Finished Extra networking commands.. lb # [ 12.355022] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' client # [ 12.380415] systemd[1]: Reached target Network. lb # [ 12.357814] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 lb # [ 12.358152] cfg80211: failed to load regulatory.db client # [ 12.389274] systemd[1]: Starting Permit User Sessions... lb # [ 12.390618] 8021q: adding VLAN 0 to HW filter on device eth1 worker1 # [ 12.383149] mousedev: PS/2 mouse device common for all mice worker2 # [ 12.402287] (udev-worker)[642]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 12.410320] (udev-worker)[646]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client: (finished: waiting for unit network-addresses-eth1.service, in 12.89 seconds) client: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; } lb # [ 12.427962] network-addresses-eth1-start[646]: adding address 192.168.1.2/24... done client # [ 12.468378] systemd[1]: Finished Permit User Sessions. worker2 # [ 12.426986] (udev-worker)[646]: Network interface NamePolicy= disabled on kernel command line. client # [ 12.480769] systemd[1]: Started Getty on tty1. lb # [ 12.443074] systemd-logind[480]: Watching system buttons on /dev/input/event0 (gpio-keys) client # [ 12.486648] systemd[1]: Reached target Login Prompts. worker2 # [ 12.459220] cfg80211: Loading compiled-in X.509 certificates for regulatory database lb # [ 12.474468] network-addresses-eth1-start[646]: adding address 2001:db8:1::2/64... done worker1 # [ 12.458534] systemd-logind[500]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) client: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; }, in 0.07 seconds) lb: waiting for unit network-addresses-eth1.service lb: waiting for the VM to finish booting lb: Guest shell says: b'Spawning backdoor root shell...\n' lb: connected to guest root shell lb: (connecting took 0.00 seconds) lb: (finished: waiting for the VM to finish booting, in 0.00 seconds) worker2 # [ 12.500013] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker2 # [ 12.500552] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' worker2 # [ 12.504267] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 worker2 # [ 12.504630] cfg80211: failed to load regulatory.db lb # [ 12.501231] dhcpcd[672]: /nix/store/1ki7bgdr0niilkg9jbkpcdaggbv0kb8a-openresolv-3.17.4/sbin/.resolvconf-wrapped: line 1250: kill: (651) - Operation not permitted lb # [ 12.503159] dhcpcd[672]: .resolvconf-wrapped: clearing stale lock pid 651 lb # [ 12.523878] systemd[1]: Finished Address configuration of eth1. lb # [ 12.622026] 8021q: adding VLAN 0 to HW filter on device eth0 lb # [ 12.609567] dhcpcd[625]: eth0: waiting for carrier lb # [ 12.610357] dhcpcd[625]: eth0: carrier acquired worker2 # [ 12.610945] systemd-logind[503]: Watching system buttons on /dev/input/event0 (gpio-keys) lb # [ 12.631052] dhcpcd[625]: DUID 00:01:00:01:32:46:1c:27:52:54:00:12:34:56 worker2 # [ 12.624383] systemd[1]: Condition check resulted in Virtio network device being skipped. lb # [ 12.635927] dhcpcd[625]: eth0: IAID 00:12:34:56 lb # [ 12.639555] dhcpcd[625]: eth0: adding address fe80::5054:ff:fe12:3456 worker2 # [ 12.629517] systemd[1]: Starting Address configuration of eth1... lb: (finished: waiting for unit network-addresses-eth1.service, in 0.17 seconds) lb: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; } worker2 # [ 12.727352] 8021q: adding VLAN 0 to HW filter on device eth0 worker2 # [ 12.716858] dhcpcd[624]: eth0: waiting for carrier worker2 # [ 12.722475] dhcpcd[624]: eth0: carrier acquired lb # [ 12.741700] systemd[1]: Finished Extra networking commands.. lb # [ 12.742627] systemd[1]: Reached target Network. worker2 # [ 12.739236] dhcpcd[624]: DUID 00:01:00:01:32:46:1c:27:52:54:00:12:34:56 lb # [ 12.751828] systemd[1]: Starting PostgreSQL Server... worker2 # [ 12.742616] dhcpcd[624]: eth0: IAID 00:12:34:56 worker2 # [ 12.743422] dhcpcd[624]: eth0: adding address fe80::5054:ff:fe12:3456[ 12.760397] 8021q: adding VLAN 0 to HW filter on device eth1 lb: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; }, in 0.09 seconds) worker2 # worker1: waiting for unit network-addresses-eth1.service worker1: waiting for the VM to finish booting lb # [ 12.759371] systemd[1]: Starting Permit User Sessions... worker1: Guest shell says: b'Spawning backdoor root shell...\n' worker1: connected to guest root shell worker1: (connecting took 0.00 seconds) worker1: (finished: waiting for the VM to finish booting, in 0.00 seconds) worker2 # [ 12.761746] network-addresses-eth1-start[673]: adding address 192.168.1.4/24... done worker2 # [ 12.773848] network-addresses-eth1-start[673]: adding address 2001:db8:1::4/64... done worker2 # [ 12.794225] systemd[1]: Finished Address configuration of eth1. client # [ 12.908662] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 lb # [ 12.855316] systemd[1]: Finished Permit User Sessions. worker1: (finished: waiting for unit network-addresses-eth1.service, in 0.10 seconds) worker1: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; } lb # [ 12.865883] systemd[1]: Started Getty on tty1. lb # [ 12.869106] systemd[1]: Reached target Login Prompts. worker2 # [ 12.881942] mousedev: PS/2 mouse device common for all mice worker1: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; }, in 0.05 seconds) worker2: waiting for unit network-addresses-eth1.service worker2: waiting for the VM to finish booting worker2: Guest shell says: b'Spawning backdoor root shell...\n' worker2: connected to guest root shell worker2: (connecting took 0.00 seconds) worker2: (finished: waiting for the VM to finish booting, in 0.00 seconds) lb # [ 12.928739] mousedev: PS/2 mouse device common for all mice worker2: (finished: waiting for unit network-addresses-eth1.service, in 0.09 seconds) worker2: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; } lb # [ 13.034009] systemd-logind[480]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) worker2: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; }, in 0.04 seconds) lb: waiting for unit niks3.service worker2 # [ 13.041832] systemd-logind[503]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) lb # [ 13.105216] postgresql-pre-start[739]: The files belonging to this database system will be owned by user "postgres". lb # [ 13.106694] postgresql-pre-start[739]: This user must also own the server process. lb # [ 13.122015] postgresql-pre-start[739]: The database cluster will be initialized with locale "en_US.UTF-8". lb # [ 13.123367] postgresql-pre-start[739]: The default database encoding has accordingly been set to "UTF8". lb # [ 13.130177] postgresql-pre-start[739]: The default text search configuration will be set to "english". lb # [ 13.131426] postgresql-pre-start[739]: Data page checksums are enabled. worker1 # [ 13.124707] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 lb # [ 13.134670] postgresql-pre-start[739]: fixing permissions on existing directory /var/lib/postgresql/18 ... ok lb # [ 13.135965] postgresql-pre-start[739]: creating subdirectories ... ok lb # [ 13.143585] postgresql-pre-start[739]: selecting dynamic shared memory implementation ... posix client # [ 13.249694] systemd[1]: Starting Virtual Console Setup... client # [ 13.264767] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 13.266504] systemd[1]: Stopped Virtual Console Setup. client # [ 13.273504] systemd[1]: Starting Virtual Console Setup... client # [ 13.306962] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. lb # [ 13.276418] postgresql-pre-start[739]: selecting default "max_connections" ... 100 worker1 # [ 13.268643] dhcpcd[635]: eth0: soliciting a DHCP lease worker1 # [ 13.276599] dhcpcd[635]: eth0: offered 10.0.2.15 from 10.0.2.2 worker1 # [ 13.284223] dhcpcd[635]: eth0: probing address 10.0.2.15/24 client # [ 13.348312] systemd-logind[423]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) client # [ 13.434772] systemd-vconsole-setup[639]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 13.438765] systemd[1]: Finished Virtual Console Setup. lb # [ 13.402716] postgresql-pre-start[739]: selecting default "shared_buffers" ... 128MB worker2 # [ 13.454429] dhcpcd[624]: eth0: soliciting a DHCP lease worker2 # [ 13.460695] dhcpcd[624]: eth0: offered 10.0.2.15 from 10.0.2.2 worker1 # [ 13.457982] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker2 # [ 13.468321] dhcpcd[624]: eth0: probing address 10.0.2.15/24 worker1 # [ 13.477162] systemd[1]: Starting Virtual Console Setup... worker1 # [ 13.501638] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 13.507958] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 13.508959] systemd[1]: Starting Virtual Console Setup... worker1 # [ 13.571098] systemd-logind[500]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) client # [ 13.708662] dhcpcd[529]: eth0: soliciting a DHCP lease client # [ 13.713347] dhcpcd[529]: eth0: offered 10.0.2.15 from 10.0.2.2 worker1 # [ 13.662270] systemd-vconsole-setup[702]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 13.720450] dhcpcd[529]: eth0: probing address 10.0.2.15/24 worker1 # [ 13.667473] systemd[1]: Finished Virtual Console Setup. client # [ 14.003350] dhcpcd[529]: eth0: soliciting an IPv6 router client # [ 14.007449] dhcpcd[529]: eth0: Router Advertisement from fe80::2 client # [ 14.010272] dhcpcd[529]: eth0: adding address fec0::5054:ff:fe12:3456/64 client # [ 14.013273] dhcpcd[529]: eth0: adding route to fec0::/64 client # [ 14.015625] dhcpcd[529]: eth0: adding default route via fe80::2 lb # [ 14.036996] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:06.0/virtio5/input/input3 worker2 # [ 14.077536] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker2 # [ 14.081701] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 14.082854] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 14.083574] systemd[1]: Stopping Virtual Console Setup... worker2 # [ 14.092638] systemd[1]: Starting Virtual Console Setup... worker2 # [ 14.109183] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 14.117974] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 14.118856] systemd[1]: Starting Virtual Console Setup... worker2 # [ 14.172137] systemd-logind[503]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) worker2 # [ 14.259060] systemd-vconsole-setup[705]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 14.264337] systemd[1]: Finished Virtual Console Setup. lb # [ 14.485457] systemd[1]: Starting Virtual Console Setup... lb # [ 14.514379] dhcpcd[625]: eth0: soliciting a DHCP lease lb # [ 14.520620] dhcpcd[625]: eth0: offered 10.0.2.15 from 10.0.2.2 lb # [ 14.528369] dhcpcd[625]: eth0: probing address 10.0.2.15/24 lb # [ 14.532692] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. lb # [ 14.558919] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 14.560556] systemd[1]: Stopped Virtual Console Setup. lb # [ 14.566753] systemd[1]: Starting Virtual Console Setup... worker1 # [ 14.574752] dhcpcd[635]: eth0: soliciting an IPv6 router worker1 # [ 14.578797] dhcpcd[635]: eth0: Router Advertisement from fe80::2 worker1 # [ 14.581823] dhcpcd[635]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker1 # [ 14.584855] dhcpcd[635]: eth0: adding route to fec0::/64 worker1 # [ 14.587375] dhcpcd[635]: eth0: adding default route via fe80::2 lb # [ 14.612661] dhcpcd[625]: eth0: soliciting an IPv6 router lb # [ 14.614211] dhcpcd[625]: eth0: Router Advertisement from fe80::2 lb # [ 14.615059] dhcpcd[625]: eth0: adding address fec0::5054:ff:fe12:3456/64 lb # [ 14.615936] dhcpcd[625]: eth0: adding route to fec0::/64 worker2 # [ 14.607189] dhcpcd[624]: eth0: soliciting an IPv6 router worker2 # [ 14.611690] dhcpcd[624]: eth0: Router Advertisement from fe80::2 lb # [ 14.623875] dhcpcd[625]: eth0: adding default route via fe80::2 worker2 # [ 14.614587] dhcpcd[624]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker2 # [ 14.617631] dhcpcd[624]: eth0: adding route to fec0::/64 worker2 # [ 14.620261] dhcpcd[624]: eth0: adding default route via fe80::2 lb # [ 14.646185] systemd-logind[480]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) lb # [ 14.871063] systemd-vconsole-setup[770]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 14.876679] systemd[1]: Finished Virtual Console Setup. lb # [ 15.210309] postgresql-pre-start[739]: selecting default time zone ... UTC lb # [ 15.213436] postgresql-pre-start[739]: creating configuration files ... ok lb # [ 15.499656] postgresql-pre-start[739]: running bootstrap script ... ok lb # [ 16.216852] postgresql-pre-start[739]: performing post-bootstrap initialization ... ok lb # [ 16.386340] postgresql-pre-start[739]: syncing data to disk ... ok lb # [ 16.388636] postgresql-pre-start[739]: initdb: warning: enabling "trust" authentication for local connections lb # [ 16.390050] postgresql-pre-start[739]: initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. lb # [ 16.392213] postgresql-pre-start[739]: Success. You can now start the database server using: lb # [ 16.393478] postgresql-pre-start[739]: pg_ctl -D /var/lib/postgresql/18 -l logfile start lb # [ 16.500614] postgres[790]: [790] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit lb # [ 16.503902] postgres[790]: [790] LOG: listening on IPv6 address "::1", port 5432 lb # [ 16.505253] postgres[790]: [790] LOG: listening on IPv4 address "127.0.0.1", port 5432 lb # [ 16.507309] postgres[790]: [790] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432" lb # [ 16.520559] postgres[799]: [799] LOG: database system was shut down at 2026-09-23 05:12:10 GMT lb # [ 16.526264] postgres[790]: [790] LOG: database system is ready to accept connections lb # [ 16.534875] systemd[1]: Started PostgreSQL Server. lb # [ 16.538434] systemd[1]: Starting PostgreSQL Setup Scripts... lb # [ 16.806043] postgresql-setup-start[819]: CREATE DATABASE lb # [ 16.851004] postgresql-setup-start[829]: CREATE ROLE lb # [ 16.869572] postgresql-setup-start[831]: ALTER DATABASE lb # [ 16.876056] systemd[1]: Finished PostgreSQL Setup Scripts. lb # [ 16.878263] systemd[1]: Reached target PostgreSQL. worker2 # [ 17.953492] dhcpcd[624]: eth0: leased 10.0.2.15 for 86400 seconds worker2 # [ 17.958318] dhcpcd[624]: eth0: adding route to 10.0.2.0/24 worker2 # [ 17.962763] dhcpcd[624]: eth0: adding default route via 10.0.2.2 worker2 # [ 18.150763] systemd[1]: Started DHCP Client. worker2 # [ 18.153392] systemd[1]: Reached target Multi-User System. worker2 # [ 18.154495] systemd[1]: Startup finished in 1.071s (kernel) + 4.412s (initrd) + 12.668s (userspace) = 18.152s. worker1 # [ 18.233674] dhcpcd[635]: eth0: leased 10.0.2.15 for 86400 seconds worker1 # [ 18.240344] dhcpcd[635]: eth0: adding route to 10.0.2.0/24 worker1 # [ 18.243079] dhcpcd[635]: eth0: adding default route via 10.0.2.2 worker1 # [ 18.445641] systemd[1]: Started DHCP Client. worker1 # [ 18.448320] systemd[1]: Reached target Multi-User System. worker1 # [ 18.450489] systemd[1]: Startup finished in 1.069s (kernel) + 5.080s (initrd) + 12.299s (userspace) = 18.449s. client # [ 19.173799] dhcpcd[529]: eth0: leased 10.0.2.15 for 86400 seconds client # [ 19.179104] dhcpcd[529]: eth0: adding route to 10.0.2.0/24 client # [ 19.186935] dhcpcd[529]: eth0: adding default route via 10.0.2.2 client # [ 19.385633] systemd[1]: Started DHCP Client. client # [ 19.390158] systemd[1]: Reached target Multi-User System. client # [ 19.391795] systemd[1]: Startup finished in 1.061s (kernel) + 5.090s (initrd) + 13.240s (userspace) = 19.391s. lb # [ 19.407639] dhcpcd[625]: eth0: leased 10.0.2.15 for 86400 seconds lb # [ 19.412889] dhcpcd[625]: eth0: adding route to 10.0.2.0/24 lb # [ 19.415612] dhcpcd[625]: eth0: adding default route via 10.0.2.2 lb # [ 19.613303] systemd[1]: Started DHCP Client. lb # [ 19.618267] systemd[1]: Reached target Network is Online. lb # [ 19.626067] systemd[1]: Started Envoy reverse proxy. lb # [ 19.632826] systemd[1]: Started mock-oidc.service. lb # [ 19.811041] mock-oidc-server[909]: Mock OIDC Server running lb # [ 19.812382] mock-oidc-server[909]: OIDC Address: 192.168.1.2:8080 lb # [ 19.813307] mock-oidc-server[909]: Issue Address: 0.0.0.0:8081 lb # [ 19.814112] mock-oidc-server[909]: Issuer: http://192.168.1.2:8080/oidc lb # [ 19.815017] mock-oidc-server[909]: JWKS: http://192.168.1.2:8080/oidc/.well-known/jwks.json lb # [ 19.816853] mock-oidc-server[909]: Discovery: http://192.168.1.2:8080/oidc/.well-known/openid-configuration lb # [ 19.818192] mock-oidc-server[909]: Issue tokens: http://0.0.0.0:8081/issue?sub=... lb # [ 20.058760] envoy[908]: [2026-09-23 05:12:14.566][908][info][main] [source/server/server.cc:467] initializing epoch 0 (base id=0, hot restart version=11.120) lb # [ 20.062107] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:469] statically linked extensions: lb # [ 20.063637] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.clusters: envoy.cluster.dns, envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.composite, envoy.clusters.dynamic_forward_proxy, envoy.clusters.dynamic_modules, envoy.clusters.mcp_multicluster, envoy.clusters.redis, envoy.clusters.reverse_connection lb # [ 20.069320] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.quic.proof_source: envoy.quic.proof_source.filter_chain lb # [ 20.071340] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.quic.client_packet_writer: envoy.quic.packet_writer.default lb # [ 20.073885] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.matching.action: envoy.matching.action.transport_socket.name, envoy.matching.actions.format_string, filter-chain-name lb # [ 20.076410] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.string_matcher: envoy.string_matcher.lua lb # [ 20.087426] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action lb # [ 20.090174] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.path.match: envoy.path.match.uri_template.uri_template_matcher lb # [ 20.094348] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.regex_engines: envoy.regex_engines.google_re2 lb # [ 20.100172] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor lb # [ 20.102985] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.network_namespace, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san lb # [ 20.139656] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.matching.transport_socket.input: envoy.matching.inputs.endpoint_metadata, envoy.matching.inputs.locality_metadata, envoy.matching.inputs.transport_socket_filter_state lb # [ 20.143713] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.tracers: envoy.tracers.datadog, envoy.tracers.dynamic_modules, envoy.tracers.fluentd, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin lb # [ 20.149684] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher lb # [ 20.153207] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.load_balancing_policies: envoy.load_balancing_policies.client_side_weighted_round_robin, envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.dynamic_modules, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.load_aware_locality, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.override_host, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset, envoy.load_balancing_policies.wrr_locality lb # [ 20.161849] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr lb # [ 20.163775] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] network.connection.client: default, envoy_internal lb # [ 20.168218] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case lb # [ 20.172926] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.matching.generic_proxy_request_input.input: envoy.matching.generic_proxy.input.host, envoy.matching.generic_proxy.input.method, envoy.matching.generic_proxy.input.path, envoy.matching.generic_proxy.input.property, envoy.matching.generic_proxy.input.request, envoy.matching.generic_proxy.input.service lb # [ 20.178270] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.quic.connection_id_generator: envoy.quic.connection_id_generator.quic_lb, envoy.quic.deterministic_connection_id_generator lb # [ 20.181128] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.resource_detectors: envoy.tracers.opentelemetry.resource_detectors.dynatrace, envoy.tracers.opentelemetry.resource_detectors.environment, envoy.tracers.opentelemetry.resource_detectors.static_config lb # [ 20.188200] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.quic.server_preferred_address: quic.server_preferred_address.datasource, quic.server_preferred_address.fixed lb # [ 20.190676] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest lb # [ 20.196974] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter lb # [ 20.198928] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher lb # [ 20.206879] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.serializers: dubbo.hessian2 lb # [ 20.208800] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter lb # [ 20.210829] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator lb # [ 20.213468] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.protocols: dubbo lb # [ 20.214980] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme lb # [ 20.219216] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.dynamic_modules, envoy.filters.udp_listener.udp_proxy lb # [ 20.224766] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.retry_priorities: envoy.retry_priorities.previous_priorities lb # [ 20.226710] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple lb # [ 20.232815] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.http.ext_proc.response_processors: envoy.extensions.http.ext_proc.save_processing_response lb # [ 20.237606] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.grpc_credentials: envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata lb # [ 20.239794] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.geoip_providers: envoy.geoip_providers.maxmind lb # [ 20.247990] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.formatter: envoy.formatter.cel, envoy.formatter.dynamic_modules, envoy.formatter.file_content, envoy.formatter.generic_secret, envoy.formatter.metadata, envoy.formatter.req_without_query lb # [ 20.251253] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts lb # [ 20.255361] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.router.cluster_specifier_plugin: envoy.router.cluster_specifier_plugin.lua, envoy.router.cluster_specifier_plugin.matcher lb # [ 20.259205] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.tls.certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret lb # [ 20.261512] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret lb # [ 20.268362] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.http.header_validators: envoy.http.header_validators.envoy_default lb # [ 20.270414] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.ssl.server_context_factory: envoy.ssl.server_context_factory.default lb # [ 20.277572] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.filters.upstream_network: envoy.filters.upstream_network.reverse_tunnel_lifecycle worker2 # [ 20.271699] systemd[1]: Reached target Network is Online. lb # [ 20.282290] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.dynamic_metadata, envoy.matching.inputs.dynamic_module_data_input, envoy.matching.inputs.filter_state, envoy.matching.inputs.local_reply, envoy.matching.inputs.network_namespace, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params worker1 # [ 20.273793] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 20.287267] systemd[1]: Starting scheduler-token.service... lb # [ 20.296001] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.fault, envoy.filters.http.a2a, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.ai_protocol_manager, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.api_key_auth, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.bandwidth_share, envoy.filters.http.basic_auth, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cache_v2, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.credential_injector, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_server, envoy.filters.http.file_system_buffer, envoy.filters.http.filter_chain, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_reverse_transcoder, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.mcp, envoy.filters.http.mcp_json_rest_bridge, envoy.filters.http.mcp_router, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.proto_api_scrubber, envoy.filters.http.proto_message_extraction, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.sse_to_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.thrift_to_metadata, envoy.filters.http.transform, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router lb # [ 20.334271] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff lb # [ 20.336802] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.upstreams: envoy.filters.connection_pools.tcp.generic lb # [ 20.338595] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router lb # [ 20.341686] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.common.key_value: envoy.key_value.file_based lb # [ 20.344395] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port lb # [ 20.346171] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.resolvers: envoy.ip, envoy.resolvers.reverse_connection lb # [ 20.347943] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation lb # [ 20.350234] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.generic_proxy.filters: envoy.filters.generic.router lb # [ 20.351962] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.health_check.event_sinks: envoy.health_check.event_sink.file lb # [ 20.354097] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable lb # [ 20.356130] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.filters.http.upstream: envoy.buffer, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.composite, envoy.filters.http.credential_injector, envoy.filters.http.ext_proc, envoy.filters.http.header_mutation, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.upstream_codec, envoy.filters.http.upstream_rbac, envoy.filters.http.wasm lb # [ 20.363159] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.dynamic_modules, envoy.tls.cert_validator.spiffe lb # [ 20.365893] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.filters.listener: envoy.filters.listener.dynamic_modules, envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.set_filter_state, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector lb # [ 20.372382] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.http.cache_v2: envoy.extensions.http.cache_v2.file_system_http_cache, envoy.extensions.http.cache_v2.simple lb # [ 20.374744] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy lb # [ 20.378893] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo, envoy.network.dns_resolver.hickory lb # [ 20.381684] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.rbac.principals: envoy.rbac.principals.mtls_authenticated lb # [ 20.383492] envoy[908]: [2026-09-23 05:12:14.569][908][info][main] [source/server/server.cc:471] envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel, envoy.access_loggers.extension_filters.process_ratelimit lb # [ 20.386464] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.health_checkers: envoy.health_checkers.dynamic_modules, envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift lb # [ 20.390134] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.tls.certificate_mappers: envoy.tls.certificate_mappers.sni, envoy.tls.certificate_mappers.static_name lb # [ 20.392500] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.rds_factory: envoy.rds_factory.default lb # [ 20.395671] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector, envoy.upstream.local_address_selector.filter_state_override lb # [ 20.401258] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router lb # [ 20.405512] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation lb # [ 20.407759] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.upstream_options: envoy.extensions.upstreams.http.reverse_tunnel.v3.ReverseTunnelUpstreamCodecOptions, envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options lb # [ 20.413884] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche lb # [ 20.416134] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso lb # [ 20.418246] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls lb # [ 20.424631] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.route_config_update_requester: envoy.route_config_update_requester.default worker2 # [ 20.413965] systemd[1]: scheduler-token.service: Deactivated successfully. lb # [ 20.426656] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.content_parsers: envoy.content_parsers.json lb # [ 20.428387] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.route.early_data_policy: envoy.route.early_data_policy.default worker2 # [ 20.418407] systemd[1]: Finished scheduler-token.service. lb # [ 20.430264] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory lb # [ 20.433987] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.ext_authz, envoy.filters.udp.session.http_capsule worker2 # [ 20.422370] systemd[1]: scheduler-token.service: Consumed 75ms CPU time over 131ms wall clock time, 3.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. lb # [ 20.437233] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.bootstrap: envoy.bootstrap.dynamic_modules, envoy.bootstrap.internal_listener, envoy.bootstrap.reverse_tunnel.downstream_socket_interface, envoy.bootstrap.reverse_tunnel.upstream_socket_interface, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface, envoy.extensions.network.socket_interface.sockmap lb # [ 20.442506] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.matching.stat_matching_data.input: stat_full_name_match_input lb # [ 20.444551] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.srds_factory: envoy.srds_factory.default lb # [ 20.446205] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] quic.http_server_connection: quic.http_server_connection.default lb # [ 20.448139] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.access_loggers: envoy.access_loggers.dynamic_modules, envoy.access_loggers.file, envoy.access_loggers.fluentd, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stats, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.fluentd_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log lb # [ 20.455197] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_mappers: envoy.tls.certificate_mappers.static_name, envoy.tls.upstream_certificate_mappers.filter_state_override lb # [ 20.458062] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.thrift_proxy.transports: auto, framed, header, unframed worker2 # [ 20.448729] systemd[1]: Starting Nix worker-protocol over gRPC... lb # [ 20.459867] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.samplers: envoy.tracers.opentelemetry.samplers.always_on, envoy.tracers.opentelemetry.samplers.cel, envoy.tracers.opentelemetry.samplers.dynatrace, envoy.tracers.opentelemetry.samplers.parent_based, envoy.tracers.opentelemetry.samplers.trace_id_ratio_based worker1 # [ 20.455220] nix-grpc-daemon[767]: ts=2026-09-23T05:12:14Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" lb # [ 20.464469] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.dynamic_modules, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.ext_proc, envoy.filters.network.generic_proxy, envoy.filters.network.geoip, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.reverse_tunnel, envoy.filters.network.reverse_tunnel_drain_aware_http_connection_manager, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_bandwidth_limit, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy lb # [ 20.478585] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.envelope, envoy.http.stateful_session.header lb # [ 20.481419] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.matching.stat_tag_matching_data.input: stat_tag_value_input worker1 # [ 20.466583] nix-grpc-daemon[767]: ts=2026-09-23T05:12:14Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc lb # [ 20.483290] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls worker1 # [ 20.469188] nix-grpc-daemon[767]: ts=2026-09-23T05:12:14Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 lb # [ 20.488870] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8 lb # [ 20.490776] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.http.forward_client_cert_details: forward_client_cert lb # [ 20.492811] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.matching.input_matchers: envoy.matching.input_matchers.generic_request_matcher, envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.dynamic_modules, envoy.matching.matchers.ip, envoy.matching.matchers.metadata_matcher, envoy.matching.matchers.runtime_fraction lb # [ 20.497864] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.http.injected_credentials: envoy.http.injected_credentials.generic, envoy.http.injected_credentials.oauth2 worker1 # [ 20.485380] nix-grpc-daemon[767]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR lb # [ 20.504464] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.quic.connection_debug_visitor: envoy.quic.connection_debug_visitor.basic, envoy.quic.connection_debug_visitor.quic_stats worker1 # [ 20.488894] nix-grpc-daemon[767]: E0000 00:00:1790140334.988170 767 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 20.507000] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.hashable_string, envoy.network.application_protocols, envoy.network.ip, envoy.network.network_namespace, envoy.network.transport_socket.http_11_proxy.info, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.ratelimit.hits_addend, envoy.router.debug_config, envoy.string, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.tcp_proxy.per_connection_idle_timeout_ms, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port worker1 # [ 20.494441] nix-grpc-daemon[767]: E0000 00:00:1790140334.997454 767 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb # [ 20.519373] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.built_in_formatters: envoy.built_in_formatters.cel, envoy.built_in_formatters.http.default, envoy.built_in_formatters.metadata, envoy.built_in_formatters.stream_info.default, envoy.built_in_formatters.xfcc_value lb # [ 20.523231] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.request_id: envoy.request_id.uuid lb # [ 20.524929] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.generic_proxy.codecs: envoy.generic_proxy.codecs.dubbo, envoy.generic_proxy.codecs.http1 lb # [ 20.527128] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.connection_handler: envoy.connection_handler.default lb # [ 20.529057] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.http.ext_proc.processing_request_modifiers: envoy.extensions.http.ext_proc.mapped_attribute_builder lb # [ 20.536119] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.dynamic_modules, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd worker1 # [ 20.522006] nix-grpc-daemon[767]: ts=2026-09-23T05:12:15Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket lb # [ 20.543098] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor worker1 # [ 20.526416] systemd[1]: Started Nix worker-protocol over gRPC. lb # [ 20.549819] envoy[908]: [2026-09-23 05:12:14.570][908][info][main] [source/server/server.cc:471] envoy.resource_monitors: envoy.resource_monitors.cgroup_memory, envoy.resource_monitors.cpu_utilization, envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource lb # [ 20.553750] envoy[908]: [2026-09-23 05:12:14.594][908][info][main] [source/server/server.cc:531] HTTP header map info: worker1 # [ 20.531649] nix-grpc-daemon[767]: E0000 00:00:1790140335.033422 809 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 20.555231] envoy[908]: [2026-09-23 05:12:14.594][908][info][main] [source/server/server.cc:534] request header map: 664 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-host,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-request-id worker1 # [ 20.551707] nix-grpc-daemon[767]: E0000 00:00:1790140335.055049 809 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb # [ 20.572184] envoy[908]: [2026-09-23 05:12:14.594][908][info][main] [source/server/server.cc:534] request trailer map: 120 bytes: worker1 # [ 20.556071] nix-grpc-daemon[767]: E0000 00:00:1790140335.058353 809 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 20.573788] envoy[908]: [2026-09-23 05:12:14.594][908][info][main] [source/server/server.cc:534] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id lb # [ 20.585852] envoy[908]: [2026-09-23 05:12:14.594][908][info][main] [source/server/server.cc:534] response trailer map: 144 bytes: grpc-message,grpc-status lb # [ 20.587692] envoy[908]: [2026-09-23 05:12:14.661][908][info][main] [source/server/server.cc:968] runtime: {} lb # [ 20.589219] envoy[908]: [2026-09-23 05:12:14.676][908][info][main] [source/server/server.cc:420] Raised file descriptor limits from 1024 to 524288. lb # [ 20.590875] envoy[908]: [2026-09-23 05:12:14.676][908][info][main] [source/server/server.cc:796] Starting admin HTTP server at 127.0.0.1:9901 lb # [ 20.592645] envoy[908]: [2026-09-23 05:12:14.676][908][info][admin] [source/server/admin/admin.cc:66] admin address: 127.0.0.1:9901 lb # [ 20.594111] envoy[908]: [2026-09-23 05:12:14.676][908][info][config] [source/server/configuration_impl.cc:181] loading tracing configuration lb # [ 20.595670] envoy[908]: [2026-09-23 05:12:14.677][908][info][config] [source/server/configuration_impl.cc:132] loading 0 static secret(s) lb # [ 20.597297] envoy[908]: [2026-09-23 05:12:14.677][908][info][config] [source/server/configuration_impl.cc:138] loading 2 cluster(s) lb # [ 20.599091] envoy[908]: [2026-09-23 05:12:14.782][908][info][config] [source/server/configuration_impl.cc:148] loading 1 listener(s) lb # [ 20.600677] envoy[908]: [2026-09-23 05:12:14.790][908][info][config] [source/server/configuration_impl.cc:164] loading stats configuration worker1 # [ 20.579660] nix-grpc-daemon[767]: E0000 00:00:1790140335.081846 809 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb # [ 20.602283] envoy[908]: [2026-09-23 05:12:14.802][908][warning][main] [source/server/server.cc:1038] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.global_downstream_max_connections` resource monitor. lb # [ 20.608166] envoy[908]: [2026-09-23 05:12:14.802][908][info][main] [source/server/server.cc:1077] starting main dispatch loop lb # [ 20.609885] envoy[908]: [2026-09-23 05:12:15.058][908][info][runtime] [source/common/runtime/runtime_impl.cc:567] RTDS has finished initialization lb # [ 20.612816] envoy[908]: [2026-09-23 05:12:15.058][908][info][upstream] [source/common/upstream/cluster_manager_impl.cc:250] cm init: all clusters initialized worker1 # [ 20.596407] nix-grpc-daemon[767]: ts=2026-09-23T05:12:15Z level=info event=scheduler_connected addr=lb:50051 lb # [ 20.615520] envoy[908]: [2026-09-23 05:12:15.058][908][info][main] [source/server/server.cc:1057] all clusters initialized. initializing init manager lb # [ 20.618120] envoy[908]: [2026-09-23 05:12:15.092][908][info][config] [source/common/listener_manager/listener_manager_impl.cc:1067] all dependencies initialized. starting workers worker1 # [ 20.602219] nix-grpc-daemon[767]: ts=2026-09-23T05:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.630571] nix-grpc-daemon[773]: ts=2026-09-23T05:12:15Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 20.641380] nix-grpc-daemon[773]: ts=2026-09-23T05:12:15Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 20.643841] nix-grpc-daemon[773]: ts=2026-09-23T05:12:15Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 20.655140] nix-grpc-daemon[773]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 20.658201] nix-grpc-daemon[773]: E0000 00:00:1790140335.154292 773 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 20.664644] nix-grpc-daemon[773]: E0000 00:00:1790140335.164079 773 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 20.686361] nix-grpc-daemon[773]: ts=2026-09-23T05:12:15Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 20.693751] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 20.700112] nix-grpc-daemon[773]: E0000 00:00:1790140335.198810 816 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 20.723952] nix-grpc-daemon[773]: E0000 00:00:1790140335.223083 816 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 20.728094] nix-grpc-daemon[773]: E0000 00:00:1790140335.226338 816 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 20.749492] nix-grpc-daemon[773]: E0000 00:00:1790140335.247650 816 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 20.766540] nix-grpc-daemon[773]: ts=2026-09-23T05:12:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.768947] nix-grpc-daemon[773]: ts=2026-09-23T05:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 20.804607] nix-grpc-daemon[767]: ts=2026-09-23T05:12:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 20.810580] nix-grpc-daemon[767]: ts=2026-09-23T05:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 20.970957] nix-grpc-daemon[773]: ts=2026-09-23T05:12:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 20.976970] nix-grpc-daemon[773]: ts=2026-09-23T05:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.011797] nix-grpc-daemon[767]: ts=2026-09-23T05:12:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.018010] nix-grpc-daemon[767]: ts=2026-09-23T05:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.178164] nix-grpc-daemon[773]: ts=2026-09-23T05:12:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.183870] nix-grpc-daemon[773]: ts=2026-09-23T05:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.218936] nix-grpc-daemon[767]: ts=2026-09-23T05:12:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.224913] nix-grpc-daemon[767]: ts=2026-09-23T05:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.384914] nix-grpc-daemon[773]: ts=2026-09-23T05:12:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.390361] nix-grpc-daemon[773]: ts=2026-09-23T05:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.425992] nix-grpc-daemon[767]: ts=2026-09-23T05:12:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.431651] nix-grpc-daemon[767]: ts=2026-09-23T05:12:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.591679] nix-grpc-daemon[773]: ts=2026-09-23T05:12:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.597133] nix-grpc-daemon[773]: ts=2026-09-23T05:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.634895] nix-grpc-daemon[767]: ts=2026-09-23T05:12:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.639318] nix-grpc-daemon[767]: ts=2026-09-23T05:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 21.800358] nix-grpc-daemon[773]: ts=2026-09-23T05:12:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 21.804679] nix-grpc-daemon[773]: ts=2026-09-23T05:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 21.835707] nix-grpc-daemon[767]: ts=2026-09-23T05:12:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 21.841466] nix-grpc-daemon[767]: ts=2026-09-23T05:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.001864] nix-grpc-daemon[773]: ts=2026-09-23T05:12:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.007535] nix-grpc-daemon[773]: ts=2026-09-23T05:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.042489] nix-grpc-daemon[767]: ts=2026-09-23T05:12:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.048371] nix-grpc-daemon[767]: ts=2026-09-23T05:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.209073] nix-grpc-daemon[773]: ts=2026-09-23T05:12:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.214874] nix-grpc-daemon[773]: ts=2026-09-23T05:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.249309] nix-grpc-daemon[767]: ts=2026-09-23T05:12:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.255160] nix-grpc-daemon[767]: ts=2026-09-23T05:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.416327] nix-grpc-daemon[773]: ts=2026-09-23T05:12:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.421937] nix-grpc-daemon[773]: ts=2026-09-23T05:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.456330] nix-grpc-daemon[767]: ts=2026-09-23T05:12:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.462034] nix-grpc-daemon[767]: ts=2026-09-23T05:12:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.623199] nix-grpc-daemon[773]: ts=2026-09-23T05:12:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.628787] nix-grpc-daemon[773]: ts=2026-09-23T05:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.664988] nix-grpc-daemon[767]: ts=2026-09-23T05:12:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.669246] nix-grpc-daemon[767]: ts=2026-09-23T05:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 22.832568] nix-grpc-daemon[773]: ts=2026-09-23T05:12:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 22.837036] nix-grpc-daemon[773]: ts=2026-09-23T05:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 22.866037] nix-grpc-daemon[767]: ts=2026-09-23T05:12:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 22.872353] nix-grpc-daemon[767]: ts=2026-09-23T05:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.034255] nix-grpc-daemon[773]: ts=2026-09-23T05:12:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.039993] nix-grpc-daemon[773]: ts=2026-09-23T05:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.073331] nix-grpc-daemon[767]: ts=2026-09-23T05:12:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.078953] nix-grpc-daemon[767]: ts=2026-09-23T05:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.241296] nix-grpc-daemon[773]: ts=2026-09-23T05:12:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.246788] nix-grpc-daemon[773]: ts=2026-09-23T05:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.279853] nix-grpc-daemon[767]: ts=2026-09-23T05:12:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.285561] nix-grpc-daemon[767]: ts=2026-09-23T05:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.448096] nix-grpc-daemon[773]: ts=2026-09-23T05:12:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.453984] nix-grpc-daemon[773]: ts=2026-09-23T05:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.486371] nix-grpc-daemon[767]: ts=2026-09-23T05:12:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.492208] nix-grpc-daemon[767]: ts=2026-09-23T05:12:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.655168] nix-grpc-daemon[773]: ts=2026-09-23T05:12:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.661360] nix-grpc-daemon[773]: ts=2026-09-23T05:12:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 23.694957] nix-grpc-daemon[767]: ts=2026-09-23T05:12:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 23.699336] nix-grpc-daemon[767]: ts=2026-09-23T05:12:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 23.864699] nix-grpc-daemon[773]: ts=2026-09-23T05:12:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 23.869191] nix-grpc-daemon[773]: ts=2026-09-23T05:12:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 23.894667] rustfs-bucket-start[953]: mb s3://farm lb # [ 23.904392] systemd[1]: Finished rustfs-bucket.service. worker1 # [ 23.895670] nix-grpc-daemon[767]: ts=2026-09-23T05:12:18Z level=info event=scheduler_connected addr=lb:50051 lb # [ 23.919213] systemd[1]: Starting niks3 server... worker1 # [ 23.901296] nix-grpc-daemon[767]: ts=2026-09-23T05:12:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.065913] nix-grpc-daemon[773]: ts=2026-09-23T05:12:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.071990] nix-grpc-daemon[773]: ts=2026-09-23T05:12:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 24.099713] postgres[977]: [977] ERROR: relation "goose_db_version" does not exist at character 36 lb # [ 24.104170] postgres[977]: [977] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC worker1 # [ 24.102181] nix-grpc-daemon[767]: ts=2026-09-23T05:12:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.107656] nix-grpc-daemon[767]: ts=2026-09-23T05:12:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 24.134022] niks3-server[970]: 2026/09/23 05:12:18 OK 20241026095416_initial_model.sql (19.31ms) lb # [ 24.138537] niks3-server[970]: 2026/09/23 05:12:18 OK 20251210153512_drop_unused_gin_index.sql (4.87ms) lb # [ 24.145259] niks3-server[970]: 2026/09/23 05:12:18 OK 20251218171726_add_pins.sql (6.56ms) lb # [ 24.150810] niks3-server[970]: 2026/09/23 05:12:18 OK 20260628120000_add_object_size_and_stats.sql (5.56ms) lb # [ 24.157204] niks3-server[970]: 2026/09/23 05:12:18 OK 20260905000000_add_claims.sql (6.32ms) lb # [ 24.161833] niks3-server[970]: 2026/09/23 05:12:18 OK 20260920000000_drop_claims.sql (4.44ms) lb # [ 24.163129] niks3-server[970]: 2026/09/23 05:12:18 goose: successfully migrated database to version: 20260920000000 lb # [ 24.168344] niks3-server[970]: 2026/09/23 05:12:18 OK 1_commit_pending_closure.sql (6.48ms) lb # [ 24.171704] niks3-server[970]: 2026/09/23 05:12:18 OK 2_object_stats_trigger.sql (3.16ms) lb # [ 24.174015] niks3-server[970]: 2026/09/23 05:12:18 goose: up to current file version: 2 lb # [ 24.175911] niks3-server[970]: 2026/09/23 05:12:18 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key lb # [ 24.205766] niks3-server[970]: 2026/09/23 05:12:18 INFO Read proxy enabled — serving cache objects from S3 lb # [ 24.210095] niks3-server[970]: 2026/09/23 05:12:18 INFO Using socket-activated listener address=0.0.0.0:5751 lb # [ 24.217095] niks3-server[970]: 2026/09/23 05:12:18 INFO systemd watchdog enabled interval=15s lb # [ 24.219748] niks3-server[970]: 2026/09/23 05:12:18 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 24.221423] systemd[1]: Started niks3 server. lb # [ 24.222794] systemd[1]: Reached target Multi-User System. lb # [ 24.223847] systemd[1]: Startup finished in 1.063s (kernel) + 5.047s (initrd) + 18.107s (userspace) = 24.219s. worker2 # [ 24.273672] nix-grpc-daemon[773]: ts=2026-09-23T05:12:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.279931] nix-grpc-daemon[773]: ts=2026-09-23T05:12:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.308627] nix-grpc-daemon[767]: ts=2026-09-23T05:12:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.314107] nix-grpc-daemon[767]: ts=2026-09-23T05:12:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.481338] nix-grpc-daemon[773]: ts=2026-09-23T05:12:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.487035] nix-grpc-daemon[773]: ts=2026-09-23T05:12:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.515070] nix-grpc-daemon[767]: ts=2026-09-23T05:12:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.520665] nix-grpc-daemon[767]: ts=2026-09-23T05:12:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.688231] nix-grpc-daemon[773]: ts=2026-09-23T05:12:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.693806] nix-grpc-daemon[773]: ts=2026-09-23T05:12:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.723492] nix-grpc-daemon[767]: ts=2026-09-23T05:12:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.727864] nix-grpc-daemon[767]: ts=2026-09-23T05:12:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.896711] nix-grpc-daemon[773]: ts=2026-09-23T05:12:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.902094] nix-grpc-daemon[773]: ts=2026-09-23T05:12:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.924292] nix-grpc-daemon[767]: ts=2026-09-23T05:12:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.930015] nix-grpc-daemon[767]: ts=2026-09-23T05:12:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: waiting for unit niks3.service, in 12.02 seconds) lb: waiting for TCP port 5751 on localhost worker2 # [ 25.103144] nix-grpc-daemon[773]: ts=2026-09-23T05:12:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.108942] nix-grpc-daemon[773]: ts=2026-09-23T05:12:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded! lb: (finished: waiting for TCP port 5751 on localhost, in 0.08 seconds) lb: waiting for unit envoy.service worker1 # [ 25.130773] nix-grpc-daemon[767]: ts=2026-09-23T05:12:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.141068] nix-grpc-daemon[767]: ts=2026-09-23T05:12:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: waiting for unit envoy.service, in 0.06 seconds) lb: waiting for TCP port 50051 on localhost lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded! lb # [ 25.234732] envoy[908]: conn peer=[::1]:41318 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=4 lb: (finished: waiting for TCP port 50051 on localhost, in 0.04 seconds) worker2 # [ 25.310281] nix-grpc-daemon[773]: ts=2026-09-23T05:12:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.318032] nix-grpc-daemon[773]: ts=2026-09-23T05:12:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.342160] nix-grpc-daemon[767]: ts=2026-09-23T05:12:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.347646] nix-grpc-daemon[767]: ts=2026-09-23T05:12:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.405183] systemd[1]: Starting scheduler-token.service... worker2 # [ 25.479213] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 25.482704] systemd[1]: Finished scheduler-token.service. worker2 # [ 25.490259] systemd[1]: scheduler-token.service: Consumed 44ms CPU time over 79ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 25.517554] nix-grpc-daemon[773]: ts=2026-09-23T05:12:20Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.521487] nix-grpc-daemon[773]: ts=2026-09-23T05:12:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.548555] nix-grpc-daemon[767]: ts=2026-09-23T05:12:20Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.555180] nix-grpc-daemon[767]: ts=2026-09-23T05:12:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.11 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.11 seconds) subtest: one node schedules and both builders hold a WorkerSession on it lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) worker2 # [ 25.722297] nix-grpc-daemon[773]: ts=2026-09-23T05:12:20Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.725811] nix-grpc-daemon[773]: ts=2026-09-23T05:12:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.757927] nix-grpc-daemon[767]: ts=2026-09-23T05:12:20Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.762181] nix-grpc-daemon[767]: ts=2026-09-23T05:12:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.928484] nix-grpc-daemon[773]: ts=2026-09-23T05:12:20Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.933563] nix-grpc-daemon[773]: ts=2026-09-23T05:12:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.958244] nix-grpc-daemon[767]: ts=2026-09-23T05:12:20Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.963593] nix-grpc-daemon[767]: ts=2026-09-23T05:12:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.134766] nix-grpc-daemon[773]: ts=2026-09-23T05:12:20Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.140260] nix-grpc-daemon[773]: ts=2026-09-23T05:12:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.164478] nix-grpc-daemon[767]: ts=2026-09-23T05:12:20Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.169653] nix-grpc-daemon[767]: ts=2026-09-23T05:12:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.341512] nix-grpc-daemon[773]: ts=2026-09-23T05:12:20Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.346848] nix-grpc-daemon[773]: ts=2026-09-23T05:12:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.370557] nix-grpc-daemon[767]: ts=2026-09-23T05:12:20Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.375888] nix-grpc-daemon[767]: ts=2026-09-23T05:12:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.548132] nix-grpc-daemon[773]: ts=2026-09-23T05:12:21Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.553625] nix-grpc-daemon[773]: ts=2026-09-23T05:12:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.576839] nix-grpc-daemon[767]: ts=2026-09-23T05:12:21Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.582390] nix-grpc-daemon[767]: ts=2026-09-23T05:12:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 26.754707] nix-grpc-daemon[773]: ts=2026-09-23T05:12:21Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.760663] nix-grpc-daemon[773]: ts=2026-09-23T05:12:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.785476] nix-grpc-daemon[767]: ts=2026-09-23T05:12:21Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.789744] nix-grpc-daemon[767]: ts=2026-09-23T05:12:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.962675] nix-grpc-daemon[773]: ts=2026-09-23T05:12:21Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.966144] nix-grpc-daemon[773]: ts=2026-09-23T05:12:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.986220] nix-grpc-daemon[767]: ts=2026-09-23T05:12:21Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.990847] nix-grpc-daemon[767]: ts=2026-09-23T05:12:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.167267] nix-grpc-daemon[773]: ts=2026-09-23T05:12:21Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.171338] nix-grpc-daemon[773]: ts=2026-09-23T05:12:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.192072] nix-grpc-daemon[767]: ts=2026-09-23T05:12:21Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.197852] nix-grpc-daemon[767]: ts=2026-09-23T05:12:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.372071] nix-grpc-daemon[773]: ts=2026-09-23T05:12:21Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.374410] nix-grpc-daemon[773]: ts=2026-09-23T05:12:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.398377] nix-grpc-daemon[767]: ts=2026-09-23T05:12:21Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.401001] nix-grpc-daemon[767]: ts=2026-09-23T05:12:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.576074] nix-grpc-daemon[773]: ts=2026-09-23T05:12:22Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.581653] nix-grpc-daemon[773]: ts=2026-09-23T05:12:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.602292] nix-grpc-daemon[767]: ts=2026-09-23T05:12:22Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.608108] nix-grpc-daemon[767]: ts=2026-09-23T05:12:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 27.783242] nix-grpc-daemon[773]: ts=2026-09-23T05:12:22Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.789320] nix-grpc-daemon[773]: ts=2026-09-23T05:12:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1 # [ 27.810941] nix-grpc-daemon[767]: ts=2026-09-23T05:12:22Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.815301] nix-grpc-daemon[767]: ts=2026-09-23T05:12:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.993151] nix-grpc-daemon[773]: ts=2026-09-23T05:12:22Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.997595] nix-grpc-daemon[773]: ts=2026-09-23T05:12:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.011876] nix-grpc-daemon[767]: ts=2026-09-23T05:12:22Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.017472] nix-grpc-daemon[767]: ts=2026-09-23T05:12:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.194476] nix-grpc-daemon[773]: ts=2026-09-23T05:12:22Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.200548] nix-grpc-daemon[773]: ts=2026-09-23T05:12:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.218474] nix-grpc-daemon[767]: ts=2026-09-23T05:12:22Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.224475] nix-grpc-daemon[767]: ts=2026-09-23T05:12:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.402060] nix-grpc-daemon[773]: ts=2026-09-23T05:12:22Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.407843] nix-grpc-daemon[773]: ts=2026-09-23T05:12:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.425147] nix-grpc-daemon[767]: ts=2026-09-23T05:12:22Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.433563] nix-grpc-daemon[767]: ts=2026-09-23T05:12:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.609275] nix-grpc-daemon[773]: ts=2026-09-23T05:12:23Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.615431] nix-grpc-daemon[773]: ts=2026-09-23T05:12:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.633110] nix-grpc-daemon[767]: ts=2026-09-23T05:12:23Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.639185] nix-grpc-daemon[767]: ts=2026-09-23T05:12:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 28.816891] nix-grpc-daemon[773]: ts=2026-09-23T05:12:23Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.823036] nix-grpc-daemon[773]: ts=2026-09-23T05:12:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.841480] nix-grpc-daemon[767]: ts=2026-09-23T05:12:23Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.847178] nix-grpc-daemon[767]: ts=2026-09-23T05:12:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker2 # [ 29.026663] nix-grpc-daemon[773]: ts=2026-09-23T05:12:23Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.031098] nix-grpc-daemon[773]: ts=2026-09-23T05:12:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.048773] nix-grpc-daemon[767]: ts=2026-09-23T05:12:23Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.055049] nix-grpc-daemon[767]: ts=2026-09-23T05:12:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.227512] nix-grpc-daemon[773]: ts=2026-09-23T05:12:23Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.231962] nix-grpc-daemon[773]: ts=2026-09-23T05:12:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.255790] nix-grpc-daemon[767]: ts=2026-09-23T05:12:23Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.261400] nix-grpc-daemon[767]: ts=2026-09-23T05:12:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.433185] nix-grpc-daemon[773]: ts=2026-09-23T05:12:23Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.437141] nix-grpc-daemon[773]: ts=2026-09-23T05:12:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.462318] nix-grpc-daemon[767]: ts=2026-09-23T05:12:23Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.468097] nix-grpc-daemon[767]: ts=2026-09-23T05:12:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.638439] nix-grpc-daemon[773]: ts=2026-09-23T05:12:24Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.644348] nix-grpc-daemon[773]: ts=2026-09-23T05:12:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.668977] nix-grpc-daemon[767]: ts=2026-09-23T05:12:24Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.675149] nix-grpc-daemon[767]: ts=2026-09-23T05:12:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.845533] nix-grpc-daemon[773]: ts=2026-09-23T05:12:24Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.851343] nix-grpc-daemon[773]: ts=2026-09-23T05:12:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 29.877513] nix-grpc-daemon[767]: ts=2026-09-23T05:12:24Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.883172] nix-grpc-daemon[767]: ts=2026-09-23T05:12:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 30.055073] nix-grpc-daemon[773]: ts=2026-09-23T05:12:24Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.059456] nix-grpc-daemon[773]: ts=2026-09-23T05:12:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.084000] nix-grpc-daemon[767]: ts=2026-09-23T05:12:24Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.088805] nix-grpc-daemon[767]: ts=2026-09-23T05:12:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.256228] nix-grpc-daemon[773]: ts=2026-09-23T05:12:24Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.261846] nix-grpc-daemon[773]: ts=2026-09-23T05:12:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.289904] nix-grpc-daemon[767]: ts=2026-09-23T05:12:24Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.295511] nix-grpc-daemon[767]: ts=2026-09-23T05:12:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.462693] nix-grpc-daemon[773]: ts=2026-09-23T05:12:24Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.465836] nix-grpc-daemon[773]: ts=2026-09-23T05:12:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.496493] nix-grpc-daemon[767]: ts=2026-09-23T05:12:24Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.501947] nix-grpc-daemon[767]: ts=2026-09-23T05:12:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.667347] nix-grpc-daemon[773]: ts=2026-09-23T05:12:25Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.673646] nix-grpc-daemon[773]: ts=2026-09-23T05:12:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.702405] nix-grpc-daemon[767]: ts=2026-09-23T05:12:25Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.706295] nix-grpc-daemon[767]: ts=2026-09-23T05:12:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.874831] nix-grpc-daemon[773]: ts=2026-09-23T05:12:25Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.882405] nix-grpc-daemon[773]: ts=2026-09-23T05:12:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.908796] nix-grpc-daemon[767]: ts=2026-09-23T05:12:25Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.913283] nix-grpc-daemon[767]: ts=2026-09-23T05:12:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2 # [ 31.084883] nix-grpc-daemon[773]: ts=2026-09-23T05:12:25Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.090183] nix-grpc-daemon[773]: ts=2026-09-23T05:12:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.114449] nix-grpc-daemon[767]: ts=2026-09-23T05:12:25Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.119989] nix-grpc-daemon[767]: ts=2026-09-23T05:12:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.291616] nix-grpc-daemon[773]: ts=2026-09-23T05:12:25Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.297259] nix-grpc-daemon[773]: ts=2026-09-23T05:12:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.320998] nix-grpc-daemon[767]: ts=2026-09-23T05:12:25Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.326669] nix-grpc-daemon[767]: ts=2026-09-23T05:12:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.498401] nix-grpc-daemon[773]: ts=2026-09-23T05:12:25Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.503935] nix-grpc-daemon[773]: ts=2026-09-23T05:12:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.527823] nix-grpc-daemon[767]: ts=2026-09-23T05:12:26Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.533536] nix-grpc-daemon[767]: ts=2026-09-23T05:12:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.705158] nix-grpc-daemon[773]: ts=2026-09-23T05:12:26Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.710754] nix-grpc-daemon[773]: ts=2026-09-23T05:12:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.739192] nix-grpc-daemon[767]: ts=2026-09-23T05:12:26Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.748287] nix-grpc-daemon[767]: ts=2026-09-23T05:12:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.911556] nix-grpc-daemon[773]: ts=2026-09-23T05:12:26Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.915862] nix-grpc-daemon[773]: ts=2026-09-23T05:12:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.942279] nix-grpc-daemon[767]: ts=2026-09-23T05:12:26Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.946564] nix-grpc-daemon[767]: ts=2026-09-23T05:12:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2 # [ 32.119295] nix-grpc-daemon[773]: ts=2026-09-23T05:12:26Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.123567] nix-grpc-daemon[773]: ts=2026-09-23T05:12:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.142507] nix-grpc-daemon[767]: ts=2026-09-23T05:12:26Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.145848] nix-grpc-daemon[767]: ts=2026-09-23T05:12:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.320327] nix-grpc-daemon[773]: ts=2026-09-23T05:12:26Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.325919] nix-grpc-daemon[773]: ts=2026-09-23T05:12:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.347091] nix-grpc-daemon[767]: ts=2026-09-23T05:12:26Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.352802] nix-grpc-daemon[767]: ts=2026-09-23T05:12:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.527507] nix-grpc-daemon[773]: ts=2026-09-23T05:12:27Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.534077] nix-grpc-daemon[773]: ts=2026-09-23T05:12:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.553655] nix-grpc-daemon[767]: ts=2026-09-23T05:12:27Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.559290] nix-grpc-daemon[767]: ts=2026-09-23T05:12:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.734968] nix-grpc-daemon[773]: ts=2026-09-23T05:12:27Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.740605] nix-grpc-daemon[773]: ts=2026-09-23T05:12:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.759808] nix-grpc-daemon[767]: ts=2026-09-23T05:12:27Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.762111] nix-grpc-daemon[767]: ts=2026-09-23T05:12:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.941752] nix-grpc-daemon[773]: ts=2026-09-23T05:12:27Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.947329] nix-grpc-daemon[773]: ts=2026-09-23T05:12:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.965294] nix-grpc-daemon[767]: ts=2026-09-23T05:12:27Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.969561] nix-grpc-daemon[767]: ts=2026-09-23T05:12:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 33.150813] nix-grpc-daemon[773]: ts=2026-09-23T05:12:27Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.155180] nix-grpc-daemon[773]: ts=2026-09-23T05:12:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.165970] nix-grpc-daemon[767]: ts=2026-09-23T05:12:27Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.171239] nix-grpc-daemon[767]: ts=2026-09-23T05:12:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.352249] nix-grpc-daemon[773]: ts=2026-09-23T05:12:27Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.358055] nix-grpc-daemon[773]: ts=2026-09-23T05:12:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.372516] nix-grpc-daemon[767]: ts=2026-09-23T05:12:27Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.378442] nix-grpc-daemon[767]: ts=2026-09-23T05:12:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.559170] nix-grpc-daemon[773]: ts=2026-09-23T05:12:28Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.564763] nix-grpc-daemon[773]: ts=2026-09-23T05:12:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.579397] nix-grpc-daemon[767]: ts=2026-09-23T05:12:28Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.584412] nix-grpc-daemon[767]: ts=2026-09-23T05:12:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.765929] nix-grpc-daemon[773]: ts=2026-09-23T05:12:28Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.771475] nix-grpc-daemon[773]: ts=2026-09-23T05:12:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.781438] nix-grpc-daemon[767]: ts=2026-09-23T05:12:28Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.787157] nix-grpc-daemon[767]: ts=2026-09-23T05:12:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.972646] nix-grpc-daemon[773]: ts=2026-09-23T05:12:28Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.977946] nix-grpc-daemon[773]: ts=2026-09-23T05:12:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.990322] nix-grpc-daemon[767]: ts=2026-09-23T05:12:28Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.994654] nix-grpc-daemon[767]: ts=2026-09-23T05:12:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2 # [ 34.180469] nix-grpc-daemon[773]: ts=2026-09-23T05:12:28Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.185257] nix-grpc-daemon[773]: ts=2026-09-23T05:12:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.191340] nix-grpc-daemon[767]: ts=2026-09-23T05:12:28Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.196909] nix-grpc-daemon[767]: ts=2026-09-23T05:12:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 34.218802] niks3-server[970]: 2026/09/23 05:12:28 INFO lead: acquired remote=192.168.1.3:39118 worker1 # [ 34.202797] nix-grpc-daemon[767]: ts=2026-09-23T05:12:28Z level=info event=scheduler_take_over worker2 # [ 34.387701] nix-grpc-daemon[773]: ts=2026-09-23T05:12:28Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.394257] nix-grpc-daemon[773]: ts=2026-09-23T05:12:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.398269] nix-grpc-daemon[767]: ts=2026-09-23T05:12:28Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.405054] nix-grpc-daemon[767]: ts=2026-09-23T05:12:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.595020] nix-grpc-daemon[773]: ts=2026-09-23T05:12:29Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.605791] nix-grpc-daemon[767]: ts=2026-09-23T05:12:29Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.613290] nix-grpc-daemon[767]: ts=2026-09-23T05:12:29Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:46736 worker1 # [ 34.623204] nix-grpc-daemon[767]: ts=2026-09-23T05:12:29Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 34.629378] nix-grpc-daemon[767]: ts=2026-09-23T05:12:29Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 34.633415] nix-grpc-daemon[767]: ts=2026-09-23T05:12:29Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:46736 worker1 # [ 34.638713] nix-grpc-daemon[767]: ts=2026-09-23T05:12:29Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open' worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.06 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.04 seconds) (finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 10.26 seconds) subtest: balancer auth: client cert, bearer token, nothing, unknown CN worker2 # [ 36.136917] nix-grpc-daemon[773]: ts=2026-09-23T05:12:30Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 36.154549] systemd[1]: Started Nix Daemon. worker2 # [ 36.350622] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 36.370722] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 77 ms (attempt 1/5) worker2 # [ 36.452518] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 74 ms (attempt 2/5) worker2 # [ 36.528045] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 194 ms (attempt 3/5) worker2 # [ 36.723643] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 320 ms (attempt 4/5) worker2 # [ 37.047092] nix-grpc-daemon[773]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 37.052537] nix-grpc-daemon[773]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 37.068989] nix-grpc-daemon[773]: ts=2026-09-23T05:12:31Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 37.083067] nix-grpc-daemon[773]: ts=2026-09-23T05:12:31Z level=debug event=path_info_miss path=5fx3y1hbxhkdng69ksrxcn3js8649kk7-nixos-system-client-test error="error: path '/nix/store/5fx3y1hbxhkdng69ksrxcn3js8649kk7-nixos-system-client-test' is required, but there is no substituter that can build it" worker2 # [ 37.096963] nix-grpc-daemon[773]: ts=2026-09-23T05:12:31Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 37.107836] nix-grpc-daemon[773]: ts=2026-09-23T05:12:31Z level=debug event=path_info_miss path=5fx3y1hbxhkdng69ksrxcn3js8649kk7-nixos-system-client-test error="error: path '/nix/store/5fx3y1hbxhkdng69ksrxcn3js8649kk7-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.08 seconds) client: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key client: (finished: must succeed: install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.04 seconds) worker2 # [ 37.445454] nix-grpc-daemon[773]: ts=2026-09-23T05:12:31Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 37.451989] nix-grpc-daemon[773]: ts=2026-09-23T05:12:31Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:59064 worker2 # [ 37.464954] nix-grpc-daemon[773]: ts=2026-09-23T05:12:31Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:59064 worker2 # [ 37.476937] nix-grpc-daemon[773]: ts=2026-09-23T05:12:31Z level=debug event=path_info_miss path=5fx3y1hbxhkdng69ksrxcn3js8649kk7-nixos-system-client-test error="error: path '/nix/store/5fx3y1hbxhkdng69ksrxcn3js8649kk7-nixos-system-client-test' is required, but there is no substituter that can build it" worker2 # [ 37.492533] nix-grpc-daemon[773]: ts=2026-09-23T05:12:31Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:59064 worker2 # [ 37.500768] nix-grpc-daemon[773]: ts=2026-09-23T05:12:31Z level=debug event=path_info_miss path=5fx3y1hbxhkdng69ksrxcn3js8649kk7-nixos-system-client-test error="error: path '/nix/store/5fx3y1hbxhkdng69ksrxcn3js8649kk7-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: rm -r /var/lib/nix-grpc-store client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.03 seconds) worker2 # [ 37.735058] nix-grpc-daemon[773]: ts=2026-09-23T05:12:32Z level=info event=denied method=QueryMissing cn=- role=none worker1 # [ 37.917267] nix-grpc-daemon[767]: ts=2026-09-23T05:12:32Z level=info event=denied method=QueryMissing cn=stranger role=none (finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 2.08 seconds) subtest: DAG build is scheduled across workers and lands in the cache client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2 worker1 # [ 38.101289] nix-grpc-daemon[767]: ts=2026-09-23T05:12:32Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 38.118704] systemd[1]: Started Nix Daemon. worker1 # [ 38.305332] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) client # don't know how to build these paths: client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker2 # [ 38.335572] nix-grpc-daemon[773]: ts=2026-09-23T05:12:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59064 worker1 # [ 38.365138] nix-grpc-daemon[767]: ts=2026-09-23T05:12:32Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=0 worker1 # [ 38.373008] nix-grpc-daemon[767]: ts=2026-09-23T05:12:32Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 38.425995] nix-grpc-daemon[767]: ts=2026-09-23T05:12:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 38.448931] nix-grpc-daemon[767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 86 ms (attempt 1/5) worker1 # [ 38.539246] nix-grpc-daemon[767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 56 ms (attempt 2/5) worker1 # [ 38.597494] nix-grpc-daemon[767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 13 ms (attempt 3/5) worker1 # [ 38.612438] nix-grpc-daemon[767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 13 ms (attempt 4/5) worker1 # [ 38.635496] nix-grpc-daemon[767]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 38.641597] nix-grpc-daemon[767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 38.658188] nix-grpc-daemon[767]: ts=2026-09-23T05:12:33Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'... worker1 # [ 38.674571] nix-grpc-daemon[767]: ts=2026-09-23T05:12:33Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 38.692133] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 38.722336] nix-grpc-daemon[767]: ts=2026-09-23T05:12:33Z level=info event=niks3_push_started pid=865 worker1 # [ 38.938060] nix-grpc-daemon[865]: time=2026-09-23T05:12:33.439Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 39.063084] nix-daemon[850]: accepted connection from pid 878, user nix-grpc-daemon (trusted) worker1 # [ 39.085882] nix-daemon[850]: reaped child process 881, status = succeeded lb # [ 39.114757] niks3-server[970]: 2026/09/23 05:12:33 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 39.119123] nix-grpc-daemon[865]: time=2026-09-23T05:12:33.620Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 39.123799] nix-grpc-daemon[865]: time=2026-09-23T05:12:33.620Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)" lb # [ 39.179880] niks3-server[970]: 2026/09/23 05:12:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign lb # [ 39.224497] niks3-server[970]: 2026/09/23 05:12:33 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls lb # [ 39.239051] niks3-server[970]: 2026/09/23 05:12:33 INFO Signed narinfos id=1 count=1 worker1 # [ 39.223302] nix-grpc-daemon[865]: time=2026-09-23T05:12:33.725Z level=INFO msg="Uploading 1 narinfos" lb # [ 39.249124] niks3-server[970]: 2026/09/23 05:12:33 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst lb # [ 39.256145] niks3-server[970]: 2026/09/23 05:12:33 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete lb # [ 39.267850] niks3-server[970]: 2026/09/23 05:12:33 INFO Completed upload id=1 worker1 # [ 39.249800] nix-grpc-daemon[865]: time=2026-09-23T05:12:33.752Z level=INFO msg="Upload complete. (308ms)" lb # [ 39.271684] niks3-server[970]: 2026/09/23 05:12:33 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo worker1 # [ 39.261391] nix-grpc-daemon[767]: ts=2026-09-23T05:12:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=0 paths=1 nar_bytes_in=504 worker1 # [ 39.276486] nix-grpc-daemon[767]: ts=2026-09-23T05:12:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 39.282146] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 39.286654] nix-daemon[850]: reaped child process 863, status = succeeded worker1 # [ 39.297423] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051... client # farm-a> node-a: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds lb # [ 40.273931] envoy[908]: conn peer=[2001:db8:1::1]:46462 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=715 tx=362 ms=1002 lb # [ 40.279813] envoy[908]: conn peer=[2001:db8:1::1]:46470 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=77 lb # [ 40.285241] envoy[908]: conn peer=[2001:db8:1::1]:46478 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=12 lb # [ 40.291184] envoy[908]: conn peer=[2001:db8:1::1]:46490 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=182 ms=29 client # farm-a> LOG-farm-a-t1 worker1 # [ 40.544229] nix-daemon[850]: reaped child process 885, status = succeeded worker1 # [ 40.664462] nix-daemon[850]: accepted connection from pid 901, user nix-grpc-daemon (trusted) worker1 # [ 40.689049] nix-daemon[850]: reaped child process 904, status = succeeded lb # [ 40.707614] niks3-server[970]: 2026/09/23 05:12:35 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 40.708218] nix-grpc-daemon[865]: time=2026-09-23T05:12:35.208Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 40.709900] nix-grpc-daemon[865]: time=2026-09-23T05:12:35.208Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)" lb # [ 40.776552] niks3-server[970]: 2026/09/23 05:12:35 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker1 # [ 40.782694] nix-grpc-daemon[865]: time=2026-09-23T05:12:35.285Z level=INFO msg="Uploading 1 narinfos" lb # [ 40.797640] niks3-server[970]: 2026/09/23 05:12:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign lb # [ 40.806214] niks3-server[970]: 2026/09/23 05:12:35 INFO Signed narinfos id=2 count=1 lb # [ 40.807722] niks3-server[970]: 2026/09/23 05:12:35 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst lb # [ 40.815061] niks3-server[970]: 2026/09/23 05:12:35 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls lb # [ 40.824795] niks3-server[970]: 2026/09/23 05:12:35 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete lb # [ 40.828079] niks3-server[970]: 2026/09/23 05:12:35 INFO Completed upload id=2 worker1 # [ 40.809435] nix-grpc-daemon[865]: time=2026-09-23T05:12:35.312Z level=INFO msg="Upload complete. (299ms)" lb # [ 40.830610] niks3-server[970]: 2026/09/23 05:12:35 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo worker1 # [ 40.825774] nix-grpc-daemon[767]: ts=2026-09-23T05:12:35Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=1 worker1 # [ 40.840251] nix-grpc-daemon[767]: ts=2026-09-23T05:12:35Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=0 worker1 # [ 40.843260] nix-grpc-daemon[767]: ts=2026-09-23T05:12:35Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 40.846953] nix-daemon[850]: reaped child process 883, status = succeeded worker1 # [ 40.878200] nix-grpc-daemon[767]: ts=2026-09-23T05:12:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 40.892381] nix-grpc-daemon[767]: ts=2026-09-23T05:12:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 40.908924] nix-grpc-daemon[865]: time=2026-09-23T05:12:35.412Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker1 # [ 40.921171] nix-grpc-daemon[767]: ts=2026-09-23T05:12:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'... worker1 # [ 40.944348] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 41.079532] nix-daemon[850]: accepted connection from pid 910, user nix-grpc-daemon (trusted) worker1 # [ 41.103019] nix-daemon[850]: reaped child process 913, status = succeeded lb # [ 41.129041] niks3-server[970]: 2026/09/23 05:12:35 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 41.121535] nix-grpc-daemon[865]: time=2026-09-23T05:12:35.624Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 41.123219] nix-grpc-daemon[865]: time=2026-09-23T05:12:35.624Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)" worker1 # [ 41.143681] nix-grpc-daemon[865]: time=2026-09-23T05:12:35.646Z level=INFO msg="Uploading 1 narinfos" lb # [ 41.162208] niks3-server[970]: 2026/09/23 05:12:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign lb # [ 41.175820] niks3-server[970]: 2026/09/23 05:12:35 INFO Signed narinfos id=3 count=1 lb # [ 41.187670] niks3-server[970]: 2026/09/23 05:12:35 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst lb # [ 41.198164] niks3-server[970]: 2026/09/23 05:12:35 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls worker1 # [ 41.191898] nix-grpc-daemon[865]: time=2026-09-23T05:12:35.695Z level=INFO msg="Upload complete. (229ms)" lb # [ 41.211717] niks3-server[970]: 2026/09/23 05:12:35 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete lb # [ 41.217916] niks3-server[970]: 2026/09/23 05:12:35 INFO Completed upload id=3 worker1 # [ 41.199869] nix-grpc-daemon[767]: ts=2026-09-23T05:12:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 41.223650] niks3-server[970]: 2026/09/23 05:12:35 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo worker1 # [ 41.219167] nix-grpc-daemon[767]: ts=2026-09-23T05:12:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 41.226200] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 41.231756] nix-daemon[850]: reaped child process 908, status = succeeded client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051... worker1 # [ 41.246378] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) client # farm-b> node-a: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-t1 worker1 # [ 42.318074] nix-daemon[850]: reaped child process 917, status = succeeded worker1 # [ 42.445151] nix-daemon[850]: accepted connection from pid 934, user nix-grpc-daemon (trusted) worker1 # [ 42.469308] nix-daemon[850]: reaped child process 937, status = succeeded lb # [ 42.486186] niks3-server[970]: 2026/09/23 05:12:36 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 42.479069] nix-grpc-daemon[865]: time=2026-09-23T05:12:36.981Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 42.480840] nix-grpc-daemon[865]: time=2026-09-23T05:12:36.981Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)" lb # [ 42.526483] niks3-server[970]: 2026/09/23 05:12:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign worker1 # [ 42.518348] nix-grpc-daemon[865]: time=2026-09-23T05:12:37.021Z level=INFO msg="Uploading 1 narinfos" lb # [ 42.543009] niks3-server[970]: 2026/09/23 05:12:37 INFO Signed narinfos id=4 count=1 lb # [ 42.557189] niks3-server[970]: 2026/09/23 05:12:37 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv lb # [ 42.567899] niks3-server[970]: 2026/09/23 05:12:37 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls lb # [ 42.582062] niks3-server[970]: 2026/09/23 05:12:37 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete lb # [ 42.587420] niks3-server[970]: 2026/09/23 05:12:37 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst lb # [ 42.596554] niks3-server[970]: 2026/09/23 05:12:37 INFO Completed upload id=4 worker1 # [ 42.577783] nix-grpc-daemon[865]: time=2026-09-23T05:12:37.081Z level=INFO msg="Upload complete. (313ms)" lb # [ 42.598740] niks3-server[970]: 2026/09/23 05:12:37 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo worker1 # [ 42.585014] nix-grpc-daemon[767]: ts=2026-09-23T05:12:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=1 worker1 # [ 42.595780] nix-daemon[850]: reaped child process 915, status = succeeded worker1 # [ 42.600956] nix-grpc-daemon[767]: ts=2026-09-23T05:12:37Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=0 worker1 # [ 42.603809] nix-grpc-daemon[767]: ts=2026-09-23T05:12:37Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 42.650403] nix-grpc-daemon[767]: ts=2026-09-23T05:12:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 42.665463] nix-grpc-daemon[767]: ts=2026-09-23T05:12:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 42.680332] nix-grpc-daemon[865]: time=2026-09-23T05:12:37.183Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'... worker1 # [ 42.692992] nix-grpc-daemon[767]: ts=2026-09-23T05:12:37Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 42.711697] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 42.865094] nix-daemon[850]: accepted connection from pid 943, user nix-grpc-daemon (trusted) worker1 # [ 42.889803] nix-daemon[850]: reaped child process 946, status = succeeded lb # [ 42.913479] niks3-server[970]: 2026/09/23 05:12:37 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 42.907024] nix-grpc-daemon[865]: time=2026-09-23T05:12:37.409Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 42.909225] nix-grpc-daemon[865]: time=2026-09-23T05:12:37.409Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)" lb # [ 42.945935] niks3-server[970]: 2026/09/23 05:12:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign worker1 # [ 42.933128] nix-grpc-daemon[865]: time=2026-09-23T05:12:37.436Z level=INFO msg="Uploading 1 narinfos" lb # [ 42.958035] niks3-server[970]: 2026/09/23 05:12:37 INFO Signed narinfos id=5 count=1 lb # [ 42.969767] niks3-server[970]: 2026/09/23 05:12:37 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst lb # [ 42.984092] niks3-server[970]: 2026/09/23 05:12:37 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls lb # [ 42.987740] niks3-server[970]: 2026/09/23 05:12:37 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete worker1 # [ 42.976082] nix-grpc-daemon[865]: time=2026-09-23T05:12:37.479Z level=INFO msg="Upload complete. (248ms)" lb # [ 42.995017] niks3-server[970]: 2026/09/23 05:12:37 INFO Completed upload id=5 lb # [ 42.999359] niks3-server[970]: 2026/09/23 05:12:37 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo worker1 # [ 42.984174] nix-grpc-daemon[767]: ts=2026-09-23T05:12:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 42.994147] nix-grpc-daemon[767]: ts=2026-09-23T05:12:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 43.000150] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 43.004572] nix-daemon[850]: reaped child process 941, status = succeeded client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051... worker1 # [ 43.016617] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) client # farm-top> node-a: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-t1 worker1 # [ 44.242576] nix-daemon[850]: reaped child process 950, status = succeeded worker1 # [ 44.342473] nix-daemon[850]: accepted connection from pid 966, user nix-grpc-daemon (trusted) worker1 # [ 44.365175] nix-daemon[850]: reaped child process 969, status = succeeded lb # [ 44.386867] niks3-server[970]: 2026/09/23 05:12:38 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 44.382570] nix-grpc-daemon[865]: time=2026-09-23T05:12:38.883Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 44.384313] nix-grpc-daemon[865]: time=2026-09-23T05:12:38.883Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)" lb # [ 44.436916] niks3-server[970]: 2026/09/23 05:12:38 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls worker1 # [ 44.428391] nix-grpc-daemon[865]: time=2026-09-23T05:12:38.931Z level=INFO msg="Uploading 1 narinfos" lb # [ 44.448120] niks3-server[970]: 2026/09/23 05:12:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign lb # [ 44.456542] niks3-server[970]: 2026/09/23 05:12:38 INFO Signed narinfos id=6 count=1 lb # [ 44.460851] niks3-server[970]: 2026/09/23 05:12:38 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker1 # [ 44.452904] nix-grpc-daemon[865]: time=2026-09-23T05:12:38.956Z level=INFO msg="Upload complete. (227ms)" lb # [ 44.471787] niks3-server[970]: 2026/09/23 05:12:38 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete lb # [ 44.481469] niks3-server[970]: 2026/09/23 05:12:38 INFO Completed upload id=6 worker1 # [ 44.459504] nix-grpc-daemon[767]: ts=2026-09-23T05:12:38Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=1 lb # [ 44.483255] niks3-server[970]: 2026/09/23 05:12:38 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst lb # [ 44.489465] niks3-server[970]: 2026/09/23 05:12:38 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo worker1 # [ 44.474748] nix-daemon[850]: reaped child process 948, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 6.58 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.18 seconds) client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.03 seconds) client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client # copying 3 paths... client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'... client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'... client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.24 seconds) (finished: subtest: DAG build is scheduled across workers and lands in the cache, in 7.13 seconds) subtest: repeat is answered Cached by the scheduler without building worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker1 # finding garbage collector roots... worker1 # deleting specified paths... worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker1 # deleting unused links... worker1 # note: hard linking is currently saving 0.0 KiB worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.22 seconds) worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker2 # finding garbage collector roots... worker2 # deleting specified paths... worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker2 # deleting unused links... worker2 # note: hard linking is currently saving 0.0 KiB worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.24 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2 worker1 # [ 45.753752] nix-grpc-daemon[767]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:46746 client # this path will be fetched (0.0 KiB download, 0.2 KiB unpacked): client # /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker2 # [ 45.789279] nix-grpc-daemon[773]: ts=2026-09-23T05:12:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59064 worker1 # [ 45.819278] nix-grpc-daemon[767]: ts=2026-09-23T05:12:40Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=1 client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 >&2, in 0.34 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 outPath, in 0.17 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: repeat is answered Cached by the scheduler without building, in 1.26 seconds) subtest: inputs already in the cache are not sent to the scheduler lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2 worker2 # [ 46.744725] nix-grpc-daemon[773]: ts=2026-09-23T05:12:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59064 client # don't know how to build these paths: client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker2 # [ 46.752152] nix-grpc-daemon[773]: ts=2026-09-23T05:12:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59064 worker1 # [ 46.760827] nix-grpc-daemon[767]: ts=2026-09-23T05:12:41Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=0 worker1 # [ 46.766317] nix-grpc-daemon[767]: ts=2026-09-23T05:12:41Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 46.806131] nix-grpc-daemon[767]: ts=2026-09-23T05:12:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 46.823668] nix-grpc-daemon[767]: ts=2026-09-23T05:12:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 client # copying 1 paths to 'lb:50051'... worker1 # [ 46.840576] nix-grpc-daemon[865]: time=2026-09-23T05:12:41.340Z level=INFO msg="All 2 paths already cached" worker1 # [ 46.847119] nix-grpc-daemon[767]: ts=2026-09-23T05:12:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'... worker1 # [ 46.867170] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 47.029141] nix-daemon[850]: accepted connection from pid 999, user nix-grpc-daemon (trusted) worker1 # [ 47.054286] nix-daemon[850]: reaped child process 1002, status = succeeded lb # [ 47.078144] niks3-server[970]: 2026/09/23 05:12:41 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 47.069734] nix-grpc-daemon[865]: time=2026-09-23T05:12:41.572Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 47.071405] nix-grpc-daemon[865]: time=2026-09-23T05:12:41.572Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)" lb # [ 47.109805] niks3-server[970]: 2026/09/23 05:12:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign worker1 # [ 47.101146] nix-grpc-daemon[865]: time=2026-09-23T05:12:41.604Z level=INFO msg="Uploading 1 narinfos" lb # [ 47.125628] niks3-server[970]: 2026/09/23 05:12:41 INFO Signed narinfos id=7 count=1 lb # [ 47.133448] niks3-server[970]: 2026/09/23 05:12:41 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst lb # [ 47.142405] niks3-server[970]: 2026/09/23 05:12:41 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls lb # [ 47.152591] niks3-server[970]: 2026/09/23 05:12:41 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete lb # [ 47.157643] niks3-server[970]: 2026/09/23 05:12:41 INFO Completed upload id=7 worker1 # [ 47.138936] nix-grpc-daemon[865]: time=2026-09-23T05:12:41.642Z level=INFO msg="Upload complete. (251ms)" lb # [ 47.162485] niks3-server[970]: 2026/09/23 05:12:41 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo worker1 # [ 47.148359] nix-grpc-daemon[767]: ts=2026-09-23T05:12:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 47.164972] nix-grpc-daemon[767]: ts=2026-09-23T05:12:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 47.172738] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 47.178595] nix-daemon[850]: reaped child process 997, status = succeeded client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051... worker1 # [ 47.191960] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) client # farm-top> node-a: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-t1b worker1 # [ 48.263654] nix-daemon[850]: reaped child process 1006, status = succeeded worker1 # [ 48.392228] nix-daemon[850]: accepted connection from pid 1022, user nix-grpc-daemon (trusted) worker1 # [ 48.415710] nix-daemon[850]: reaped child process 1025, status = succeeded lb # [ 48.437600] niks3-server[970]: 2026/09/23 05:12:42 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 48.432463] nix-grpc-daemon[865]: time=2026-09-23T05:12:42.933Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 48.434196] nix-grpc-daemon[865]: time=2026-09-23T05:12:42.933Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)" lb # [ 48.489286] niks3-server[970]: 2026/09/23 05:12:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign worker1 # [ 48.481828] nix-grpc-daemon[865]: time=2026-09-23T05:12:42.984Z level=INFO msg="Uploading 1 narinfos" lb # [ 48.503250] niks3-server[970]: 2026/09/23 05:12:43 INFO Signed narinfos id=8 count=1 lb # [ 48.511035] niks3-server[970]: 2026/09/23 05:12:43 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst lb # [ 48.523864] niks3-server[970]: 2026/09/23 05:12:43 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv lb # [ 48.536566] niks3-server[970]: 2026/09/23 05:12:43 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls worker1 # [ 48.523223] nix-grpc-daemon[865]: time=2026-09-23T05:12:43.026Z level=INFO msg="Upload complete. (316ms)" lb # [ 48.543686] niks3-server[970]: 2026/09/23 05:12:43 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete lb # [ 48.549230] niks3-server[970]: 2026/09/23 05:12:43 INFO Completed upload id=8 worker1 # [ 48.530733] nix-grpc-daemon[767]: ts=2026-09-23T05:12:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=1 lb # [ 48.551985] niks3-server[970]: 2026/09/23 05:12:43 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo worker1 # [ 48.543602] nix-daemon[850]: reaped child process 1004, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.06 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.15 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds) (finished: subtest: inputs already in the cache are not sent to the scheduler, in 2.58 seconds) subtest: outputs read back through any worker carry the cache signature client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2 worker1 # [ 49.121552] nix-grpc-daemon[767]: ts=2026-09-23T05:12:43Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:46746 client # don't know how to build these paths: client # /nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker1 # [ 49.133359] nix-grpc-daemon[767]: ts=2026-09-23T05:12:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 49.161104] nix-grpc-daemon[767]: ts=2026-09-23T05:12:43Z level=debug event=want client=4 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv cached=0 worker1 # [ 49.167733] nix-grpc-daemon[767]: ts=2026-09-23T05:12:43Z level=debug event=assigned drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 49.182005] nix-grpc-daemon[773]: ts=2026-09-23T05:12:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 49.199817] nix-grpc-daemon[773]: ts=2026-09-23T05:12:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59064 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' to 'lb:50051'... worker2 # [ 49.217968] nix-grpc-daemon[773]: ts=2026-09-23T05:12:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 49.234918] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 49.268169] nix-grpc-daemon[773]: ts=2026-09-23T05:12:43Z level=info event=niks3_push_started pid=870 worker2 # [ 49.449253] nix-grpc-daemon[870]: time=2026-09-23T05:12:43.946Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 49.563531] nix-daemon[842]: accepted connection from pid 883, user nix-grpc-daemon (trusted) worker2 # [ 49.583057] nix-daemon[842]: reaped child process 886, status = succeeded lb # [ 49.603560] niks3-server[970]: 2026/09/23 05:12:44 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 49.605978] nix-grpc-daemon[870]: time=2026-09-23T05:12:44.104Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 49.607645] nix-grpc-daemon[870]: time=2026-09-23T05:12:44.104Z level=INFO msg="Uploading 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv (512B)" lb # [ 49.647500] niks3-server[970]: 2026/09/23 05:12:44 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.ls worker2 # [ 49.648272] nix-grpc-daemon[870]: time=2026-09-23T05:12:44.147Z level=INFO msg="Uploading 1 narinfos" lb # [ 49.661364] niks3-server[970]: 2026/09/23 05:12:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign lb # [ 49.672863] niks3-server[970]: 2026/09/23 05:12:44 INFO Signed narinfos id=9 count=1 lb # [ 49.675642] niks3-server[970]: 2026/09/23 05:12:44 INFO Registered completed upload object_key=nar/13mhmmy2qay13imbbdi287vhryz1hwnyg0w6agd9s0ph37wnsqll.nar.zst lb # [ 49.687016] niks3-server[970]: 2026/09/23 05:12:44 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete lb # [ 49.698671] niks3-server[970]: 2026/09/23 05:12:44 INFO Completed upload id=9 worker2 # [ 49.687690] nix-grpc-daemon[870]: time=2026-09-23T05:12:44.186Z level=INFO msg="Upload complete. (239ms)" lb # [ 49.704982] niks3-server[970]: 2026/09/23 05:12:44 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.narinfo worker2 # [ 49.700515] nix-grpc-daemon[773]: ts=2026-09-23T05:12:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=0 paths=1 nar_bytes_in=512 worker2 # [ 49.716865] nix-grpc-daemon[773]: ts=2026-09-23T05:12:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 49.723285] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 49.729133] nix-daemon[842]: reaped child process 868, status = succeeded client # building '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' on lb:50051... client # farm-a> worker2: building 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker2 # [ 49.743370] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) lb # [ 50.274963] envoy[908]: conn peer=[2001:db8:1::1]:46506 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9523 tx=4411 ms=6393 lb # [ 50.280271] envoy[908]: conn peer=[2001:db8:1::1]:36144 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=503 ms=90 lb # [ 50.285089] envoy[908]: conn peer=[2001:db8:1::1]:36148 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4185 tx=1916 ms=1815 client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-sig worker2 # [ 50.789094] nix-daemon[842]: reaped child process 890, status = succeeded worker2 # [ 50.900129] nix-daemon[842]: accepted connection from pid 906, user nix-grpc-daemon (trusted) worker2 # [ 50.925826] nix-daemon[842]: reaped child process 909, status = succeeded lb # [ 50.933828] niks3-server[970]: 2026/09/23 05:12:45 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 50.936628] nix-grpc-daemon[870]: time=2026-09-23T05:12:45.433Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 50.938172] nix-grpc-daemon[870]: time=2026-09-23T05:12:45.433Z level=INFO msg="Uploading 2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a (128B)" lb # [ 50.969622] niks3-server[970]: 2026/09/23 05:12:45 INFO Registered completed upload object_key=log/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker2 # [ 50.988363] nix-grpc-daemon[870]: time=2026-09-23T05:12:45.487Z level=INFO msg="Uploading 1 narinfos" lb # [ 50.991304] niks3-server[970]: 2026/09/23 05:12:45 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst lb # [ 51.006222] niks3-server[970]: 2026/09/23 05:12:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign lb # [ 51.014718] niks3-server[970]: 2026/09/23 05:12:45 INFO Signed narinfos id=10 count=1 lb # [ 51.024258] niks3-server[970]: 2026/09/23 05:12:45 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.ls lb # [ 51.027765] niks3-server[970]: 2026/09/23 05:12:45 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete lb # [ 51.036946] niks3-server[970]: 2026/09/23 05:12:45 INFO Completed upload id=10 worker2 # [ 51.025080] nix-grpc-daemon[870]: time=2026-09-23T05:12:45.524Z level=INFO msg="Upload complete. (269ms)" lb # [ 51.039074] niks3-server[970]: 2026/09/23 05:12:45 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.narinfo worker2 # [ 51.037453] nix-grpc-daemon[773]: ts=2026-09-23T05:12:45Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=1 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv assign_id=5 outputs=1 worker1 # [ 51.033418] nix-grpc-daemon[767]: ts=2026-09-23T05:12:45Z level=debug event=want client=4 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv cached=0 worker1 # [ 51.039866] nix-grpc-daemon[767]: ts=2026-09-23T05:12:45Z level=debug event=assigned drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 51.059695] nix-daemon[842]: reaped child process 888, status = succeeded worker2 # [ 51.095465] nix-grpc-daemon[773]: ts=2026-09-23T05:12:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 51.111626] nix-grpc-daemon[773]: ts=2026-09-23T05:12:45Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 51.124720] nix-grpc-daemon[870]: time=2026-09-23T05:12:45.624Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' to 'lb:50051'... worker2 # [ 51.137905] nix-grpc-daemon[773]: ts=2026-09-23T05:12:45Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 51.154325] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 51.317168] nix-daemon[842]: accepted connection from pid 913, user nix-grpc-daemon (trusted) worker2 # [ 51.343575] nix-daemon[842]: reaped child process 916, status = succeeded lb # [ 51.350951] niks3-server[970]: 2026/09/23 05:12:45 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 51.351554] nix-grpc-daemon[870]: time=2026-09-23T05:12:45.850Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 51.353668] nix-grpc-daemon[870]: time=2026-09-23T05:12:45.850Z level=INFO msg="Uploading z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv (624B)" lb # [ 51.387162] niks3-server[970]: 2026/09/23 05:12:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign worker2 # [ 51.388766] nix-grpc-daemon[870]: time=2026-09-23T05:12:45.887Z level=INFO msg="Uploading 1 narinfos" lb # [ 51.405227] niks3-server[970]: 2026/09/23 05:12:45 INFO Signed narinfos id=11 count=1 lb # [ 51.407808] niks3-server[970]: 2026/09/23 05:12:45 INFO Registered completed upload object_key=nar/0cnhkgsbmxv5hj7zw3fa59qfxikw6251mlhz2gz1v6pgn8c8shwp.nar.zst lb # [ 51.424597] niks3-server[970]: 2026/09/23 05:12:45 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.ls lb # [ 51.433742] niks3-server[970]: 2026/09/23 05:12:45 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete worker2 # [ 51.421346] nix-grpc-daemon[870]: time=2026-09-23T05:12:45.920Z level=INFO msg="Upload complete. (248ms)" lb # [ 51.440973] niks3-server[970]: 2026/09/23 05:12:45 INFO Completed upload id=11 lb # [ 51.443211] niks3-server[970]: 2026/09/23 05:12:45 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.narinfo worker2 # [ 51.432610] nix-grpc-daemon[773]: ts=2026-09-23T05:12:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=0 paths=1 nar_bytes_in=624 worker2 # [ 51.448921] nix-grpc-daemon[773]: ts=2026-09-23T05:12:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 51.455780] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 51.462448] nix-daemon[842]: reaped child process 911, status = succeeded client # building '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' on lb:50051... client # farm-b> worker2: building z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker2 # [ 51.478513] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-sig worker2 # [ 52.114823] nix-daemon[842]: reaped child process 920, status = succeeded worker2 # [ 52.228248] nix-daemon[842]: accepted connection from pid 936, user nix-grpc-daemon (trusted) worker2 # [ 52.248471] nix-daemon[842]: reaped child process 939, status = succeeded lb # [ 52.267090] niks3-server[970]: 2026/09/23 05:12:46 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 52.268530] nix-grpc-daemon[870]: time=2026-09-23T05:12:46.766Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 52.270205] nix-grpc-daemon[870]: time=2026-09-23T05:12:46.766Z level=INFO msg="Uploading h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b (176B)" lb # [ 52.306642] niks3-server[970]: 2026/09/23 05:12:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign worker2 # [ 52.307637] nix-grpc-daemon[870]: time=2026-09-23T05:12:46.806Z level=INFO msg="Uploading 1 narinfos" lb # [ 52.322586] niks3-server[970]: 2026/09/23 05:12:46 INFO Signed narinfos id=12 count=1 lb # [ 52.336349] niks3-server[970]: 2026/09/23 05:12:46 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.ls lb # [ 52.346697] niks3-server[970]: 2026/09/23 05:12:46 INFO Registered completed upload object_key=log/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv lb # [ 52.354661] niks3-server[970]: 2026/09/23 05:12:46 INFO Registered completed upload object_key=nar/0f75gpfkjj0g9xi2113yvvpdw0l16spc2f1xik1ifnzfbrvx4pfk.nar.zst worker2 # [ 52.356888] nix-grpc-daemon[870]: time=2026-09-23T05:12:46.856Z level=INFO msg="Upload complete. (266ms)" lb # [ 52.368141] niks3-server[970]: 2026/09/23 05:12:46 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete lb # [ 52.378084] niks3-server[970]: 2026/09/23 05:12:46 INFO Completed upload id=12 lb # [ 52.381141] niks3-server[970]: 2026/09/23 05:12:46 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.narinfo worker2 # [ 52.369441] nix-grpc-daemon[773]: ts=2026-09-23T05:12:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=0 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv assign_id=6 outputs=1 worker1 # [ 52.372612] nix-grpc-daemon[767]: ts=2026-09-23T05:12:46Z level=debug event=want client=4 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv cached=0 worker1 # [ 52.372947] nix-grpc-daemon[767]: ts=2026-09-23T05:12:46Z level=debug event=assigned drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 52.392254] nix-daemon[842]: reaped child process 918, status = succeeded worker2 # [ 52.426684] nix-grpc-daemon[773]: ts=2026-09-23T05:12:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 52.444474] nix-grpc-daemon[773]: ts=2026-09-23T05:12:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 52.460775] nix-grpc-daemon[870]: time=2026-09-23T05:12:46.960Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' to 'lb:50051'... worker2 # [ 52.473493] nix-grpc-daemon[773]: ts=2026-09-23T05:12:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 52.488923] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 52.656415] nix-daemon[842]: accepted connection from pid 943, user nix-grpc-daemon (trusted) lb # [ 52.694075] niks3-server[970]: 2026/09/23 05:12:47 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 52.687161] nix-daemon[842]: reaped child process 946, status = succeeded worker2 # [ 52.693345] nix-grpc-daemon[870]: time=2026-09-23T05:12:47.192Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 52.695664] nix-grpc-daemon[870]: time=2026-09-23T05:12:47.192Z level=INFO msg="Uploading knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv (752B)" lb # [ 52.726060] niks3-server[970]: 2026/09/23 05:12:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign worker2 # [ 52.726444] nix-grpc-daemon[870]: time=2026-09-23T05:12:47.225Z level=INFO msg="Uploading 1 narinfos" lb # [ 52.743019] niks3-server[970]: 2026/09/23 05:12:47 INFO Signed narinfos id=13 count=1 lb # [ 52.749856] niks3-server[970]: 2026/09/23 05:12:47 INFO Registered completed upload object_key=nar/0lpagyrngav67x80d1hq0njjlkmqdkvqawraz8dsp36ag50v6iwf.nar.zst lb # [ 52.758710] niks3-server[970]: 2026/09/23 05:12:47 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.ls lb # [ 52.769020] niks3-server[970]: 2026/09/23 05:12:47 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete worker2 # [ 52.764389] nix-grpc-daemon[870]: time=2026-09-23T05:12:47.263Z level=INFO msg="Upload complete. (256ms)" lb # [ 52.775937] niks3-server[970]: 2026/09/23 05:12:47 INFO Completed upload id=13 lb # [ 52.780729] niks3-server[970]: 2026/09/23 05:12:47 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.narinfo worker2 # [ 52.772441] nix-grpc-daemon[773]: ts=2026-09-23T05:12:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=0 paths=1 nar_bytes_in=752 worker2 # [ 52.791251] nix-grpc-daemon[773]: ts=2026-09-23T05:12:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 52.798332] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 52.803708] nix-daemon[842]: reaped child process 941, status = succeeded client # building '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' on lb:50051... worker2 # [ 52.817708] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) client # farm-top> worker2: building knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-sig worker2 # [ 54.336633] nix-daemon[842]: reaped child process 950, status = succeeded worker2 # [ 54.414994] nix-daemon[842]: accepted connection from pid 966, user nix-grpc-daemon (trusted) worker2 # [ 54.439697] nix-daemon[842]: reaped child process 969, status = succeeded lb # [ 54.449656] niks3-server[970]: 2026/09/23 05:12:48 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 54.450674] nix-grpc-daemon[870]: time=2026-09-23T05:12:48.948Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 54.452418] nix-grpc-daemon[870]: time=2026-09-23T05:12:48.948Z level=INFO msg="Uploading 4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top (232B)" lb # [ 54.489129] niks3-server[970]: 2026/09/23 05:12:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign worker2 # [ 54.488818] nix-grpc-daemon[870]: time=2026-09-23T05:12:48.987Z level=INFO msg="Uploading 1 narinfos" lb # [ 54.501579] niks3-server[970]: 2026/09/23 05:12:49 INFO Signed narinfos id=14 count=1 lb # [ 54.511063] niks3-server[970]: 2026/09/23 05:12:49 INFO Registered completed upload object_key=nar/07ar364rwqqniwi2nkdpqlgv7ws7702n5lqz703ackfmssrh40v8.nar.zst lb # [ 54.517917] niks3-server[970]: 2026/09/23 05:12:49 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.ls lb # [ 54.521789] niks3-server[970]: 2026/09/23 05:12:49 INFO Registered completed upload object_key=log/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv lb # [ 54.526441] niks3-server[970]: 2026/09/23 05:12:49 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete lb # [ 54.533054] niks3-server[970]: 2026/09/23 05:12:49 INFO Completed upload id=14 worker2 # [ 54.523186] nix-grpc-daemon[870]: time=2026-09-23T05:12:49.022Z level=INFO msg="Upload complete. (289ms)" lb # [ 54.536557] niks3-server[970]: 2026/09/23 05:12:49 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.narinfo worker2 # [ 54.536699] nix-grpc-daemon[773]: ts=2026-09-23T05:12:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=1 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv assign_id=7 outputs=1 worker2 # [ 54.556760] nix-daemon[842]: reaped child process 948, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig >&2, in 5.68 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag sig outPath, in 0.20 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker1 # [ 54.921916] nix-grpc-daemon[767]: ts=2026-09-23T05:12:49Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:34970 client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked): client # /nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a client # /nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b client # /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker1 # [ 54.963902] nix-grpc-daemon[767]: ts=2026-09-23T05:12:49Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:34970 worker1 # [ 54.973643] nix-grpc-daemon[767]: copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'http://lb:5751'... worker1 # [ 55.002746] nix-grpc-daemon[767]: copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'http://lb:5751'... worker1 # [ 55.026073] nix-grpc-daemon[767]: copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.30 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker2 # [ 55.280246] nix-grpc-daemon[773]: ts=2026-09-23T05:12:49Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:59150 worker2 # [ 55.323818] nix-grpc-daemon[773]: ts=2026-09-23T05:12:49Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:59150 client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&ca-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.31 seconds) client: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top client # finding garbage collector roots... client # removing stale temporary roots file "/nix/var/nix/temproots/788" client # removing stale temporary roots file "/nix/var/nix/temproots/846" client # removing stale temporary roots file "/nix/var/nix/temproots/867" client # removing stale temporary roots file "/nix/var/nix/temproots/882" client # removing stale temporary roots file "/nix/var/nix/temproots/838" client # removing stale temporary roots file "/nix/var/nix/temproots/826" client # removing stale temporary roots file "/nix/var/nix/temproots/859" client # removing stale temporary roots file "/nix/var/nix/temproots/804" client # deleting specified paths... client # deleting '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' client # deleting unused links... client # note: hard linking is currently saving 0.0 KiB worker2 # [ 55.757917] nix-grpc-daemon[773]: ts=2026-09-23T05:12:50Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 55.768732] nix-grpc-daemon[773]: ts=2026-09-23T05:12:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 55.782540] nix-grpc-daemon[870]: time=2026-09-23T05:12:50.280Z level=INFO msg="All 1 paths already cached" worker2 # [ 55.792649] nix-grpc-daemon[773]: ts=2026-09-23T05:12:50Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59064 worker1 # [ 55.787486] nix-grpc-daemon[767]: ts=2026-09-23T05:12:50Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:46746 client # copying 3 paths... worker2 # [ 55.810848] nix-grpc-daemon[773]: ts=2026-09-23T05:12:50Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59064 client # copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'grpc://lb:50051'... worker2 # [ 55.836634] nix-grpc-daemon[773]: ts=2026-09-23T05:12:50Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 55.844984] nix-grpc-daemon[773]: ts=2026-09-23T05:12:50Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=0 paths=1 nar_bytes_out=232 worker2 # [ 55.855458] nix-grpc-daemon[773]: ts=2026-09-23T05:12:50Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59064 worker1 # [ 55.853017] nix-grpc-daemon[767]: ts=2026-09-23T05:12:50Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:46746 worker2 # [ 55.861152] nix-grpc-daemon[773]: ts=2026-09-23T05:12:50Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=0 paths=1 nar_bytes_out=176 worker1 # [ 55.861246] nix-grpc-daemon[767]: ts=2026-09-23T05:12:50Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=0 paths=1 nar_bytes_out=128 client # copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'grpc://lb:50051'... client # copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'grpc://lb:50051'... client: (finished: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.54 seconds) (finished: subtest: outputs read back through any worker carry the cache signature, in 7.03 seconds) subtest: two clients wanting the same drv build it once client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup client # [ 56.033384] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup. client # Running as unit: dedup1.service; invocation ID: 00e76a387dee412d81d760288b05e110 client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 0.06 seconds) client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup worker1 # [ 56.166054] nix-grpc-daemon[767]: ts=2026-09-23T05:12:50Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:46746 client # [ 56.231536] nix[935]: don't know how to build these paths: client # [ 56.233899] nix[935]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker1 # [ 56.187009] nix-grpc-daemon[767]: ts=2026-09-23T05:12:50Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 worker1 # [ 56.194926] nix-grpc-daemon[767]: ts=2026-09-23T05:12:50Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 56.233943] nix-grpc-daemon[767]: ts=2026-09-23T05:12:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 56.251194] nix-grpc-daemon[767]: ts=2026-09-23T05:12:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 client # [ 56.324270] nix[935]: copying 1 paths to 'lb:50051'... client # [ 56.330187] nix[935]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'... worker1 # [ 56.271683] nix-grpc-daemon[767]: ts=2026-09-23T05:12:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 56.288581] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 56.465196] nix-daemon[850]: accepted connection from pid 1045, user nix-grpc-daemon (trusted) worker1 # [ 56.484861] nix-daemon[850]: reaped child process 1048, status = succeeded lb # [ 56.512806] niks3-server[970]: 2026/09/23 05:12:51 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 56.504951] nix-grpc-daemon[865]: time=2026-09-23T05:12:51.007Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 56.506487] nix-grpc-daemon[865]: time=2026-09-23T05:12:51.007Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)" lb # [ 56.540797] niks3-server[970]: 2026/09/23 05:12:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign lb # [ 56.546686] niks3-server[970]: 2026/09/23 05:12:51 INFO Signed narinfos id=15 count=1 worker1 # [ 56.527654] nix-grpc-daemon[865]: time=2026-09-23T05:12:51.030Z level=INFO msg="Uploading 1 narinfos" lb # [ 56.557941] niks3-server[970]: 2026/09/23 05:12:51 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst lb # [ 56.566037] niks3-server[970]: 2026/09/23 05:12:51 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls lb # [ 56.574764] niks3-server[970]: 2026/09/23 05:12:51 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete worker1 # [ 56.561204] nix-grpc-daemon[865]: time=2026-09-23T05:12:51.064Z level=INFO msg="Upload complete. (253ms)" lb # [ 56.581838] niks3-server[970]: 2026/09/23 05:12:51 INFO Completed upload id=15 lb # [ 56.586035] niks3-server[970]: 2026/09/23 05:12:51 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo worker1 # [ 56.568192] nix-grpc-daemon[767]: ts=2026-09-23T05:12:51Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=0 paths=1 nar_bytes_in=432 worker1 # [ 56.584207] nix-grpc-daemon[767]: ts=2026-09-23T05:12:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 56.591916] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 56.597916] nix-daemon[850]: reaped child process 1043, status = succeeded client # [ 56.668967] nix[935]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... worker1 # [ 56.615341] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) client # [ 57.109273] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup. client # Running as unit: dedup2.service; invocation ID: 731bafb446804ed5aa6f9b2b505830c2 client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag dedup, in 1.08 seconds) worker2 # [ 57.267658] nix-grpc-daemon[773]: ts=2026-09-23T05:12:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59064 client # [ 57.324524] nix[951]: don't know how to build these paths: client # [ 57.325529] nix[951]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker1 # [ 57.271968] nix-grpc-daemon[767]: ts=2026-09-23T05:12:51Z level=debug event=want client=6 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 worker1 # [ 57.278688] nix-grpc-daemon[767]: ts=2026-09-23T05:12:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 client # [ 57.340826] nix[951]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... client: must succeed: sleep 2 lb # [ 60.274635] envoy[908]: conn peer=[2001:db8:1::1]:36156 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9515 tx=4377 ms=5417 lb # [ 60.279756] envoy[908]: conn peer=[2001:db8:1::1]:34304 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=410 ms=83 lb # [ 60.284724] envoy[908]: conn peer=[2001:db8:1::1]:34306 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=376 ms=71 lb # [ 60.290800] envoy[908]: conn peer=[2001:db8:1::1]:34294 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1051 tx=2913 ms=164 lb # [ 60.295847] envoy[908]: conn peer=[2001:db8:1::1]:34310 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=329 ms=58 client: (finished: must succeed: sleep 2, in 2.03 seconds) client: waiting for success: ! systemctl is-active dedup1 dedup2 worker1 # [ 60.726455] nix-daemon[850]: reaped child process 1052, status = succeeded worker1 # [ 60.801919] nix-daemon[850]: accepted connection from pid 1084, user nix-grpc-daemon (trusted) worker1 # [ 60.824391] nix-daemon[850]: reaped child process 1087, status = succeeded lb # [ 60.848838] niks3-server[970]: 2026/09/23 05:12:55 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 60.844492] nix-grpc-daemon[865]: time=2026-09-23T05:12:55.344Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 60.846170] nix-grpc-daemon[865]: time=2026-09-23T05:12:55.344Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)" lb # [ 60.903614] niks3-server[970]: 2026/09/23 05:12:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign worker1 # [ 60.891094] nix-grpc-daemon[865]: time=2026-09-23T05:12:55.394Z level=INFO msg="Uploading 1 narinfos" lb # [ 60.922698] niks3-server[970]: 2026/09/23 05:12:55 INFO Signed narinfos id=16 count=1 lb # [ 60.929713] niks3-server[970]: 2026/09/23 05:12:55 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls lb # [ 60.937147] niks3-server[970]: 2026/09/23 05:12:55 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst lb # [ 60.945897] niks3-server[970]: 2026/09/23 05:12:55 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv lb # [ 60.956736] niks3-server[970]: 2026/09/23 05:12:55 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete worker1 # [ 60.937884] nix-grpc-daemon[865]: time=2026-09-23T05:12:55.441Z level=INFO msg="Upload complete. (308ms)" lb # [ 60.966569] niks3-server[970]: 2026/09/23 05:12:55 INFO Completed upload id=16 lb # [ 60.969888] niks3-server[970]: 2026/09/23 05:12:55 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo worker1 # [ 60.953308] nix-grpc-daemon[767]: ts=2026-09-23T05:12:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 worker1 # [ 60.973288] nix-grpc-daemon[767]: ts=2026-09-23T05:12:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 worker1 # [ 60.984428] nix-daemon[850]: reaped child process 1050, status = succeeded client # [ 61.084462] systemd[1]: dedup1.service: Deactivated successfully. client # [ 61.088385] systemd[1]: dedup1.service: Consumed 179ms CPU time over 5.054s wall clock time, 41.5M memory peak, 7.4K incoming IP traffic, 8.5K outgoing IP traffic. client # [ 61.096559] systemd[1]: dedup2.service: Deactivated successfully. client # [ 61.098138] systemd[1]: dedup2.service: Consumed 204ms CPU time over 3.984s wall clock time, 37.2M memory peak, 6.1K incoming IP traffic, 5.7K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.14 seconds) client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; } client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.07 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.08 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.08 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: two clients wanting the same drv build it once, in 6.13 seconds) subtest: --timeout and --max-silent-time reach the builder worker2 # [ 62.297792] nix-grpc-daemon[773]: ts=2026-09-23T05:12:56Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59064 worker1 # [ 62.303034] nix-grpc-daemon[767]: ts=2026-09-23T05:12:56Z level=debug event=want client=7 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv cached=0 worker1 # [ 62.309464] nix-grpc-daemon[767]: ts=2026-09-23T05:12:56Z level=debug event=assigned drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 62.359338] nix-grpc-daemon[773]: ts=2026-09-23T05:12:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 62.377280] nix-grpc-daemon[773]: ts=2026-09-23T05:12:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 62.393930] nix-grpc-daemon[773]: ts=2026-09-23T05:12:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 62.410949] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 62.581679] nix-daemon[842]: accepted connection from pid 994, user nix-grpc-daemon (trusted) worker2 # [ 62.605231] nix-daemon[842]: reaped child process 997, status = succeeded lb # [ 62.613084] niks3-server[970]: 2026/09/23 05:12:57 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 62.613412] nix-grpc-daemon[870]: time=2026-09-23T05:12:57.112Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 62.614907] nix-grpc-daemon[870]: time=2026-09-23T05:12:57.112Z level=INFO msg="Uploading sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv (424B)" lb # [ 62.651092] niks3-server[970]: 2026/09/23 05:12:57 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.ls lb # [ 62.663865] niks3-server[970]: 2026/09/23 05:12:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign worker2 # [ 62.659493] nix-grpc-daemon[870]: time=2026-09-23T05:12:57.158Z level=INFO msg="Uploading 1 narinfos" lb # [ 62.675798] niks3-server[970]: 2026/09/23 05:12:57 INFO Signed narinfos id=17 count=1 lb # [ 62.683357] niks3-server[970]: 2026/09/23 05:12:57 INFO Registered completed upload object_key=nar/1m12xmj6yzmb9y2j8ly0yvh5fyzy1nqk7ajhqwr4wn3b5h0bsch5.nar.zst lb # [ 62.701115] niks3-server[970]: 2026/09/23 05:12:57 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete lb # [ 62.709294] niks3-server[970]: 2026/09/23 05:12:57 INFO Completed upload id=17 lb # [ 62.711158] niks3-server[970]: 2026/09/23 05:12:57 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.narinfo worker2 # [ 62.698198] nix-grpc-daemon[870]: time=2026-09-23T05:12:57.197Z level=INFO msg="Upload complete. (267ms)" worker2 # [ 62.711428] nix-grpc-daemon[773]: ts=2026-09-23T05:12:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=0 paths=1 nar_bytes_in=424 worker2 # [ 62.730921] nix-grpc-daemon[773]: ts=2026-09-23T05:12:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 62.737851] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 62.743273] nix-daemon[842]: reaped child process 992, status = succeeded worker2 # [ 62.759183] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 68.575166] nix-grpc-daemon[773]: ts=2026-09-23T05:13:03Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=5 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv assign_id=9 outputs=0 worker2 # [ 68.595014] nix-daemon[842]: reaped child process 999, status = succeeded worker2 # [ 68.596447] nix-daemon[842]: reaped child process 1001, status = succeeded worker1 # [ 68.857502] nix-grpc-daemon[767]: ts=2026-09-23T05:13:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 68.874875] nix-grpc-daemon[767]: ts=2026-09-23T05:13:03Z level=debug event=want client=8 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv cached=0 worker1 # [ 68.882237] nix-grpc-daemon[767]: ts=2026-09-23T05:13:03Z level=debug event=assigned drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 68.892318] nix-grpc-daemon[767]: ts=2026-09-23T05:13:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 68.903885] nix-grpc-daemon[767]: ts=2026-09-23T05:13:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 68.919263] nix-grpc-daemon[767]: ts=2026-09-23T05:13:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 68.935113] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 69.091404] nix-daemon[850]: accepted connection from pid 1104, user nix-grpc-daemon (trusted) worker1 # [ 69.115049] nix-daemon[850]: reaped child process 1107, status = succeeded lb # [ 69.137918] niks3-server[970]: 2026/09/23 05:13:03 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 69.129457] nix-grpc-daemon[865]: time=2026-09-23T05:13:03.632Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 69.130958] nix-grpc-daemon[865]: time=2026-09-23T05:13:03.632Z level=INFO msg="Uploading pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv (424B)" lb # [ 69.167532] niks3-server[970]: 2026/09/23 05:13:03 INFO Registered completed upload object_key=nar/14f5yrnllfav3svmw4v4svw15r4zlf5yv1p972jy14mmq6gdw943.nar.zst lb # [ 69.190152] niks3-server[970]: 2026/09/23 05:13:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign worker1 # [ 69.178221] nix-grpc-daemon[865]: time=2026-09-23T05:13:03.681Z level=INFO msg="Uploading 1 narinfos" lb # [ 69.200903] niks3-server[970]: 2026/09/23 05:13:03 INFO Signed narinfos id=18 count=1 lb # [ 69.210342] niks3-server[970]: 2026/09/23 05:13:03 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.ls lb # [ 69.224556] niks3-server[970]: 2026/09/23 05:13:03 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete lb # [ 69.233006] niks3-server[970]: 2026/09/23 05:13:03 INFO Completed upload id=18 worker1 # [ 69.213241] nix-grpc-daemon[865]: time=2026-09-23T05:13:03.716Z level=INFO msg="Upload complete. (261ms)" lb # [ 69.237818] niks3-server[970]: 2026/09/23 05:13:03 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.narinfo worker1 # [ 69.224358] nix-grpc-daemon[767]: ts=2026-09-23T05:13:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=0 paths=1 nar_bytes_in=424 worker1 # [ 69.242989] nix-grpc-daemon[767]: ts=2026-09-23T05:13:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 69.256405] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 69.267180] nix-daemon[850]: reaped child process 1102, status = succeeded worker1 # [ 69.277753] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) lb # [ 70.276294] envoy[908]: conn peer=[2001:db8:1::1]:34318 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3018 tx=1536 ms=4808 lb # [ 70.276691] envoy[908]: conn peer=[2001:db8:1::1]:34326 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1430 tx=1197 ms=3738 lb # [ 70.276827] envoy[908]: conn peer=[2001:db8:1::1]:47390 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3001 tx=954 ms=6345 worker1 # [ 75.294241] nix-grpc-daemon[767]: ts=2026-09-23T05:13:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=6 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv assign_id=10 outputs=0 worker1 # [ 75.313927] nix-daemon[850]: reaped child process 1109, status = succeeded worker1 # [ 75.315056] nix-daemon[850]: reaped child process 1111, status = succeeded (finished: subtest: --timeout and --max-silent-time reach the builder, in 13.46 seconds) subtest: the first client leaving does not fail the second client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share client # [ 75.646584] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share. client # Running as unit: share1.service; invocation ID: 2245124843634344a28df86e4a733d17 client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 0.09 seconds) client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share client # [ 75.861147] nix[1025]: don't know how to build these paths: client # [ 75.862305] nix[1025]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker2 # [ 75.810750] nix-grpc-daemon[773]: ts=2026-09-23T05:13:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59064 worker1 # [ 75.815553] nix-grpc-daemon[767]: ts=2026-09-23T05:13:10Z level=debug event=want client=9 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 worker2 # [ 75.828680] nix-grpc-daemon[773]: ts=2026-09-23T05:13:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 worker1 # [ 75.822075] nix-grpc-daemon[767]: ts=2026-09-23T05:13:10Z level=debug event=assigned drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 75.846674] nix-grpc-daemon[773]: ts=2026-09-23T05:13:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59064 client # [ 75.909274] nix[1025]: copying 1 paths to 'lb:50051'... client # [ 75.913093] nix[1025]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' to 'lb:50051'... worker2 # [ 75.864118] nix-grpc-daemon[773]: ts=2026-09-23T05:13:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 75.879331] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 76.041820] nix-daemon[842]: accepted connection from pid 1024, user nix-grpc-daemon (trusted) worker2 # [ 76.065747] nix-daemon[842]: reaped child process 1027, status = succeeded lb # [ 76.075289] niks3-server[970]: 2026/09/23 05:13:10 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 76.076451] nix-grpc-daemon[870]: time=2026-09-23T05:13:10.575Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 76.077979] nix-grpc-daemon[870]: time=2026-09-23T05:13:10.575Z level=INFO msg="Uploading rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv (432B)" lb # [ 76.112352] niks3-server[970]: 2026/09/23 05:13:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign worker2 # [ 76.110977] nix-grpc-daemon[870]: time=2026-09-23T05:13:10.610Z level=INFO msg="Uploading 1 narinfos" lb # [ 76.128219] niks3-server[970]: 2026/09/23 05:13:10 INFO Signed narinfos id=19 count=1 lb # [ 76.134713] niks3-server[970]: 2026/09/23 05:13:10 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.ls lb # [ 76.146703] niks3-server[970]: 2026/09/23 05:13:10 INFO Registered completed upload object_key=nar/0chdjpc9an2ckdblh08lnyrmzvw3h8qs76cn5ydwjwvgwkrm731l.nar.zst lb # [ 76.164350] niks3-server[970]: 2026/09/23 05:13:10 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete lb # [ 76.172210] niks3-server[970]: 2026/09/23 05:13:10 INFO Completed upload id=19 worker2 # [ 76.160819] nix-grpc-daemon[870]: time=2026-09-23T05:13:10.660Z level=INFO msg="Upload complete. (262ms)" lb # [ 76.181741] niks3-server[970]: 2026/09/23 05:13:10 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.narinfo worker2 # [ 76.171952] nix-grpc-daemon[773]: ts=2026-09-23T05:13:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=0 paths=1 nar_bytes_in=432 worker2 # [ 76.186792] nix-grpc-daemon[773]: ts=2026-09-23T05:13:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 76.198375] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 76.206189] nix-daemon[842]: reaped child process 1022, status = succeeded client # [ 76.266556] nix[1025]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... worker2 # [ 76.219679] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) client # [ 76.755792] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share. client # Running as unit: share2.service; invocation ID: 10fc821103c3486dbb5551a0698eba0f client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag share, in 1.12 seconds) worker2 # [ 76.937812] nix-grpc-daemon[773]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59064 client # [ 77.003265] nix[1041]: this derivation will be built: worker2 # [ 76.955440] nix-grpc-daemon[773]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59064 client # [ 77.013621] nix[1041]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker1 # [ 76.965081] nix-grpc-daemon[767]: ts=2026-09-23T05:13:11Z level=debug event=want client=10 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 worker2 # [ 76.976909] nix-grpc-daemon[773]: ts=2026-09-23T05:13:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 client # [ 77.033525] nix[1041]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... client: must succeed: sleep 2; systemctl kill -s INT share1 client # [ 80.143765] systemd[1]: share1.service: Sent signal SIGINT to main process 1025 (nix) on client request. client # [ 80.153235] nix[1025]: shutting downrxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.09 seconds) client: waiting for success: ! systemctl is-active share1 client # [ 80.167441] nix[1025]: error: interrupted by the user client # [ 80.224186] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE client # [ 80.226304] systemd[1]: share1.service: Failed with result 'exit-code'. client # [ 80.229023] systemd[1]: share1.service: Consumed 181ms CPU time over 4.582s wall clock time, 41.4M memory peak, 6.3K incoming IP traffic, 8.2K outgoing IP traffic. lb # [ 80.275976] envoy[908]: conn peer=[2001:db8:1::1]:47396 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3028 tx=952 ms=6500 lb # [ 80.276488] envoy[908]: conn peer=[2001:db8:1::1]:35120 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2990 tx=814 ms=4315 client: (finished: waiting for success: ! systemctl is-active share1, in 1.12 seconds) client: waiting for success: ! systemctl is-active share2 worker2 # [ 81.690266] nix-daemon[842]: reaped child process 1031, status = succeeded worker2 # [ 81.821533] nix-daemon[842]: accepted connection from pid 1065, user nix-grpc-daemon (trusted) worker2 # [ 81.843687] nix-daemon[842]: reaped child process 1068, status = succeeded lb # [ 81.853799] niks3-server[970]: 2026/09/23 05:13:16 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 81.856434] nix-grpc-daemon[870]: time=2026-09-23T05:13:16.355Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 81.858094] nix-grpc-daemon[870]: time=2026-09-23T05:13:16.355Z level=INFO msg="Uploading fjww0fd4rwk1s35jav3cyal62l24n356-slow-share (120B)" lb # [ 81.905457] niks3-server[970]: 2026/09/23 05:13:16 INFO Registered completed upload object_key=log/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker2 # [ 81.905825] nix-grpc-daemon[870]: time=2026-09-23T05:13:16.404Z level=INFO msg="Uploading 1 narinfos" lb # [ 81.918075] niks3-server[970]: 2026/09/23 05:13:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign lb # [ 81.925136] niks3-server[970]: 2026/09/23 05:13:16 INFO Signed narinfos id=20 count=1 lb # [ 81.932223] niks3-server[970]: 2026/09/23 05:13:16 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst lb # [ 81.935424] niks3-server[970]: 2026/09/23 05:13:16 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete worker2 # [ 81.930512] nix-grpc-daemon[870]: time=2026-09-23T05:13:16.429Z level=INFO msg="Upload complete. (296ms)" lb # [ 81.941431] niks3-server[970]: 2026/09/23 05:13:16 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.ls lb # [ 81.948846] niks3-server[970]: 2026/09/23 05:13:16 INFO Completed upload id=20 lb # [ 81.951750] niks3-server[970]: 2026/09/23 05:13:16 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.narinfo worker2 # [ 81.943920] nix-grpc-daemon[773]: ts=2026-09-23T05:13:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=5 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=1 worker2 # [ 81.960384] nix-grpc-daemon[773]: ts=2026-09-23T05:13:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=4 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=1 worker2 # [ 81.966552] nix-daemon[842]: reaped child process 1029, status = succeeded client # [ 82.069200] systemd[1]: share2.service: Deactivated successfully. client # [ 82.073229] systemd[1]: share2.service: Consumed 204ms CPU time over 5.313s wall clock time, 35.3M memory peak, 6.5K incoming IP traffic, 5.9K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active share2, in 1.12 seconds) client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; } client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.06 seconds) (finished: subtest: the first client leaving does not fail the second, in 7.15 seconds) subtest: build hook: nix-daemon with builders = grpc://lb client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log client # [ 82.862101] systemd[1]: Started Nix Daemon. client # [ 82.989540] nix-daemon[1093]: accepted connection from pid 1090, user root (trusted) worker1 # [ 83.087941] nix-grpc-daemon[767]: ts=2026-09-23T05:13:17Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 83.094636] nix-grpc-daemon[767]: ts=2026-09-23T05:13:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 83.114338] nix-grpc-daemon[767]: ts=2026-09-23T05:13:17Z level=debug event=want client=11 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=0 worker1 # [ 83.121612] nix-grpc-daemon[767]: ts=2026-09-23T05:13:17Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 83.131007] nix-grpc-daemon[767]: ts=2026-09-23T05:13:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 83.142560] nix-grpc-daemon[767]: ts=2026-09-23T05:13:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 83.158208] nix-grpc-daemon[767]: ts=2026-09-23T05:13:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 83.173833] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 83.326468] nix-daemon[850]: accepted connection from pid 1156, user nix-grpc-daemon (trusted) worker1 # [ 83.349948] nix-daemon[850]: reaped child process 1159, status = succeeded lb # [ 83.373253] niks3-server[970]: 2026/09/23 05:13:17 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 83.364375] nix-grpc-daemon[865]: time=2026-09-23T05:13:17.866Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 83.365902] nix-grpc-daemon[865]: time=2026-09-23T05:13:17.867Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)" lb # [ 83.403444] niks3-server[970]: 2026/09/23 05:13:17 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst worker1 # [ 83.396988] nix-grpc-daemon[865]: time=2026-09-23T05:13:17.900Z level=INFO msg="Uploading 1 narinfos" lb # [ 83.418330] niks3-server[970]: 2026/09/23 05:13:17 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign lb # [ 83.431048] niks3-server[970]: 2026/09/23 05:13:17 INFO Signed narinfos id=21 count=1 lb # [ 83.438570] niks3-server[970]: 2026/09/23 05:13:17 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls lb # [ 83.449398] niks3-server[970]: 2026/09/23 05:13:17 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete lb # [ 83.459833] niks3-server[970]: 2026/09/23 05:13:17 INFO Completed upload id=21 worker1 # [ 83.439906] nix-grpc-daemon[865]: time=2026-09-23T05:13:17.943Z level=INFO msg="Upload complete. (249ms)" lb # [ 83.464033] niks3-server[970]: 2026/09/23 05:13:17 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo worker1 # [ 83.452172] nix-grpc-daemon[767]: ts=2026-09-23T05:13:17Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 83.468392] nix-grpc-daemon[767]: ts=2026-09-23T05:13:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 83.474996] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 83.480821] nix-daemon[850]: reaped child process 1154, status = succeeded worker1 # [ 83.492459] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 84.776828] nix-daemon[850]: reaped child process 1163, status = succeeded worker1 # [ 84.872003] nix-daemon[850]: accepted connection from pid 1179, user nix-grpc-daemon (trusted) worker1 # [ 84.896425] nix-daemon[850]: reaped child process 1182, status = succeeded lb # [ 84.913938] niks3-server[970]: 2026/09/23 05:13:19 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 84.904921] nix-grpc-daemon[865]: time=2026-09-23T05:13:19.407Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 84.906475] nix-grpc-daemon[865]: time=2026-09-23T05:13:19.407Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)" lb # [ 84.962593] niks3-server[970]: 2026/09/23 05:13:19 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls worker1 # [ 84.953893] nix-grpc-daemon[865]: time=2026-09-23T05:13:19.456Z level=INFO msg="Uploading 1 narinfos" lb # [ 84.975788] niks3-server[970]: 2026/09/23 05:13:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign lb # [ 84.985611] niks3-server[970]: 2026/09/23 05:13:19 INFO Signed narinfos id=22 count=1 lb # [ 84.991252] niks3-server[970]: 2026/09/23 05:13:19 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker1 # [ 84.982534] nix-grpc-daemon[865]: time=2026-09-23T05:13:19.485Z level=INFO msg="Upload complete. (276ms)" lb # [ 85.003176] niks3-server[970]: 2026/09/23 05:13:19 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst lb # [ 85.010056] niks3-server[970]: 2026/09/23 05:13:19 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete lb # [ 85.014053] niks3-server[970]: 2026/09/23 05:13:19 INFO Completed upload id=22 worker1 # [ 84.993111] nix-grpc-daemon[767]: ts=2026-09-23T05:13:19Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=12 outputs=1 lb # [ 85.016833] niks3-server[970]: 2026/09/23 05:13:19 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo worker1 # [ 85.010427] nix-daemon[850]: reaped child process 1161, status = succeeded worker2 # [ 85.050178] nix-grpc-daemon[773]: ts=2026-09-23T05:13:19Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 85.059239] nix-grpc-daemon[773]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'... worker2 # [ 85.101734] nix-grpc-daemon[773]: ts=2026-09-23T05:13:19Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 85.109005] nix-grpc-daemon[773]: ts=2026-09-23T05:13:19Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=0 paths=1 nar_bytes_out=128 worker2 # [ 85.393694] nix-grpc-daemon[773]: ts=2026-09-23T05:13:19Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:59064 worker1 # [ 85.389615] nix-grpc-daemon[767]: ts=2026-09-23T05:13:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 85.400673] nix-grpc-daemon[865]: time=2026-09-23T05:13:19.903Z level=INFO msg="All 1 paths already cached" worker1 # [ 85.419915] nix-grpc-daemon[767]: ts=2026-09-23T05:13:19Z level=debug event=want client=12 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=0 worker1 # [ 85.426845] nix-grpc-daemon[767]: ts=2026-09-23T05:13:19Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 85.435902] nix-grpc-daemon[767]: ts=2026-09-23T05:13:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 85.447865] nix-grpc-daemon[767]: ts=2026-09-23T05:13:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 85.462683] nix-grpc-daemon[865]: time=2026-09-23T05:13:19.965Z level=INFO msg="All 1 paths already cached" worker1 # [ 85.472815] nix-grpc-daemon[767]: ts=2026-09-23T05:13:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 85.487078] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 85.649161] nix-daemon[850]: accepted connection from pid 1188, user nix-grpc-daemon (trusted) worker1 # [ 85.672855] nix-daemon[850]: reaped child process 1191, status = succeeded lb # [ 85.690275] niks3-server[970]: 2026/09/23 05:13:20 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 85.680987] nix-grpc-daemon[865]: time=2026-09-23T05:13:20.183Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 85.683239] nix-grpc-daemon[865]: time=2026-09-23T05:13:20.183Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)" lb # [ 85.727579] niks3-server[970]: 2026/09/23 05:13:20 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst worker1 # [ 85.716463] nix-grpc-daemon[865]: time=2026-09-23T05:13:20.219Z level=INFO msg="Uploading 1 narinfos" lb # [ 85.743216] niks3-server[970]: 2026/09/23 05:13:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign lb # [ 85.755791] niks3-server[970]: 2026/09/23 05:13:20 INFO Signed narinfos id=23 count=1 lb # [ 85.762087] niks3-server[970]: 2026/09/23 05:13:20 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls worker1 # [ 85.742353] nix-grpc-daemon[865]: time=2026-09-23T05:13:20.245Z level=INFO msg="Upload complete. (238ms)" lb # [ 85.772396] niks3-server[970]: 2026/09/23 05:13:20 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete worker1 # [ 85.753599] nix-grpc-daemon[767]: ts=2026-09-23T05:13:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 85.777211] niks3-server[970]: 2026/09/23 05:13:20 INFO Completed upload id=23 lb # [ 85.779609] niks3-server[970]: 2026/09/23 05:13:20 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo worker1 # [ 85.768666] nix-grpc-daemon[767]: ts=2026-09-23T05:13:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 85.775309] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 85.781519] nix-daemon[850]: reaped child process 1186, status = succeeded worker1 # [ 85.794961] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 87.233352] nix-daemon[850]: reaped child process 1195, status = succeeded worker1 # [ 87.319657] nix-daemon[850]: accepted connection from pid 1211, user nix-grpc-daemon (trusted) worker1 # [ 87.339232] nix-daemon[850]: reaped child process 1214, status = succeeded lb # [ 87.365700] niks3-server[970]: 2026/09/23 05:13:21 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 87.356414] nix-grpc-daemon[865]: time=2026-09-23T05:13:21.859Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 87.357895] nix-grpc-daemon[865]: time=2026-09-23T05:13:21.859Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)" lb # [ 87.406696] niks3-server[970]: 2026/09/23 05:13:21 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst worker1 # [ 87.404261] nix-grpc-daemon[865]: time=2026-09-23T05:13:21.907Z level=INFO msg="Uploading 1 narinfos" lb # [ 87.423283] niks3-server[970]: 2026/09/23 05:13:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign lb # [ 87.432376] niks3-server[970]: 2026/09/23 05:13:21 INFO Signed narinfos id=24 count=1 lb # [ 87.440576] niks3-server[970]: 2026/09/23 05:13:21 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls lb # [ 87.448411] niks3-server[970]: 2026/09/23 05:13:21 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker1 # [ 87.442278] nix-grpc-daemon[865]: time=2026-09-23T05:13:21.945Z level=INFO msg="Upload complete. (237ms)" lb # [ 87.461500] niks3-server[970]: 2026/09/23 05:13:21 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete lb # [ 87.465589] niks3-server[970]: 2026/09/23 05:13:21 INFO Completed upload id=24 lb # [ 87.470684] niks3-server[970]: 2026/09/23 05:13:21 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo worker1 # [ 87.457174] nix-grpc-daemon[767]: ts=2026-09-23T05:13:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=13 outputs=1 worker2 # [ 87.470835] nix-grpc-daemon[773]: ts=2026-09-23T05:13:21Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59064 worker1 # [ 87.473081] nix-daemon[850]: reaped child process 1193, status = succeeded worker2 # [ 87.483398] nix-grpc-daemon[773]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'... worker1 # [ 87.513224] nix-grpc-daemon[767]: ts=2026-09-23T05:13:22Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 87.520619] nix-grpc-daemon[767]: ts=2026-09-23T05:13:22Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=0 paths=1 nar_bytes_out=176 worker1 # [ 87.757880] nix-grpc-daemon[767]: ts=2026-09-23T05:13:22Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 87.765549] nix-grpc-daemon[767]: ts=2026-09-23T05:13:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 87.773382] nix-grpc-daemon[865]: time=2026-09-23T05:13:22.274Z level=INFO msg="All 2 paths already cached" worker1 # [ 87.786477] nix-grpc-daemon[767]: ts=2026-09-23T05:13:22Z level=debug event=want client=13 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=0 worker1 # [ 87.793865] nix-grpc-daemon[767]: ts=2026-09-23T05:13:22Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 87.833878] nix-grpc-daemon[767]: ts=2026-09-23T05:13:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 87.851016] nix-grpc-daemon[767]: ts=2026-09-23T05:13:22Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 87.866192] nix-grpc-daemon[865]: time=2026-09-23T05:13:22.369Z level=INFO msg="All 2 paths already cached" worker1 # [ 87.876953] nix-grpc-daemon[767]: ts=2026-09-23T05:13:22Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 87.891174] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 88.045765] nix-daemon[850]: accepted connection from pid 1220, user nix-grpc-daemon (trusted) worker1 # [ 88.067719] nix-daemon[850]: reaped child process 1223, status = succeeded lb # [ 88.084060] niks3-server[970]: 2026/09/23 05:13:22 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 88.075198] nix-grpc-daemon[865]: time=2026-09-23T05:13:22.578Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 88.076944] nix-grpc-daemon[865]: time=2026-09-23T05:13:22.578Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)" lb # [ 88.117094] niks3-server[970]: 2026/09/23 05:13:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign worker1 # [ 88.104138] nix-grpc-daemon[865]: time=2026-09-23T05:13:22.607Z level=INFO msg="Uploading 1 narinfos" lb # [ 88.131613] niks3-server[970]: 2026/09/23 05:13:22 INFO Signed narinfos id=25 count=1 lb # [ 88.135812] niks3-server[970]: 2026/09/23 05:13:22 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls worker1 # [ 88.132964] nix-grpc-daemon[865]: time=2026-09-23T05:13:22.636Z level=INFO msg="Upload complete. (224ms)" lb # [ 88.152620] niks3-server[970]: 2026/09/23 05:13:22 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst worker1 # [ 88.140363] nix-grpc-daemon[767]: ts=2026-09-23T05:13:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 88.159984] niks3-server[970]: 2026/09/23 05:13:22 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete lb # [ 88.168252] niks3-server[970]: 2026/09/23 05:13:22 INFO Completed upload id=25 lb # [ 88.170739] niks3-server[970]: 2026/09/23 05:13:22 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo worker1 # [ 88.156365] nix-grpc-daemon[767]: ts=2026-09-23T05:13:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 88.163009] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 88.168189] nix-daemon[850]: reaped child process 1218, status = succeeded worker1 # [ 88.180419] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 89.055864] nix-daemon[850]: reaped child process 1227, status = succeeded worker1 # [ 89.118826] nix-daemon[850]: accepted connection from pid 1243, user nix-grpc-daemon (trusted) worker1 # [ 89.142505] nix-daemon[850]: reaped child process 1246, status = succeeded lb # [ 89.163784] niks3-server[970]: 2026/09/23 05:13:23 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 89.244966] nix-grpc-daemon[865]: time=2026-09-23T05:13:23.745Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 89.250773] nix-grpc-daemon[865]: time=2026-09-23T05:13:23.745Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)" lb # [ 89.302928] niks3-server[970]: 2026/09/23 05:13:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign worker1 # [ 89.294309] nix-grpc-daemon[865]: time=2026-09-23T05:13:23.797Z level=INFO msg="Uploading 1 narinfos" lb # [ 89.319534] niks3-server[970]: 2026/09/23 05:13:23 INFO Signed narinfos id=26 count=1 lb # [ 89.328287] niks3-server[970]: 2026/09/23 05:13:23 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls lb # [ 89.340986] niks3-server[970]: 2026/09/23 05:13:23 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst worker1 # [ 89.330056] nix-grpc-daemon[865]: time=2026-09-23T05:13:23.833Z level=INFO msg="Upload complete. (337ms)" lb # [ 89.353523] niks3-server[970]: 2026/09/23 05:13:23 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv lb # [ 89.359937] niks3-server[970]: 2026/09/23 05:13:23 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete lb # [ 89.365481] niks3-server[970]: 2026/09/23 05:13:23 INFO Completed upload id=26 worker1 # [ 89.342184] nix-grpc-daemon[767]: ts=2026-09-23T05:13:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=14 outputs=1 lb # [ 89.369073] niks3-server[970]: 2026/09/23 05:13:23 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo worker2 # [ 89.369317] nix-grpc-daemon[773]: ts=2026-09-23T05:13:23Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59064 worker1 # [ 89.364514] nix-daemon[850]: reaped child process 1225, status = succeeded worker2 # [ 89.383020] nix-grpc-daemon[773]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'... worker2 # [ 89.422238] nix-grpc-daemon[773]: ts=2026-09-23T05:13:23Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 89.429433] nix-grpc-daemon[773]: ts=2026-09-23T05:13:23Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=0 paths=1 nar_bytes_out=232 client # [ 89.602830] nix-daemon[1093]: reaped child process 1095, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag hook 2>/tmp/hook.log, in 6.90 seconds) client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.04 seconds) client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.05 seconds) client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds) (finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 7.02 seconds) subtest: an input only one worker has still reaches the builder client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only client # this derivation will be built: client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag w1only, in 0.30 seconds) client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.12 seconds) worker1: must succeed: nix-store --import < /tmp/shared/inp.closure lb # [ 90.276394] envoy[908]: conn peer=[2001:db8:1::1]:35130 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1605 tx=1596 ms=5058 lb # [ 90.276695] envoy[908]: conn peer=[2001:db8:1::1]:35100 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3445 tx=1943 ms=2072 lb # [ 90.276797] envoy[908]: conn peer=[2001:db8:1::1]:35116 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=333 ms=70 lb # [ 90.276908] envoy[908]: conn peer=[2001:db8:1::1]:35132 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3914 tx=2156 ms=2153 lb # [ 90.277022] envoy[908]: conn peer=[2001:db8:1::1]:35136 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=375 ms=27 lb # [ 90.277118] envoy[908]: conn peer=[2001:db8:1::1]:35146 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4352 tx=2412 ms=1720 lb # [ 90.277226] envoy[908]: conn peer=[2001:db8:1::1]:35160 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=412 tx=406 ms=69 worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.20 seconds) worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.03 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2 client # [ 90.583235] nix-daemon[1093]: accepted connection from pid 1176, user root (trusted) client # this derivation will be built: client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker1 # [ 90.679124] nix-grpc-daemon[767]: ts=2026-09-23T05:13:25Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 90.687652] nix-grpc-daemon[767]: ts=2026-09-23T05:13:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 90.851099] nix-daemon[850]: accepted connection from pid 1254, user nix-grpc-daemon (trusted) worker1 # [ 90.875301] nix-daemon[850]: reaped child process 1257, status = succeeded lb # [ 90.899963] niks3-server[970]: 2026/09/23 05:13:25 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 90.894086] nix-grpc-daemon[865]: time=2026-09-23T05:13:25.396Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 90.895878] nix-grpc-daemon[865]: time=2026-09-23T05:13:25.396Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)" lb # [ 90.932579] niks3-server[970]: 2026/09/23 05:13:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign worker1 # [ 90.925206] nix-grpc-daemon[865]: time=2026-09-23T05:13:25.428Z level=INFO msg="Uploading 1 narinfos" lb # [ 90.948965] niks3-server[970]: 2026/09/23 05:13:25 INFO Signed narinfos id=27 count=1 lb # [ 90.956881] niks3-server[970]: 2026/09/23 05:13:25 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls lb # [ 90.969535] niks3-server[970]: 2026/09/23 05:13:25 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete lb # [ 90.978490] niks3-server[970]: 2026/09/23 05:13:25 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst lb # [ 90.989960] niks3-server[970]: 2026/09/23 05:13:25 INFO Completed upload id=27 worker1 # [ 90.971328] nix-grpc-daemon[865]: time=2026-09-23T05:13:25.474Z level=INFO msg="Upload complete. (284ms)" lb # [ 91.002999] niks3-server[970]: 2026/09/23 05:13:25 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo worker1 # [ 90.999090] nix-grpc-daemon[767]: ts=2026-09-23T05:13:25Z level=debug event=want client=14 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=0 worker1 # [ 91.005412] nix-grpc-daemon[767]: ts=2026-09-23T05:13:25Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 91.056813] nix-grpc-daemon[773]: ts=2026-09-23T05:13:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 91.073006] nix-grpc-daemon[773]: ts=2026-09-23T05:13:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59064 client # copying 2 paths to 'lb:50051'... client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'... worker2 # [ 91.093065] nix-grpc-daemon[773]: ts=2026-09-23T05:13:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59064 client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'... worker2 # [ 91.108322] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 91.279512] nix-daemon[842]: accepted connection from pid 1078, user nix-grpc-daemon (trusted) worker2 # [ 91.297745] nix-daemon[842]: reaped child process 1081, status = succeeded lb # [ 91.314691] niks3-server[970]: 2026/09/23 05:13:25 INFO Received uploads request method=POST path=/api/pending_closures lb # [ 91.324061] niks3-server[970]: 2026/09/23 05:13:25 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 91.326830] nix-grpc-daemon[870]: time=2026-09-23T05:13:25.823Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker2 # [ 91.329768] nix-grpc-daemon[870]: time=2026-09-23T05:13:25.824Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)" worker2 # [ 91.332503] nix-grpc-daemon[870]: time=2026-09-23T05:13:25.824Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)" lb # [ 91.372452] niks3-server[970]: 2026/09/23 05:13:25 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst lb # [ 91.386023] niks3-server[970]: 2026/09/23 05:13:25 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst worker2 # [ 91.388583] nix-grpc-daemon[870]: time=2026-09-23T05:13:25.887Z level=INFO msg="Uploading 3 narinfos" lb # [ 91.401392] niks3-server[970]: 2026/09/23 05:13:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign lb # [ 91.409319] niks3-server[970]: 2026/09/23 05:13:25 INFO Signed narinfos id=28 count=1 lb # [ 91.414566] niks3-server[970]: 2026/09/23 05:13:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign lb # [ 91.421998] niks3-server[970]: 2026/09/23 05:13:25 INFO Signed narinfos id=29 count=2 lb # [ 91.427471] niks3-server[970]: 2026/09/23 05:13:25 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls lb # [ 91.433130] niks3-server[970]: 2026/09/23 05:13:25 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls lb # [ 91.444194] niks3-server[970]: 2026/09/23 05:13:25 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete lb # [ 91.447244] niks3-server[970]: 2026/09/23 05:13:25 INFO Completed upload id=28 lb # [ 91.453840] niks3-server[970]: 2026/09/23 05:13:25 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete lb # [ 91.457079] niks3-server[970]: 2026/09/23 05:13:25 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo lb # [ 91.460918] niks3-server[970]: 2026/09/23 05:13:25 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo lb # [ 91.465073] niks3-server[970]: 2026/09/23 05:13:25 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo lb # [ 91.470618] niks3-server[970]: 2026/09/23 05:13:25 INFO Completed upload id=29 worker2 # [ 91.458210] nix-grpc-daemon[870]: time=2026-09-23T05:13:25.957Z level=INFO msg="Upload complete. (329ms)" worker2 # [ 91.468941] nix-grpc-daemon[773]: ts=2026-09-23T05:13:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=0 paths=2 nar_bytes_in=984 worker2 # [ 91.486892] nix-grpc-daemon[773]: ts=2026-09-23T05:13:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 91.500370] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 91.512641] nix-daemon[842]: reaped child process 1076, status = succeeded worker2 # [ 91.523319] nix-grpc-daemon[773]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'... client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051... worker2 # [ 91.539893] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) client # uses-input-w1onlya> worker2: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 92.371549] nix-daemon[842]: reaped child process 1086, status = succeeded worker2 # [ 92.474257] nix-daemon[842]: accepted connection from pid 1102, user nix-grpc-daemon (trusted) worker2 # [ 92.491308] nix-daemon[842]: reaped child process 1105, status = succeeded lb # [ 92.510018] niks3-server[970]: 2026/09/23 05:13:27 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 92.511361] nix-grpc-daemon[870]: time=2026-09-23T05:13:27.009Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 92.537853] niks3-server[970]: 2026/09/23 05:13:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign worker2 # [ 92.537068] nix-grpc-daemon[870]: time=2026-09-23T05:13:27.036Z level=INFO msg="Uploading 1 narinfos" lb # [ 92.549412] niks3-server[970]: 2026/09/23 05:13:27 INFO Signed narinfos id=30 count=1 lb # [ 92.558011] niks3-server[970]: 2026/09/23 05:13:27 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls lb # [ 92.568478] niks3-server[970]: 2026/09/23 05:13:27 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker2 # [ 92.569023] nix-grpc-daemon[870]: time=2026-09-23T05:13:27.068Z level=INFO msg="Upload complete. (208ms)" lb # [ 92.580969] niks3-server[970]: 2026/09/23 05:13:27 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete lb # [ 92.586430] niks3-server[970]: 2026/09/23 05:13:27 INFO Completed upload id=30 lb # [ 92.589518] niks3-server[970]: 2026/09/23 05:13:27 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo worker2 # [ 92.580882] nix-grpc-daemon[773]: ts=2026-09-23T05:13:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=15 outputs=1 worker2 # [ 92.592453] nix-daemon[842]: reaped child process 1083, status = succeeded client # copying 1 paths... worker2 # [ 92.627836] nix-grpc-daemon[773]: ts=2026-09-23T05:13:27Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59064 client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'... worker2 # [ 92.662536] nix-grpc-daemon[773]: ts=2026-09-23T05:13:27Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 92.669538] nix-grpc-daemon[773]: ts=2026-09-23T05:13:27Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=0 paths=1 nar_bytes_out=120 client # [ 92.848503] nix-daemon[1093]: reaped child process 1179, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt a >&2, in 2.46 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2 client # [ 93.017800] nix-daemon[1093]: accepted connection from pid 1197, user root (trusted) client # this derivation will be built: client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker1 # [ 93.109260] nix-grpc-daemon[767]: ts=2026-09-23T05:13:27Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:46746 client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker1 # [ 93.119393] nix-grpc-daemon[767]: ts=2026-09-23T05:13:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 93.131238] nix-grpc-daemon[865]: time=2026-09-23T05:13:27.633Z level=INFO msg="All 1 paths already cached" worker1 # [ 93.150804] nix-grpc-daemon[767]: ts=2026-09-23T05:13:27Z level=debug event=want client=15 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=0 worker1 # [ 93.158875] nix-grpc-daemon[767]: ts=2026-09-23T05:13:27Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 93.168202] nix-grpc-daemon[767]: ts=2026-09-23T05:13:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 93.179396] nix-grpc-daemon[767]: ts=2026-09-23T05:13:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:46746 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'... worker1 # [ 93.199117] nix-grpc-daemon[767]: ts=2026-09-23T05:13:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 93.214410] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 93.234438] nix-grpc-daemon[767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 14 ms (attempt 1/5) worker1 # [ 93.255673] nix-grpc-daemon[767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 56 ms (attempt 2/5) worker1 # [ 93.320114] nix-grpc-daemon[767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 251 ms (attempt 3/5) worker1 # [ 93.573485] nix-grpc-daemon[767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 574 ms (attempt 4/5) worker1 # [ 94.158785] nix-grpc-daemon[767]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 94.167960] nix-grpc-daemon[767]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 94.175570] nix-grpc-daemon[767]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'... worker1 # [ 94.343203] nix-daemon[850]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker1 # [ 94.362932] nix-daemon[850]: reaped child process 1273, status = succeeded lb # [ 94.391368] niks3-server[970]: 2026/09/23 05:13:28 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 94.383615] nix-grpc-daemon[865]: time=2026-09-23T05:13:28.886Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 94.385959] nix-grpc-daemon[865]: time=2026-09-23T05:13:28.886Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)" lb # [ 94.425005] niks3-server[970]: 2026/09/23 05:13:28 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst worker1 # [ 94.419143] nix-grpc-daemon[865]: time=2026-09-23T05:13:28.922Z level=INFO msg="Uploading 1 narinfos" lb # [ 94.440480] niks3-server[970]: 2026/09/23 05:13:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign lb # [ 94.452250] niks3-server[970]: 2026/09/23 05:13:28 INFO Signed narinfos id=31 count=1 lb # [ 94.457905] niks3-server[970]: 2026/09/23 05:13:28 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls worker1 # [ 94.448678] nix-grpc-daemon[865]: time=2026-09-23T05:13:28.951Z level=INFO msg="Upload complete. (259ms)" lb # [ 94.468955] niks3-server[970]: 2026/09/23 05:13:28 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete lb # [ 94.474789] niks3-server[970]: 2026/09/23 05:13:28 INFO Completed upload id=31 lb # [ 94.478857] niks3-server[970]: 2026/09/23 05:13:28 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo worker1 # [ 94.460816] nix-grpc-daemon[767]: ts=2026-09-23T05:13:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=1 paths=1 nar_bytes_in=568 worker1 # [ 94.475482] nix-grpc-daemon[767]: ts=2026-09-23T05:13:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 94.484008] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) worker1 # [ 94.499165] nix-daemon[850]: reaped child process 1263, status = succeeded worker1 # [ 94.505424] nix-daemon[850]: accepted connection from pid 767, user nix-grpc-daemon (trusted) client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051... client # uses-input-w1onlyb> node-a: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 95.972529] nix-daemon[850]: reaped child process 1277, status = succeeded worker1 # [ 96.104958] nix-daemon[850]: accepted connection from pid 1293, user nix-grpc-daemon (trusted) worker1 # [ 96.126546] nix-daemon[850]: reaped child process 1296, status = succeeded lb # [ 96.149680] niks3-server[970]: 2026/09/23 05:13:30 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 96.142331] nix-grpc-daemon[865]: time=2026-09-23T05:13:30.643Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 96.181967] niks3-server[970]: 2026/09/23 05:13:30 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls worker1 # [ 96.177063] nix-grpc-daemon[865]: time=2026-09-23T05:13:30.680Z level=INFO msg="Uploading 1 narinfos" lb # [ 96.198581] niks3-server[970]: 2026/09/23 05:13:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign lb # [ 96.208622] niks3-server[970]: 2026/09/23 05:13:30 INFO Signed narinfos id=32 count=1 lb # [ 96.214131] niks3-server[970]: 2026/09/23 05:13:30 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker1 # [ 96.207579] nix-grpc-daemon[865]: time=2026-09-23T05:13:30.710Z level=INFO msg="Upload complete. (287ms)" lb # [ 96.226902] niks3-server[970]: 2026/09/23 05:13:30 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete lb # [ 96.234552] niks3-server[970]: 2026/09/23 05:13:30 INFO Completed upload id=32 client # copying 1 paths... lb # [ 96.238141] niks3-server[970]: 2026/09/23 05:13:30 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo worker1 # [ 96.224643] nix-grpc-daemon[767]: ts=2026-09-23T05:13:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:46746 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=16 outputs=1 worker2 # [ 96.238053] nix-grpc-daemon[773]: ts=2026-09-23T05:13:30Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 96.245879] nix-grpc-daemon[773]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'... worker1 # [ 96.240227] nix-daemon[850]: reaped child process 1275, status = succeeded client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'... worker2 # [ 96.288457] nix-grpc-daemon[773]: ts=2026-09-23T05:13:30Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 96.295423] nix-grpc-daemon[773]: ts=2026-09-23T05:13:30Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=0 paths=1 nar_bytes_out=120 client # [ 96.475813] nix-daemon[1093]: reaped child process 1200, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.63 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2 client # [ 96.644606] nix-daemon[1093]: accepted connection from pid 1218, user root (trusted) client # this derivation will be built: client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker2 # [ 96.744431] nix-grpc-daemon[773]: ts=2026-09-23T05:13:31Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:59064 client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux'... worker2 # [ 96.752210] nix-grpc-daemon[773]: ts=2026-09-23T05:13:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 96.760110] nix-grpc-daemon[870]: time=2026-09-23T05:13:31.257Z level=INFO msg="All 1 paths already cached" worker1 # [ 96.761376] nix-grpc-daemon[767]: ts=2026-09-23T05:13:31Z level=debug event=want client=16 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=0 worker1 # [ 96.768659] nix-grpc-daemon[767]: ts=2026-09-23T05:13:31Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 96.822014] nix-grpc-daemon[773]: ts=2026-09-23T05:13:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 96.837802] nix-grpc-daemon[773]: ts=2026-09-23T05:13:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59064 client # copying 1 paths to 'lb:50051'... worker2 # [ 96.852534] nix-grpc-daemon[870]: time=2026-09-23T05:13:31.351Z level=INFO msg="All 1 paths already cached" client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'... worker2 # [ 96.860142] nix-grpc-daemon[773]: ts=2026-09-23T05:13:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 96.875680] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 97.035955] nix-daemon[842]: accepted connection from pid 1110, user nix-grpc-daemon (trusted) worker2 # [ 97.058645] nix-daemon[842]: reaped child process 1113, status = succeeded lb # [ 97.071923] niks3-server[970]: 2026/09/23 05:13:31 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 97.072776] nix-grpc-daemon[870]: time=2026-09-23T05:13:31.572Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 97.074876] nix-grpc-daemon[870]: time=2026-09-23T05:13:31.573Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)" lb # [ 97.109290] niks3-server[970]: 2026/09/23 05:13:31 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls worker2 # [ 97.111170] nix-grpc-daemon[870]: time=2026-09-23T05:13:31.610Z level=INFO msg="Uploading 1 narinfos" lb # [ 97.123392] niks3-server[970]: 2026/09/23 05:13:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign lb # [ 97.135990] niks3-server[970]: 2026/09/23 05:13:31 INFO Signed narinfos id=33 count=1 lb # [ 97.141911] niks3-server[970]: 2026/09/23 05:13:31 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst worker2 # [ 97.143449] nix-grpc-daemon[870]: time=2026-09-23T05:13:31.642Z level=INFO msg="Upload complete. (248ms)" lb # [ 97.155328] niks3-server[970]: 2026/09/23 05:13:31 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete lb # [ 97.158752] niks3-server[970]: 2026/09/23 05:13:31 INFO Completed upload id=33 lb # [ 97.161834] niks3-server[970]: 2026/09/23 05:13:31 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.narinfo worker2 # [ 97.157042] nix-grpc-daemon[773]: ts=2026-09-23T05:13:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=0 paths=1 nar_bytes_in=568 worker2 # [ 97.172545] nix-grpc-daemon[773]: ts=2026-09-23T05:13:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 97.179198] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) worker2 # [ 97.185390] nix-daemon[842]: reaped child process 1108, status = succeeded client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051... worker2 # [ 97.199940] nix-daemon[842]: accepted connection from pid 773, user nix-grpc-daemon (trusted) client # uses-input-w1onlyc> worker2: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 98.697094] nix-daemon[842]: reaped child process 1117, status = succeeded worker2 # [ 98.764971] nix-daemon[842]: accepted connection from pid 1133, user nix-grpc-daemon (trusted) worker2 # [ 98.786146] nix-daemon[842]: reaped child process 1136, status = succeeded lb # [ 98.795336] niks3-server[970]: 2026/09/23 05:13:33 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 98.798804] nix-grpc-daemon[870]: time=2026-09-23T05:13:33.295Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 98.828564] niks3-server[970]: 2026/09/23 05:13:33 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls worker2 # [ 98.830870] nix-grpc-daemon[870]: time=2026-09-23T05:13:33.330Z level=INFO msg="Uploading 1 narinfos" lb # [ 98.840682] niks3-server[970]: 2026/09/23 05:13:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign lb # [ 98.853153] niks3-server[970]: 2026/09/23 05:13:33 INFO Signed narinfos id=34 count=1 lb # [ 98.860178] niks3-server[970]: 2026/09/23 05:13:33 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker2 # [ 98.864900] nix-grpc-daemon[870]: time=2026-09-23T05:13:33.364Z level=INFO msg="Upload complete. (253ms)" lb # [ 98.875349] niks3-server[970]: 2026/09/23 05:13:33 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete lb # [ 98.882465] niks3-server[970]: 2026/09/23 05:13:33 INFO Completed upload id=34 lb # [ 98.885929] niks3-server[970]: 2026/09/23 05:13:33 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo worker2 # [ 98.877026] nix-grpc-daemon[773]: ts=2026-09-23T05:13:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=17 outputs=1 worker2 # [ 98.892488] nix-daemon[842]: reaped child process 1115, status = succeeded client # copying 1 paths... worker1 # [ 98.917004] nix-grpc-daemon[767]: ts=2026-09-23T05:13:33Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:46746 worker1 # [ 98.930579] nix-grpc-daemon[767]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'... client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'... worker2 # [ 98.983833] nix-grpc-daemon[773]: ts=2026-09-23T05:13:33Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59064 worker2 # [ 98.991366] nix-grpc-daemon[773]: ts=2026-09-23T05:13:33Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:59064 duration_s=0 paths=1 nar_bytes_out=120 client # [ 99.123152] nix-daemon[1093]: reaped child process 1221, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.64 seconds) (finished: subtest: an input only one worker has still reaches the builder, in 9.39 seconds) subtest: upload whose reference the worker lacks is completed from the cache client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache client # this derivation will be built: client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A input --argstr tag viacache, in 0.34 seconds) client: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache client # this derivation will be built: client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/xwnww2wldfcdimh1fkyvzhwrccjdhpmk-dep.nix -A referrer --argstr tag viacache, in 0.29 seconds) client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.12 seconds) worker1: must succeed: nix-store --import < /tmp/shared/ref.closure worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.19 seconds) worker2 # [ 100.098593] systemd[1]: Stopping Nix worker-protocol over gRPC... lb # [ 100.277059] envoy[908]: conn peer=[2001:db8:1::1]:35162 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3891 tx=2066 ms=2039 lb # [ 100.282214] envoy[908]: conn peer=[2001:db8:1::1]:48492 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=69 lb # [ 100.287004] envoy[908]: conn peer=[2001:db8:1::1]:48504 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3757 tx=2122 ms=3237 lb # [ 100.291952] envoy[908]: conn peer=[2001:db8:1::1]:48506 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=328 ms=71 lb # [ 100.296604] envoy[908]: conn peer=[2001:db8:1::1]:48522 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3758 tx=2126 ms=2263 lb # [ 100.301509] envoy[908]: conn peer=[2001:db8:1::1]:48532 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=328 ms=29 worker2 # [ 100.724365] nix-grpc-daemon[773]: ts=2026-09-23T05:13:35Z level=info event=signal_exit worker1 # [ 102.209074] nix-grpc-daemon[767]: ts=2026-09-23T05:13:36Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:46736 worker1 # [ 102.209407] nix-grpc-daemon[767]: ts=2026-09-23T05:13:36Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 102.225192] nix-grpc-daemon[773]: ts=2026-09-23T05:13:36Z level=info event=scheduler_restarting worker2 # [ 102.250389] nix-grpc-daemon[773]: ts=2026-09-23T05:13:36Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 102.270462] nix-daemon[842]: reaped child process 844, status = succeeded worker2 # [ 102.889016] nix-grpc-daemon[773]: E0000 00:00:1790140417.388181 819 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true worker2 # [ 102.897965] nix-grpc-daemon[773]: E0000 00:00:1790140417.397172 819 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker2 # [ 104.132821] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 104.137778] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 104.143802] systemd[1]: nix-grpc-daemon.service: Consumed 2.792s CPU time over 1min 23.701s wall clock time, 60.6M memory peak, 80.6K incoming IP traffic, 185.8K outgoing IP traffic. worker2 # [ 104.159015] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker2 # [ 104.168430] systemd[1]: Closed Nix worker-protocol over gRPC. worker2 # [ 104.170151] systemd[1]: nix-grpc-daemon.socket: Consumed 719us CPU time over 1min 33.862s wall clock time, 256K memory peak, 93.5K incoming IP traffic, 81.8K outgoing IP traffic. lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.07 seconds) client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2 worker1 # [ 104.459398] nix-grpc-daemon[767]: ts=2026-09-23T05:13:38Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:46746 client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache worker1 # [ 104.466932] nix-grpc-daemon[767]: ts=2026-09-23T05:13:38Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:46746 client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.25 seconds) worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.03 seconds) worker2 # [ 104.597904] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 104.662008] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 105.129556] systemd[1]: Starting scheduler-token.service... worker2 # [ 105.222828] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 105.233149] systemd[1]: Finished scheduler-token.service. worker2 # [ 105.235119] systemd[1]: scheduler-token.service: Consumed 62ms CPU time over 97ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 105.242507] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 105.392575] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:39Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 105.395149] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:39Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 105.399288] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:39Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 105.404056] nix-grpc-daemon[1160]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 105.405511] nix-grpc-daemon[1160]: E0000 00:00:1790140419.903431 1160 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 105.452094] nix-grpc-daemon[1160]: E0000 00:00:1790140419.909882 1160 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 105.454551] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:39Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 105.460470] nix-grpc-daemon[1160]: E0000 00:00:1790140419.920793 1200 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 105.463347] nix-grpc-daemon[1160]: E0000 00:00:1790140419.939621 1200 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 105.466003] nix-grpc-daemon[1160]: E0000 00:00:1790140419.939795 1200 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 105.471559] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 105.474587] nix-grpc-daemon[1160]: E0000 00:00:1790140419.958910 1200 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 105.477235] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:39Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 105.469967] nix-grpc-daemon[767]: ts=2026-09-23T05:13:39Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:46736 worker1 # [ 105.476197] nix-grpc-daemon[767]: ts=2026-09-23T05:13:39Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 105.542668] nix-grpc-daemon[767]: ts=2026-09-23T05:13:40Z level=info event=signal_exit lb # [ 106.230876] niks3-server[970]: 2026/09/23 05:13:40 INFO lead: released remote=192.168.1.3:39118 worker1 # [ 107.203750] nix-grpc-daemon[767]: ts=2026-09-23T05:13:41Z level=info event=scheduler_restarting worker2 # [ 107.215873] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:41Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 107.218512] nix-grpc-daemon[767]: ts=2026-09-23T05:13:41Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:46736 worker1 # [ 107.224502] nix-grpc-daemon[767]: ts=2026-09-23T05:13:41Z level=info event=worker_gone addr=192.168.1.4:50051 worker1 # [ 107.229075] nix-grpc-daemon[767]: ts=2026-09-23T05:13:41Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 107.233936] nix-grpc-daemon[767]: ts=2026-09-23T05:13:41Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:46736 worker1 # [ 107.239482] nix-grpc-daemon[767]: ts=2026-09-23T05:13:41Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 107.269466] nix-daemon[850]: reaped child process 852, status = succeeded worker2 # [ 107.415817] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:41Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 107.422382] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:41Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 107.441695] nix-grpc-daemon[767]: ts=2026-09-23T05:13:41Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 107.623568] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 107.629447] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 107.731876] nix-grpc-daemon[767]: E0000 00:00:1790140422.234748 813 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 19, started true, shutdown true worker1 # [ 107.738861] nix-grpc-daemon[767]: E0000 00:00:1790140422.235140 813 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker2 # [ 107.830416] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 107.836215] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 108.037901] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 108.043513] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 108.063289] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 108.071626] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 108.079975] systemd[1]: nix-grpc-daemon.service: Consumed 3.899s CPU time over 1min 27.803s wall clock time, 69.3M memory peak, 115.5K incoming IP traffic, 233.1K outgoing IP traffic. worker1 # [ 108.095331] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker1 # [ 108.101837] systemd[1]: Closed Nix worker-protocol over gRPC. worker1 # [ 108.103589] systemd[1]: nix-grpc-daemon.socket: Consumed 808us CPU time over 1min 37.371s wall clock time, 256K memory peak, 125.6K incoming IP traffic, 106.6K outgoing IP traffic. lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.08 seconds) client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2 worker2 # [ 108.246227] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 108.251672] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 108.443683] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:42Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:37296 worker2 # [ 108.461248] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:42Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 108.470740] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:42Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 108.481372] nix-daemon[842]: accepted connection from pid 1160, user nix-grpc-daemon (trusted) worker2 # [ 108.497495] nix-grpc-daemon[1160]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 93 ms (attempt 1/5) worker2 # [ 108.598121] nix-grpc-daemon[1160]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 164 ms (attempt 2/5) worker2 # [ 108.656719] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 108.662445] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 108.764649] nix-grpc-daemon[1160]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 392 ms (attempt 3/5) worker2 # [ 108.863480] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 108.869277] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 109.070466] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 109.075866] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 109.158895] nix-grpc-daemon[1160]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 648 ms (attempt 4/5) worker2 # [ 109.279268] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 109.284491] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 109.480733] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:43Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 109.486538] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:43Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 109.687519] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 109.692958] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 109.810585] nix-grpc-daemon[1160]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 109.817354] nix-grpc-daemon[1160]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org client # copying 2 paths... client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'... worker2 # [ 109.844651] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37296 client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'... worker2 # [ 109.866583] nix-daemon[842]: accepted connection from pid 1160, user nix-grpc-daemon (trusted) worker2 # [ 109.899606] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:44Z level=info event=niks3_push_started pid=1221 worker2 # [ 109.925830] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 109.930666] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 110.062641] nix-grpc-daemon[1221]: time=2026-09-23T05:13:44.561Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 110.114405] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 110.118556] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 110.182614] nix-daemon[842]: accepted connection from pid 1234, user nix-grpc-daemon (trusted) worker2 # [ 110.206319] nix-daemon[842]: reaped child process 1237, status = succeeded lb # [ 110.224970] niks3-server[970]: 2026/09/23 05:13:44 INFO Received uploads request method=POST path=/api/pending_closures lb # [ 110.236502] niks3-server[970]: 2026/09/23 05:13:44 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 110.238321] nix-grpc-daemon[1221]: time=2026-09-23T05:13:44.735Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker2 # [ 110.239963] nix-grpc-daemon[1221]: time=2026-09-23T05:13:44.735Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)" worker2 # [ 110.244574] nix-grpc-daemon[1221]: time=2026-09-23T05:13:44.735Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)" lb # [ 110.279904] envoy[908]: conn peer=[2001:db8:1::4]:52702 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=53547 tx=3126 ms=81470 lb # [ 110.293985] envoy[908]: conn peer=[2001:db8:1::1]:43496 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=23 lb # [ 110.303415] envoy[908]: conn peer=[2001:db8:1::3]:47868 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=11705 tx=3444 ms=86827 lb # [ 110.317150] niks3-server[970]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls worker2 # [ 110.319375] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:44Z level=info event=scheduler_connected addr=lb:50051 lb # [ 110.331317] niks3-server[970]: 2026/09/23 05:13:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign worker2 # [ 110.323331] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:44Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 110.346369] niks3-server[970]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls lb # [ 110.357444] niks3-server[970]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst lb # [ 110.362699] niks3-server[970]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst lb # [ 110.367726] niks3-server[970]: 2026/09/23 05:13:44 INFO Signed narinfos id=35 count=1 worker2 # [ 110.360154] nix-grpc-daemon[1221]: time=2026-09-23T05:13:44.857Z level=INFO msg="Uploading 3 narinfos" lb # [ 110.369883] niks3-server[970]: 2026/09/23 05:13:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign lb # [ 110.374868] niks3-server[970]: 2026/09/23 05:13:44 INFO Signed narinfos id=36 count=2 lb # [ 110.401019] niks3-server[970]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo lb # [ 110.406799] niks3-server[970]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo lb # [ 110.411583] niks3-server[970]: 2026/09/23 05:13:44 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete lb # [ 110.419727] niks3-server[970]: 2026/09/23 05:13:44 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo lb # [ 110.425619] niks3-server[970]: 2026/09/23 05:13:44 INFO Completed upload id=35 lb # [ 110.426895] niks3-server[970]: 2026/09/23 05:13:44 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete lb # [ 110.429849] niks3-server[970]: 2026/09/23 05:13:44 INFO Completed upload id=36 lb # [ 110.432669] niks3-server[970]: 2026/09/23 05:13:44 INFO lead: acquired remote=192.168.1.4:46736 worker2 # [ 110.420390] nix-grpc-daemon[1221]: time=2026-09-23T05:13:44.919Z level=INFO msg="Upload complete. (353ms)" worker2 # [ 110.432527] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:44Z level=info event=scheduler_take_over worker2 # [ 110.440564] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37296 duration_s=0 paths=2 nar_bytes_in=312 worker2 # [ 110.461562] nix-daemon[842]: reaped child process 1219, status = succeeded client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 2.27 seconds) worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.03 seconds) worker2 # [ 110.520806] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 110.539502] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:45Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 110.544302] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:45Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:37304 worker2 # [ 110.549321] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:45Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 110.573303] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 110.583550] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 110.730991] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:45Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 110.733459] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:45Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 110.737620] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:45Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 110.742105] nix-grpc-daemon[1322]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 110.743588] nix-grpc-daemon[1322]: E0000 00:00:1790140425.245451 1322 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 110.780314] nix-grpc-daemon[1322]: E0000 00:00:1790140425.252152 1322 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 110.795192] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:45Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 110.797680] nix-grpc-daemon[1322]: E0000 00:00:1790140425.262433 1362 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 110.800227] nix-grpc-daemon[1322]: E0000 00:00:1790140425.279451 1362 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 110.802460] nix-grpc-daemon[1322]: E0000 00:00:1790140425.279723 1362 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 110.804872] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 110.805698] nix-grpc-daemon[1322]: E0000 00:00:1790140425.297319 1362 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 110.814112] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 110.824104] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:45Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:37304 worker2 # [ 110.830818] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:45Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.15 seconds) (finished: subtest: upload whose reference the worker lacks is completed from the cache, in 12.84 seconds) subtest: a niks3 restart does not cost the leader its role lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb # [ 112.069673] niks3-server[970]: 2026/09/23 05:13:46 INFO Shutdown signal received, draining in-flight requests timeout=10s lb # [ 112.070249] systemd[1]: Stopping niks3 server... worker1 # [ 112.058216] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:46Z level=info event=scheduler_lock_lost status=200 lb # [ 112.080157] niks3-server[970]: 2026/09/23 05:13:46 INFO lead: released remote=192.168.1.4:46736 worker2 # [ 112.067768] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:46Z level=info event=scheduler_lock_lost status=200 lb # [ 112.102378] systemd[1]: niks3.service: Deactivated successfully. lb # [ 112.104322] systemd[1]: Stopped niks3 server. lb # [ 112.105055] systemd[1]: niks3.service: Consumed 689ms CPU time over 1min 28.183s wall clock time, 27.7M memory peak, 98K incoming IP traffic, 141.1K outgoing IP traffic. lb # [ 112.115344] systemd[1]: Starting niks3 server... lb # [ 112.179717] niks3-server[1111]: 2026/09/23 05:13:46 goose: no migrations to run. current version: 20260920000000 lb # [ 112.186456] niks3-server[1111]: 2026/09/23 05:13:46 OK 1_commit_pending_closure.sql (5.07ms) lb # [ 112.190356] niks3-server[1111]: 2026/09/23 05:13:46 OK 2_object_stats_trigger.sql (3.5ms) lb # [ 112.191561] niks3-server[1111]: 2026/09/23 05:13:46 goose: up to current file version: 2 lb # [ 112.193672] niks3-server[1111]: 2026/09/23 05:13:46 INFO Loaded signing key name=farm-test-1 path=/nix/store/3qiyvklx8qgcqqsb1swilx6drs1cdj8m-key lb # [ 112.205098] niks3-server[1111]: 2026/09/23 05:13:46 INFO Read proxy enabled — serving cache objects from S3 lb # [ 112.206524] niks3-server[1111]: 2026/09/23 05:13:46 INFO Using socket-activated listener address=0.0.0.0:5751 lb # [ 112.211947] niks3-server[1111]: 2026/09/23 05:13:46 INFO systemd watchdog enabled interval=15s lb # [ 112.213490] niks3-server[1111]: 2026/09/23 05:13:46 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 112.215347] systemd[1]: Started niks3 server. lb: waiting for unit niks3.service lb: (finished: waiting for unit niks3.service, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield worker2: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.07 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2 worker1 # [ 112.702710] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:47Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:48992 worker1 # [ 112.714431] nix-daemon[850]: accepted connection from pid 1322, user nix-grpc-daemon (trusted) client # don't know how to build these paths: client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker2 # [ 112.753520] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:37296 worker2 # [ 112.788173] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:47Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=0 worker2 # [ 112.794368] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:47Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 112.842828] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37296 worker2 # [ 112.861163] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:37296 client # copying 1 paths to 'lb:50051'... worker2 # [ 112.878887] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37296 client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'... worker2 # [ 112.896753] nix-daemon[842]: accepted connection from pid 1160, user nix-grpc-daemon (trusted) worker2 # [ 113.057643] nix-daemon[842]: accepted connection from pid 1254, user nix-grpc-daemon (trusted) lb # [ 113.087183] niks3-server[1111]: 2026/09/23 05:13:47 INFO lead: acquired remote=192.168.1.4:39538 worker2 # [ 113.085618] nix-daemon[842]: reaped child process 1257, status = succeeded worker2 # [ 113.118949] nix-grpc-daemon[1221]: time=2026-09-23T05:13:47.617Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 113.121338] nix-grpc-daemon[1221]: time=2026-09-23T05:13:47.617Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)" lb # [ 113.093436] niks3-server[1111]: 2026/09/23 05:13:47 INFO Received uploads request method=POST path=/api/pending_closures lb # [ 113.163444] niks3-server[1111]: 2026/09/23 05:13:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign lb # [ 113.189619] niks3-server[1111]: 2026/09/23 05:13:47 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst lb # [ 113.208097] niks3-server[1111]: 2026/09/23 05:13:47 INFO Signed narinfos id=37 count=1 lb # [ 113.216385] niks3-server[1111]: 2026/09/23 05:13:47 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls worker2 # [ 113.203784] nix-grpc-daemon[1221]: time=2026-09-23T05:13:47.701Z level=INFO msg="Uploading 1 narinfos" lb # [ 113.224421] niks3-server[1111]: 2026/09/23 05:13:47 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete lb # [ 113.232738] niks3-server[1111]: 2026/09/23 05:13:47 INFO Completed upload id=37 lb # [ 113.240390] niks3-server[1111]: 2026/09/23 05:13:47 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.narinfo worker2 # [ 113.223098] nix-grpc-daemon[1221]: time=2026-09-23T05:13:47.722Z level=INFO msg="Upload complete. (307ms)" worker2 # [ 113.238606] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37296 duration_s=0 paths=1 nar_bytes_in=544 worker2 # [ 113.252525] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37296 worker2 # [ 113.266518] nix-daemon[842]: accepted connection from pid 1160, user nix-grpc-daemon (trusted) worker2 # [ 113.273117] nix-daemon[842]: reaped child process 1252, status = succeeded client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051... client # farm-a> worker2: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker2 # [ 113.286756] nix-daemon[842]: accepted connection from pid 1160, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-after-niks3-restart worker2 # [ 114.126196] nix-daemon[842]: reaped child process 1263, status = succeeded worker2 # [ 114.231127] nix-daemon[842]: accepted connection from pid 1279, user nix-grpc-daemon (trusted) worker2 # [ 114.251880] nix-daemon[842]: reaped child process 1282, status = succeeded lb # [ 114.270095] niks3-server[1111]: 2026/09/23 05:13:48 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 114.275639] nix-grpc-daemon[1221]: time=2026-09-23T05:13:48.772Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 114.277416] nix-grpc-daemon[1221]: time=2026-09-23T05:13:48.772Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)" lb # [ 114.316973] niks3-server[1111]: 2026/09/23 05:13:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign lb # [ 114.334826] niks3-server[1111]: 2026/09/23 05:13:48 INFO Signed narinfos id=38 count=1 worker2 # [ 114.329702] nix-grpc-daemon[1221]: time=2026-09-23T05:13:48.828Z level=INFO msg="Uploading 1 narinfos" lb # [ 114.348667] niks3-server[1111]: 2026/09/23 05:13:48 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst lb # [ 114.361347] niks3-server[1111]: 2026/09/23 05:13:48 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv lb # [ 114.373013] niks3-server[1111]: 2026/09/23 05:13:48 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls lb # [ 114.375851] niks3-server[1111]: 2026/09/23 05:13:48 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete lb # [ 114.380465] niks3-server[1111]: 2026/09/23 05:13:48 INFO Completed upload id=38 worker2 # [ 114.370481] nix-grpc-daemon[1221]: time=2026-09-23T05:13:48.869Z level=INFO msg="Upload complete. (269ms)" lb # [ 114.383002] niks3-server[1111]: 2026/09/23 05:13:48 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.narinfo worker2 # [ 114.397280] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37296 duration_s=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=1 worker2 # [ 114.408585] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:48Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=0 worker2 # [ 114.415876] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:48Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 114.420576] nix-daemon[842]: reaped child process 1261, status = succeeded worker2 # [ 114.442762] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37296 worker2 # [ 114.459756] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:37296 worker2 # [ 114.478399] nix-grpc-daemon[1221]: time=2026-09-23T05:13:48.977Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker2 # [ 114.490221] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37296 client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'... worker2 # [ 114.507774] nix-daemon[842]: accepted connection from pid 1160, user nix-grpc-daemon (trusted) worker2 # [ 114.668156] nix-daemon[842]: accepted connection from pid 1288, user nix-grpc-daemon (trusted) worker2 # [ 114.693510] nix-daemon[842]: reaped child process 1291, status = succeeded lb # [ 114.700649] niks3-server[1111]: 2026/09/23 05:13:49 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 114.699786] nix-grpc-daemon[1221]: time=2026-09-23T05:13:49.198Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 114.701647] nix-grpc-daemon[1221]: time=2026-09-23T05:13:49.198Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)" lb # [ 114.728998] niks3-server[1111]: 2026/09/23 05:13:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign worker2 # [ 114.725619] nix-grpc-daemon[1221]: time=2026-09-23T05:13:49.224Z level=INFO msg="Uploading 1 narinfos" lb # [ 114.740983] niks3-server[1111]: 2026/09/23 05:13:49 INFO Signed narinfos id=39 count=1 lb # [ 114.752763] niks3-server[1111]: 2026/09/23 05:13:49 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst lb # [ 114.772451] niks3-server[1111]: 2026/09/23 05:13:49 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls lb # [ 114.775925] niks3-server[1111]: 2026/09/23 05:13:49 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete lb # [ 114.781315] niks3-server[1111]: 2026/09/23 05:13:49 INFO Completed upload id=39 worker2 # [ 114.771314] nix-grpc-daemon[1221]: time=2026-09-23T05:13:49.270Z level=INFO msg="Upload complete. (244ms)" lb # [ 114.785700] niks3-server[1111]: 2026/09/23 05:13:49 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo worker2 # [ 114.784505] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37296 duration_s=0 paths=1 nar_bytes_in=656 worker2 # [ 114.793898] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37296 worker2 # [ 114.806705] nix-daemon[842]: accepted connection from pid 1160, user nix-grpc-daemon (trusted) worker2 # [ 114.813947] nix-daemon[842]: reaped child process 1286, status = succeeded client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051... client # farm-b> worker2: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker2 # [ 114.829194] nix-daemon[842]: accepted connection from pid 1160, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-after-niks3-restart worker2 # [ 115.463303] nix-daemon[842]: reaped child process 1295, status = succeeded worker2 # [ 115.563799] nix-daemon[842]: accepted connection from pid 1311, user nix-grpc-daemon (trusted) worker2 # [ 115.589241] nix-daemon[842]: reaped child process 1314, status = succeeded lb # [ 115.597833] niks3-server[1111]: 2026/09/23 05:13:50 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 115.600606] nix-grpc-daemon[1221]: time=2026-09-23T05:13:50.099Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 115.602278] nix-grpc-daemon[1221]: time=2026-09-23T05:13:50.099Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)" lb # [ 115.648976] niks3-server[1111]: 2026/09/23 05:13:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign worker2 # [ 115.648254] nix-grpc-daemon[1221]: time=2026-09-23T05:13:50.147Z level=INFO msg="Uploading 1 narinfos" lb # [ 115.661020] niks3-server[1111]: 2026/09/23 05:13:50 INFO Signed narinfos id=40 count=1 lb # [ 115.669576] niks3-server[1111]: 2026/09/23 05:13:50 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls lb # [ 115.677167] niks3-server[1111]: 2026/09/23 05:13:50 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv lb # [ 115.689253] niks3-server[1111]: 2026/09/23 05:13:50 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete lb # [ 115.696869] niks3-server[1111]: 2026/09/23 05:13:50 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst lb # [ 115.702509] niks3-server[1111]: 2026/09/23 05:13:50 INFO Completed upload id=40 worker2 # [ 115.692641] nix-grpc-daemon[1221]: time=2026-09-23T05:13:50.191Z level=INFO msg="Upload complete. (252ms)" lb # [ 115.706363] niks3-server[1111]: 2026/09/23 05:13:50 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.narinfo worker2 # [ 115.717450] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37296 duration_s=0 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=1 worker2 # [ 115.734660] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:50Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=0 worker2 # [ 115.738143] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:50Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 115.742284] nix-daemon[842]: reaped child process 1293, status = succeeded worker2 # [ 115.758821] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37296 worker2 # [ 115.774280] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:37296 worker2 # [ 115.789755] nix-grpc-daemon[1221]: time=2026-09-23T05:13:50.289Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'... worker2 # [ 115.801715] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37296 worker2 # [ 115.816403] nix-daemon[842]: accepted connection from pid 1160, user nix-grpc-daemon (trusted) worker2 # [ 115.974053] nix-daemon[842]: accepted connection from pid 1320, user nix-grpc-daemon (trusted) worker2 # [ 115.998080] nix-daemon[842]: reaped child process 1323, status = succeeded lb # [ 116.009295] niks3-server[1111]: 2026/09/23 05:13:50 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 116.008392] nix-grpc-daemon[1221]: time=2026-09-23T05:13:50.507Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 116.010558] nix-grpc-daemon[1221]: time=2026-09-23T05:13:50.507Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)" worker2 # [ 116.042849] nix-grpc-daemon[1221]: time=2026-09-23T05:13:50.541Z level=INFO msg="Uploading 1 narinfos" lb # [ 116.046586] niks3-server[1111]: 2026/09/23 05:13:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign lb # [ 116.059187] niks3-server[1111]: 2026/09/23 05:13:50 INFO Signed narinfos id=41 count=1 lb # [ 116.068263] niks3-server[1111]: 2026/09/23 05:13:50 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst worker2 # [ 116.075617] nix-grpc-daemon[1221]: time=2026-09-23T05:13:50.575Z level=INFO msg="Upload complete. (242ms)" lb # [ 116.085908] niks3-server[1111]: 2026/09/23 05:13:50 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls lb # [ 116.093809] niks3-server[1111]: 2026/09/23 05:13:50 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete worker2 # [ 116.084345] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:37296 duration_s=0 paths=1 nar_bytes_in=784 lb # [ 116.099400] niks3-server[1111]: 2026/09/23 05:13:50 INFO Completed upload id=41 lb # [ 116.102739] niks3-server[1111]: 2026/09/23 05:13:50 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo worker2 # [ 116.100405] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37296 worker2 # [ 116.109718] nix-daemon[842]: accepted connection from pid 1160, user nix-grpc-daemon (trusted) worker2 # [ 116.115203] nix-daemon[842]: reaped child process 1318, status = succeeded client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051... worker2 # [ 116.127200] nix-daemon[842]: accepted connection from pid 1160, user nix-grpc-daemon (trusted) client # farm-top> worker2: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-after-niks3-restart worker2 # [ 117.757391] nix-daemon[842]: reaped child process 1327, status = succeeded worker2 # [ 117.861557] nix-daemon[842]: accepted connection from pid 1343, user nix-grpc-daemon (trusted) worker2 # [ 117.884744] nix-daemon[842]: reaped child process 1346, status = succeeded lb # [ 117.906494] niks3-server[1111]: 2026/09/23 05:13:52 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 117.909093] nix-grpc-daemon[1221]: time=2026-09-23T05:13:52.407Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 117.910667] nix-grpc-daemon[1221]: time=2026-09-23T05:13:52.407Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)" lb # [ 117.958887] niks3-server[1111]: 2026/09/23 05:13:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign worker2 # [ 117.958236] nix-grpc-daemon[1221]: time=2026-09-23T05:13:52.457Z level=INFO msg="Uploading 1 narinfos" lb # [ 117.972322] niks3-server[1111]: 2026/09/23 05:13:52 INFO Signed narinfos id=42 count=1 lb # [ 117.980779] niks3-server[1111]: 2026/09/23 05:13:52 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst lb # [ 117.990399] niks3-server[1111]: 2026/09/23 05:13:52 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv lb # [ 117.997947] niks3-server[1111]: 2026/09/23 05:13:52 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls lb # [ 118.007893] niks3-server[1111]: 2026/09/23 05:13:52 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete worker2 # [ 118.002022] nix-grpc-daemon[1221]: time=2026-09-23T05:13:52.501Z level=INFO msg="Upload complete. (260ms)" lb # [ 118.015270] niks3-server[1111]: 2026/09/23 05:13:52 INFO Completed upload id=42 lb # [ 118.018244] niks3-server[1111]: 2026/09/23 05:13:52 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo worker2 # [ 118.018866] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:37296 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=1 worker2 # [ 118.045722] nix-daemon[842]: reaped child process 1325, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart >&2, in 5.59 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag after-niks3-restart outPath, in 0.20 seconds) (finished: subtest: a niks3 restart does not cost the leader its role, in 6.34 seconds) subtest: scheduler down: the other node takes the lock and keeps it lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2 # [ 118.526503] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 119.424889] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:53Z level=info event=signal_exit lb # [ 120.097337] niks3-server[1111]: 2026/09/23 05:13:54 INFO lead: released remote=192.168.1.4:39538 lb # [ 120.279236] envoy[908]: conn peer=[2001:db8:1::1]:43502 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1011 tx=305 ms=2012 lb # [ 120.284574] envoy[908]: conn peer=[2001:db8:1::1]:46834 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9719 tx=4434 ms=5325 worker1 # [ 121.077225] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:55Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 121.090151] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:55Z level=info event=scheduler_restarting worker2 # [ 121.095509] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:55Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:37304 worker2 # [ 121.100917] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:55Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 121.105679] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:55Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:37304 worker2 # [ 121.111132] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:55Z level=info event=worker_gone addr=192.168.1.3:50051 worker2 # [ 121.115645] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:55Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 121.141116] nix-daemon[842]: reaped child process 1213, status = succeeded worker1 # [ 121.276858] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:55Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 121.282609] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 121.310698] nix-grpc-daemon[1160]: ts=2026-09-23T05:13:55Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 121.478891] nix-grpc-daemon[1160]: E0000 00:00:1790140435.977814 1206 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true worker2 # [ 121.486078] nix-grpc-daemon[1160]: E0000 00:00:1790140435.978749 1206 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker1 # [ 121.483335] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:55Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 121.488909] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:55Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 121.689759] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:56Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 121.695194] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 121.896143] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:56Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 121.901568] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 122.104706] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:56Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 122.108934] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 122.305685] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:56Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 122.311101] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:56Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 122.374832] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 122.379831] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 122.388581] systemd[1]: nix-grpc-daemon.service: Consumed 1.411s CPU time over 17.136s wall clock time, 39.1M memory peak, 49.2K incoming IP traffic, 94.4K outgoing IP traffic. worker2 # [ 122.400577] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker2 # [ 122.405720] systemd[1]: Closed Nix worker-protocol over gRPC. worker2 # [ 122.407485] systemd[1]: nix-grpc-daemon.socket: Consumed 801us CPU time over 17.814s wall clock time, 256K memory peak, 38.6K incoming IP traffic, 29K outgoing IP traffic. worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.08 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1 # [ 122.512361] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 122.519287] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds) worker1 # [ 122.717715] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 122.723252] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 122.924119] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 122.929514] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:57Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 123.064134] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:57Z level=info event=scheduler_take_over lb # [ 123.083624] niks3-server[1111]: 2026/09/23 05:13:57 INFO lead: acquired remote=192.168.1.3:57136 worker1 # [ 123.131568] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 123.141529] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:57Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:50830 worker1 # [ 123.147934] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:57Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2 worker1 # [ 123.835667] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:58Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:48992 client # don't know how to build these paths: client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker1 # [ 123.846429] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48992 worker1 # [ 123.861757] nix-grpc-daemon[1322]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 90 ms (attempt 1/5) worker1 # [ 123.961498] nix-grpc-daemon[1322]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 158 ms (attempt 2/5) worker1 # [ 124.124057] nix-grpc-daemon[1322]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 145 ms (attempt 3/5) worker1 # [ 124.279826] nix-grpc-daemon[1322]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 470 ms (attempt 4/5) worker1 # [ 124.751556] nix-grpc-daemon[1322]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 124.758694] nix-grpc-daemon[1322]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 124.792138] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:59Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=0 worker1 # [ 124.798609] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:59Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 124.846022] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48992 worker1 # [ 124.863000] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48992 client # copying 1 paths to 'lb:50051'... worker1 # [ 124.878202] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48992 client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'... worker1 # [ 124.895683] nix-daemon[850]: accepted connection from pid 1322, user nix-grpc-daemon (trusted) worker1 # [ 124.924149] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:59Z level=info event=niks3_push_started pid=1399 worker1 # [ 125.088841] nix-grpc-daemon[1399]: time=2026-09-23T05:13:59.591Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 125.203906] nix-daemon[850]: accepted connection from pid 1412, user nix-grpc-daemon (trusted) worker1 # [ 125.228645] nix-daemon[850]: reaped child process 1415, status = succeeded lb # [ 125.249200] niks3-server[1111]: 2026/09/23 05:13:59 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 125.243058] nix-grpc-daemon[1399]: time=2026-09-23T05:13:59.745Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 125.247148] nix-grpc-daemon[1399]: time=2026-09-23T05:13:59.745Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)" lb # [ 125.295058] niks3-server[1111]: 2026/09/23 05:13:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign lb # [ 125.313803] niks3-server[1111]: 2026/09/23 05:13:59 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst lb # [ 125.329739] niks3-server[1111]: 2026/09/23 05:13:59 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls lb # [ 125.346603] niks3-server[1111]: 2026/09/23 05:13:59 INFO Signed narinfos id=43 count=1 worker1 # [ 125.329819] nix-grpc-daemon[1399]: time=2026-09-23T05:13:59.831Z level=INFO msg="Uploading 1 narinfos" lb # [ 125.362987] niks3-server[1111]: 2026/09/23 05:13:59 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete lb # [ 125.379544] niks3-server[1111]: 2026/09/23 05:13:59 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo lb # [ 125.386090] niks3-server[1111]: 2026/09/23 05:13:59 INFO Completed upload id=43 worker1 # [ 125.367683] nix-grpc-daemon[1399]: time=2026-09-23T05:13:59.870Z level=INFO msg="Upload complete. (275ms)" worker1 # [ 125.384217] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48992 duration_s=0 paths=1 nar_bytes_in=520 worker1 # [ 125.400181] nix-grpc-daemon[1322]: ts=2026-09-23T05:13:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48992 worker1 # [ 125.407827] nix-daemon[850]: accepted connection from pid 1322, user nix-grpc-daemon (trusted) worker1 # [ 125.413810] nix-daemon[850]: reaped child process 1397, status = succeeded client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051... client # farm-a> node-a: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker1 # [ 125.430251] nix-daemon[850]: accepted connection from pid 1322, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-on-standby worker1 # [ 126.471106] nix-daemon[850]: reaped child process 1419, status = succeeded worker1 # [ 126.585269] nix-daemon[850]: accepted connection from pid 1435, user nix-grpc-daemon (trusted) worker1 # [ 126.608047] nix-daemon[850]: reaped child process 1438, status = succeeded lb # [ 126.625148] niks3-server[1111]: 2026/09/23 05:14:01 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 126.620462] nix-grpc-daemon[1399]: time=2026-09-23T05:14:01.121Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 126.621993] nix-grpc-daemon[1399]: time=2026-09-23T05:14:01.121Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)" lb # [ 126.666472] niks3-server[1111]: 2026/09/23 05:14:01 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv lb # [ 126.688102] niks3-server[1111]: 2026/09/23 05:14:01 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst worker1 # [ 126.683254] nix-grpc-daemon[1399]: time=2026-09-23T05:14:01.186Z level=INFO msg="Uploading 1 narinfos" lb # [ 126.701674] niks3-server[1111]: 2026/09/23 05:14:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign lb # [ 126.710742] niks3-server[1111]: 2026/09/23 05:14:01 INFO Signed narinfos id=44 count=1 lb # [ 126.719204] niks3-server[1111]: 2026/09/23 05:14:01 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete lb # [ 126.729237] niks3-server[1111]: 2026/09/23 05:14:01 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls lb # [ 126.734911] niks3-server[1111]: 2026/09/23 05:14:01 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo lb # [ 126.739308] niks3-server[1111]: 2026/09/23 05:14:01 INFO Completed upload id=44 worker1 # [ 126.720732] nix-grpc-daemon[1399]: time=2026-09-23T05:14:01.223Z level=INFO msg="Upload complete. (277ms)" worker1 # [ 126.743834] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48992 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=1 worker1 # [ 126.760851] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:01Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=0 worker1 # [ 126.766583] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:01Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 126.770802] nix-daemon[850]: reaped child process 1417, status = succeeded worker1 # [ 126.786081] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48992 worker1 # [ 126.801807] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:01Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48992 worker1 # [ 126.816962] nix-grpc-daemon[1399]: time=2026-09-23T05:14:01.320Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'... worker1 # [ 126.828603] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:01Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48992 worker1 # [ 126.842966] nix-daemon[850]: accepted connection from pid 1322, user nix-grpc-daemon (trusted) worker1 # [ 127.002060] nix-daemon[850]: accepted connection from pid 1444, user nix-grpc-daemon (trusted) worker1 # [ 127.024870] nix-daemon[850]: reaped child process 1447, status = succeeded lb # [ 127.040699] niks3-server[1111]: 2026/09/23 05:14:01 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 127.032248] nix-grpc-daemon[1399]: time=2026-09-23T05:14:01.535Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 127.034379] nix-grpc-daemon[1399]: time=2026-09-23T05:14:01.535Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)" worker1 # [ 127.055431] nix-grpc-daemon[1399]: time=2026-09-23T05:14:01.558Z level=INFO msg="Uploading 1 narinfos" lb # [ 127.074504] niks3-server[1111]: 2026/09/23 05:14:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign lb # [ 127.087442] niks3-server[1111]: 2026/09/23 05:14:01 INFO Signed narinfos id=45 count=1 lb # [ 127.096467] niks3-server[1111]: 2026/09/23 05:14:01 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls worker1 # [ 127.090932] nix-grpc-daemon[1399]: time=2026-09-23T05:14:01.594Z level=INFO msg="Upload complete. (231ms)" lb # [ 127.109706] niks3-server[1111]: 2026/09/23 05:14:01 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst lb # [ 127.117751] niks3-server[1111]: 2026/09/23 05:14:01 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete lb # [ 127.126986] niks3-server[1111]: 2026/09/23 05:14:01 INFO Completed upload id=45 lb # [ 127.130647] niks3-server[1111]: 2026/09/23 05:14:01 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo worker1 # [ 127.100317] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48992 duration_s=0 paths=1 nar_bytes_in=640 worker1 # [ 127.116993] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:01Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48992 worker1 # [ 127.123378] nix-daemon[850]: accepted connection from pid 1322, user nix-grpc-daemon (trusted) worker1 # [ 127.132868] nix-daemon[850]: reaped child process 1442, status = succeeded client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051... worker1 # [ 127.143175] nix-daemon[850]: accepted connection from pid 1322, user nix-grpc-daemon (trusted) client # farm-b> node-a: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-on-standby worker1 # [ 128.206338] nix-daemon[850]: reaped child process 1451, status = succeeded worker1 # [ 128.321205] nix-daemon[850]: accepted connection from pid 1467, user nix-grpc-daemon (trusted) worker1 # [ 128.343441] nix-daemon[850]: reaped child process 1470, status = succeeded lb # [ 128.366701] niks3-server[1111]: 2026/09/23 05:14:02 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 128.359612] nix-grpc-daemon[1399]: time=2026-09-23T05:14:02.861Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 128.361661] nix-grpc-daemon[1399]: time=2026-09-23T05:14:02.861Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)" lb # [ 128.409608] niks3-server[1111]: 2026/09/23 05:14:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign worker1 # [ 128.401416] nix-grpc-daemon[1399]: time=2026-09-23T05:14:02.904Z level=INFO msg="Uploading 1 narinfos" lb # [ 128.424530] niks3-server[1111]: 2026/09/23 05:14:02 INFO Signed narinfos id=46 count=1 lb # [ 128.435382] niks3-server[1111]: 2026/09/23 05:14:02 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv lb # [ 128.448211] niks3-server[1111]: 2026/09/23 05:14:02 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls lb # [ 128.451524] niks3-server[1111]: 2026/09/23 05:14:02 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst worker1 # [ 128.445779] nix-grpc-daemon[1399]: time=2026-09-23T05:14:02.949Z level=INFO msg="Upload complete. (292ms)" lb # [ 128.465882] niks3-server[1111]: 2026/09/23 05:14:02 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete lb # [ 128.469826] niks3-server[1111]: 2026/09/23 05:14:02 INFO Completed upload id=46 lb # [ 128.473166] niks3-server[1111]: 2026/09/23 05:14:02 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo worker1 # [ 128.463760] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48992 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=1 worker1 # [ 128.484405] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:02Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=0 worker1 # [ 128.491055] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:02Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 128.496659] nix-daemon[850]: reaped child process 1449, status = succeeded worker1 # [ 128.522554] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48992 worker1 # [ 128.539238] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48992 worker1 # [ 128.555363] nix-grpc-daemon[1399]: time=2026-09-23T05:14:03.058Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'... worker1 # [ 128.567464] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48992 worker1 # [ 128.581759] nix-daemon[850]: accepted connection from pid 1322, user nix-grpc-daemon (trusted) worker1 # [ 128.746893] nix-daemon[850]: accepted connection from pid 1476, user nix-grpc-daemon (trusted) worker1 # [ 128.768416] nix-daemon[850]: reaped child process 1479, status = succeeded lb # [ 128.791854] niks3-server[1111]: 2026/09/23 05:14:03 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 128.782467] nix-grpc-daemon[1399]: time=2026-09-23T05:14:03.285Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 128.784235] nix-grpc-daemon[1399]: time=2026-09-23T05:14:03.285Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)" lb # [ 128.824287] niks3-server[1111]: 2026/09/23 05:14:03 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst worker1 # [ 128.816252] nix-grpc-daemon[1399]: time=2026-09-23T05:14:03.319Z level=INFO msg="Uploading 1 narinfos" lb # [ 128.840416] niks3-server[1111]: 2026/09/23 05:14:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign lb # [ 128.850127] niks3-server[1111]: 2026/09/23 05:14:03 INFO Signed narinfos id=47 count=1 lb # [ 128.855770] niks3-server[1111]: 2026/09/23 05:14:03 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls lb # [ 128.866082] niks3-server[1111]: 2026/09/23 05:14:03 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete lb # [ 128.872156] niks3-server[1111]: 2026/09/23 05:14:03 INFO Completed upload id=47 lb # [ 128.875120] niks3-server[1111]: 2026/09/23 05:14:03 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo worker1 # [ 128.850889] nix-grpc-daemon[1399]: time=2026-09-23T05:14:03.354Z level=INFO msg="Upload complete. (251ms)" worker1 # [ 128.860622] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48992 duration_s=0 paths=1 nar_bytes_in=768 worker1 # [ 128.880186] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48992 worker1 # [ 128.887375] nix-daemon[850]: accepted connection from pid 1322, user nix-grpc-daemon (trusted) worker1 # [ 128.893242] nix-daemon[850]: reaped child process 1474, status = succeeded client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051... worker1 # [ 128.907659] nix-daemon[850]: accepted connection from pid 1322, user nix-grpc-daemon (trusted) client # farm-top> node-a: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-on-standby lb # [ 130.278050] envoy[908]: conn peer=[2001:db8:1::4]:52110 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=13940 tx=1040 ms=15824 worker1 # [ 130.418954] nix-daemon[850]: reaped child process 1483, status = succeeded worker1 # [ 130.494192] nix-daemon[850]: accepted connection from pid 1499, user nix-grpc-daemon (trusted) worker1 # [ 130.516488] nix-daemon[850]: reaped child process 1502, status = succeeded lb # [ 130.533559] niks3-server[1111]: 2026/09/23 05:14:05 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 130.528499] nix-grpc-daemon[1399]: time=2026-09-23T05:14:05.029Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 130.530122] nix-grpc-daemon[1399]: time=2026-09-23T05:14:05.029Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)" lb # [ 130.583428] niks3-server[1111]: 2026/09/23 05:14:05 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker1 # [ 130.575623] nix-grpc-daemon[1399]: time=2026-09-23T05:14:05.078Z level=INFO msg="Uploading 1 narinfos" lb # [ 130.599348] niks3-server[1111]: 2026/09/23 05:14:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign lb # [ 130.611634] niks3-server[1111]: 2026/09/23 05:14:05 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls worker1 # [ 130.603108] nix-grpc-daemon[1399]: time=2026-09-23T05:14:05.106Z level=INFO msg="Upload complete. (283ms)" lb # [ 130.623881] niks3-server[1111]: 2026/09/23 05:14:05 INFO Signed narinfos id=48 count=1 lb # [ 130.632629] niks3-server[1111]: 2026/09/23 05:14:05 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete lb # [ 130.637696] niks3-server[1111]: 2026/09/23 05:14:05 INFO Completed upload id=48 worker1 # [ 130.617322] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48992 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=1 lb # [ 130.641750] niks3-server[1111]: 2026/09/23 05:14:05 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo lb # [ 130.645654] niks3-server[1111]: 2026/09/23 05:14:05 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst worker1 # [ 130.638309] nix-daemon[850]: reaped child process 1481, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby >&2, in 7.05 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag on-standby outPath, in 0.19 seconds) worker2 # [ 130.939047] systemd[1]: Listening on Nix worker-protocol over gRPC. worker2 # [ 130.948514] systemd[1]: Starting scheduler-token.service... worker2 # [ 131.018612] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 131.022853] systemd[1]: Finished scheduler-token.service. worker2 # [ 131.036832] systemd[1]: scheduler-token.service: Consumed 43ms CPU time over 73ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 131.039894] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 131.155575] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:05Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 131.158067] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:05Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 131.161979] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:05Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 131.168061] nix-grpc-daemon[1367]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 131.169556] nix-grpc-daemon[1367]: E0000 00:00:1790140445.665868 1367 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 131.213610] nix-grpc-daemon[1367]: E0000 00:00:1790140445.672356 1367 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 131.217824] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:05Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 131.220097] nix-grpc-daemon[1367]: E0000 00:00:1790140445.680281 1408 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 131.222646] nix-grpc-daemon[1367]: E0000 00:00:1790140445.697815 1408 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 131.224967] nix-grpc-daemon[1367]: E0000 00:00:1790140445.697993 1408 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 131.227322] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 131.228178] nix-grpc-daemon[1367]: E0000 00:00:1790140445.716299 1408 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 131.236281] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:05Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 131.239463] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:05Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 131.244383] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:05Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:50830 worker1 # [ 131.250972] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:05Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: scheduler down: the other node takes the lock and keeps it, in 13.27 seconds) subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs client # [ 131.649817] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs. client # Running as unit: rs.service; invocation ID: 48d4e5b2bc20411782e0ed1aa9cf24a4 client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag rs, in 0.09 seconds) worker1 # [ 131.794972] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:06Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:48992 client # [ 131.862634] nix[1345]: don't know how to build these paths: client # [ 131.863756] nix[1345]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker1 # [ 131.812891] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:06Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=0 worker1 # [ 131.820242] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:06Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 131.857660] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48992 worker1 # [ 131.873922] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48992 client # [ 131.946719] nix[1345]: copying 1 paths to 'lb:50051'... worker1 # [ 131.890369] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48992 client # [ 131.952112] nix[1345]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'... worker1 # [ 131.905644] nix-daemon[850]: accepted connection from pid 1322, user nix-grpc-daemon (trusted) worker1 # [ 132.072727] nix-daemon[850]: accepted connection from pid 1515, user nix-grpc-daemon (trusted) worker1 # [ 132.092901] nix-daemon[850]: reaped child process 1518, status = succeeded lb # [ 132.118156] niks3-server[1111]: 2026/09/23 05:14:06 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 132.109579] nix-grpc-daemon[1399]: time=2026-09-23T05:14:06.612Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 132.111248] nix-grpc-daemon[1399]: time=2026-09-23T05:14:06.612Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)" worker1 # [ 132.138928] nix-grpc-daemon[1399]: time=2026-09-23T05:14:06.641Z level=INFO msg="Uploading 1 narinfos" lb # [ 132.149195] niks3-server[1111]: 2026/09/23 05:14:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign lb # [ 132.165857] niks3-server[1111]: 2026/09/23 05:14:06 INFO Signed narinfos id=49 count=1 lb # [ 132.173491] niks3-server[1111]: 2026/09/23 05:14:06 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls lb # [ 132.181303] niks3-server[1111]: 2026/09/23 05:14:06 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst lb # [ 132.192818] niks3-server[1111]: 2026/09/23 05:14:06 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete lb # [ 132.199024] niks3-server[1111]: 2026/09/23 05:14:06 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo worker1 # [ 132.180880] nix-grpc-daemon[1399]: time=2026-09-23T05:14:06.684Z level=INFO msg="Upload complete. (257ms)" lb # [ 132.203499] niks3-server[1111]: 2026/09/23 05:14:06 INFO Completed upload id=49 worker1 # [ 132.191800] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48992 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 132.208620] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48992 worker1 # [ 132.216833] nix-daemon[850]: accepted connection from pid 1322, user nix-grpc-daemon (trusted) worker1 # [ 132.227569] nix-daemon[850]: reaped child process 1513, status = succeeded client # [ 132.290792] nix[1345]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051... worker1 # [ 132.235495] nix-daemon[850]: accepted connection from pid 1322, user nix-grpc-daemon (trusted) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.03 seconds) worker2: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true worker2 # [ 134.441676] systemd-journald[264]: Received client request to rotate journal, rotating. worker2 # [ 134.461228] systemd-journald[264]: Vacuuming done, freed 0B of archived journals from /var/log/journal/99399b7496db48009c4418bf81fcde58. worker2: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.08 seconds) worker1: must succeed: systemctl restart --no-block nix-grpc-daemon.service worker1 # [ 134.493068] systemd[1]: Stopping Nix worker-protocol over gRPC... worker1: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.05 seconds) client: must succeed: sleep 5.0 worker1 # [ 134.767141] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:09Z level=info event=draining client: (finished: must succeed: sleep 5.0, in 5.05 seconds) worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=' worker1 # [ 139.916341] nix-daemon[850]: reaped child process 1522, status = succeeded worker1 # [ 140.008425] nix-daemon[850]: accepted connection from pid 1563, user nix-grpc-daemon (trusted) worker1 # [ 140.032124] nix-daemon[850]: reaped child process 1566, status = succeeded lb # [ 140.051023] niks3-server[1111]: 2026/09/23 05:14:14 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 140.044480] nix-grpc-daemon[1399]: time=2026-09-23T05:14:14.545Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 140.046246] nix-grpc-daemon[1399]: time=2026-09-23T05:14:14.545Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)" lb # [ 140.102999] niks3-server[1111]: 2026/09/23 05:14:14 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls lb # [ 140.113028] niks3-server[1111]: 2026/09/23 05:14:14 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst worker1 # [ 140.104765] nix-grpc-daemon[1399]: time=2026-09-23T05:14:14.607Z level=INFO msg="Uploading 1 narinfos" lb # [ 140.126551] niks3-server[1111]: 2026/09/23 05:14:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign lb # [ 140.134185] niks3-server[1111]: 2026/09/23 05:14:14 INFO Signed narinfos id=50 count=1 lb # [ 140.138908] niks3-server[1111]: 2026/09/23 05:14:14 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv lb # [ 140.149541] niks3-server[1111]: 2026/09/23 05:14:14 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete worker1 # [ 140.129514] nix-grpc-daemon[1399]: time=2026-09-23T05:14:14.632Z level=INFO msg="Upload complete. (279ms)" lb # [ 140.151949] niks3-server[1111]: 2026/09/23 05:14:14 INFO Completed upload id=50 lb # [ 140.154668] niks3-server[1111]: 2026/09/23 05:14:14 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo worker1 # [ 140.143708] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48992 duration_s=7 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=1 worker1 # [ 140.166435] nix-daemon[850]: reaped child process 1520, status = succeeded client # [ 140.237299] systemd[1]: rs.service: Deactivated successfully. client # [ 140.242512] systemd[1]: rs.service: Consumed 198ms CPU time over 8.588s wall clock time, 41.4M memory peak, 7.3K incoming IP traffic, 8.4K outgoing IP traffic. lb # [ 140.278051] envoy[908]: conn peer=[2001:db8:1::1]:51674 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9636 tx=4485 ms=6798 lb # [ 140.283135] envoy[908]: conn peer=[2001:db8:1::1]:58600 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2979 tx=1540 ms=8363 worker1 # [ 140.769471] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:15Z level=info event=signal_exit lb # [ 141.089443] niks3-server[1111]: 2026/09/23 05:14:15 INFO lead: released remote=192.168.1.3:57136 lb # [ 141.192321] niks3-server[1111]: 2026/09/23 05:14:15 INFO lead: acquired remote=192.168.1.4:48538 worker2 # [ 141.187823] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:15Z level=info event=scheduler_take_over worker2 # [ 142.078375] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:16Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 142.074777] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:16Z level=info event=scheduler_restarting worker1 # [ 142.080044] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:16Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 142.086445] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:16Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:50830 worker1 # [ 142.091593] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:16Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 142.096976] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:16Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:50830 worker1 # [ 142.102129] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:16Z level=info event=worker_gone addr=192.168.1.4:50051 worker1 # [ 142.129578] nix-daemon[850]: reaped child process 1375, status = succeeded worker1 # [ 142.133313] nix-grpc-daemon[1322]: E0000 00:00:1790140456.636349 1368 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true worker1 # [ 142.162607] nix-grpc-daemon[1322]: E0000 00:00:1790140456.665692 1368 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true worker2 # [ 142.281296] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 142.286737] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 142.293305] nix-grpc-daemon[1322]: ts=2026-09-23T05:14:16Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 142.488151] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 142.506542] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:17Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 142.511375] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:17Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:59168 worker2 # [ 142.517147] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:17Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.26 seconds) client: waiting for success: ! systemctl is-active rs client: (finished: waiting for success: ! systemctl is-active rs, in 0.07 seconds) client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; } client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.06 seconds) client: must fail: journalctl -u rs | grep -q 'reconnecting' client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) worker1 # [ 143.383726] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 143.390565] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 143.397148] systemd[1]: nix-grpc-daemon.service: Consumed 1.679s CPU time over 32.805s wall clock time, 38.9M memory peak, 48.1K incoming IP traffic, 90K outgoing IP traffic. worker1 # [ 143.415988] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 143.557311] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:18Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 143.559497] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:18Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 143.563513] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:18Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 143.569298] nix-grpc-daemon[1573]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 143.570819] nix-grpc-daemon[1573]: E0000 00:00:1790140458.071875 1573 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 143.612718] nix-grpc-daemon[1573]: E0000 00:00:1790140458.078203 1573 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 143.615663] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:18Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 143.617697] nix-grpc-daemon[1573]: E0000 00:00:1790140458.084565 1616 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 143.620069] nix-grpc-daemon[1573]: E0000 00:00:1790140458.100335 1616 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 143.624131] nix-grpc-daemon[1573]: E0000 00:00:1790140458.100512 1616 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 143.626565] nix-grpc-daemon[1573]: E0000 00:00:1790140458.115015 1616 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 143.630540] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 143.633306] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 143.645632] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:18Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:59168 worker2 # [ 143.645958] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:18Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 12.99 seconds) subtest: scheduler restart mid-build: build finishes, no double build client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid client # [ 144.644913] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid. client # Running as unit: mid.service; invocation ID: f8af968362394d8f9117c8fa3c1a64b3 client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag mid, in 0.09 seconds) worker2 # [ 144.815114] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59180 worker2 # [ 144.830443] nix-daemon[842]: accepted connection from pid 1367, user nix-grpc-daemon (trusted) client # [ 144.898292] nix[1381]: don't know how to build these paths: client # [ 144.900942] nix[1381]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker2 # [ 144.865686] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:19Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=0 worker2 # [ 144.873143] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:19Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 144.919470] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59180 worker2 # [ 144.937339] nix-grpc-daemon[1367]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 87 ms (attempt 1/5) worker2 # [ 145.033781] nix-grpc-daemon[1367]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 93 ms (attempt 2/5) worker2 # [ 145.130286] nix-grpc-daemon[1367]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 343 ms (attempt 3/5) worker2 # [ 145.475445] nix-grpc-daemon[1367]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 1 ms (attempt 4/5) worker2 # [ 145.490066] nix-grpc-daemon[1367]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 145.495059] nix-grpc-daemon[1367]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 145.509148] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:20Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59180 client # [ 145.572083] nix[1381]: copying 1 paths to 'lb:50051'... client # [ 145.577771] nix[1381]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'... worker2 # [ 145.529043] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:20Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59180 worker2 # [ 145.544725] nix-daemon[842]: accepted connection from pid 1367, user nix-grpc-daemon (trusted) worker2 # [ 145.581587] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:20Z level=info event=niks3_push_started pid=1496 worker2 # [ 145.730018] nix-grpc-daemon[1496]: time=2026-09-23T05:14:20.228Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 145.847553] nix-daemon[842]: accepted connection from pid 1509, user nix-grpc-daemon (trusted) lb # [ 145.887711] niks3-server[1111]: 2026/09/23 05:14:20 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 145.881984] nix-daemon[842]: reaped child process 1512, status = succeeded worker2 # [ 145.890564] nix-grpc-daemon[1496]: time=2026-09-23T05:14:20.388Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 145.895561] nix-grpc-daemon[1496]: time=2026-09-23T05:14:20.388Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)" lb # [ 145.928643] niks3-server[1111]: 2026/09/23 05:14:20 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst worker2 # [ 145.929785] nix-grpc-daemon[1496]: time=2026-09-23T05:14:20.428Z level=INFO msg="Uploading 1 narinfos" lb # [ 145.941412] niks3-server[1111]: 2026/09/23 05:14:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign lb # [ 145.951479] niks3-server[1111]: 2026/09/23 05:14:20 INFO Signed narinfos id=51 count=1 lb # [ 145.957289] niks3-server[1111]: 2026/09/23 05:14:20 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls worker2 # [ 145.957273] nix-grpc-daemon[1496]: time=2026-09-23T05:14:20.456Z level=INFO msg="Upload complete. (223ms)" lb # [ 145.966827] niks3-server[1111]: 2026/09/23 05:14:20 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete lb # [ 145.974476] niks3-server[1111]: 2026/09/23 05:14:20 INFO Completed upload id=51 worker2 # [ 145.963562] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59180 duration_s=0 paths=1 nar_bytes_in=424 lb # [ 145.978396] niks3-server[1111]: 2026/09/23 05:14:20 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo worker2 # [ 145.974110] nix-grpc-daemon[1367]: ts=2026-09-23T05:14:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59180 worker2 # [ 145.978558] nix-daemon[842]: accepted connection from pid 1367, user nix-grpc-daemon (trusted) worker2 # [ 145.982252] nix-daemon[842]: reaped child process 1494, status = succeeded worker2 # [ 145.990969] nix-daemon[842]: accepted connection from pid 1367, user nix-grpc-daemon (trusted) client # [ 146.041603] nix[1381]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051... lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service worker2 # [ 147.365755] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1367 (nix-grpc-daemon) on client request. worker2 # [ 147.367384] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request. lb # [ 147.401471] niks3-server[1111]: 2026/09/23 05:14:21 INFO lead: released remote=192.168.1.4:48538 worker2 # [ 147.400485] systemd[1]: Starting scheduler-token.service... worker2 # [ 147.401353] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL worker2 # [ 147.402843] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'. client # [ 147.450674] nix[1381]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again worker1 # [ 147.392380] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:21Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination" client # [ 147.457304] nix[1381]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting worker2 # [ 147.419388] systemd[1]: nix-grpc-daemon.service: Consumed 526ms CPU time over 16.368s wall clock time, 34.7M memory peak, 15.2K incoming IP traffic, 25.3K outgoing IP traffic. worker2 # [ 147.451565] nix-daemon[842]: reaped child process 1486, status = succeeded worker2 # [ 147.453527] nix-daemon[842]: reaped child process 1518, status = succeeded worker2 # [ 147.460432] nix-daemon[1520]: unexpected Nix daemon error: error: interrupted by the user worker2 # [ 147.471687] nix-daemon[842]: reaped child process 1520, status = failed with exit code 1 worker2 # [ 147.495616] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 147.500155] systemd[1]: Finished scheduler-token.service. worker2 # [ 147.500997] systemd[1]: scheduler-token.service: Consumed 26ms CPU time over 98ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 147.502904] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1. worker2 # [ 147.506500] systemd[1]: Starting scheduler-token.service... worker2 # [ 147.553580] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 147.555349] systemd[1]: Finished scheduler-token.service. worker2 # [ 147.558997] systemd[1]: scheduler-token.service: Consumed 25ms CPU time over 49ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 147.567756] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 147.592929] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:22Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 147.669843] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:22Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 147.672195] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:22Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 147.676293] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:22Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 147.682911] nix-grpc-daemon[1559]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 147.704189] nix-grpc-daemon[1559]: E0000 00:00:1790140462.180082 1559 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 147.714589] niks3-server[1111]: 2026/09/23 05:14:22 INFO lead: acquired remote=192.168.1.4:54964 worker2 # [ 147.706703] nix-grpc-daemon[1559]: E0000 00:00:1790140462.182209 1559 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.40 seconds) client: must succeed: sleep 5.0 worker2 # [ 147.745733] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:22Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 147.748658] nix-grpc-daemon[1559]: E0000 00:00:1790140462.202145 1602 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 147.751590] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:22Z level=info event=scheduler_take_over worker2 # [ 147.752987] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:22Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:59210 worker2 # [ 147.758137] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:22Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker2 # [ 147.762759] nix-grpc-daemon[1559]: E0000 00:00:1790140462.228895 1602 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 147.766327] nix-grpc-daemon[1559]: E0000 00:00:1790140462.229087 1602 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 147.769448] nix-grpc-daemon[1559]: E0000 00:00:1790140462.244189 1602 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 147.772456] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 147.773788] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:22Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 147.777726] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:22Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 147.779209] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:22Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:59210 worker2 # [ 147.781356] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:22Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker2 # [ 147.914862] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:22Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=0 worker2 # [ 147.921916] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:22Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 147.970298] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59968 worker1 # [ 147.986292] nix-daemon[850]: accepted connection from pid 1573, user nix-grpc-daemon (trusted) worker1 # [ 148.007958] nix-grpc-daemon[1573]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 13 ms (attempt 1/5) worker1 # [ 148.027311] nix-grpc-daemon[1573]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 74 ms (attempt 2/5) worker1 # [ 148.109254] nix-grpc-daemon[1573]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 243 ms (attempt 3/5) worker1 # [ 148.354935] nix-grpc-daemon[1573]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 252 ms (attempt 4/5) worker1 # [ 148.610948] nix-grpc-daemon[1573]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 148.617991] nix-grpc-daemon[1573]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 148.636526] nix-grpc-daemon[1573]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' from 'http://lb:5751'... worker1 # [ 148.667637] nix-daemon[850]: accepted connection from pid 1573, user nix-grpc-daemon (trusted) client # [ 148.747680] nix[1381]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051... worker1 # [ 148.692119] nix-daemon[850]: accepted connection from pid 1573, user nix-grpc-daemon (trusted) lb # [ 150.278835] envoy[908]: conn peer=[2001:db8:1::3]:53972 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=2717 tx=949 ms=31472 lb # [ 150.284409] envoy[908]: conn peer=[2001:db8:1::4]:51394 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=2722 tx=393 ms=16177 client: (finished: must succeed: sleep 5.0, in 5.06 seconds) client: waiting for success: ! systemctl is-active mid worker1 # [ 153.123600] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:27Z level=info event=niks3_push_started pid=1676 worker1 # [ 153.178080] nix-daemon[850]: reaped child process 1651, status = succeeded worker1 # [ 153.304618] nix-grpc-daemon[1676]: time=2026-09-23T05:14:27.807Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 153.417698] nix-daemon[850]: accepted connection from pid 1689, user nix-grpc-daemon (trusted) worker1 # [ 153.438626] nix-daemon[850]: reaped child process 1692, status = succeeded lb # [ 153.461337] niks3-server[1111]: 2026/09/23 05:14:27 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 153.451434] nix-grpc-daemon[1676]: time=2026-09-23T05:14:27.953Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 153.455166] nix-grpc-daemon[1676]: time=2026-09-23T05:14:27.953Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)" lb # [ 153.499902] niks3-server[1111]: 2026/09/23 05:14:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign lb # [ 153.529417] niks3-server[1111]: 2026/09/23 05:14:28 INFO Signed narinfos id=52 count=1 worker1 # [ 153.511659] nix-grpc-daemon[1676]: time=2026-09-23T05:14:28.014Z level=INFO msg="Uploading 1 narinfos" lb # [ 153.536330] niks3-server[1111]: 2026/09/23 05:14:28 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv lb # [ 153.547185] niks3-server[1111]: 2026/09/23 05:14:28 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls lb # [ 153.555392] niks3-server[1111]: 2026/09/23 05:14:28 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete lb # [ 153.562210] niks3-server[1111]: 2026/09/23 05:14:28 INFO Completed upload id=52 worker1 # [ 153.541841] nix-grpc-daemon[1676]: time=2026-09-23T05:14:28.045Z level=INFO msg="Upload complete. (232ms)" lb # [ 153.567660] niks3-server[1111]: 2026/09/23 05:14:28 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.narinfo lb # [ 153.571945] niks3-server[1111]: 2026/09/23 05:14:28 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst worker1 # [ 153.552340] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:28Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59968 duration_s=5 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=1 worker1 # [ 153.573343] nix-daemon[850]: reaped child process 1649, status = succeeded client # [ 153.656243] systemd[1]: mid.service: Deactivated successfully. client # [ 153.660554] systemd[1]: mid.service: Consumed 210ms CPU time over 9.016s wall clock time, 43.5M memory peak, 8.7K incoming IP traffic, 10.5K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active mid, in 1.12 seconds) client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; } client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.06 seconds) (finished: subtest: scheduler restart mid-build: build finishes, no double build, in 9.69 seconds) subtest: interrupting the client stops the build on the worker client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr client # [ 154.308552] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr. client # Running as unit: intr.service; invocation ID: 8df96aefc6e84bc2add6dca953d10a0d client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag intr, in 0.05 seconds) worker1 # [ 154.423226] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:28Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59968 client # [ 154.490486] nix[1417]: don't know how to build these paths: client # [ 154.491578] nix[1417]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker2 # [ 154.459241] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:28Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=0 worker2 # [ 154.459644] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:28Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 154.518222] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 154.533091] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 154.558216] nix-grpc-daemon[1559]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 12 ms (attempt 1/5) worker2 # [ 154.576880] nix-grpc-daemon[1559]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 66 ms (attempt 2/5) worker2 # [ 154.650675] nix-grpc-daemon[1559]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 111 ms (attempt 3/5) worker2 # [ 154.763805] nix-grpc-daemon[1559]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 38 ms (attempt 4/5) worker2 # [ 154.804371] nix-grpc-daemon[1559]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 154.810480] nix-grpc-daemon[1559]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 154.828540] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59586 client # [ 154.893576] nix[1417]: copying 1 paths to 'lb:50051'... worker2 # [ 154.843924] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:29Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 client # [ 154.898178] nix[1417]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'... worker2 # [ 154.858779] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 154.896269] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:29Z level=info event=niks3_push_started pid=1636 worker2 # [ 155.035151] nix-grpc-daemon[1636]: time=2026-09-23T05:14:29.534Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker2.key ca="" worker2 # [ 155.141635] nix-daemon[842]: accepted connection from pid 1648, user nix-grpc-daemon (trusted) worker2 # [ 155.162845] nix-daemon[842]: reaped child process 1651, status = succeeded lb # [ 155.174994] niks3-server[1111]: 2026/09/23 05:14:29 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 155.179141] nix-grpc-daemon[1636]: time=2026-09-23T05:14:29.677Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 155.182612] nix-grpc-daemon[1636]: time=2026-09-23T05:14:29.677Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)" lb # [ 155.218111] niks3-server[1111]: 2026/09/23 05:14:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign worker2 # [ 155.220982] nix-grpc-daemon[1636]: time=2026-09-23T05:14:29.719Z level=INFO msg="Uploading 1 narinfos" lb # [ 155.234759] niks3-server[1111]: 2026/09/23 05:14:29 INFO Signed narinfos id=53 count=1 lb # [ 155.241735] niks3-server[1111]: 2026/09/23 05:14:29 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls lb # [ 155.252328] niks3-server[1111]: 2026/09/23 05:14:29 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst lb # [ 155.255831] niks3-server[1111]: 2026/09/23 05:14:29 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete worker2 # [ 155.252321] nix-grpc-daemon[1636]: time=2026-09-23T05:14:29.751Z level=INFO msg="Upload complete. (212ms)" lb # [ 155.263961] niks3-server[1111]: 2026/09/23 05:14:29 INFO Completed upload id=53 lb # [ 155.266221] niks3-server[1111]: 2026/09/23 05:14:29 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo worker2 # [ 155.260111] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:29Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=0 paths=1 nar_bytes_in=424 worker2 # [ 155.274919] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 155.281379] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 155.286719] nix-daemon[842]: reaped child process 1634, status = succeeded client # [ 155.346204] nix[1417]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051... worker2 # [ 155.299512] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) client: must succeed: systemctl kill -s INT intr worker2 # [ 156.750568] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:31Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv client # [ 156.797348] systemd[1]: intr.service: Sent signal SIGINT to main process 1417 (nix) on client request. client # [ 156.807087] nix[1417]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client # [ 156.818556] nix[1417]: error: interrupted by the user client: (finished: must succeed: systemctl kill -s INT intr, in 0.08 seconds) client # [ 156.845234] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE client # [ 156.850811] systemd[1]: intr.service: Failed with result 'exit-code'. client # [ 156.853646] systemd[1]: intr.service: Consumed 169ms CPU time over 2.540s wall clock time, 41.4M memory peak, 6.4K incoming IP traffic, 8.3K outgoing IP traffic. worker2 # [ 156.921554] nix-daemon[1655]: unexpected Nix daemon error: error: interrupted by the user worker2 # [ 156.932640] nix-daemon[842]: reaped child process 1653, status = succeeded worker2 # [ 156.934345] nix-daemon[842]: reaped child process 1655, status = failed with exit code 1 (finished: subtest: interrupting the client stops the build on the worker, in 3.82 seconds) subtest: a deploy mid-build drains: the build finishes, the new generation takes over client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw client # [ 158.157931] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw. client # Running as unit: sw.service; invocation ID: c59c77d087d342488b0b4e47f82ccf09 client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix --argstr tag sw, in 0.09 seconds) worker1 # [ 158.304579] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:32Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59968 client # [ 158.370332] nix[1438]: don't know how to build these paths: client # [ 158.371381] nix[1438]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker2 # [ 158.335086] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:32Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=0 worker2 # [ 158.341108] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:32Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 158.372929] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59968 worker1 # [ 158.384773] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59968 client # [ 158.455354] nix[1438]: copying 1 paths to 'lb:50051'... worker1 # [ 158.396882] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59968 client # [ 158.458030] nix[1438]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'... worker1 # [ 158.412367] nix-daemon[850]: accepted connection from pid 1573, user nix-grpc-daemon (trusted) worker1 # [ 158.570894] nix-daemon[850]: accepted connection from pid 1720, user nix-grpc-daemon (trusted) worker1 # [ 158.590649] nix-daemon[850]: reaped child process 1723, status = succeeded lb # [ 158.615428] niks3-server[1111]: 2026/09/23 05:14:33 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 158.608570] nix-grpc-daemon[1676]: time=2026-09-23T05:14:33.110Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 158.611367] nix-grpc-daemon[1676]: time=2026-09-23T05:14:33.110Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)" lb # [ 158.647578] niks3-server[1111]: 2026/09/23 05:14:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign worker1 # [ 158.641390] nix-grpc-daemon[1676]: time=2026-09-23T05:14:33.143Z level=INFO msg="Uploading 1 narinfos" lb # [ 158.663011] niks3-server[1111]: 2026/09/23 05:14:33 INFO Signed narinfos id=54 count=1 lb # [ 158.671600] niks3-server[1111]: 2026/09/23 05:14:33 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst lb # [ 158.684388] niks3-server[1111]: 2026/09/23 05:14:33 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls lb # [ 158.697615] niks3-server[1111]: 2026/09/23 05:14:33 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete lb # [ 158.703553] niks3-server[1111]: 2026/09/23 05:14:33 INFO Completed upload id=54 worker1 # [ 158.678654] nix-grpc-daemon[1676]: time=2026-09-23T05:14:33.181Z level=INFO msg="Upload complete. (249ms)" lb # [ 158.707465] niks3-server[1111]: 2026/09/23 05:14:33 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo worker1 # [ 158.689468] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:33Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59968 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 158.704335] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:33Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59968 worker1 # [ 158.711449] nix-daemon[850]: accepted connection from pid 1573, user nix-grpc-daemon (trusted) worker1 # [ 158.717034] nix-daemon[850]: reaped child process 1718, status = succeeded client # [ 158.786721] nix[1438]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051... worker1 # [ 158.731890] nix-daemon[850]: accepted connection from pid 1573, user nix-grpc-daemon (trusted) lb # [ 160.278711] envoy[908]: conn peer=[2001:db8:1::1]:33554 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3940 tx=1966 ms=8780 lb # [ 160.284557] envoy[908]: conn peer=[2001:db8:1::1]:39032 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3013 tx=827 ms=2349 worker1: must succeed: systemctl reload nix-grpc-daemon.service worker1 # [ 160.698016] systemd[1]: Reloading Nix worker-protocol over gRPC... worker1 # [ 160.744119] systemd[1]: Reloaded Nix worker-protocol over gRPC. worker1: (finished: must succeed: systemctl reload nix-grpc-daemon.service, in 0.08 seconds) worker1: must succeed: pgrep -f 'read -t [9]0 x' worker1: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.05 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2 client # don't know how to build these paths: client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker2 # [ 161.024472] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 161.030726] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 161.056146] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:35Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=0 worker2 # [ 161.062042] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:35Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 161.106925] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 161.127190] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:35Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59586 client # copying 1 paths to 'lb:50051'... worker2 # [ 161.142841] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:35Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'... worker2 # [ 161.166518] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 161.327926] nix-daemon[842]: accepted connection from pid 1710, user nix-grpc-daemon (trusted) worker2 # [ 161.347591] nix-daemon[842]: reaped child process 1713, status = succeeded lb # [ 161.365816] niks3-server[1111]: 2026/09/23 05:14:35 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 161.364349] nix-grpc-daemon[1636]: time=2026-09-23T05:14:35.862Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 161.366136] nix-grpc-daemon[1636]: time=2026-09-23T05:14:35.862Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)" lb # [ 161.399167] niks3-server[1111]: 2026/09/23 05:14:35 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst lb # [ 161.417243] niks3-server[1111]: 2026/09/23 05:14:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign worker2 # [ 161.413267] nix-grpc-daemon[1636]: time=2026-09-23T05:14:35.911Z level=INFO msg="Uploading 1 narinfos" lb # [ 161.433505] niks3-server[1111]: 2026/09/23 05:14:35 INFO Signed narinfos id=55 count=1 lb # [ 161.447989] niks3-server[1111]: 2026/09/23 05:14:35 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls lb # [ 161.459843] niks3-server[1111]: 2026/09/23 05:14:35 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete worker2 # [ 161.457194] nix-grpc-daemon[1636]: time=2026-09-23T05:14:35.956Z level=INFO msg="Upload complete. (273ms)" lb # [ 161.469365] niks3-server[1111]: 2026/09/23 05:14:35 INFO Completed upload id=55 lb # [ 161.472233] niks3-server[1111]: 2026/09/23 05:14:35 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo worker2 # [ 161.468314] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=0 paths=1 nar_bytes_in=528 worker2 # [ 161.487042] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 161.499725] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 161.503685] nix-daemon[842]: reaped child process 1708, status = succeeded client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051... client # farm-a> worker2: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker2 # [ 161.522328] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker1 # [ 161.585038] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:36Z level=info event=draining client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-during-drain worker2 # [ 163.193939] nix-daemon[842]: reaped child process 1717, status = succeeded worker2 # [ 163.313762] nix-daemon[842]: accepted connection from pid 1733, user nix-grpc-daemon (trusted) worker2 # [ 163.333338] nix-daemon[842]: reaped child process 1736, status = succeeded lb # [ 163.348784] niks3-server[1111]: 2026/09/23 05:14:37 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 163.352478] nix-grpc-daemon[1636]: time=2026-09-23T05:14:37.850Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 163.354421] nix-grpc-daemon[1636]: time=2026-09-23T05:14:37.850Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)" lb # [ 163.394545] niks3-server[1111]: 2026/09/23 05:14:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign worker2 # [ 163.393083] nix-grpc-daemon[1636]: time=2026-09-23T05:14:37.892Z level=INFO msg="Uploading 1 narinfos" lb # [ 163.409768] niks3-server[1111]: 2026/09/23 05:14:37 INFO Signed narinfos id=56 count=1 lb # [ 163.420187] niks3-server[1111]: 2026/09/23 05:14:37 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst lb # [ 163.431057] niks3-server[1111]: 2026/09/23 05:14:37 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls lb # [ 163.438515] niks3-server[1111]: 2026/09/23 05:14:37 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker2 # [ 163.434205] nix-grpc-daemon[1636]: time=2026-09-23T05:14:37.933Z level=INFO msg="Upload complete. (298ms)" lb # [ 163.444811] niks3-server[1111]: 2026/09/23 05:14:37 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete lb # [ 163.453149] niks3-server[1111]: 2026/09/23 05:14:37 INFO Completed upload id=56 lb # [ 163.456000] niks3-server[1111]: 2026/09/23 05:14:37 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo worker2 # [ 163.449424] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:37Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=1 worker2 # [ 163.471476] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:37Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=0 worker2 # [ 163.476445] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:37Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 163.481399] nix-daemon[842]: reaped child process 1715, status = succeeded worker2 # [ 163.498355] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 163.513384] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:38Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 163.527679] nix-grpc-daemon[1636]: time=2026-09-23T05:14:38.027Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker2 # [ 163.538654] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:38Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'... worker2 # [ 163.561034] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 163.724145] nix-daemon[842]: accepted connection from pid 1742, user nix-grpc-daemon (trusted) worker2 # [ 163.745326] nix-daemon[842]: reaped child process 1745, status = succeeded lb # [ 163.753854] niks3-server[1111]: 2026/09/23 05:14:38 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 163.752266] nix-grpc-daemon[1636]: time=2026-09-23T05:14:38.251Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 163.754832] nix-grpc-daemon[1636]: time=2026-09-23T05:14:38.251Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)" lb # [ 163.787835] niks3-server[1111]: 2026/09/23 05:14:38 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst worker2 # [ 163.789019] nix-grpc-daemon[1636]: time=2026-09-23T05:14:38.288Z level=INFO msg="Uploading 1 narinfos" lb # [ 163.804606] niks3-server[1111]: 2026/09/23 05:14:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign lb # [ 163.815641] niks3-server[1111]: 2026/09/23 05:14:38 INFO Signed narinfos id=57 count=1 lb # [ 163.821922] niks3-server[1111]: 2026/09/23 05:14:38 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls lb # [ 163.828876] niks3-server[1111]: 2026/09/23 05:14:38 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete lb # [ 163.837898] niks3-server[1111]: 2026/09/23 05:14:38 INFO Completed upload id=57 worker2 # [ 163.826306] nix-grpc-daemon[1636]: time=2026-09-23T05:14:38.325Z level=INFO msg="Upload complete. (247ms)" lb # [ 163.840497] niks3-server[1111]: 2026/09/23 05:14:38 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo worker2 # [ 163.836322] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:38Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=0 paths=1 nar_bytes_in=640 worker2 # [ 163.856592] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:38Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 163.863676] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 163.870107] nix-daemon[842]: reaped child process 1740, status = succeeded client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051... worker2 # [ 163.883492] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) client # farm-b> worker2: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-during-drain worker2 # [ 165.168214] nix-daemon[842]: reaped child process 1749, status = succeeded worker2 # [ 165.267185] nix-daemon[842]: accepted connection from pid 1765, user nix-grpc-daemon (trusted) worker2 # [ 165.285167] nix-daemon[842]: reaped child process 1768, status = succeeded lb # [ 165.302109] niks3-server[1111]: 2026/09/23 05:14:39 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 165.302465] nix-grpc-daemon[1636]: time=2026-09-23T05:14:39.800Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 165.304093] nix-grpc-daemon[1636]: time=2026-09-23T05:14:39.800Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)" lb # [ 165.345931] niks3-server[1111]: 2026/09/23 05:14:39 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls worker2 # [ 165.347502] nix-grpc-daemon[1636]: time=2026-09-23T05:14:39.846Z level=INFO msg="Uploading 1 narinfos" lb # [ 165.359516] niks3-server[1111]: 2026/09/23 05:14:39 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign lb # [ 165.369469] niks3-server[1111]: 2026/09/23 05:14:39 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst lb # [ 165.381318] niks3-server[1111]: 2026/09/23 05:14:39 INFO Signed narinfos id=58 count=1 lb # [ 165.383205] niks3-server[1111]: 2026/09/23 05:14:39 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv lb # [ 165.388072] niks3-server[1111]: 2026/09/23 05:14:39 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete lb # [ 165.392374] niks3-server[1111]: 2026/09/23 05:14:39 INFO Completed upload id=58 worker2 # [ 165.382513] nix-grpc-daemon[1636]: time=2026-09-23T05:14:39.881Z level=INFO msg="Upload complete. (285ms)" lb # [ 165.394697] niks3-server[1111]: 2026/09/23 05:14:39 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo worker2 # [ 165.404833] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=1 worker2 # [ 165.416296] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:39Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=0 worker2 # [ 165.424304] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:39Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 165.429458] nix-daemon[842]: reaped child process 1747, status = succeeded worker2 # [ 165.446380] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:39Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 165.467257] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:39Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 165.481268] nix-grpc-daemon[1636]: time=2026-09-23T05:14:39.980Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... worker2 # [ 165.492112] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:39Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'... worker2 # [ 165.514722] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 165.674214] nix-daemon[842]: accepted connection from pid 1774, user nix-grpc-daemon (trusted) worker2 # [ 165.692820] nix-daemon[842]: reaped child process 1777, status = succeeded lb # [ 165.708079] niks3-server[1111]: 2026/09/23 05:14:40 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 165.706396] nix-grpc-daemon[1636]: time=2026-09-23T05:14:40.205Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 165.708228] nix-grpc-daemon[1636]: time=2026-09-23T05:14:40.205Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)" lb # [ 165.736824] niks3-server[1111]: 2026/09/23 05:14:40 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls lb # [ 165.747574] niks3-server[1111]: 2026/09/23 05:14:40 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign worker2 # [ 165.744741] nix-grpc-daemon[1636]: time=2026-09-23T05:14:40.243Z level=INFO msg="Uploading 1 narinfos" lb # [ 165.762526] niks3-server[1111]: 2026/09/23 05:14:40 INFO Signed narinfos id=59 count=1 lb # [ 165.770386] niks3-server[1111]: 2026/09/23 05:14:40 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst lb # [ 165.782381] niks3-server[1111]: 2026/09/23 05:14:40 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete lb # [ 165.788965] niks3-server[1111]: 2026/09/23 05:14:40 INFO Completed upload id=59 lb # [ 165.790989] niks3-server[1111]: 2026/09/23 05:14:40 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo worker2 # [ 165.777064] nix-grpc-daemon[1636]: time=2026-09-23T05:14:40.276Z level=INFO msg="Upload complete. (244ms)" worker2 # [ 165.788431] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:40Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=0 paths=1 nar_bytes_in=768 worker2 # [ 165.804955] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 165.812740] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 165.824408] nix-daemon[842]: reaped child process 1772, status = succeeded client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051... client # farm-top> worker2: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker2 # [ 165.833836] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-during-drain worker2 # [ 166.724382] nix-daemon[842]: reaped child process 1781, status = succeeded worker2 # [ 166.810658] nix-daemon[842]: accepted connection from pid 1797, user nix-grpc-daemon (trusted) worker2 # [ 166.832136] nix-daemon[842]: reaped child process 1800, status = succeeded lb # [ 166.846841] niks3-server[1111]: 2026/09/23 05:14:41 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 166.848751] nix-grpc-daemon[1636]: time=2026-09-23T05:14:41.346Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 166.850655] nix-grpc-daemon[1636]: time=2026-09-23T05:14:41.346Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)" lb # [ 166.886510] niks3-server[1111]: 2026/09/23 05:14:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign worker2 # [ 166.886246] nix-grpc-daemon[1636]: time=2026-09-23T05:14:41.385Z level=INFO msg="Uploading 1 narinfos" lb # [ 166.901414] niks3-server[1111]: 2026/09/23 05:14:41 INFO Signed narinfos id=60 count=1 lb # [ 166.909335] niks3-server[1111]: 2026/09/23 05:14:41 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst lb # [ 166.920991] niks3-server[1111]: 2026/09/23 05:14:41 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker2 # [ 166.917252] nix-grpc-daemon[1636]: time=2026-09-23T05:14:41.416Z level=INFO msg="Upload complete. (273ms)" lb # [ 166.929320] niks3-server[1111]: 2026/09/23 05:14:41 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls lb # [ 166.932695] niks3-server[1111]: 2026/09/23 05:14:41 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete lb # [ 166.937365] niks3-server[1111]: 2026/09/23 05:14:41 INFO Completed upload id=60 lb # [ 166.940276] niks3-server[1111]: 2026/09/23 05:14:41 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo worker2 # [ 166.929106] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:41Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=1 worker2 # [ 166.951125] nix-daemon[842]: reaped child process 1779, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain >&2, in 6.18 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag during-drain outPath, in 0.18 seconds) worker1: must succeed: pkill -f 'read -t [9]0 x' worker1: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.05 seconds) client: waiting for success: ! systemctl is-active sw worker1 # [ 167.404892] nix-daemon[850]: reaped child process 1727, status = succeeded worker1 # [ 167.515028] nix-daemon[850]: accepted connection from pid 1768, user nix-grpc-daemon (trusted) worker1 # [ 167.535122] nix-daemon[850]: reaped child process 1771, status = succeeded lb # [ 167.556710] niks3-server[1111]: 2026/09/23 05:14:42 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 167.547730] nix-grpc-daemon[1676]: time=2026-09-23T05:14:42.050Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 167.549501] nix-grpc-daemon[1676]: time=2026-09-23T05:14:42.050Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)" lb # [ 167.597160] niks3-server[1111]: 2026/09/23 05:14:42 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker1 # [ 167.589010] nix-grpc-daemon[1676]: time=2026-09-23T05:14:42.091Z level=INFO msg="Uploading 1 narinfos" lb # [ 167.611209] niks3-server[1111]: 2026/09/23 05:14:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign lb # [ 167.620272] niks3-server[1111]: 2026/09/23 05:14:42 INFO Signed narinfos id=61 count=1 lb # [ 167.628568] niks3-server[1111]: 2026/09/23 05:14:42 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst lb # [ 167.636469] niks3-server[1111]: 2026/09/23 05:14:42 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls worker1 # [ 167.614271] nix-grpc-daemon[1676]: time=2026-09-23T05:14:42.117Z level=INFO msg="Upload complete. (264ms)" lb # [ 167.639371] niks3-server[1111]: 2026/09/23 05:14:42 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete lb # [ 167.645861] niks3-server[1111]: 2026/09/23 05:14:42 INFO Completed upload id=61 lb # [ 167.648560] niks3-server[1111]: 2026/09/23 05:14:42 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo worker1 # [ 167.626536] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:42Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59968 duration_s=8 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=1 worker1 # [ 167.645612] nix-daemon[850]: reaped child process 1725, status = succeeded client # [ 167.722815] systemd[1]: sw.service: Deactivated successfully. client # [ 167.728211] systemd[1]: sw.service: Consumed 195ms CPU time over 9.566s wall clock time, 39.2M memory peak, 7.3K incoming IP traffic, 8.4K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active sw, in 1.10 seconds) client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; } client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.05 seconds) worker1: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service worker1: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 0.06 seconds) lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker1 # [ 168.589731] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:43Z level=info event=signal_exit lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb # [ 170.278345] envoy[908]: conn peer=[2001:db8:1::1]:42220 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9621 tx=4401 ms=5934 lb # [ 170.283365] envoy[908]: conn peer=[2001:db8:1::1]:39042 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2978 tx=1513 ms=9340 lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 170.602016] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:45Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:59210 worker1 # [ 170.595740] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:45Z level=info event=scheduler_restarting worker2 # [ 170.602613] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:45Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 170.619746] nix-grpc-daemon[1573]: ts=2026-09-23T05:14:45Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1 # [ 170.642953] nix-daemon[850]: reaped child process 1642, status = succeeded worker1 # [ 171.463567] nix-grpc-daemon[1573]: E0000 00:00:1790140485.966611 1619 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 16, started true, shutdown true worker1 # [ 171.470542] nix-grpc-daemon[1573]: E0000 00:00:1790140485.966829 1619 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker1 # [ 171.611234] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 171.631452] systemd[1]: nix-grpc-daemon.service: Consumed 864ms CPU time over 28.204s wall clock time, 35.8M memory peak, 21.9K incoming IP traffic, 28.5K outgoing IP traffic. lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1 # [ 172.608809] systemd[1]: Starting Nix worker-protocol over gRPC... lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) worker1 # [ 172.776809] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:47Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 172.778815] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:47Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 172.783004] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:47Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 172.788057] nix-grpc-daemon[1783]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 172.789524] nix-grpc-daemon[1783]: E0000 00:00:1790140487.291083 1783 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 172.832075] nix-grpc-daemon[1783]: E0000 00:00:1790140487.297358 1783 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 172.834387] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:47Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 172.841942] nix-grpc-daemon[1783]: E0000 00:00:1790140487.309132 1823 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 172.844799] nix-grpc-daemon[1783]: E0000 00:00:1790140487.326079 1823 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 172.847241] nix-grpc-daemon[1783]: E0000 00:00:1790140487.326214 1823 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 172.852886] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 172.855485] nix-grpc-daemon[1783]: E0000 00:00:1790140487.344650 1823 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 172.858019] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:47Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 172.866312] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:47Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:59210 worker2 # [ 172.872960] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:47Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 5.42 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 15.92 seconds) subtest: queue gauges show a burst while every build blocks lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) client: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]' client # [ 174.249351] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key" --eval-store auto --impure --expr "map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ \"q0\" \"q1\" \"q2\" \"q3\" ]". client # Running as unit: burst.service; invocation ID: 256dca65abf1442f9f95760379855550 client: (finished: must succeed: systemd-run --unit burst nix build --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --impure --expr 'map (tag: import /nix/store/bzzab28rfcdvvdzhgyw7sy9agp2z9pdd-slow.nix { inherit tag; }) [ "q0" "q1" "q2" "q3" ]', in 0.09 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics client # [ 174.457519] nix[1494]: don't know how to build these paths: client # [ 174.458505] nix[1494]: /nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv client # [ 174.459551] nix[1494]: /nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv worker2 # [ 174.407582] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:48Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59586 client # [ 174.460657] nix[1494]: /nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv client # [ 174.462150] nix[1494]: /nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) worker2 # [ 174.422807] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:48Z level=debug event=want client=5 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv cached=0 worker2 # [ 174.428923] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:48Z level=debug event=assigned drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 174.434006] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:48Z level=debug event=want client=5 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv cached=0 worker2 # [ 174.438382] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:48Z level=debug event=assigned drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv worker=192.168.1.3:50051 clients=1 worker2 # [ 174.442925] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:48Z level=debug event=want client=5 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv cached=0 worker2 # [ 174.446888] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:48Z level=debug event=want client=5 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv cached=0 worker2 # [ 174.470643] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker1 # [ 174.467909] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker2 # [ 174.486592] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59586 worker1 # [ 174.482580] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) client # [ 174.548979] nix[1494]: copying 1 paths to 'lb:50051'... worker2 # [ 174.500963] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 client # [ 174.554365] nix[1494]: copying path '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' to 'lb:50051'... worker2 # [ 174.516854] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker1 # [ 174.507015] nix-grpc-daemon[1783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 78 ms (attempt 1/5) worker1 # [ 174.594586] nix-grpc-daemon[1783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 62 ms (attempt 2/5) worker2 # [ 174.664861] nix-daemon[842]: accepted connection from pid 1815, user nix-grpc-daemon (trusted) worker1 # [ 174.658551] nix-grpc-daemon[1783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 334 ms (attempt 3/5) worker2 # [ 174.685185] nix-daemon[842]: reaped child process 1818, status = succeeded lb # [ 174.705768] niks3-server[1111]: 2026/09/23 05:14:49 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 174.704105] nix-grpc-daemon[1636]: time=2026-09-23T05:14:49.202Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 174.705797] nix-grpc-daemon[1636]: time=2026-09-23T05:14:49.202Z level=INFO msg="Uploading d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv (416B)" lb # [ 174.748084] niks3-server[1111]: 2026/09/23 05:14:49 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.ls worker2 # [ 174.746290] nix-grpc-daemon[1636]: time=2026-09-23T05:14:49.245Z level=INFO msg="Uploading 1 narinfos" lb # [ 174.759741] niks3-server[1111]: 2026/09/23 05:14:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign lb # [ 174.770787] niks3-server[1111]: 2026/09/23 05:14:49 INFO Signed narinfos id=62 count=1 lb # [ 174.776281] niks3-server[1111]: 2026/09/23 05:14:49 INFO Registered completed upload object_key=nar/06ismykzsix0w29371dm1hbzzf3ljf9jn7dyh8skp5wqfkkw0vlk.nar.zst lb # [ 174.792371] niks3-server[1111]: 2026/09/23 05:14:49 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete lb # [ 174.799833] niks3-server[1111]: 2026/09/23 05:14:49 INFO Completed upload id=62 worker2 # [ 174.789115] nix-grpc-daemon[1636]: time=2026-09-23T05:14:49.288Z level=INFO msg="Upload complete. (257ms)" lb # [ 174.802621] niks3-server[1111]: 2026/09/23 05:14:49 INFO Registered completed upload object_key=d4rrlfallgfbcpmjnsn604l2mgff4hgx.narinfo worker2 # [ 174.801006] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:49Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=0 paths=1 nar_bytes_in=416 worker2 # [ 174.816531] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:49Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 174.826993] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 174.835047] nix-daemon[842]: reaped child process 1813, status = succeeded worker2 # [ 174.847831] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) client # [ 174.896468] nix[1494]: building '/nix/store/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv' on lb:50051... worker1 # [ 174.994982] nix-grpc-daemon[1783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 386 ms (attempt 4/5) worker1 # [ 175.383687] nix-grpc-daemon[1783]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 175.391629] nix-grpc-daemon[1783]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 175.406414] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:49Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54454 client # [ 175.481340] nix[1494]: copying 1 paths to 'lb:50051'... worker1 # [ 175.424125] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:49Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 client # [ 175.485469] nix[1494]: copying path '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' to 'lb:50051'... worker1 # [ 175.439131] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 175.477891] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:49Z level=info event=niks3_push_started pid=1844 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) worker1 # [ 175.623848] nix-grpc-daemon[1844]: time=2026-09-23T05:14:50.124Z level=INFO msg="Configuring client TLS" cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.pem key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/worker1.key ca="" worker1 # [ 175.746556] nix-daemon[850]: accepted connection from pid 1861, user nix-grpc-daemon (trusted) worker1 # [ 175.764241] nix-daemon[850]: reaped child process 1864, status = succeeded lb # [ 175.791834] niks3-server[1111]: 2026/09/23 05:14:50 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 175.786515] nix-grpc-daemon[1844]: time=2026-09-23T05:14:50.288Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 175.790853] nix-grpc-daemon[1844]: time=2026-09-23T05:14:50.288Z level=INFO msg="Uploading dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv (416B)" lb # [ 175.839023] niks3-server[1111]: 2026/09/23 05:14:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign worker1 # [ 175.830563] nix-grpc-daemon[1844]: time=2026-09-23T05:14:50.332Z level=INFO msg="Uploading 1 narinfos" lb # [ 175.852646] niks3-server[1111]: 2026/09/23 05:14:50 INFO Registered completed upload object_key=nar/06mdwsnf16qmfi86y8r33asi4vcrp5z7kdxqki87psgfpgvvpdki.nar.zst lb # [ 175.861785] niks3-server[1111]: 2026/09/23 05:14:50 INFO Signed narinfos id=63 count=1 lb # [ 175.868639] niks3-server[1111]: 2026/09/23 05:14:50 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.ls lb # [ 175.871494] niks3-server[1111]: 2026/09/23 05:14:50 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete worker1 # [ 175.856934] nix-grpc-daemon[1844]: time=2026-09-23T05:14:50.360Z level=INFO msg="Upload complete. (234ms)" lb # [ 175.878859] niks3-server[1111]: 2026/09/23 05:14:50 INFO Completed upload id=63 lb # [ 175.881336] niks3-server[1111]: 2026/09/23 05:14:50 INFO Registered completed upload object_key=dspy0axfmddglppg9zj9blmxlzhfs40x.narinfo worker1 # [ 175.867940] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=0 paths=1 nar_bytes_in=416 worker1 # [ 175.881925] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 175.887935] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 175.893074] nix-daemon[850]: reaped child process 1842, status = succeeded client # [ 175.961305] nix[1494]: building '/nix/store/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv' on lb:50051... worker1 # [ 175.905962] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker2 # [ 175.916659] nix-daemon[842]: reaped child process 1822, status = succeeded worker2 # [ 176.018264] nix-daemon[842]: accepted connection from pid 1846, user nix-grpc-daemon (trusted) worker2 # [ 176.038248] nix-daemon[842]: reaped child process 1849, status = succeeded lb # [ 176.052934] niks3-server[1111]: 2026/09/23 05:14:50 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 176.055373] nix-grpc-daemon[1636]: time=2026-09-23T05:14:50.552Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 176.057114] nix-grpc-daemon[1636]: time=2026-09-23T05:14:50.552Z level=INFO msg="Uploading k6zivhvzgmlyxfmy64fzd759bxlhf9b5-slow-q0 (120B)" lb # [ 176.098084] niks3-server[1111]: 2026/09/23 05:14:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign worker2 # [ 176.093171] nix-grpc-daemon[1636]: time=2026-09-23T05:14:50.592Z level=INFO msg="Uploading 1 narinfos" lb # [ 176.110205] niks3-server[1111]: 2026/09/23 05:14:50 INFO Signed narinfos id=64 count=1 lb # [ 176.115631] niks3-server[1111]: 2026/09/23 05:14:50 INFO Registered completed upload object_key=nar/0xr809jj32p5n6mnc6i7lqsz48wrm5f9jp14m77k724041711iq7.nar.zst lb # [ 176.122876] niks3-server[1111]: 2026/09/23 05:14:50 INFO Registered completed upload object_key=log/d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv worker2 # [ 176.120811] nix-grpc-daemon[1636]: time=2026-09-23T05:14:50.620Z level=INFO msg="Upload complete. (258ms)" lb # [ 176.131800] niks3-server[1111]: 2026/09/23 05:14:50 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.ls lb # [ 176.136209] niks3-server[1111]: 2026/09/23 05:14:50 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete lb # [ 176.140378] niks3-server[1111]: 2026/09/23 05:14:50 INFO Completed upload id=64 worker2 # [ 176.131344] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=1 drv=d4rrlfallgfbcpmjnsn604l2mgff4hgx-slow-q0.drv assign_id=7 outputs=1 lb # [ 176.145457] niks3-server[1111]: 2026/09/23 05:14:50 INFO Registered completed upload object_key=k6zivhvzgmlyxfmy64fzd759bxlhf9b5.narinfo worker2 # [ 176.148234] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:50Z level=debug event=assigned drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 176.156837] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 176.161382] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 176.165844] nix-daemon[842]: reaped child process 1820, status = succeeded client # [ 176.219417] nix[1494]: copying 1 paths to 'lb:50051'... worker2 # [ 176.169134] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 client # [ 176.221983] nix[1494]: copying path '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' to 'lb:50051'... worker2 # [ 176.178614] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 176.308226] nix-daemon[842]: accepted connection from pid 1853, user nix-grpc-daemon (trusted) worker2 # [ 176.329003] nix-daemon[842]: reaped child process 1856, status = succeeded lb # [ 176.337890] niks3-server[1111]: 2026/09/23 05:14:50 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 176.336432] nix-grpc-daemon[1636]: time=2026-09-23T05:14:50.835Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 176.338017] nix-grpc-daemon[1636]: time=2026-09-23T05:14:50.835Z level=INFO msg="Uploading jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv (416B)" lb # [ 176.370071] niks3-server[1111]: 2026/09/23 05:14:50 INFO Registered completed upload object_key=nar/1fyfjzlx8vymsfmdv5cv8ymgjkr18fwm737qvys29plwwiwmyvwr.nar.zst lb # [ 176.385219] niks3-server[1111]: 2026/09/23 05:14:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign worker2 # [ 176.379903] nix-grpc-daemon[1636]: time=2026-09-23T05:14:50.879Z level=INFO msg="Uploading 1 narinfos" lb # [ 176.393996] niks3-server[1111]: 2026/09/23 05:14:50 INFO Signed narinfos id=65 count=1 lb # [ 176.399008] niks3-server[1111]: 2026/09/23 05:14:50 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.ls worker2 # [ 176.397193] nix-grpc-daemon[1636]: time=2026-09-23T05:14:50.896Z level=INFO msg="Upload complete. (209ms)" lb # [ 176.408527] niks3-server[1111]: 2026/09/23 05:14:50 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete lb # [ 176.412314] niks3-server[1111]: 2026/09/23 05:14:50 INFO Completed upload id=65 lb # [ 176.416079] niks3-server[1111]: 2026/09/23 05:14:50 INFO Registered completed upload object_key=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy.narinfo worker2 # [ 176.404299] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=0 paths=1 nar_bytes_in=416 worker2 # [ 176.418035] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 176.423701] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 176.428280] nix-daemon[842]: reaped child process 1851, status = succeeded worker2 # [ 176.437845] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) client # [ 176.487900] nix[1494]: building '/nix/store/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv' on lb:50051... worker1 # [ 178.137227] nix-daemon[850]: reaped child process 1868, status = succeeded worker1 # [ 178.255912] nix-daemon[850]: accepted connection from pid 1893, user nix-grpc-daemon (trusted) worker1 # [ 178.273295] nix-daemon[850]: reaped child process 1896, status = succeeded lb # [ 178.298350] niks3-server[1111]: 2026/09/23 05:14:52 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 178.290508] nix-grpc-daemon[1844]: time=2026-09-23T05:14:52.791Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 178.291992] nix-grpc-daemon[1844]: time=2026-09-23T05:14:52.791Z level=INFO msg="Uploading y3qkgrgq2zzpgvi2rf5chq94i627wkdb-slow-q3 (120B)" lb # [ 178.338897] niks3-server[1111]: 2026/09/23 05:14:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign worker1 # [ 178.330398] nix-grpc-daemon[1844]: time=2026-09-23T05:14:52.833Z level=INFO msg="Uploading 1 narinfos" lb # [ 178.355775] niks3-server[1111]: 2026/09/23 05:14:52 INFO Signed narinfos id=66 count=1 worker2 # [ 178.354174] nix-daemon[842]: reaped child process 1860, status = succeeded lb # [ 178.365489] niks3-server[1111]: 2026/09/23 05:14:52 INFO Registered completed upload object_key=nar/1v1grzkvw3yj09x0dkdm2avihvqvnri156qzp768hk85zbw7lkn9.nar.zst lb # [ 178.375065] niks3-server[1111]: 2026/09/23 05:14:52 INFO Registered completed upload object_key=log/dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv lb # [ 178.384112] niks3-server[1111]: 2026/09/23 05:14:52 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.ls lb # [ 178.387715] niks3-server[1111]: 2026/09/23 05:14:52 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete lb # [ 178.396231] niks3-server[1111]: 2026/09/23 05:14:52 INFO Completed upload id=66 lb # [ 178.398776] niks3-server[1111]: 2026/09/23 05:14:52 INFO Registered completed upload object_key=y3qkgrgq2zzpgvi2rf5chq94i627wkdb.narinfo worker1 # [ 178.375616] nix-grpc-daemon[1844]: time=2026-09-23T05:14:52.878Z level=INFO msg="Upload complete. (258ms)" worker2 # [ 178.396598] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:52Z level=debug event=assigned drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 178.392602] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:52Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=2 drv=dspy0axfmddglppg9zj9blmxlzhfs40x-slow-q3.drv assign_id=8 outputs=1 worker1 # [ 178.407908] nix-daemon[850]: reaped child process 1866, status = succeeded worker2 # [ 178.433182] nix-daemon[842]: accepted connection from pid 1885, user nix-grpc-daemon (trusted) worker1 # [ 178.433868] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker2 # [ 178.452829] nix-daemon[842]: reaped child process 1888, status = succeeded lb # [ 178.468000] niks3-server[1111]: 2026/09/23 05:14:52 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 178.455605] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:52Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54454 worker2 # [ 178.468459] nix-grpc-daemon[1636]: time=2026-09-23T05:14:52.966Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 178.470169] nix-grpc-daemon[1636]: time=2026-09-23T05:14:52.966Z level=INFO msg="Uploading gjyi8j90pk5flrsqqdhj6h132pcb7n33-slow-q2 (120B)" worker1 # [ 178.471309] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:52Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 client # [ 178.533156] nix[1494]: copying 1 paths to 'lb:50051'... client # [ 178.533443] nix[1494]: copying path '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' to 'lb:50051'... worker1 # [ 178.492935] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) lb # [ 178.514332] niks3-server[1111]: 2026/09/23 05:14:53 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.ls worker2 # [ 178.520367] nix-grpc-daemon[1636]: time=2026-09-23T05:14:53.019Z level=INFO msg="Uploading 1 narinfos" lb # [ 178.523001] niks3-server[1111]: 2026/09/23 05:14:53 INFO Registered completed upload object_key=nar/15imdx7zyy6mzjjv6qvdzywh0d9w1n6ikc0rq4rq95aw9x6swfzj.nar.zst lb # [ 178.533774] niks3-server[1111]: 2026/09/23 05:14:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign lb # [ 178.541378] niks3-server[1111]: 2026/09/23 05:14:53 INFO Signed narinfos id=67 count=1 lb # [ 178.548272] niks3-server[1111]: 2026/09/23 05:14:53 INFO Registered completed upload object_key=log/jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv worker2 # [ 178.544663] nix-grpc-daemon[1636]: time=2026-09-23T05:14:53.044Z level=INFO msg="Upload complete. (291ms)" lb # [ 178.555182] niks3-server[1111]: 2026/09/23 05:14:53 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete lb # [ 178.559740] niks3-server[1111]: 2026/09/23 05:14:53 INFO Completed upload id=67 lb # [ 178.563536] niks3-server[1111]: 2026/09/23 05:14:53 INFO Registered completed upload object_key=gjyi8j90pk5flrsqqdhj6h132pcb7n33.narinfo worker2 # [ 178.556363] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:53Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=2 drv=jgwqjvc2sfm39r12g67n9dhiqhi4z7xy-slow-q2.drv assign_id=9 outputs=1 worker2 # [ 178.572716] nix-daemon[842]: reaped child process 1858, status = succeeded worker1 # [ 178.653844] nix-daemon[850]: accepted connection from pid 1900, user nix-grpc-daemon (trusted) worker1 # [ 178.673233] nix-daemon[850]: reaped child process 1903, status = succeeded lb # [ 178.697815] niks3-server[1111]: 2026/09/23 05:14:53 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 178.687272] nix-grpc-daemon[1844]: time=2026-09-23T05:14:53.190Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 178.689092] nix-grpc-daemon[1844]: time=2026-09-23T05:14:53.190Z level=INFO msg="Uploading pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv (416B)" lb # [ 178.725257] niks3-server[1111]: 2026/09/23 05:14:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign worker1 # [ 178.713122] nix-grpc-daemon[1844]: time=2026-09-23T05:14:53.216Z level=INFO msg="Uploading 1 narinfos" lb # [ 178.737917] niks3-server[1111]: 2026/09/23 05:14:53 INFO Signed narinfos id=68 count=1 lb # [ 178.747876] niks3-server[1111]: 2026/09/23 05:14:53 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.ls lb # [ 178.756853] niks3-server[1111]: 2026/09/23 05:14:53 INFO Registered completed upload object_key=nar/1k21jrz411xc9gkca2pnr9mrjq1s3bg3zhsam3nnv4j066dgzgm7.nar.zst worker1 # [ 178.750451] nix-grpc-daemon[1844]: time=2026-09-23T05:14:53.253Z level=INFO msg="Upload complete. (240ms)" lb # [ 178.769782] niks3-server[1111]: 2026/09/23 05:14:53 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete lb # [ 178.779439] niks3-server[1111]: 2026/09/23 05:14:53 INFO Completed upload id=68 worker1 # [ 178.761323] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=0 paths=1 nar_bytes_in=416 lb # [ 178.782073] niks3-server[1111]: 2026/09/23 05:14:53 INFO Registered completed upload object_key=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd.narinfo worker1 # [ 178.775190] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 178.782304] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 178.787475] nix-daemon[850]: reaped child process 1898, status = succeeded client # [ 178.856917] nix[1494]: building '/nix/store/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv' on lb:50051... worker1 # [ 178.800252] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) lb # [ 180.280359] envoy[908]: conn peer=[2001:db8:1::3]:47526 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1020 tx=482 ms=26979 worker1 # [ 180.431230] nix-daemon[850]: reaped child process 1907, status = succeeded worker1 # [ 180.533204] nix-daemon[850]: accepted connection from pid 1932, user nix-grpc-daemon (trusted) worker1 # [ 180.550799] nix-daemon[850]: reaped child process 1935, status = succeeded lb # [ 180.578188] niks3-server[1111]: 2026/09/23 05:14:55 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 180.568406] nix-grpc-daemon[1844]: time=2026-09-23T05:14:55.070Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 180.570098] nix-grpc-daemon[1844]: time=2026-09-23T05:14:55.070Z level=INFO msg="Uploading l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx-slow-q1 (120B)" lb # [ 180.618780] niks3-server[1111]: 2026/09/23 05:14:55 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.ls worker1 # [ 180.610066] nix-grpc-daemon[1844]: time=2026-09-23T05:14:55.113Z level=INFO msg="Uploading 1 narinfos" lb # [ 180.633500] niks3-server[1111]: 2026/09/23 05:14:55 INFO Registered completed upload object_key=nar/0c1c367r3msa4fz6imwp70zxi4fxvrkb86v3hv921wsrf13yd8v3.nar.zst lb # [ 180.643187] niks3-server[1111]: 2026/09/23 05:14:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign lb # [ 180.652412] niks3-server[1111]: 2026/09/23 05:14:55 INFO Signed narinfos id=69 count=1 worker1 # [ 180.633329] nix-grpc-daemon[1844]: time=2026-09-23T05:14:55.136Z level=INFO msg="Upload complete. (220ms)" lb # [ 180.654369] niks3-server[1111]: 2026/09/23 05:14:55 INFO Registered completed upload object_key=log/pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv lb # [ 180.660379] niks3-server[1111]: 2026/09/23 05:14:55 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete lb # [ 180.663907] niks3-server[1111]: 2026/09/23 05:14:55 INFO Completed upload id=69 worker1 # [ 180.644894] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:55Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=1 drv=pgfqjrvqz7k2z3ab3gk4xjy7130b5jcd-slow-q1.drv assign_id=10 outputs=1 lb # [ 180.667585] niks3-server[1111]: 2026/09/23 05:14:55 INFO Registered completed upload object_key=l3s7m1wb77d0cgdpbn5bcb47ibrwjvmx.narinfo worker1 # [ 180.657976] nix-daemon[850]: reaped child process 1905, status = succeeded client # [ 180.738904] systemd[1]: burst.service: Deactivated successfully. client # [ 180.744165] systemd[1]: burst.service: Consumed 236ms CPU time over 6.490s wall clock time, 57.6M memory peak, 14.1K incoming IP traffic, 18.9K outgoing IP traffic. client: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; } client: (finished: must succeed: systemctl show -p Result --value burst | grep -qx success || { journalctl -u burst >&2; false; }, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: queue gauges show a burst while every build blocks, in 8.03 seconds) subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1 worker2 # [ 182.209881] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:56Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 182.220790] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 182.263130] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:56Z level=debug event=want client=6 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=0 worker2 # [ 182.269449] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:56Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=1 worker2 # [ 182.275416] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:56Z level=debug event=want client=6 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=0 worker2 # [ 182.280754] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:56Z level=debug event=want client=6 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=0 worker1 # [ 182.303146] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:56Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 182.319334] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 182.334268] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 182.349357] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 182.511998] nix-daemon[850]: accepted connection from pid 1943, user nix-grpc-daemon (trusted) worker1 # [ 182.532133] nix-daemon[850]: reaped child process 1946, status = succeeded lb # [ 182.557085] niks3-server[1111]: 2026/09/23 05:14:57 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 182.546315] nix-grpc-daemon[1844]: time=2026-09-23T05:14:57.048Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 182.548444] nix-grpc-daemon[1844]: time=2026-09-23T05:14:57.048Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)" lb # [ 182.587153] niks3-server[1111]: 2026/09/23 05:14:57 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst worker1 # [ 182.576065] nix-grpc-daemon[1844]: time=2026-09-23T05:14:57.079Z level=INFO msg="Uploading 1 narinfos" lb # [ 182.595900] niks3-server[1111]: 2026/09/23 05:14:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign lb # [ 182.603044] niks3-server[1111]: 2026/09/23 05:14:57 INFO Signed narinfos id=70 count=1 lb # [ 182.609930] niks3-server[1111]: 2026/09/23 05:14:57 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls lb # [ 182.619039] niks3-server[1111]: 2026/09/23 05:14:57 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete lb # [ 182.624107] niks3-server[1111]: 2026/09/23 05:14:57 INFO Completed upload id=70 lb # [ 182.628677] niks3-server[1111]: 2026/09/23 05:14:57 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo worker1 # [ 182.605356] nix-grpc-daemon[1844]: time=2026-09-23T05:14:57.108Z level=INFO msg="Upload complete. (238ms)" worker1 # [ 182.616300] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 182.628698] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 182.637332] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 182.642729] nix-daemon[850]: reaped child process 1941, status = succeeded worker1 # [ 182.652517] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 183.558217] nix-daemon[850]: reaped child process 1950, status = succeeded worker1 # [ 183.634273] nix-daemon[850]: accepted connection from pid 1966, user nix-grpc-daemon (trusted) worker1 # [ 183.653517] nix-daemon[850]: reaped child process 1969, status = succeeded lb # [ 183.679421] niks3-server[1111]: 2026/09/23 05:14:58 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 183.670127] nix-grpc-daemon[1844]: time=2026-09-23T05:14:58.172Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 183.671737] nix-grpc-daemon[1844]: time=2026-09-23T05:14:58.172Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)" lb # [ 183.724947] niks3-server[1111]: 2026/09/23 05:14:58 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker1 # [ 183.714290] nix-grpc-daemon[1844]: time=2026-09-23T05:14:58.217Z level=INFO msg="Uploading 1 narinfos" lb # [ 183.737874] niks3-server[1111]: 2026/09/23 05:14:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign lb # [ 183.747882] niks3-server[1111]: 2026/09/23 05:14:58 INFO Signed narinfos id=71 count=1 lb # [ 183.756293] niks3-server[1111]: 2026/09/23 05:14:58 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst worker1 # [ 183.743242] nix-grpc-daemon[1844]: time=2026-09-23T05:14:58.246Z level=INFO msg="Upload complete. (279ms)" lb # [ 183.766230] niks3-server[1111]: 2026/09/23 05:14:58 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls lb # [ 183.772941] niks3-server[1111]: 2026/09/23 05:14:58 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete lb # [ 183.778456] niks3-server[1111]: 2026/09/23 05:14:58 INFO Completed upload id=71 worker2 # [ 183.764298] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:58Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 183.757262] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:58Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=11 outputs=1 lb # [ 183.781415] niks3-server[1111]: 2026/09/23 05:14:58 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo worker2 # [ 183.780161] nix-grpc-daemon[1559]: ts=2026-09-23T05:14:58Z level=debug event=want client=6 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=0 worker1 # [ 183.779378] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 183.786723] nix-daemon[850]: reaped child process 1948, status = succeeded worker1 # [ 183.790041] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:58Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 183.803278] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:58Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 183.816075] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 183.951607] nix-daemon[850]: accepted connection from pid 1973, user nix-grpc-daemon (trusted) worker1 # [ 183.969353] nix-daemon[850]: reaped child process 1976, status = succeeded lb # [ 183.993634] niks3-server[1111]: 2026/09/23 05:14:58 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 183.983485] nix-grpc-daemon[1844]: time=2026-09-23T05:14:58.486Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 183.985360] nix-grpc-daemon[1844]: time=2026-09-23T05:14:58.486Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)" lb # [ 184.022246] niks3-server[1111]: 2026/09/23 05:14:58 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst worker1 # [ 184.021488] nix-grpc-daemon[1844]: time=2026-09-23T05:14:58.524Z level=INFO msg="Uploading 1 narinfos" lb # [ 184.034666] niks3-server[1111]: 2026/09/23 05:14:58 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign lb # [ 184.045460] niks3-server[1111]: 2026/09/23 05:14:58 INFO Signed narinfos id=72 count=1 lb # [ 184.051668] niks3-server[1111]: 2026/09/23 05:14:58 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls worker1 # [ 184.043447] nix-grpc-daemon[1844]: time=2026-09-23T05:14:58.546Z level=INFO msg="Upload complete. (215ms)" lb # [ 184.063541] niks3-server[1111]: 2026/09/23 05:14:58 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete lb # [ 184.073699] niks3-server[1111]: 2026/09/23 05:14:58 INFO Completed upload id=72 worker1 # [ 184.052357] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=0 paths=1 nar_bytes_in=512 lb # [ 184.077339] niks3-server[1111]: 2026/09/23 05:14:58 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.narinfo worker1 # [ 184.071100] nix-grpc-daemon[1783]: ts=2026-09-23T05:14:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 184.078238] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 184.083482] nix-daemon[850]: reaped child process 1971, status = succeeded worker1 # [ 184.094931] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 185.346474] nix-daemon[850]: reaped child process 1980, status = succeeded worker1 # [ 185.473968] nix-daemon[850]: accepted connection from pid 1996, user nix-grpc-daemon (trusted) worker1 # [ 185.501103] nix-daemon[850]: reaped child process 1999, status = succeeded lb # [ 185.521551] niks3-server[1111]: 2026/09/23 05:15:00 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 185.516502] nix-grpc-daemon[1844]: time=2026-09-23T05:15:00.017Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 185.517966] nix-grpc-daemon[1844]: time=2026-09-23T05:15:00.017Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)" lb # [ 185.598330] niks3-server[1111]: 2026/09/23 05:15:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign worker1 # [ 185.591361] nix-grpc-daemon[1844]: time=2026-09-23T05:15:00.093Z level=INFO msg="Uploading 1 narinfos" lb # [ 185.617542] niks3-server[1111]: 2026/09/23 05:15:00 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst lb # [ 185.627023] niks3-server[1111]: 2026/09/23 05:15:00 INFO Signed narinfos id=73 count=1 lb # [ 185.632697] niks3-server[1111]: 2026/09/23 05:15:00 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker1 # [ 185.628644] nix-grpc-daemon[1844]: time=2026-09-23T05:15:00.131Z level=INFO msg="Upload complete. (320ms)" lb # [ 185.648797] niks3-server[1111]: 2026/09/23 05:15:00 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls lb # [ 185.652519] niks3-server[1111]: 2026/09/23 05:15:00 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete lb # [ 185.659313] niks3-server[1111]: 2026/09/23 05:15:00 INFO Completed upload id=73 lb # [ 185.663313] niks3-server[1111]: 2026/09/23 05:15:00 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo worker2 # [ 185.651472] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:00Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 185.641664] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:00Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=12 outputs=1 worker2 # [ 185.670432] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:00Z level=debug event=want client=6 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=0 worker1 # [ 185.664457] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 185.680633] nix-daemon[850]: reaped child process 1978, status = succeeded worker1 # [ 185.683610] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 185.698209] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 185.714515] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 185.909895] nix-daemon[850]: accepted connection from pid 2003, user nix-grpc-daemon (trusted) worker1 # [ 185.932692] nix-daemon[850]: reaped child process 2006, status = succeeded lb # [ 185.956718] niks3-server[1111]: 2026/09/23 05:15:00 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 185.948660] nix-grpc-daemon[1844]: time=2026-09-23T05:15:00.451Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 185.950184] nix-grpc-daemon[1844]: time=2026-09-23T05:15:00.451Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)" lb # [ 185.999662] niks3-server[1111]: 2026/09/23 05:15:00 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls lb # [ 186.015842] niks3-server[1111]: 2026/09/23 05:15:00 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign worker1 # [ 186.005456] nix-grpc-daemon[1844]: time=2026-09-23T05:15:00.508Z level=INFO msg="Uploading 1 narinfos" lb # [ 186.030980] niks3-server[1111]: 2026/09/23 05:15:00 INFO Signed narinfos id=74 count=1 lb # [ 186.037767] niks3-server[1111]: 2026/09/23 05:15:00 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst lb # [ 186.054222] niks3-server[1111]: 2026/09/23 05:15:00 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete lb # [ 186.062110] niks3-server[1111]: 2026/09/23 05:15:00 INFO Completed upload id=74 worker1 # [ 186.042138] nix-grpc-daemon[1844]: time=2026-09-23T05:15:00.545Z level=INFO msg="Upload complete. (310ms)" lb # [ 186.065120] niks3-server[1111]: 2026/09/23 05:15:00 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo worker1 # [ 186.052486] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:00Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 186.076372] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:00Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 186.089149] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 186.096558] nix-daemon[850]: reaped child process 2001, status = succeeded worker1 # [ 186.119891] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 187.849189] nix-daemon[850]: reaped child process 2010, status = succeeded worker1 # [ 187.970956] nix-daemon[850]: accepted connection from pid 2026, user nix-grpc-daemon (trusted) worker1 # [ 187.994379] nix-daemon[850]: reaped child process 2029, status = succeeded lb # [ 188.023268] niks3-server[1111]: 2026/09/23 05:15:02 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 188.020265] nix-grpc-daemon[1844]: time=2026-09-23T05:15:02.520Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 188.021983] nix-grpc-daemon[1844]: time=2026-09-23T05:15:02.521Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)" lb # [ 188.083425] niks3-server[1111]: 2026/09/23 05:15:02 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign worker1 # [ 188.075504] nix-grpc-daemon[1844]: time=2026-09-23T05:15:02.578Z level=INFO msg="Uploading 1 narinfos" lb # [ 188.096886] niks3-server[1111]: 2026/09/23 05:15:02 INFO Signed narinfos id=75 count=1 lb # [ 188.113503] niks3-server[1111]: 2026/09/23 05:15:02 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv lb # [ 188.128349] niks3-server[1111]: 2026/09/23 05:15:02 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls lb # [ 188.137085] niks3-server[1111]: 2026/09/23 05:15:02 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst lb # [ 188.148647] niks3-server[1111]: 2026/09/23 05:15:02 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete lb # [ 188.156071] niks3-server[1111]: 2026/09/23 05:15:02 INFO Completed upload id=75 worker1 # [ 188.139897] nix-grpc-daemon[1844]: time=2026-09-23T05:15:02.640Z level=INFO msg="Upload complete. (402ms)" lb # [ 188.161670] niks3-server[1111]: 2026/09/23 05:15:02 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo worker2 # [ 188.159989] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:02Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 188.157933] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:02Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=2 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=13 outputs=1 worker2 # [ 188.177013] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:02Z level=debug event=want client=6 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=0 worker1 # [ 188.176519] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:02Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 188.196732] nix-daemon[850]: reaped child process 2008, status = succeeded worker1 # [ 188.204877] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:02Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 188.218478] nix-grpc-daemon[1844]: time=2026-09-23T05:15:02.721Z level=INFO msg="All 1 paths already cached" worker1 # [ 188.230203] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:02Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 188.247674] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 188.458943] nix-daemon[850]: accepted connection from pid 2033, user nix-grpc-daemon (trusted) worker1 # [ 188.479882] nix-daemon[850]: reaped child process 2036, status = succeeded lb # [ 188.509062] niks3-server[1111]: 2026/09/23 05:15:03 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 188.501669] nix-grpc-daemon[1844]: time=2026-09-23T05:15:03.004Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 188.503677] nix-grpc-daemon[1844]: time=2026-09-23T05:15:03.004Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)" lb # [ 188.557475] niks3-server[1111]: 2026/09/23 05:15:03 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign worker1 # [ 188.546275] nix-grpc-daemon[1844]: time=2026-09-23T05:15:03.049Z level=INFO msg="Uploading 1 narinfos" lb # [ 188.571873] niks3-server[1111]: 2026/09/23 05:15:03 INFO Signed narinfos id=76 count=1 lb # [ 188.578863] niks3-server[1111]: 2026/09/23 05:15:03 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls lb # [ 188.591635] niks3-server[1111]: 2026/09/23 05:15:03 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst lb # [ 188.602751] niks3-server[1111]: 2026/09/23 05:15:03 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete worker1 # [ 188.588698] nix-grpc-daemon[1844]: time=2026-09-23T05:15:03.091Z level=INFO msg="Upload complete. (321ms)" lb # [ 188.610490] niks3-server[1111]: 2026/09/23 05:15:03 INFO Completed upload id=76 lb # [ 188.612938] niks3-server[1111]: 2026/09/23 05:15:03 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo worker1 # [ 188.600483] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:03Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 188.617282] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 188.636869] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 188.645137] nix-daemon[850]: reaped child process 2031, status = succeeded worker1 # [ 188.666806] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 189.831408] nix-daemon[850]: reaped child process 2040, status = succeeded worker1 # [ 189.926063] nix-daemon[850]: accepted connection from pid 2056, user nix-grpc-daemon (trusted) worker1 # [ 189.953775] nix-daemon[850]: reaped child process 2059, status = succeeded lb # [ 189.979615] niks3-server[1111]: 2026/09/23 05:15:04 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 189.971857] nix-grpc-daemon[1844]: time=2026-09-23T05:15:04.473Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 189.975401] nix-grpc-daemon[1844]: time=2026-09-23T05:15:04.473Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)" lb # [ 190.046614] niks3-server[1111]: 2026/09/23 05:15:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign worker1 # [ 190.041009] nix-grpc-daemon[1844]: time=2026-09-23T05:15:04.543Z level=INFO msg="Uploading 1 narinfos" lb # [ 190.061827] niks3-server[1111]: 2026/09/23 05:15:04 INFO Signed narinfos id=77 count=1 lb # [ 190.073343] niks3-server[1111]: 2026/09/23 05:15:04 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst lb # [ 190.085658] niks3-server[1111]: 2026/09/23 05:15:04 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv lb # [ 190.094507] niks3-server[1111]: 2026/09/23 05:15:04 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls lb # [ 190.106345] niks3-server[1111]: 2026/09/23 05:15:04 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete worker1 # [ 190.086968] nix-grpc-daemon[1844]: time=2026-09-23T05:15:04.590Z level=INFO msg="Upload complete. (405ms)" lb # [ 190.110657] niks3-server[1111]: 2026/09/23 05:15:04 INFO Completed upload id=77 lb # [ 190.113075] niks3-server[1111]: 2026/09/23 05:15:04 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo worker1 # [ 190.101573] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:04Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=14 outputs=1 worker2 # [ 190.119283] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:04Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=1 worker2 # [ 190.128341] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:04Z level=debug event=want client=6 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=0 worker1 # [ 190.128780] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 190.138956] nix-daemon[850]: reaped child process 2038, status = succeeded worker1 # [ 190.151120] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:04Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 190.172988] nix-grpc-daemon[1844]: time=2026-09-23T05:15:04.676Z level=INFO msg="All 1 paths already cached" worker1 # [ 190.187762] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:04Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 190.204944] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) lb # [ 190.282084] envoy[908]: conn peer=[2001:db8:1::1]:49350 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9740 tx=5201 ms=6263 worker1 # [ 190.381037] nix-daemon[850]: accepted connection from pid 2063, user nix-grpc-daemon (trusted) worker1 # [ 190.406491] nix-daemon[850]: reaped child process 2066, status = succeeded lb # [ 190.429372] niks3-server[1111]: 2026/09/23 05:15:04 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 190.426413] nix-grpc-daemon[1844]: time=2026-09-23T05:15:04.929Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 190.428390] nix-grpc-daemon[1844]: time=2026-09-23T05:15:04.929Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)" lb # [ 190.480657] niks3-server[1111]: 2026/09/23 05:15:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign worker1 # [ 190.470540] nix-grpc-daemon[1844]: time=2026-09-23T05:15:04.973Z level=INFO msg="Uploading 1 narinfos" lb # [ 190.496421] niks3-server[1111]: 2026/09/23 05:15:04 INFO Signed narinfos id=78 count=1 lb # [ 190.503944] niks3-server[1111]: 2026/09/23 05:15:05 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst lb # [ 190.514225] niks3-server[1111]: 2026/09/23 05:15:05 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls worker1 # [ 190.516272] nix-grpc-daemon[1844]: time=2026-09-23T05:15:05.019Z level=INFO msg="Upload complete. (293ms)" lb # [ 190.529230] niks3-server[1111]: 2026/09/23 05:15:05 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete lb # [ 190.540436] niks3-server[1111]: 2026/09/23 05:15:05 INFO Completed upload id=78 lb # [ 190.542486] niks3-server[1111]: 2026/09/23 05:15:05 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo worker1 # [ 190.528504] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:05Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 190.545195] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 190.560448] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 190.572526] nix-daemon[850]: reaped child process 2061, status = succeeded worker1 # [ 190.594769] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 191.503890] nix-daemon[850]: reaped child process 2070, status = succeeded worker1 # [ 191.617085] nix-daemon[850]: accepted connection from pid 2086, user nix-grpc-daemon (trusted) worker1 # [ 191.637233] nix-daemon[850]: reaped child process 2089, status = succeeded lb # [ 191.662223] niks3-server[1111]: 2026/09/23 05:15:06 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 191.655186] nix-grpc-daemon[1844]: time=2026-09-23T05:15:06.156Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 191.660087] nix-grpc-daemon[1844]: time=2026-09-23T05:15:06.156Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)" worker1 # [ 191.700819] nix-grpc-daemon[1844]: time=2026-09-23T05:15:06.203Z level=INFO msg="Uploading 1 narinfos" lb # [ 191.710374] niks3-server[1111]: 2026/09/23 05:15:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign lb # [ 191.725199] niks3-server[1111]: 2026/09/23 05:15:06 INFO Signed narinfos id=79 count=1 lb # [ 191.734134] niks3-server[1111]: 2026/09/23 05:15:06 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv lb # [ 191.745931] niks3-server[1111]: 2026/09/23 05:15:06 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls lb # [ 191.753018] niks3-server[1111]: 2026/09/23 05:15:06 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst worker1 # [ 191.742364] nix-grpc-daemon[1844]: time=2026-09-23T05:15:06.245Z level=INFO msg="Upload complete. (331ms)" lb # [ 191.762599] niks3-server[1111]: 2026/09/23 05:15:06 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete lb # [ 191.766592] niks3-server[1111]: 2026/09/23 05:15:06 INFO Completed upload id=79 lb # [ 191.770285] niks3-server[1111]: 2026/09/23 05:15:06 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo worker2 # [ 191.761888] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:06Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 191.757221] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:06Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=15 outputs=1 worker1 # [ 191.775913] nix-daemon[850]: reaped child process 2068, status = succeeded worker1 # [ 191.799840] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker2 # [ 191.812147] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:06Z level=debug event=want client=6 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=0 worker1 # [ 191.817743] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:06Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 191.832440] nix-grpc-daemon[1844]: time=2026-09-23T05:15:06.335Z level=INFO msg="All 1 paths already cached" worker1 # [ 191.841633] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 191.857036] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 192.017712] nix-daemon[850]: accepted connection from pid 2093, user nix-grpc-daemon (trusted) worker1 # [ 192.038845] nix-daemon[850]: reaped child process 2096, status = succeeded lb # [ 192.058951] niks3-server[1111]: 2026/09/23 05:15:06 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 192.049214] nix-grpc-daemon[1844]: time=2026-09-23T05:15:06.552Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 192.050753] nix-grpc-daemon[1844]: time=2026-09-23T05:15:06.552Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)" lb # [ 192.092709] niks3-server[1111]: 2026/09/23 05:15:06 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls worker1 # [ 192.084589] nix-grpc-daemon[1844]: time=2026-09-23T05:15:06.587Z level=INFO msg="Uploading 1 narinfos" lb # [ 192.105425] niks3-server[1111]: 2026/09/23 05:15:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/80/sign lb # [ 192.117134] niks3-server[1111]: 2026/09/23 05:15:06 INFO Signed narinfos id=80 count=1 lb # [ 192.123272] niks3-server[1111]: 2026/09/23 05:15:06 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst lb # [ 192.134242] niks3-server[1111]: 2026/09/23 05:15:06 INFO Received complete upload request method=POST path=/api/pending_closures/80/complete lb # [ 192.143729] niks3-server[1111]: 2026/09/23 05:15:06 INFO Completed upload id=80 worker1 # [ 192.123873] nix-grpc-daemon[1844]: time=2026-09-23T05:15:06.627Z level=INFO msg="Upload complete. (248ms)" lb # [ 192.147548] niks3-server[1111]: 2026/09/23 05:15:06 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo worker1 # [ 192.136315] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 192.152362] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 192.163978] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 192.167102] nix-daemon[850]: reaped child process 2091, status = succeeded worker1 # [ 192.184317] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 193.288541] nix-daemon[850]: reaped child process 2100, status = succeeded worker1 # [ 193.361379] nix-daemon[850]: accepted connection from pid 2116, user nix-grpc-daemon (trusted) worker1 # [ 193.378868] nix-daemon[850]: reaped child process 2119, status = succeeded lb # [ 193.405261] niks3-server[1111]: 2026/09/23 05:15:07 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 193.396409] nix-grpc-daemon[1844]: time=2026-09-23T05:15:07.898Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 193.398052] nix-grpc-daemon[1844]: time=2026-09-23T05:15:07.898Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)" lb # [ 193.448727] niks3-server[1111]: 2026/09/23 05:15:07 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst worker1 # [ 193.449322] nix-grpc-daemon[1844]: time=2026-09-23T05:15:07.952Z level=INFO msg="Uploading 1 narinfos" lb # [ 193.461512] niks3-server[1111]: 2026/09/23 05:15:07 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv lb # [ 193.473294] niks3-server[1111]: 2026/09/23 05:15:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/81/sign lb # [ 193.482456] niks3-server[1111]: 2026/09/23 05:15:07 INFO Signed narinfos id=81 count=1 lb # [ 193.488989] niks3-server[1111]: 2026/09/23 05:15:07 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls worker1 # [ 193.481657] nix-grpc-daemon[1844]: time=2026-09-23T05:15:07.984Z level=INFO msg="Upload complete. (285ms)" lb # [ 193.502479] niks3-server[1111]: 2026/09/23 05:15:08 INFO Received complete upload request method=POST path=/api/pending_closures/81/complete lb # [ 193.510326] niks3-server[1111]: 2026/09/23 05:15:08 INFO Completed upload id=81 worker2 # [ 193.499540] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:07Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=1 lb # [ 193.512831] niks3-server[1111]: 2026/09/23 05:15:08 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo worker1 # [ 193.497157] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=16 outputs=1 worker2 # [ 193.513229] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:08Z level=debug event=want client=6 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=0 worker1 # [ 193.515691] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 193.523215] nix-daemon[850]: reaped child process 2098, status = succeeded worker1 # [ 193.531988] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:08Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 193.545558] nix-grpc-daemon[1844]: time=2026-09-23T05:15:08.048Z level=INFO msg="All 2 paths already cached" worker1 # [ 193.555817] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:08Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 193.569473] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 193.712617] nix-daemon[850]: accepted connection from pid 2123, user nix-grpc-daemon (trusted) worker1 # [ 193.729424] nix-daemon[850]: reaped child process 2126, status = succeeded lb # [ 193.755190] niks3-server[1111]: 2026/09/23 05:15:08 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 193.746389] nix-grpc-daemon[1844]: time=2026-09-23T05:15:08.249Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 193.748097] nix-grpc-daemon[1844]: time=2026-09-23T05:15:08.249Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)" lb # [ 193.785548] niks3-server[1111]: 2026/09/23 05:15:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/82/sign worker1 # [ 193.775635] nix-grpc-daemon[1844]: time=2026-09-23T05:15:08.278Z level=INFO msg="Uploading 1 narinfos" lb # [ 193.799651] niks3-server[1111]: 2026/09/23 05:15:08 INFO Signed narinfos id=82 count=1 lb # [ 193.807025] niks3-server[1111]: 2026/09/23 05:15:08 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst lb # [ 193.815469] niks3-server[1111]: 2026/09/23 05:15:08 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls lb # [ 193.822804] niks3-server[1111]: 2026/09/23 05:15:08 INFO Received complete upload request method=POST path=/api/pending_closures/82/complete lb # [ 193.828135] niks3-server[1111]: 2026/09/23 05:15:08 INFO Completed upload id=82 worker1 # [ 193.808186] nix-grpc-daemon[1844]: time=2026-09-23T05:15:08.311Z level=INFO msg="Upload complete. (225ms)" lb # [ 193.830384] niks3-server[1111]: 2026/09/23 05:15:08 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo worker1 # [ 193.816281] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 193.831667] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 193.838928] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 193.843777] nix-daemon[850]: reaped child process 2121, status = succeeded worker1 # [ 193.856292] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 194.965015] nix-daemon[850]: reaped child process 2130, status = succeeded worker1 # [ 195.033748] nix-daemon[850]: accepted connection from pid 2146, user nix-grpc-daemon (trusted) worker1 # [ 195.051448] nix-daemon[850]: reaped child process 2149, status = succeeded lb # [ 195.078384] niks3-server[1111]: 2026/09/23 05:15:09 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 195.069890] nix-grpc-daemon[1844]: time=2026-09-23T05:15:09.572Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 195.071669] nix-grpc-daemon[1844]: time=2026-09-23T05:15:09.572Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)" lb # [ 195.125485] niks3-server[1111]: 2026/09/23 05:15:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/83/sign worker1 # [ 195.115153] nix-grpc-daemon[1844]: time=2026-09-23T05:15:09.618Z level=INFO msg="Uploading 1 narinfos" lb # [ 195.139428] niks3-server[1111]: 2026/09/23 05:15:09 INFO Signed narinfos id=83 count=1 lb # [ 195.147505] niks3-server[1111]: 2026/09/23 05:15:09 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls lb # [ 195.155987] niks3-server[1111]: 2026/09/23 05:15:09 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst lb # [ 195.159589] niks3-server[1111]: 2026/09/23 05:15:09 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv lb # [ 195.166266] niks3-server[1111]: 2026/09/23 05:15:09 INFO Received complete upload request method=POST path=/api/pending_closures/83/complete lb # [ 195.173695] niks3-server[1111]: 2026/09/23 05:15:09 INFO Completed upload id=83 worker1 # [ 195.153827] nix-grpc-daemon[1844]: time=2026-09-23T05:15:09.657Z level=INFO msg="Upload complete. (287ms)" lb # [ 195.175538] niks3-server[1111]: 2026/09/23 05:15:09 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo worker2 # [ 195.172323] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:09Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 195.169148] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=17 outputs=1 worker1 # [ 195.185630] nix-daemon[850]: reaped child process 2128, status = succeeded worker1 # [ 195.209361] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:09Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 195.224902] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:09Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 195.238708] nix-grpc-daemon[1844]: time=2026-09-23T05:15:09.741Z level=INFO msg="All 2 paths already cached" worker1 # [ 195.248737] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:09Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 195.262386] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 195.413172] nix-daemon[850]: accepted connection from pid 2153, user nix-grpc-daemon (trusted) worker1 # [ 195.430395] nix-daemon[850]: reaped child process 2156, status = succeeded lb # [ 195.455767] niks3-server[1111]: 2026/09/23 05:15:09 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 195.445640] nix-grpc-daemon[1844]: time=2026-09-23T05:15:09.948Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 195.447330] nix-grpc-daemon[1844]: time=2026-09-23T05:15:09.948Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)" worker1 # [ 195.476371] nix-grpc-daemon[1844]: time=2026-09-23T05:15:09.979Z level=INFO msg="Uploading 1 narinfos" lb # [ 195.484498] niks3-server[1111]: 2026/09/23 05:15:09 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls lb # [ 195.498938] niks3-server[1111]: 2026/09/23 05:15:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/84/sign lb # [ 195.508932] niks3-server[1111]: 2026/09/23 05:15:10 INFO Signed narinfos id=84 count=1 worker1 # [ 195.499592] nix-grpc-daemon[1844]: time=2026-09-23T05:15:10.002Z level=INFO msg="Upload complete. (221ms)" lb # [ 195.519303] niks3-server[1111]: 2026/09/23 05:15:10 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst worker1 # [ 195.508318] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 195.530905] niks3-server[1111]: 2026/09/23 05:15:10 INFO Received complete upload request method=POST path=/api/pending_closures/84/complete lb # [ 195.538123] niks3-server[1111]: 2026/09/23 05:15:10 INFO Completed upload id=84 lb # [ 195.541196] niks3-server[1111]: 2026/09/23 05:15:10 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo worker1 # [ 195.520678] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 195.531023] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 195.534603] nix-daemon[850]: reaped child process 2151, status = succeeded worker1 # [ 195.544751] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 196.850856] nix-daemon[850]: reaped child process 2160, status = succeeded worker1 # [ 196.930995] nix-daemon[850]: accepted connection from pid 2176, user nix-grpc-daemon (trusted) worker1 # [ 196.949050] nix-daemon[850]: reaped child process 2179, status = succeeded lb # [ 196.974990] niks3-server[1111]: 2026/09/23 05:15:11 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 196.967607] nix-grpc-daemon[1844]: time=2026-09-23T05:15:11.469Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 196.969615] nix-grpc-daemon[1844]: time=2026-09-23T05:15:11.469Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)" lb # [ 197.011534] niks3-server[1111]: 2026/09/23 05:15:11 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls worker1 # [ 197.008752] nix-grpc-daemon[1844]: time=2026-09-23T05:15:11.511Z level=INFO msg="Uploading 1 narinfos" lb # [ 197.028285] niks3-server[1111]: 2026/09/23 05:15:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/85/sign lb # [ 197.037446] niks3-server[1111]: 2026/09/23 05:15:11 INFO Signed narinfos id=85 count=1 lb # [ 197.039980] niks3-server[1111]: 2026/09/23 05:15:11 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst lb # [ 197.047969] niks3-server[1111]: 2026/09/23 05:15:11 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker1 # [ 197.039022] nix-grpc-daemon[1844]: time=2026-09-23T05:15:11.542Z level=INFO msg="Upload complete. (276ms)" lb # [ 197.060104] niks3-server[1111]: 2026/09/23 05:15:11 INFO Received complete upload request method=POST path=/api/pending_closures/85/complete lb # [ 197.063282] niks3-server[1111]: 2026/09/23 05:15:11 INFO Completed upload id=85 lb # [ 197.066263] niks3-server[1111]: 2026/09/23 05:15:11 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo worker2 # [ 197.057757] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:11Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 197.052834] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:11Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=18 outputs=1 worker1 # [ 197.068615] nix-daemon[850]: reaped child process 2158, status = succeeded worker1 # [ 197.097517] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 197.112432] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:11Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 197.126047] nix-grpc-daemon[1844]: time=2026-09-23T05:15:11.629Z level=INFO msg="All 2 paths already cached" worker1 # [ 197.135692] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:11Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 197.150509] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 197.302770] nix-daemon[850]: accepted connection from pid 2183, user nix-grpc-daemon (trusted) worker1 # [ 197.321488] nix-daemon[850]: reaped child process 2186, status = succeeded lb # [ 197.343603] niks3-server[1111]: 2026/09/23 05:15:11 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 197.333705] nix-grpc-daemon[1844]: time=2026-09-23T05:15:11.836Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 197.335291] nix-grpc-daemon[1844]: time=2026-09-23T05:15:11.836Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)" lb # [ 197.374733] niks3-server[1111]: 2026/09/23 05:15:11 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls worker1 # [ 197.373606] nix-grpc-daemon[1844]: time=2026-09-23T05:15:11.876Z level=INFO msg="Uploading 1 narinfos" lb # [ 197.383897] niks3-server[1111]: 2026/09/23 05:15:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/86/sign lb # [ 197.396947] niks3-server[1111]: 2026/09/23 05:15:11 INFO Signed narinfos id=86 count=1 lb # [ 197.404761] niks3-server[1111]: 2026/09/23 05:15:11 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst lb # [ 197.418608] niks3-server[1111]: 2026/09/23 05:15:11 INFO Received complete upload request method=POST path=/api/pending_closures/86/complete worker1 # [ 197.403946] nix-grpc-daemon[1844]: time=2026-09-23T05:15:11.907Z level=INFO msg="Upload complete. (239ms)" lb # [ 197.426001] niks3-server[1111]: 2026/09/23 05:15:11 INFO Completed upload id=86 lb # [ 197.430121] niks3-server[1111]: 2026/09/23 05:15:11 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo worker1 # [ 197.416571] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 197.429706] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 worker1 # [ 197.435995] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 197.440996] nix-daemon[850]: reaped child process 2181, status = succeeded worker1 # [ 197.453547] nix-daemon[850]: accepted connection from pid 1783, user nix-grpc-daemon (trusted) worker1 # [ 198.090704] nix-daemon[850]: reaped child process 2190, status = succeeded worker1 # [ 198.183686] nix-daemon[850]: accepted connection from pid 2206, user nix-grpc-daemon (trusted) worker1 # [ 198.198469] nix-daemon[850]: reaped child process 2209, status = succeeded lb # [ 198.226100] niks3-server[1111]: 2026/09/23 05:15:12 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 198.218671] nix-grpc-daemon[1844]: time=2026-09-23T05:15:12.720Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 198.220208] nix-grpc-daemon[1844]: time=2026-09-23T05:15:12.720Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)" lb # [ 198.265501] niks3-server[1111]: 2026/09/23 05:15:12 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls lb # [ 198.277000] niks3-server[1111]: 2026/09/23 05:15:12 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst worker1 # [ 198.270338] nix-grpc-daemon[1844]: time=2026-09-23T05:15:12.773Z level=INFO msg="Uploading 1 narinfos" lb # [ 198.290544] niks3-server[1111]: 2026/09/23 05:15:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/87/sign lb # [ 198.300109] niks3-server[1111]: 2026/09/23 05:15:12 INFO Signed narinfos id=87 count=1 lb # [ 198.302791] niks3-server[1111]: 2026/09/23 05:15:12 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv lb # [ 198.315209] niks3-server[1111]: 2026/09/23 05:15:12 INFO Received complete upload request method=POST path=/api/pending_closures/87/complete worker1 # [ 198.301107] nix-grpc-daemon[1844]: time=2026-09-23T05:15:12.804Z level=INFO msg="Upload complete. (239ms)" lb # [ 198.322940] niks3-server[1111]: 2026/09/23 05:15:12 INFO Completed upload id=87 lb # [ 198.325044] niks3-server[1111]: 2026/09/23 05:15:12 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo worker1 # [ 198.312962] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:54454 duration_s=0 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=19 outputs=1 worker1 # [ 198.326769] nix-daemon[850]: reaped child process 2188, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 16.45 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]' client # [ 198.629897] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features "[\"gpu\"]". client # Running as unit: gpu.service; invocation ID: ca7eda90d7034d30b84190d4c0e4aee3 client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.04 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2 # [ 198.777667] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:13Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:59586 client # [ 198.835531] nix[1568]: don't know how to build these paths: client # [ 198.836555] nix[1568]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv worker2 # [ 198.791887] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:13Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 198.811813] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:13Z level=debug event=want client=7 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=0 client # [ 198.862361] nix[1568]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip}, waiting up to 30s for a worker worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.10 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) client: waiting for success: systemctl show -p Result gpu | grep -q exit-code lb # [ 200.280703] envoy[908]: conn peer=[2001:db8:1::1]:45426 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27213 tx=12447 ms=16174 client # [ 228.869387] nix[1568]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, the workers offer {benchmark,big-parallel,gccarch-armv8-a,kvm,nixos-test,vip} client # [ 228.883289] nix[1568]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed client # [ 228.915286] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE client # [ 228.923535] systemd[1]: gpu.service: Failed with result 'exit-code'. client # [ 228.927485] systemd[1]: gpu.service: Consumed 198ms CPU time over 30.290s wall clock time, 35.4M memory peak, 5.6K incoming IP traffic, 5.2K outgoing IP traffic. client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.06 seconds) client: must succeed: journalctl -u gpu -o cat client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) (finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 47.27 seconds) subtest: low disk drains a worker and builds go to the other worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill lb # [ 230.277753] envoy[908]: conn peer=[2001:db8:1::1]:35584 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=931 tx=614 ms=30070 worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.19 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free' worker1 # [ 230.834308] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:45Z level=info event=unhealthy reason=min_free worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.10 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2 worker1 # [ 231.768276] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:46Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:54454 client # don't know how to build these paths: client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker1 # [ 231.778160] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:54454 worker2 # [ 231.816429] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:46Z level=debug event=want client=8 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=0 worker2 # [ 231.822085] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:46Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 231.863378] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 231.878734] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59586 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'... worker2 # [ 231.894775] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 231.913491] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 232.086272] nix-daemon[842]: accepted connection from pid 1934, user nix-grpc-daemon (trusted) worker2 # [ 232.107242] nix-daemon[842]: reaped child process 1937, status = succeeded lb # [ 232.125024] niks3-server[1111]: 2026/09/23 05:15:46 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 232.126408] nix-grpc-daemon[1636]: time=2026-09-23T05:15:46.624Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 232.127966] nix-grpc-daemon[1636]: time=2026-09-23T05:15:46.624Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)" lb # [ 232.161652] niks3-server[1111]: 2026/09/23 05:15:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/88/sign worker2 # [ 232.158382] nix-grpc-daemon[1636]: time=2026-09-23T05:15:46.657Z level=INFO msg="Uploading 1 narinfos" lb # [ 232.177054] niks3-server[1111]: 2026/09/23 05:15:46 INFO Signed narinfos id=88 count=1 lb # [ 232.195190] niks3-server[1111]: 2026/09/23 05:15:46 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls lb # [ 232.203449] niks3-server[1111]: 2026/09/23 05:15:46 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst lb # [ 232.223246] niks3-server[1111]: 2026/09/23 05:15:46 INFO Received complete upload request method=POST path=/api/pending_closures/88/complete lb # [ 232.230504] niks3-server[1111]: 2026/09/23 05:15:46 INFO Completed upload id=88 worker2 # [ 232.219405] nix-grpc-daemon[1636]: time=2026-09-23T05:15:46.718Z level=INFO msg="Upload complete. (286ms)" lb # [ 232.233614] niks3-server[1111]: 2026/09/23 05:15:46 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo worker2 # [ 232.229955] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:46Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=0 paths=1 nar_bytes_in=512 worker2 # [ 232.245058] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 232.258179] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 232.265246] nix-daemon[842]: reaped child process 1932, status = succeeded client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051... client # farm-a> worker2: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker2 # [ 232.279089] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-drain worker2 # [ 233.790449] nix-daemon[842]: reaped child process 1941, status = succeeded worker2 # [ 233.861629] nix-daemon[842]: accepted connection from pid 1957, user nix-grpc-daemon (trusted) worker2 # [ 233.883021] nix-daemon[842]: reaped child process 1960, status = succeeded lb # [ 233.891473] niks3-server[1111]: 2026/09/23 05:15:48 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 233.895165] nix-grpc-daemon[1636]: time=2026-09-23T05:15:48.392Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 233.896840] nix-grpc-daemon[1636]: time=2026-09-23T05:15:48.392Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)" lb # [ 233.933827] niks3-server[1111]: 2026/09/23 05:15:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/89/sign worker2 # [ 233.932550] nix-grpc-daemon[1636]: time=2026-09-23T05:15:48.431Z level=INFO msg="Uploading 1 narinfos" lb # [ 233.950108] niks3-server[1111]: 2026/09/23 05:15:48 INFO Signed narinfos id=89 count=1 lb # [ 233.956473] niks3-server[1111]: 2026/09/23 05:15:48 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv lb # [ 233.963401] niks3-server[1111]: 2026/09/23 05:15:48 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst lb # [ 233.971332] niks3-server[1111]: 2026/09/23 05:15:48 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls worker2 # [ 233.966782] nix-grpc-daemon[1636]: time=2026-09-23T05:15:48.466Z level=INFO msg="Upload complete. (275ms)" lb # [ 233.977408] niks3-server[1111]: 2026/09/23 05:15:48 INFO Received complete upload request method=POST path=/api/pending_closures/89/complete lb # [ 233.981460] niks3-server[1111]: 2026/09/23 05:15:48 INFO Completed upload id=89 lb # [ 233.983168] niks3-server[1111]: 2026/09/23 05:15:48 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo worker2 # [ 233.978486] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=20 outputs=1 worker2 # [ 234.000954] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:48Z level=debug event=want client=8 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=0 worker2 # [ 234.006186] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:48Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 234.010915] nix-daemon[842]: reaped child process 1939, status = succeeded worker2 # [ 234.030345] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 234.045332] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:48Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 234.060254] nix-grpc-daemon[1636]: time=2026-09-23T05:15:48.559Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker2 # [ 234.071759] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:48Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'... worker2 # [ 234.088150] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 234.255254] nix-daemon[842]: accepted connection from pid 1966, user nix-grpc-daemon (trusted) worker2 # [ 234.272244] nix-daemon[842]: reaped child process 1969, status = succeeded lb # [ 234.288689] niks3-server[1111]: 2026/09/23 05:15:48 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 234.287230] nix-grpc-daemon[1636]: time=2026-09-23T05:15:48.786Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 234.289008] nix-grpc-daemon[1636]: time=2026-09-23T05:15:48.786Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)" lb # [ 234.316920] niks3-server[1111]: 2026/09/23 05:15:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/90/sign worker2 # [ 234.313477] nix-grpc-daemon[1636]: time=2026-09-23T05:15:48.812Z level=INFO msg="Uploading 1 narinfos" lb # [ 234.330303] niks3-server[1111]: 2026/09/23 05:15:48 INFO Signed narinfos id=90 count=1 lb # [ 234.336741] niks3-server[1111]: 2026/09/23 05:15:48 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls lb # [ 234.349518] niks3-server[1111]: 2026/09/23 05:15:48 INFO Received complete upload request method=POST path=/api/pending_closures/90/complete worker2 # [ 234.344674] nix-grpc-daemon[1636]: time=2026-09-23T05:15:48.844Z level=INFO msg="Upload complete. (238ms)" lb # [ 234.359353] niks3-server[1111]: 2026/09/23 05:15:48 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst lb # [ 234.365300] niks3-server[1111]: 2026/09/23 05:15:48 INFO Completed upload id=90 worker2 # [ 234.356196] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=0 paths=1 nar_bytes_in=632 lb # [ 234.368844] niks3-server[1111]: 2026/09/23 05:15:48 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo worker2 # [ 234.369534] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 234.376586] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 234.381875] nix-daemon[842]: reaped child process 1964, status = succeeded client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051... worker2 # [ 234.394675] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) client # farm-b> worker2: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-drain worker2 # [ 235.300941] nix-daemon[842]: reaped child process 1973, status = succeeded worker2 # [ 235.372317] nix-daemon[842]: accepted connection from pid 1989, user nix-grpc-daemon (trusted) worker2 # [ 235.394649] nix-daemon[842]: reaped child process 1992, status = succeeded lb # [ 235.406541] niks3-server[1111]: 2026/09/23 05:15:49 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 235.404508] nix-grpc-daemon[1636]: time=2026-09-23T05:15:49.903Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 235.406135] nix-grpc-daemon[1636]: time=2026-09-23T05:15:49.903Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)" lb # [ 235.447076] niks3-server[1111]: 2026/09/23 05:15:49 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls worker2 # [ 235.458213] nix-grpc-daemon[1636]: time=2026-09-23T05:15:49.957Z level=INFO msg="Uploading 1 narinfos" lb # [ 235.462267] niks3-server[1111]: 2026/09/23 05:15:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/91/sign lb # [ 235.476387] niks3-server[1111]: 2026/09/23 05:15:49 INFO Signed narinfos id=91 count=1 lb # [ 235.482320] niks3-server[1111]: 2026/09/23 05:15:49 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst lb # [ 235.493229] niks3-server[1111]: 2026/09/23 05:15:49 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv lb # [ 235.502175] niks3-server[1111]: 2026/09/23 05:15:50 INFO Received complete upload request method=POST path=/api/pending_closures/91/complete lb # [ 235.512206] niks3-server[1111]: 2026/09/23 05:15:50 INFO Completed upload id=91 worker2 # [ 235.500537] nix-grpc-daemon[1636]: time=2026-09-23T05:15:49.999Z level=INFO msg="Upload complete. (296ms)" lb # [ 235.514518] niks3-server[1111]: 2026/09/23 05:15:50 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo worker2 # [ 235.511684] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:50Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=21 outputs=1 worker2 # [ 235.532107] nix-daemon[842]: reaped child process 1971, status = succeeded worker2 # [ 235.558874] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:50Z level=debug event=want client=8 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=0 worker2 # [ 235.565735] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:50Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 235.576182] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 235.587747] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 235.601094] nix-grpc-daemon[1636]: time=2026-09-23T05:15:50.100Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'... worker2 # [ 235.612490] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:50Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 235.629534] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 235.783557] nix-daemon[842]: accepted connection from pid 1998, user nix-grpc-daemon (trusted) worker2 # [ 235.802820] nix-daemon[842]: reaped child process 2001, status = succeeded lb # [ 235.817213] niks3-server[1111]: 2026/09/23 05:15:50 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 235.815639] nix-grpc-daemon[1636]: time=2026-09-23T05:15:50.314Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 235.817437] nix-grpc-daemon[1636]: time=2026-09-23T05:15:50.314Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)" lb # [ 235.850340] niks3-server[1111]: 2026/09/23 05:15:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/92/sign worker2 # [ 235.846883] nix-grpc-daemon[1636]: time=2026-09-23T05:15:50.346Z level=INFO msg="Uploading 1 narinfos" lb # [ 235.864529] niks3-server[1111]: 2026/09/23 05:15:50 INFO Signed narinfos id=92 count=1 lb # [ 235.870125] niks3-server[1111]: 2026/09/23 05:15:50 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls worker2 # [ 235.873825] nix-grpc-daemon[1636]: time=2026-09-23T05:15:50.373Z level=INFO msg="Upload complete. (227ms)" lb # [ 235.884739] niks3-server[1111]: 2026/09/23 05:15:50 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst worker2 # [ 235.882708] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:50Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=0 paths=1 nar_bytes_in=760 lb # [ 235.894148] niks3-server[1111]: 2026/09/23 05:15:50 INFO Received complete upload request method=POST path=/api/pending_closures/92/complete lb # [ 235.898880] niks3-server[1111]: 2026/09/23 05:15:50 INFO Completed upload id=92 lb # [ 235.902147] niks3-server[1111]: 2026/09/23 05:15:50 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo worker2 # [ 235.897450] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:50Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 worker2 # [ 235.903897] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker2 # [ 235.909214] nix-daemon[842]: reaped child process 1996, status = succeeded client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051... worker2 # [ 235.920114] nix-daemon[842]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) client # farm-top> worker2: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-drain worker2 # [ 236.812332] nix-daemon[842]: reaped child process 2005, status = succeeded worker2 # [ 236.896235] nix-daemon[842]: accepted connection from pid 2021, user nix-grpc-daemon (trusted) worker2 # [ 236.917681] nix-daemon[842]: reaped child process 2024, status = succeeded lb # [ 236.926777] niks3-server[1111]: 2026/09/23 05:15:51 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 236.926573] nix-grpc-daemon[1636]: time=2026-09-23T05:15:51.425Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 236.930292] nix-grpc-daemon[1636]: time=2026-09-23T05:15:51.425Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)" lb # [ 236.973517] niks3-server[1111]: 2026/09/23 05:15:51 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst worker2 # [ 236.973065] nix-grpc-daemon[1636]: time=2026-09-23T05:15:51.472Z level=INFO msg="Uploading 1 narinfos" lb # [ 236.987278] niks3-server[1111]: 2026/09/23 05:15:51 INFO Received sign narinfos request method=POST path=/api/pending_closures/93/sign lb # [ 236.995832] niks3-server[1111]: 2026/09/23 05:15:51 INFO Signed narinfos id=93 count=1 lb # [ 237.000972] niks3-server[1111]: 2026/09/23 05:15:51 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls lb # [ 237.003883] niks3-server[1111]: 2026/09/23 05:15:51 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv lb # [ 237.013768] niks3-server[1111]: 2026/09/23 05:15:51 INFO Received complete upload request method=POST path=/api/pending_closures/93/complete lb # [ 237.017475] niks3-server[1111]: 2026/09/23 05:15:51 INFO Completed upload id=93 worker2 # [ 237.007267] nix-grpc-daemon[1636]: time=2026-09-23T05:15:51.506Z level=INFO msg="Upload complete. (274ms)" lb # [ 237.020368] niks3-server[1111]: 2026/09/23 05:15:51 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo worker2 # [ 237.021130] nix-grpc-daemon[1559]: ts=2026-09-23T05:15:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:59586 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=22 outputs=1 worker2 # [ 237.038306] nix-daemon[842]: reaped child process 2003, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.pem&client-key=/nix/store/hshhzk1ggy1xxf00y8n24b7qg938ckjz-farm-certs/ci.key' --eval-store auto -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain >&2, in 5.55 seconds) client: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/vr4gwv4y4d4pl04zngqql7lcz1lvgab1-job.nix --argstr tag drain outPath, in 0.19 seconds) worker1: must succeed: rm /nix/.rw-store/fill worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.15 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free' worker1 # [ 237.844687] nix-grpc-daemon[1783]: ts=2026-09-23T05:15:52Z level=info event=healthy reason=min_free worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.10 seconds) (finished: subtest: low disk drains a worker and builds go to the other, in 9.37 seconds) (finished: run the VM test script, in 239.66 seconds) test script finished in 239.91s cleanup kill QemuMachine (pid 51) client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) client # [2026-09-23T05:15:52Z INFO virtiofsd] Client disconnected, shutting down client # [2026-09-23T05:15:53Z INFO virtiofsd] Client disconnected, shutting down client # [2026-09-23T05:15:53Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 49) lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) lb # [2026-09-23T05:15:53Z INFO virtiofsd] Client disconnected, shutting down lb # [2026-09-23T05:15:53Z INFO virtiofsd] Client disconnected, shutting down lb # [2026-09-23T05:15:53Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 50) worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) worker1 # [2026-09-23T05:15:53Z INFO virtiofsd] Client disconnected, shutting down worker1 # [2026-09-23T05:15:53Z INFO virtiofsd] Client disconnected, shutting down worker1 # [2026-09-23T05:15:53Z INFO virtiofsd] Client disconnected, shutting down kill QemuMachine (pid 52) worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/3fl7bdkdxk2k4nssy1d1161isbzn6bsr-python3-3.14.7/bin/python3.14) worker2 # [2026-09-23T05:15:54Z INFO virtiofsd] Client disconnected, shutting down worker2 # [2026-09-23T05:15:54Z INFO virtiofsd] Client disconnected, shutting down worker2 # [2026-09-23T05:15:54Z INFO virtiofsd] Client disconnected, shutting down (finished: cleanup, in 1.28 seconds) additionally exposed symbols: client, lb, worker1, worker2, vlan1, start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh