tribuchet: building on eliza Machine state will be reset. To keep it, pass --keep-machine-state start all VLans (finished: start all VLans, in 0.01 seconds) Test will time out and terminate in 900.0 seconds run the VM test script start all VMs client: starting vm lb: starting vm worker1: starting vm worker2: starting vm worker1 # Disk image does not exist, creating the virtualisation disk image... worker1: QEMU running (pid 51) worker2 # Disk image does not exist, creating the virtualisation disk image... worker2: QEMU running (pid 52) worker1 # Formatting '/build/vm-state-worker1/tmp.dXiobIir6P', fmt=raw size=1073741824 client # Disk image does not exist, creating the virtualisation disk image... lb # Disk image does not exist, creating the virtualisation disk image... lb: QEMU running (pid 49) worker2 # Formatting '/build/vm-state-worker2/tmp.M9cLLboM0W', fmt=raw size=1073741824 client: QEMU running (pid 50) worker1 # mke2fs 1.47.4 (6-Mar-2025) worker2 # mke2fs 1.47.4 (6-Mar-2025) client # Formatting '/build/vm-state-client/tmp.tezcDG5AML', fmt=raw size=1073741824 lb # Formatting '/build/vm-state-lb/tmp.r3reIge0GV', fmt=raw size=1073741824 lb # mke2fs 1.47.4 (6-Mar-2025) worker2 # Discarding device blocks: 0/262144 done worker1 # Discarding device blocks: 0/262144 done (finished: start all VMs, in 0.51 seconds) client # mke2fs 1.47.4 (6-Mar-2025) lb # Discarding device blocks: 0/262144 done worker2 # Creating filesystem with 262144 4k blocks and 65536 inodes worker1 # Creating filesystem with 262144 4k blocks and 65536 inodes lb # Creating filesystem with 262144 4k blocks and 65536 inodes worker2 # Filesystem UUID: 1901d721-ca8c-4062-ae20-6ee1aeb92fa2 worker1 # Filesystem UUID: b7494a4a-0c71-4154-867d-11aa1912d093 lb # Filesystem UUID: 81debc9d-e7d6-4829-8d96-4b8ab9939e27 worker2 # Superblock backups stored on blocks: worker1 # Superblock backups stored on blocks: lb # Superblock backups stored on blocks: worker2 # 32768, 98304, 163840, 229376 worker1 # 32768, 98304, 163840, 229376 lb # 32768, 98304, 163840, 229376 worker2 # worker1 # lb # worker2 # Allocating group tables: 0/8 done client: waiting for unit network-addresses-eth1.service lb # Allocating group tables: 0/8 done worker2 # Writing inode tables: 0/8 done client: waiting for the VM to finish booting lb # Writing inode tables: 0/8 done worker2 # Creating journal (8192 blocks): done client # Discarding device blocks: 0/262144 done worker1 # Allocating group tables: 0/8 done worker2 # Writing superblocks and filesystem accounting information: 0/8 done client # Creating filesystem with 262144 4k blocks and 65536 inodes lb # Creating journal (8192 blocks): done client # Filesystem UUID: bbd434e5-d1a0-43bf-a370-65bcbd9a72f5 lb # Writing superblocks and filesystem accounting information: 0/8 done worker2 # worker1 # Writing inode tables: 0/8 done client # Superblock backups stored on blocks: lb # worker2 # Virtualisation disk image created. worker1 # Creating journal (8192 blocks): done client # 32768, 98304, 163840, 229376 worker1 # Writing superblocks and filesystem accounting information: 0/8 done client # worker1 # client # Allocating group tables: 0/8 done worker1 # Virtualisation disk image created. client # Writing inode tables: 0/8 done lb # Virtualisation disk image created. client # Creating journal (8192 blocks): done client # Writing superblocks and filesystem accounting information: 0/8 done client # client # Virtualisation disk image created. worker1 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker1 # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026 worker1 # [ 0.000000] KASLR enabled worker1 # [ 0.000000] random: crng init done worker1 # [ 0.000000] Machine model: linux,dummy-virt worker1 # [ 0.000000] efi: UEFI not found. worker1 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker1 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker1 # [ 0.000000] Zone ranges: worker1 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] DMA32 empty worker1 # [ 0.000000] Normal empty worker1 # [ 0.000000] Device empty worker1 # [ 0.000000] Movable zone start for each node worker1 # [ 0.000000] Early memory node ranges worker1 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker1 # [ 0.000000] psci: probing for conduit method from DT. worker1 # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker1 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker1 # [ 0.000000] psci: Trusted OS migration not required worker1 # [ 0.000000] psci: SMC Calling Convention v1.1 worker1 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker1 # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296 worker1 # [ 0.000000] Detected PIPT I-cache on CPU0 worker1 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker1 # [ 0.000000] CPU features: detected: GICv3 CPU interface worker2 # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker1 # [ 0.000000] CPU features: detected: Spectre-v4 worker1 # [ 0.000000] CPU features: detected: Spectre-BHB worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker2 # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026 worker2 # [ 0.000000] KASLR enabled worker1 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker2 # [ 0.000000] random: crng init done worker1 # [ 0.000000] alternatives: applying boot alternatives worker2 # [ 0.000000] Machine model: linux,dummy-virt worker2 # [ 0.000000] efi: UEFI not found. worker2 # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker2 # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker2 # [ 0.000000] Zone ranges: worker1 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/hl1ynmh5snhcjq7al2fl9h42d6chq0rx-nixos-system-worker1-test/init regInfo=/nix/store/z498f4bnd0bm1y7j2vivsc2m8rqscg04-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker2 # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] DMA32 empty worker2 # [ 0.000000] Normal empty worker2 # [ 0.000000] Device empty worker2 # [ 0.000000] Movable zone start for each node worker1 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/z498f4bnd0bm1y7j2vivsc2m8rqscg04-closure-info/registration", will be passed to user space. worker2 # [ 0.000000] Early memory node ranges worker1 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker2 # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker2 # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker1 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker2 # [ 0.000000] psci: probing for conduit method from DT. worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB worker2 # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker1 # [ 0.000000] software IO TLB: area num 1. worker2 # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker1 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker2 # [ 0.000000] psci: Trusted OS migration not required worker2 # [ 0.000000] psci: SMC Calling Convention v1.1 worker1 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker1 # [ 0.000000] Fallback order for Node 0: 0 worker2 # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker1 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker2 # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296 worker1 # [ 0.000000] Policy zone: DMA worker2 # [ 0.000000] Detected PIPT I-cache on CPU0 worker1 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off worker1 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 worker2 # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker2 # [ 0.000000] CPU features: detected: GICv3 CPU interface worker1 # [ 0.000000] allocated 3145728 bytes of page_ext worker2 # [ 0.000000] CPU features: detected: Spectre-v4 worker1 # [ 0.000000] ftrace: allocating 74885 entries in 294 pages worker2 # [ 0.000000] CPU features: detected: Spectre-BHB lb # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker1 # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker1 # [ 0.000000] rcu: Hierarchical RCU implementation. worker2 # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker1 # [ 0.000000] rcu: RCU event tracing is enabled. lb # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026 worker2 # [ 0.000000] alternatives: applying boot alternatives lb # [ 0.000000] KASLR enabled worker1 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. lb # [ 0.000000] random: crng init done worker1 # [ 0.000000] Trampoline variant of Tasks RCU enabled. lb # [ 0.000000] Machine model: linux,dummy-virt lb # [ 0.000000] efi: UEFI not found. worker1 # [ 0.000000] Rude variant of Tasks RCU enabled. worker1 # [ 0.000000] Tracing variant of Tasks RCU enabled. lb # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker1 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. lb # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/02j81afdj8xzihnmsy7rp3hq1zfspx7w-nixos-system-worker2-test/init regInfo=/nix/store/b12amaslgxhmmal4fdpicv0pb8fqziw4-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker1 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 lb # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] lb # [ 0.000000] Zone ranges: worker1 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] lb # [ 0.000000] DMA32 empty worker2 # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/b12amaslgxhmmal4fdpicv0pb8fqziw4-closure-info/registration", will be passed to user space. lb # [ 0.000000] Normal empty worker1 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] Device empty worker2 # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes lb # [ 0.000000] Movable zone start for each node lb # [ 0.000000] Early memory node ranges worker1 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker1 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 lb # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] GICv3: 256 SPIs implemented worker2 # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) lb # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000000] GICv3: 0 Extended SPIs implemented worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB worker1 # [ 0.000000] Root IRQ handler: gic_handle_irq lb # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker2 # [ 0.000000] software IO TLB: area num 1. worker1 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI lb # [ 0.000000] psci: probing for conduit method from DT. worker2 # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker1 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 lb # [ 0.000000] psci: PSCIv1.3 detected in firmware. lb # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker2 # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker1 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 lb # [ 0.000000] psci: Trusted OS migration not required worker2 # [ 0.000000] Fallback order for Node 0: 0 worker1 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] lb # [ 0.000000] psci: SMC Calling Convention v1.1 worker2 # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker2 # [ 0.000000] Policy zone: DMA lb # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker2 # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off lb # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296 worker1 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) client # [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0xc00fac40] worker2 # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 lb # [ 0.000000] Detected PIPT I-cache on CPU0 worker1 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker2 # [ 0.000000] allocated 3145728 bytes of page_ext lb # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker2 # [ 0.000000] ftrace: allocating 74885 entries in 294 pages worker1 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 client # [ 0.000000] Linux version 6.18.50 (nixbld@localhost) (gcc (GCC) 15.3.0, GNU ld (GNU Binutils) 2.46) #1-NixOS SMP PREEMPT Mon Sep 7 15:23:00 UTC 2026 client # [ 0.000000] KASLR enabled lb # [ 0.000000] CPU features: detected: GICv3 CPU interface worker2 # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker1 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. client # [ 0.000000] random: crng init done lb # [ 0.000000] CPU features: detected: Spectre-v4 worker2 # [ 0.000000] rcu: Hierarchical RCU implementation. client # [ 0.000000] Machine model: linux,dummy-virt lb # [ 0.000000] CPU features: detected: Spectre-BHB worker2 # [ 0.000000] rcu: RCU event tracing is enabled. client # [ 0.000000] efi: UEFI not found. worker1 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker2 # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. client # [ 0.000000] OF: reserved mem: Reserved memory: No reserved-memory node in the DT worker1 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). lb # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 worker2 # [ 0.000000] Trampoline variant of Tasks RCU enabled. client # [ 0.000000] NUMA: Faking a node at [mem 0x0000000040000000-0x000000009fffffff] worker2 # [ 0.000000] Rude variant of Tasks RCU enabled. lb # [ 0.000000] alternatives: applying boot alternatives worker1 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns client # [ 0.000000] NODE_DATA(0) allocated [mem 0x9fdec700-0x9fdefe7f] worker2 # [ 0.000000] Tracing variant of Tasks RCU enabled. client # [ 0.000000] Zone ranges: worker1 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker2 # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. client # [ 0.000000] DMA [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.000038] arm-pv: using stolen time PV client # [ 0.000000] DMA32 empty worker2 # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 client # [ 0.000000] Normal empty client # [ 0.000000] Device empty worker1 # [ 0.000793] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) client # [ 0.000000] Movable zone start for each node worker2 # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.000989] Console: colour dummy device 80x25 client # [ 0.000000] Early memory node ranges lb # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/mmnw0b9xdzgvlxb4mzaa4gjdldj887b1-nixos-system-lb-test/init regInfo=/nix/store/mmqsdvj4977m1gdgnvij2v3i1inliqqm-closure-info/registration console=ttyAMA0,115200n8 console=tty0 worker1 # [ 0.000998] printk: legacy console [tty0] enabled worker2 # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.000000] node 0: [mem 0x0000000040000000-0x000000009fffffff] client # [ 0.000000] Initmem setup node 0 [mem 0x0000000040000000-0x000000009fffffff] worker1 # [ 0.001203] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) lb # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/mmqsdvj4977m1gdgnvij2v3i1inliqqm-closure-info/registration", will be passed to user space. worker2 # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. client # [ 0.000000] cma: Reserved 32 MiB at 0x000000009c400000 worker1 # [ 0.001210] pid_max: default: 32768 minimum: 301 worker2 # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 lb # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes client # [ 0.000000] psci: probing for conduit method from DT. worker2 # [ 0.000000] GICv3: 256 SPIs implemented worker1 # [ 0.001287] LSM: initializing lsm=capability,landlock,yama,bpf,ima client # [ 0.000000] psci: PSCIv1.3 detected in firmware. worker2 # [ 0.000000] GICv3: 0 Extended SPIs implemented worker1 # [ 0.001507] landlock: Up and running. lb # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker1 # [ 0.001510] Yama: becoming mindful. worker2 # [ 0.000000] Root IRQ handler: gic_handle_irq client # [ 0.000000] psci: Using standard PSCI v0.2 function IDs worker1 # [ 0.002214] LSM support for eBPF active client # [ 0.000000] psci: Trusted OS migration not required worker2 # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI lb # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker2 # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 client # [ 0.000000] psci: SMC Calling Convention v1.1 worker1 # [ 0.002375] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB lb # [ 0.000000] software IO TLB: area num 1. worker2 # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 client # [ 0.000000] smccc: KVM: hypervisor services detected (0x00000000 0x00000000 0x00000000 0x00000003) worker1 # [ 0.002406] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] lb # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB client # [ 0.000000] percpu: Embedded 76 pages/cpu s186648 r8192 d116456 u311296 worker1 # [ 0.003641] cacheinfo: Unable to detect cache hierarchy for CPU 0 client # [ 0.000000] Detected PIPT I-cache on CPU0 lb # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) worker1 # [ 0.004453] rcu: Hierarchical SRCU implementation. lb # [ 0.000000] Fallback order for Node 0: 0 worker1 # [ 0.004459] rcu: Max phase no-delay instances is 1000. client # [ 0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm) worker1 # [ 0.005724] fsl-mc MSI: its@8080000 domain created lb # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 worker2 # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) client # [ 0.000000] CPU features: detected: GICv3 CPU interface lb # [ 0.000000] Policy zone: DMA worker1 # [ 0.005818] EFI services will not be available. worker2 # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 client # [ 0.000000] CPU features: detected: Spectre-v4 worker1 # [ 0.005897] smp: Bringing up secondary CPUs ... lb # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off client # [ 0.000000] CPU features: detected: Spectre-BHB worker1 # [ 0.005907] smp: Brought up 1 node, 1 CPU worker2 # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 lb # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 client # [ 0.000000] CPU features: detected: AmpereOne erratum AC03_CPU_38 worker1 # [ 0.005910] SMP: Total of 1 processors activated. worker2 # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. lb # [ 0.000000] allocated 3145728 bytes of page_ext worker1 # [ 0.005917] CPU: All CPU(s) started at EL1 client # [ 0.000000] CPU features: detected: AmpereOne erratum AC04_CPU_23 lb # [ 0.000000] ftrace: allocating 74885 entries in 294 pages worker1 # [ 0.005930] CPU features: detected: Branch Target Identification client # [ 0.000000] alternatives: applying boot alternatives worker2 # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns lb # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker1 # [ 0.005935] CPU features: detected: ARMv8.4 Translation Table Level worker2 # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). lb # [ 0.000000] rcu: Hierarchical RCU implementation. lb # [ 0.000000] rcu: RCU event tracing is enabled. worker1 # [ 0.005939] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker2 # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns lb # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker1 # [ 0.005942] CPU features: detected: Data cache clean to the PoU not required for I/D coherence lb # [ 0.000000] Trampoline variant of Tasks RCU enabled. worker2 # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns lb # [ 0.000000] Rude variant of Tasks RCU enabled. worker1 # [ 0.005945] CPU features: detected: Common not Private translations worker2 # [ 0.000033] arm-pv: using stolen time PV client # [ 0.000000] Kernel command line: console=ttyAMA0 console=tty0 panic=1 boot.panic_on_fail clocksource=acpi_pm root=fstab loglevel=7 net.ifnames=0 lsm=landlock,yama,bpf init=/nix/store/k6qpk00c7qa52lid1b7zqxvp2lykb521-nixos-system-client-test/init regInfo=/nix/store/55zbzi2q0wlj27byxrsggqmmpdy1hwi9-closure-info/registration console=ttyAMA0,115200n8 console=tty0 lb # [ 0.000000] Tracing variant of Tasks RCU enabled. worker1 # [ 0.005949] CPU features: detected: CRC32 instructions worker2 # [ 0.000585] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) lb # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker1 # [ 0.005951] CPU features: detected: Data cache clean to Point of Deep Persistence worker2 # [ 0.000785] Console: colour dummy device 80x25 client # [ 0.000000] Unknown kernel command line parameters "regInfo=/nix/store/55zbzi2q0wlj27byxrsggqmmpdy1hwi9-closure-info/registration", will be passed to user space. lb # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 worker2 # [ 0.000793] printk: legacy console [tty0] enabled worker1 # [ 0.005955] CPU features: detected: Data cache clean to Point of Persistence client # [ 0.000000] printk: log buffer data + meta data: 131072 + 458752 = 589824 bytes worker1 # [ 0.005958] CPU features: detected: Data independent timing control (DIT) lb # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.000991] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) client # [ 0.000000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) worker1 # [ 0.005962] CPU features: detected: E0PD worker2 # [ 0.000998] pid_max: default: 32768 minimum: 301 lb # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.005964] CPU features: detected: Enhanced Counter Virtualization client # [ 0.000000] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes, linear) worker2 # [ 0.001079] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker2 # [ 0.001293] landlock: Up and running. worker1 # [ 0.005967] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 1MB lb # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.001296] Yama: becoming mindful. client # [ 0.000000] software IO TLB: area num 1. lb # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker1 # [ 0.005970] CPU features: detected: Enhanced Virtualization Traps worker2 # [ 0.001872] LSM support for eBPF active lb # [ 0.000000] GICv3: 256 SPIs implemented client # [ 0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 2MB worker1 # [ 0.005973] CPU features: detected: Fine Grained Traps worker2 # [ 0.002024] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) lb # [ 0.000000] GICv3: 0 Extended SPIs implemented client # [ 0.000000] software IO TLB: mapped [mem 0x000000009c000000-0x000000009c200000] (2MB) lb # [ 0.000000] Root IRQ handler: gic_handle_irq worker1 # [ 0.005978] CPU features: detected: Generic authentication (architected QARMA5 algorithm) client # [ 0.000000] Fallback order for Node 0: 0 worker2 # [ 0.002056] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) lb # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI worker1 # [ 0.005983] CPU features: detected: RCpc load-acquire (LDAPR) worker2 # [ 0.003194] cacheinfo: Unable to detect cache hierarchy for CPU 0 client # [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 393216 lb # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 worker1 # [ 0.005986] CPU features: detected: LSE atomic instructions client # [ 0.000000] Policy zone: DMA worker2 # [ 0.003959] rcu: Hierarchical SRCU implementation. lb # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 worker1 # [ 0.005989] CPU features: detected: Privileged Access Never worker2 # [ 0.003963] rcu: Max phase no-delay instances is 1000. client # [ 0.000000] mem auto-init: stack:all(zero), heap alloc:on, heap free:off lb # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] worker1 # [ 0.005992] CPU features: detected: PMUv3 worker2 # [ 0.005271] fsl-mc MSI: its@8080000 domain created client # [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 worker1 # [ 0.005994] CPU features: detected: RAS Extension Support worker2 # [ 0.005367] EFI services will not be available. lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449f0000 (indirect, esz 8, psz 64K, shr 1) client # [ 0.000000] allocated 3145728 bytes of page_ext worker1 # [ 0.005997] CPU features: detected: RASv1p1 Extension Support worker2 # [ 0.005446] smp: Bringing up secondary CPUs ... client # [ 0.000000] ftrace: allocating 74885 entries in 294 pages worker2 # [ 0.005456] smp: Brought up 1 node, 1 CPU worker1 # [ 0.006000] CPU features: detected: Random Number Generator lb # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @44a00000 (flat, esz 8, psz 64K, shr 1) client # [ 0.000000] ftrace: allocated 294 pages with 4 groups worker2 # [ 0.005460] SMP: Total of 1 processors activated. worker1 # [ 0.006002] CPU features: detected: Speculation barrier (SB) client # [ 0.000000] rcu: Hierarchical RCU implementation. lb # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 worker2 # [ 0.005466] CPU: All CPU(s) started at EL1 worker1 # [ 0.006005] CPU features: detected: Stage-2 Force Write-Back client # [ 0.000000] rcu: RCU event tracing is enabled. worker2 # [ 0.005478] CPU features: detected: Branch Target Identification lb # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker1 # [ 0.006008] CPU features: detected: TLB range maintenance instructions client # [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=384 to nr_cpu_ids=1. worker2 # [ 0.005483] CPU features: detected: ARMv8.4 Translation Table Level lb # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. client # [ 0.000000] Trampoline variant of Tasks RCU enabled. worker1 # [ 0.006013] CPU features: detected: Speculative Store Bypassing Safe (SSBS) client # [ 0.000000] Rude variant of Tasks RCU enabled. worker2 # [ 0.005487] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker1 # [ 0.006057] alternatives: applying system-wide alternatives lb # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns client # [ 0.000000] Tracing variant of Tasks RCU enabled. worker1 # [ 0.009268] CPU features: detected: BBM Level 2 without TLB conflict abort lb # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker2 # [ 0.005490] CPU features: detected: Data cache clean to the PoU not required for I/D coherence client # [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. worker2 # [ 0.005494] CPU features: detected: Common not Private translations client # [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1 lb # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker2 # [ 0.005497] CPU features: detected: CRC32 instructions worker1 # [ 0.009509] Memory: 1406508K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123772K reserved, 32768K cma-reserved) worker1 # [ 0.009981] devtmpfs: initialized client # [ 0.000000] RCU Tasks: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. lb # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker2 # [ 0.005500] CPU features: detected: Data cache clean to Point of Deep Persistence lb # [ 0.000033] arm-pv: using stolen time PV worker1 # [ 0.011885] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) worker2 # [ 0.005504] CPU features: detected: Data cache clean to Point of Persistence client # [ 0.000000] RCU Tasks Rude: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker1 # [ 0.011911] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). worker2 # [ 0.005507] CPU features: detected: Data independent timing control (DIT) lb # [ 0.000484] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) client # [ 0.000000] RCU Tasks Trace: Setting shift to 0 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=1. worker2 # [ 0.005510] CPU features: detected: E0PD lb # [ 0.000840] Console: colour dummy device 80x25 worker1 # [ 0.012118] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL lb # [ 0.000848] printk: legacy console [tty0] enabled client # [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 worker2 # [ 0.005513] CPU features: detected: Enhanced Counter Virtualization worker1 # [ 0.012170] 0 pages in range for non-PLT usage client # [ 0.000000] GICv3: 256 SPIs implemented worker1 # [ 0.012171] 508288 pages in range for PLT usage worker2 # [ 0.005516] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) client # [ 0.000000] GICv3: 0 Extended SPIs implemented lb # [ 0.001051] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker1 # [ 0.012314] pinctrl core: initialized pinctrl subsystem client # [ 0.000000] Root IRQ handler: gic_handle_irq worker2 # [ 0.005520] CPU features: detected: Enhanced Virtualization Traps lb # [ 0.001058] pid_max: default: 32768 minimum: 301 worker1 # [ 0.013160] DMI not present or invalid. client # [ 0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI worker2 # [ 0.005523] CPU features: detected: Fine Grained Traps lb # [ 0.001161] LSM: initializing lsm=capability,landlock,yama,bpf,ima worker1 # [ 0.016741] NET: Registered PF_NETLINK/PF_ROUTE protocol family client # [ 0.000000] GICv3: GICD_CTLR.DS=1, SCR_EL3.FIQ=0 lb # [ 0.001356] landlock: Up and running. worker2 # [ 0.005528] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker1 # [ 0.019069] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations lb # [ 0.001359] Yama: becoming mindful. client # [ 0.000000] GICv3: CPU0: found redistributor 0 region 0:0x00000000080a0000 lb # [ 0.001895] LSM support for eBPF active worker2 # [ 0.005533] CPU features: detected: RCpc load-acquire (LDAPR) client # [ 0.000000] ITS [mem 0x08080000-0x0809ffff] worker1 # [ 0.019266] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations worker2 # [ 0.005536] CPU features: detected: LSE atomic instructions lb # [ 0.002057] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Devices @449e0000 (indirect, esz 8, psz 64K, shr 1) worker2 # [ 0.005540] CPU features: detected: Privileged Access Never worker1 # [ 0.019487] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations lb # [ 0.002087] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.005542] CPU features: detected: PMUv3 worker1 # [ 0.019514] audit: initializing netlink subsys (disabled) client # [ 0.000000] ITS@0x0000000008080000: allocated 8192 Interrupt Collections @449f0000 (flat, esz 8, psz 64K, shr 1) worker2 # [ 0.005545] CPU features: detected: RAS Extension Support lb # [ 0.003276] cacheinfo: Unable to detect cache hierarchy for CPU 0 worker1 # [ 0.020364] thermal_sys: Registered thermal governor 'fair_share' client # [ 0.000000] GICv3: using LPI property table @0x0000000044a10000 lb # [ 0.004032] rcu: Hierarchical SRCU implementation. worker2 # [ 0.005547] CPU features: detected: RASv1p1 Extension Support worker1 # [ 0.020367] thermal_sys: Registered thermal governor 'bang_bang' lb # [ 0.004036] rcu: Max phase no-delay instances is 1000. worker2 # [ 0.005550] CPU features: detected: Random Number Generator client # [ 0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000044a20000 worker1 # [ 0.020371] thermal_sys: Registered thermal governor 'step_wise' lb # [ 0.005293] fsl-mc MSI: its@8080000 domain created worker2 # [ 0.005553] CPU features: detected: Speculation barrier (SB) client # [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. lb # [ 0.005393] EFI services will not be available. worker1 # [ 0.020376] thermal_sys: Registered thermal governor 'user_space' worker2 # [ 0.005556] CPU features: detected: Stage-2 Force Write-Back lb # [ 0.005473] smp: Bringing up secondary CPUs ... worker2 # [ 0.005559] CPU features: detected: TLB range maintenance instructions client # [ 0.000000] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns lb # [ 0.005482] smp: Brought up 1 node, 1 CPU lb # [ 0.005485] SMP: Total of 1 processors activated. worker1 # [ 0.020379] thermal_sys: Registered thermal governor 'power_allocator' client # [ 0.000000] arch_timer: cp15 timer running at 1000.00MHz (virt). worker2 # [ 0.005564] CPU features: detected: Speculative Store Bypassing Safe (SSBS) lb # [ 0.005491] CPU: All CPU(s) started at EL1 worker2 # [ 0.005607] alternatives: applying system-wide alternatives worker1 # [ 0.020418] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 lb # [ 0.005503] CPU features: detected: Branch Target Identification client # [ 0.000000] clocksource: arch_sys_counter: mask: 0x1fffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns worker1 # [ 0.020427] cpuidle: using governor ladder worker2 # [ 0.008813] CPU features: detected: BBM Level 2 without TLB conflict abort lb # [ 0.005508] CPU features: detected: ARMv8.4 Translation Table Level worker1 # [ 0.020433] cpuidle: using governor menu client # [ 0.000000] sched_clock: 61 bits at 1000MHz, resolution 1ns, wraps every 4398046511103ns worker1 # [ 0.020643] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. client # [ 0.000035] arm-pv: using stolen time PV lb # [ 0.005511] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker2 # [ 0.009011] Memory: 1406716K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123772K reserved, 32768K cma-reserved) worker1 # [ 0.020661] ASID allocator initialised with 65536 entries worker2 # [ 0.009465] devtmpfs: initialized lb # [ 0.005515] CPU features: detected: Data cache clean to the PoU not required for I/D coherence worker1 # [ 0.021944] Serial: AMBA PL011 UART driver client # [ 0.000413] kfence: initialized - using 2097152 bytes for 255 objects at 0x(____ptrval____)-0x(____ptrval____) worker2 # [ 0.011324] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) client # [ 0.000618] Console: colour dummy device 80x25 lb # [ 0.005518] CPU features: detected: Common not Private translations worker1 # [ 0.027395] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 client # [ 0.000627] printk: legacy console [tty0] enabled lb # [ 0.005522] CPU features: detected: CRC32 instructions worker1 # [ 0.027564] printk: console [ttyAMA0] enabled worker2 # [ 0.011346] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). lb # [ 0.005525] CPU features: detected: Data cache clean to Point of Deep Persistence worker2 # [ 0.011554] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL client # [ 0.000817] Calibrating delay loop (skipped), value calculated using timer frequency.. 2000.00 BogoMIPS (lpj=4000000) worker2 # [ 0.011559] 0 pages in range for non-PLT usage client # [ 0.000824] pid_max: default: 32768 minimum: 301 worker2 # [ 0.011560] 508288 pages in range for PLT usage client # [ 0.000896] LSM: initializing lsm=capability,landlock,yama,bpf,ima lb # [ 0.005529] CPU features: detected: Data cache clean to Point of Persistence worker2 # [ 0.011665] pinctrl core: initialized pinctrl subsystem client # [ 0.001057] landlock: Up and running. worker2 # [ 0.012465] DMI not present or invalid. client # [ 0.001059] Yama: becoming mindful. lb # [ 0.005532] CPU features: detected: Data independent timing control (DIT) client # [ 0.001516] LSM support for eBPF active lb # [ 0.005535] CPU features: detected: E0PD worker2 # [ 0.015619] NET: Registered PF_NETLINK/PF_ROUTE protocol family client # [ 0.001670] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) lb # [ 0.005538] CPU features: detected: Enhanced Counter Virtualization worker2 # [ 0.018105] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations lb # [ 0.005540] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) client # [ 0.001702] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) worker2 # [ 0.018316] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations client # [ 0.002808] cacheinfo: Unable to detect cache hierarchy for CPU 0 lb # [ 0.005544] CPU features: detected: Enhanced Virtualization Traps worker2 # [ 0.018532] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations lb # [ 0.005547] CPU features: detected: Fine Grained Traps client # [ 0.003621] rcu: Hierarchical SRCU implementation. worker1 # [ 0.147155] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages worker2 # [ 0.018567] audit: initializing netlink subsys (disabled) client # [ 0.003626] rcu: Max phase no-delay instances is 1000. lb # [ 0.005550] CPU features: detected: Generic authentication (architected QARMA5 algorithm) worker1 # [ 0.147178] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker2 # [ 0.019214] thermal_sys: Registered thermal governor 'fair_share' client # [ 0.004850] fsl-mc MSI: its@8080000 domain created lb # [ 0.005556] CPU features: detected: RCpc load-acquire (LDAPR) client # [ 0.004965] EFI services will not be available. worker2 # [ 0.019216] thermal_sys: Registered thermal governor 'bang_bang' worker1 # [ 0.147183] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages lb # [ 0.005559] CPU features: detected: LSE atomic instructions client # [ 0.005054] smp: Bringing up secondary CPUs ... worker2 # [ 0.019220] thermal_sys: Registered thermal governor 'step_wise' worker1 # [ 0.147188] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page client # [ 0.005064] smp: Brought up 1 node, 1 CPU lb # [ 0.005562] CPU features: detected: Privileged Access Never worker2 # [ 0.019226] thermal_sys: Registered thermal governor 'user_space' client # [ 0.005068] SMP: Total of 1 processors activated. worker1 # [ 0.147192] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages lb # [ 0.005564] CPU features: detected: PMUv3 client # [ 0.005074] CPU: All CPU(s) started at EL1 lb # [ 0.005567] CPU features: detected: RAS Extension Support worker1 # [ 0.147196] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page client # [ 0.005087] CPU features: detected: Branch Target Identification lb # [ 0.005570] CPU features: detected: RASv1p1 Extension Support worker2 # [ 0.019229] thermal_sys: Registered thermal governor 'power_allocator' worker1 # [ 0.147201] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages client # [ 0.005092] CPU features: detected: ARMv8.4 Translation Table Level lb # [ 0.005572] CPU features: detected: Random Number Generator worker1 # [ 0.147205] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker2 # [ 0.019256] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 lb # [ 0.005575] CPU features: detected: Speculation barrier (SB) client # [ 0.005095] CPU features: detected: Instruction cache invalidation not required for I/D coherence worker2 # [ 0.019265] cpuidle: using governor ladder lb # [ 0.005578] CPU features: detected: Stage-2 Force Write-Back worker2 # [ 0.019271] cpuidle: using governor menu client # [ 0.005098] CPU features: detected: Data cache clean to the PoU not required for I/D coherence lb # [ 0.005581] CPU features: detected: TLB range maintenance instructions worker2 # [ 0.019475] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. worker1 # [ 0.154534] fbcon: Taking over console client # [ 0.005102] CPU features: detected: Common not Private translations worker2 # [ 0.019493] ASID allocator initialised with 65536 entries lb # [ 0.005585] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker1 # [ 0.154554] ACPI: Interpreter disabled. client # [ 0.005105] CPU features: detected: CRC32 instructions worker2 # [ 0.020729] Serial: AMBA PL011 UART driver lb # [ 0.005627] alternatives: applying system-wide alternatives client # [ 0.005108] CPU features: detected: Data cache clean to Point of Deep Persistence worker2 # [ 0.026177] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 lb # [ 0.008866] CPU features: detected: BBM Level 2 without TLB conflict abort worker2 # [ 0.026351] printk: console [ttyAMA0] enabled client # [ 0.005112] CPU features: detected: Data cache clean to Point of Persistence client # [ 0.005115] CPU features: detected: Data independent timing control (DIT) lb # [ 0.009050] Memory: 1406508K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123764K reserved, 32768K cma-reserved) client # [ 0.005118] CPU features: detected: E0PD lb # [ 0.009508] devtmpfs: initialized client # [ 0.005121] CPU features: detected: Enhanced Counter Virtualization lb # [ 0.011360] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) client # [ 0.005124] CPU features: detected: Enhanced Counter Virtualization (CNTPOFF) lb # [ 0.011387] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). client # [ 0.005128] CPU features: detected: Enhanced Virtualization Traps lb # [ 0.011593] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL client # [ 0.005131] CPU features: detected: Fine Grained Traps lb # [ 0.011599] 0 pages in range for non-PLT usage client # [ 0.005135] CPU features: detected: Generic authentication (architected QARMA5 algorithm) lb # [ 0.011600] 508288 pages in range for PLT usage client # [ 0.005140] CPU features: detected: RCpc load-acquire (LDAPR) lb # [ 0.011733] pinctrl core: initialized pinctrl subsystem lb # [ 0.012514] DMI not present or invalid. client # [ 0.005143] CPU features: detected: LSE atomic instructions worker1 # [ 0.163598] iommu: Default domain type: Translated worker2 # [ 0.143879] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages lb # [ 0.015655] NET: Registered PF_NETLINK/PF_ROUTE protocol family client # [ 0.005146] CPU features: detected: Privileged Access Never worker1 # [ 0.163610] iommu: DMA domain TLB invalidation policy: strict mode worker2 # [ 0.143902] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page client # [ 0.005149] CPU features: detected: PMUv3 worker1 # [ 0.163967] SCSI subsystem initialized lb # [ 0.018043] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations worker2 # [ 0.143908] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages lb # [ 0.018232] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations worker2 # [ 0.143912] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page worker1 # [ 0.165934] usbcore: registered new interface driver usbfs client # [ 0.005152] CPU features: detected: RAS Extension Support worker1 # [ 0.165965] usbcore: registered new interface driver hub lb # [ 0.018447] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker2 # [ 0.143917] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages client # [ 0.005155] CPU features: detected: RASv1p1 Extension Support worker1 # [ 0.165983] usbcore: registered new device driver usb lb # [ 0.018475] audit: initializing netlink subsys (disabled) client # [ 0.005157] CPU features: detected: Random Number Generator worker2 # [ 0.143921] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page worker1 # [ 0.166265] pps_core: LinuxPPS API ver. 1 registered lb # [ 0.019089] thermal_sys: Registered thermal governor 'fair_share' client # [ 0.005160] CPU features: detected: Speculation barrier (SB) worker2 # [ 0.143925] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages lb # [ 0.019091] thermal_sys: Registered thermal governor 'bang_bang' client # [ 0.005163] CPU features: detected: Stage-2 Force Write-Back worker1 # [ 0.166271] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti worker2 # [ 0.143930] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page worker1 # [ 0.166282] PTP clock support registered lb # [ 0.019094] thermal_sys: Registered thermal governor 'step_wise' client # [ 0.005166] CPU features: detected: TLB range maintenance instructions worker1 # [ 0.166331] EDAC MC: Ver: 3.0.0 lb # [ 0.019100] thermal_sys: Registered thermal governor 'user_space' client # [ 0.005171] CPU features: detected: Speculative Store Bypassing Safe (SSBS) worker1 # [ 0.170859] scmi_core: SCMI protocol bus registered lb # [ 0.019103] thermal_sys: Registered thermal governor 'power_allocator' client # [ 0.005217] alternatives: applying system-wide alternatives worker2 # [ 0.151287] fbcon: Taking over console worker2 # [ 0.151307] ACPI: Interpreter disabled. worker1 # [ 0.171848] FPGA manager framework client # [ 0.008407] CPU features: detected: BBM Level 2 without TLB conflict abort lb # [ 0.019130] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 worker1 # [ 0.172871] vgaarb: loaded lb # [ 0.019139] cpuidle: using governor ladder lb # [ 0.019145] cpuidle: using governor menu worker1 # [ 0.173548] clocksource: Switched to clocksource arch_sys_counter client # [ 0.008583] Memory: 1406516K/1572864K available (24448K kernel code, 7090K rwdata, 26348K rodata, 4736K init, 1107K bss, 123784K reserved, 32768K cma-reserved) lb # [ 0.019343] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. client # [ 0.009060] devtmpfs: initialized lb # [ 0.019362] ASID allocator initialised with 65536 entries lb # [ 0.020560] Serial: AMBA PL011 UART driver client # [ 0.010958] posixtimers hash table entries: 512 (order: 1, 8192 bytes, linear) lb # [ 0.026051] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 client # [ 0.010982] futex hash table entries: 256 (16384 bytes on 1 NUMA nodes, total 16 KiB, linear). lb # [ 0.026242] printk: console [ttyAMA0] enabled client # [ 0.011180] 2G module region forced by RANDOMIZE_MODULE_REGION_FULL client # [ 0.011185] 0 pages in range for non-PLT usage client # [ 0.011186] 508288 pages in range for PLT usage client # [ 0.011294] pinctrl core: initialized pinctrl subsystem client # [ 0.012055] DMI not present or invalid. worker1 # [ 0.179779] VFS: Disk quotas dquot_6.6.0 client # [ 0.015429] NET: Registered PF_NETLINK/PF_ROUTE protocol family worker2 # [ 0.160379] iommu: Default domain type: Translated worker1 # [ 0.179842] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) client # [ 0.017814] DMA: preallocated 256 KiB GFP_KERNEL pool for atomic allocations worker2 # [ 0.160393] iommu: DMA domain TLB invalidation policy: strict mode worker1 # [ 0.181572] netfs: FS-Cache loaded client # [ 0.018008] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations worker2 # [ 0.160766] SCSI subsystem initialized worker1 # [ 0.181691] pnp: PnP ACPI: disabled client # [ 0.018226] DMA: preallocated 256 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations worker2 # [ 0.162765] usbcore: registered new interface driver usbfs client # [ 0.018254] audit: initializing netlink subsys (disabled) worker2 # [ 0.162796] usbcore: registered new interface driver hub client # [ 0.018882] thermal_sys: Registered thermal governor 'fair_share' worker2 # [ 0.162814] usbcore: registered new device driver usb client # [ 0.018884] thermal_sys: Registered thermal governor 'bang_bang' worker2 # [ 0.163084] pps_core: LinuxPPS API ver. 1 registered client # [ 0.018888] thermal_sys: Registered thermal governor 'step_wise' worker2 # [ 0.163090] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti client # [ 0.018894] thermal_sys: Registered thermal governor 'user_space' lb # [ 0.146831] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages worker2 # [ 0.163100] PTP clock support registered client # [ 0.018897] thermal_sys: Registered thermal governor 'power_allocator' worker2 # [ 0.163148] EDAC MC: Ver: 3.0.0 lb # [ 0.146854] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page lb # [ 0.146859] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages client # [ 0.018932] audit: type=2000 audit(0.016:1): state=initialized audit_enabled=0 res=1 worker2 # [ 0.167791] scmi_core: SCMI protocol bus registered client # [ 0.018940] cpuidle: using governor ladder lb # [ 0.146864] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page client # [ 0.018946] cpuidle: using governor menu worker2 # [ 0.168832] FPGA manager framework lb # [ 0.146868] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages worker2 # [ 0.169834] vgaarb: loaded client # [ 0.019151] hw-breakpoint: found 16 breakpoint and 4 watchpoint registers. worker1 # [ 0.185586] NET: Registered PF_INET protocol family lb # [ 0.146872] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page client # [ 0.019168] ASID allocator initialised with 65536 entries worker2 # [ 0.170481] clocksource: Switched to clocksource arch_sys_counter worker1 # [ 0.185838] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) client # [ 0.020394] Serial: AMBA PL011 UART driver lb # [ 0.146877] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages lb # [ 0.146881] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page client # [ 0.025669] 9000000.pl011: ttyAMA0 at MMIO 0x9000000 (irq = 13, base_baud = 0) is a PL011 rev1 client # [ 0.025826] printk: console [ttyAMA0] enabled lb # [ 0.154394] fbcon: Taking over console lb # [ 0.154412] ACPI: Interpreter disabled. lb # [ 0.156340] iommu: Default domain type: Translated lb # [ 0.156350] iommu: DMA domain TLB invalidation policy: strict mode worker2 # [ 0.176614] VFS: Disk quotas dquot_6.6.0 lb # [ 0.158082] SCSI subsystem initialized worker2 # [ 0.176672] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) worker2 # [ 0.178382] netfs: FS-Cache loaded worker2 # [ 0.178958] pnp: PnP ACPI: disabled client # [ 0.145582] HugeTLB: registered 1.00 GiB page size, pre-allocated 0 pages client # [ 0.145607] HugeTLB: 0 KiB vmemmap can be freed for a 1.00 GiB page worker2 # [ 0.182469] NET: Registered PF_INET protocol family client # [ 0.145613] HugeTLB: registered 32.0 MiB page size, pre-allocated 0 pages lb # [ 0.163180] usbcore: registered new interface driver usbfs lb # [ 0.163213] usbcore: registered new interface driver hub client # [ 0.145617] HugeTLB: 0 KiB vmemmap can be freed for a 32.0 MiB page lb # [ 0.163239] usbcore: registered new device driver usb client # [ 0.145621] HugeTLB: registered 2.00 MiB page size, pre-allocated 0 pages lb # [ 0.163536] pps_core: LinuxPPS API ver. 1 registered client # [ 0.145625] HugeTLB: 0 KiB vmemmap can be freed for a 2.00 MiB page lb # [ 0.163542] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti client # [ 0.145630] HugeTLB: registered 64.0 KiB page size, pre-allocated 0 pages lb # [ 0.163553] PTP clock support registered worker2 # [ 0.183364] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) client # [ 0.145634] HugeTLB: 0 KiB vmemmap can be freed for a 64.0 KiB page lb # [ 0.163602] EDAC MC: Ver: 3.0.0 lb # [ 0.168252] scmi_core: SCMI protocol bus registered lb # [ 0.169242] FPGA manager framework client # [ 0.153170] fbcon: Taking over console lb # [ 0.170222] vgaarb: loaded client # [ 0.153189] ACPI: Interpreter disabled. lb # [ 0.170864] clocksource: Switched to clocksource arch_sys_counter lb # [ 0.175212] VFS: Disk quotas dquot_6.6.0 lb # [ 0.175250] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) client # [ 0.161867] iommu: Default domain type: Translated lb # [ 0.179083] netfs: FS-Cache loaded client # [ 0.161882] iommu: DMA domain TLB invalidation policy: strict mode lb # [ 0.179222] pnp: PnP ACPI: disabled client # [ 0.162327] SCSI subsystem initialized client # [ 0.164329] usbcore: registered new interface driver usbfs client # [ 0.164362] usbcore: registered new interface driver hub client # [ 0.164381] usbcore: registered new device driver usb client # [ 0.164708] pps_core: LinuxPPS API ver. 1 registered client # [ 0.164715] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti client # [ 0.164725] PTP clock support registered client # [ 0.164784] EDAC MC: Ver: 3.0.0 client # [ 0.169467] scmi_core: SCMI protocol bus registered lb # [ 0.183211] NET: Registered PF_INET protocol family client # [ 0.170475] FPGA manager framework worker1 # [ 0.217855] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) client # [ 0.171457] vgaarb: loaded lb # [ 0.183481] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker1 # [ 0.217916] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) client # [ 0.172094] clocksource: Switched to clocksource arch_sys_counter worker1 # [ 0.217952] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) worker1 # [ 0.218042] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) worker1 # [ 0.218186] TCP: Hash tables configured (established 16384 bind 16384) worker1 # [ 0.218283] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) worker1 # [ 0.218323] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.176715] VFS: Disk quotas dquot_6.6.0 worker1 # [ 0.218362] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.176778] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) worker1 # [ 0.218462] NET: Registered PF_UNIX/PF_LOCAL protocol family worker1 # [ 0.218485] NET: Registered PF_XDP protocol family client # [ 0.178499] netfs: FS-Cache loaded worker1 # [ 0.218504] PCI: CLS 0 bytes, default 64 client # [ 0.178641] pnp: PnP ACPI: disabled worker1 # [ 0.218767] Trying to unpack rootfs image as initramfs... worker2 # [ 0.214463] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) client # [ 0.182498] NET: Registered PF_INET protocol family client # [ 0.182769] IP idents hash table entries: 32768 (order: 6, 262144 bytes, linear) worker2 # [ 0.215748] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) worker2 # [ 0.215799] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) worker2 # [ 0.215892] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) worker2 # [ 0.216034] TCP: Hash tables configured (established 16384 bind 16384) worker1 # [ 0.241605] kvm [1]: HYP mode not available worker2 # [ 0.216134] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) worker2 # [ 0.216174] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) worker2 # [ 0.216213] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) worker2 # [ 0.216370] NET: Registered PF_UNIX/PF_LOCAL protocol family worker2 # [ 0.216399] NET: Registered PF_XDP protocol family worker2 # [ 0.216420] PCI: CLS 0 bytes, default 64 worker2 # [ 0.216673] Trying to unpack rootfs image as initramfs... lb # [ 0.216081] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) lb # [ 0.216141] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) lb # [ 0.216173] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) lb # [ 0.216269] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) lb # [ 0.216412] TCP: Hash tables configured (established 16384 bind 16384) worker2 # [ 0.238562] kvm [1]: HYP mode not available lb # [ 0.216510] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) lb # [ 0.216549] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.216587] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) lb # [ 0.216697] NET: Registered PF_UNIX/PF_LOCAL protocol family lb # [ 0.216728] NET: Registered PF_XDP protocol family lb # [ 0.216745] PCI: CLS 0 bytes, default 64 lb # [ 0.216998] Trying to unpack rootfs image as initramfs... client # [ 0.214450] tcp_listen_portaddr_hash hash table entries: 1024 (order: 2, 16384 bytes, linear) client # [ 0.214515] Table-perturb hash table entries: 65536 (order: 6, 262144 bytes, linear) client # [ 0.214552] TCP established hash table entries: 16384 (order: 5, 131072 bytes, linear) lb # [ 0.232576] kvm [1]: HYP mode not available client # [ 0.214646] TCP bind hash table entries: 16384 (order: 7, 524288 bytes, linear) client # [ 0.214791] TCP: Hash tables configured (established 16384 bind 16384) client # [ 0.214887] MPTCP token hash table entries: 2048 (order: 4, 49152 bytes, linear) client # [ 0.214930] UDP hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.214968] UDP-Lite hash table entries: 1024 (order: 4, 65536 bytes, linear) client # [ 0.215088] NET: Registered PF_UNIX/PF_LOCAL protocol family client # [ 0.215119] NET: Registered PF_XDP protocol family client # [ 0.215142] PCI: CLS 0 bytes, default 64 client # [ 0.215409] Trying to unpack rootfs image as initramfs... client # [ 0.230099] kvm [1]: HYP mode not available worker1 # [ 0.442129] Initialise system trusted keyrings worker1 # [ 0.442912] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker1 # [ 0.444274] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker1 # [ 0.445057] 9p: Installing v9fs 9p2000 file system support worker2 # [ 0.432530] Initialise system trusted keyrings worker2 # [ 0.433316] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker2 # [ 0.434661] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker2 # [ 0.435437] 9p: Installing v9fs 9p2000 file system support worker1 # [ 0.473861] Key type asymmetric registered worker1 # [ 0.473896] Asymmetric key parser 'x509' registered lb # [ 0.435508] Initialise system trusted keyrings worker1 # [ 0.473976] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) lb # [ 0.436258] workingset: timestamp_bits=42 max_order=19 bucket_order=0 worker1 # [ 0.476109] io scheduler mq-deadline registered worker1 # [ 0.476168] io scheduler kyber registered lb # [ 0.437617] squashfs: version 4.0 (2009/01/31) Phillip Lougher client # [ 0.421507] Initialise system trusted keyrings client # [ 0.422267] workingset: timestamp_bits=42 max_order=19 bucket_order=0 lb # [ 0.438381] 9p: Installing v9fs 9p2000 file system support client # [ 0.423613] squashfs: version 4.0 (2009/01/31) Phillip Lougher worker2 # [ 0.459444] Key type asymmetric registered worker2 # [ 0.459477] Asymmetric key parser 'x509' registered worker2 # [ 0.459555] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker2 # [ 0.461710] io scheduler mq-deadline registered worker2 # [ 0.461722] io scheduler kyber registered client # [ 0.424415] 9p: Installing v9fs 9p2000 file system support worker1 # [ 0.485750] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker1 # [ 0.487227] ledtrig-cpu: registered to indicate activity on CPUs worker1 # [ 0.487657] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker1 # [ 0.487677] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker1 # [ 0.487689] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker1 # [ 0.487697] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker1 # [ 0.487728] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker2 # [ 0.474988] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker1 # [ 0.487756] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker1 # [ 0.487849] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker1 # [ 0.487859] pci_bus 0000:00: root bus resource [bus 00-ff] worker1 # [ 0.487865] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker1 # [ 0.487870] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.487875] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.487933] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker1 # [ 0.488397] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.476452] ledtrig-cpu: registered to indicate activity on CPUs worker1 # [ 0.488581] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.476887] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker1 # [ 0.488604] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.488633] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.476905] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker1 # [ 0.488649] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.476918] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker1 # [ 0.489115] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.467168] Key type asymmetric registered worker2 # [ 0.476926] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker1 # [ 0.489298] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.467204] Asymmetric key parser 'x509' registered worker1 # [ 0.489314] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.476947] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits client # [ 0.452401] Key type asymmetric registered lb # [ 0.467285] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker1 # [ 0.489343] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.452435] Asymmetric key parser 'x509' registered lb # [ 0.469446] io scheduler mq-deadline registered worker2 # [ 0.476971] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] lb # [ 0.469458] io scheduler kyber registered client # [ 0.452516] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 242) worker2 # [ 0.477053] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 client # [ 0.454620] io scheduler mq-deadline registered worker1 # [ 0.509017] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint worker2 # [ 0.477063] pci_bus 0000:00: root bus resource [bus 00-ff] client # [ 0.454634] io scheduler kyber registered worker1 # [ 0.509206] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f] worker2 # [ 0.477069] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] worker1 # [ 0.509222] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.477074] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.509251] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.477079] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker2 # [ 0.477138] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint worker2 # [ 0.477580] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker2 # [ 0.477777] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.477794] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.477824] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.477840] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker2 # [ 0.478312] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint lb # [ 0.479040] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker2 # [ 0.478538] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.478555] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.464313] pl061_gpio 9030000.pl061: PL061 GPIO chip registered worker2 # [ 0.478585] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.479051] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint lb # [ 0.480525] ledtrig-cpu: registered to indicate activity on CPUs worker2 # [ 0.479234] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f] worker2 # [ 0.479250] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.480958] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: client # [ 0.465914] ledtrig-cpu: registered to indicate activity on CPUs worker2 # [ 0.479279] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.480976] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 client # [ 0.466338] pci-host-generic 4010000000.pcie: host bridge /pcie@10000000 ranges: worker2 # [ 0.479746] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint lb # [ 0.480998] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 client # [ 0.466355] pci-host-generic 4010000000.pcie: IO 0x003eff0000..0x003effffff -> 0x0000000000 worker2 # [ 0.479928] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.517913] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint worker2 # [ 0.479943] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.481007] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 client # [ 0.466369] pci-host-generic 4010000000.pcie: MEM 0x0010000000..0x003efeffff -> 0x0010000000 worker1 # [ 0.518131] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.479976] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.481027] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker1 # [ 0.518148] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.466378] pci-host-generic 4010000000.pcie: MEM 0x8000000000..0xffffffffff -> 0x8000000000 worker2 # [ 0.480463] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint worker1 # [ 0.518181] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.481055] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] client # [ 0.466409] pci-host-generic 4010000000.pcie: Memory resource size exceeds max for 32 bits worker2 # [ 0.480646] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.518652] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint worker2 # [ 0.480662] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.481179] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 client # [ 0.466437] pci-host-generic 4010000000.pcie: ECAM at [mem 0x4010000000-0x401fffffff] for [bus 00-ff] worker1 # [ 0.518837] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.481189] pci_bus 0000:00: root bus resource [bus 00-ff] worker2 # [ 0.480692] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.466527] pci-host-generic 4010000000.pcie: PCI host bridge to bus 0000:00 worker1 # [ 0.518853] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.481195] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] client # [ 0.466537] pci_bus 0000:00: root bus resource [bus 00-ff] worker2 # [ 0.481141] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker1 # [ 0.518882] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.481200] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker2 # [ 0.481323] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f] client # [ 0.466543] pci_bus 0000:00: root bus resource [io 0x0000-0xffff] lb # [ 0.481205] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.519331] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker2 # [ 0.481339] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.466548] pci_bus 0000:00: root bus resource [mem 0x10000000-0x3efeffff] worker1 # [ 0.519515] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f] worker2 # [ 0.481369] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.481303] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint client # [ 0.466553] pci_bus 0000:00: root bus resource [mem 0x8000000000-0xffffffffff] worker1 # [ 0.519530] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.481849] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.466612] pci 0000:00:00.0: [1b36:0008] type 00 class 0x060000 conventional PCI endpoint lb # [ 0.481760] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.519560] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.482033] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.481950] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.482049] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.520056] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint client # [ 0.467049] pci 0000:00:01.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint lb # [ 0.481966] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.467238] pci 0000:00:01.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.520273] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.482079] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.481995] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.467255] pci 0000:00:01.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.520290] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.482095] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref] lb # [ 0.482011] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.467285] pci 0000:00:01.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.520319] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.482465] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint client # [ 0.467301] pci 0000:00:01.0: ROM [mem 0x00000000-0x0003ffff pref] worker1 # [ 0.520335] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref] lb # [ 0.482648] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] client # [ 0.467768] pci 0000:00:02.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.520809] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint lb # [ 0.482663] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.467952] pci 0000:00:02.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.520997] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.482692] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.467969] pci 0000:00:02.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.521027] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.467999] pci 0000:00:02.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.521476] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker1 # [ 0.521678] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.501716] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint worker1 # [ 0.521707] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.501905] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f] lb # [ 0.501921] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.522096] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint lb # [ 0.501950] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.522276] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff] lb # [ 0.502397] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint worker1 # [ 0.522521] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint lb # [ 0.502579] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.522858] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] worker2 # [ 0.522673] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint lb # [ 0.502594] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.522876] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.522899] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.502626] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.522906] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.522930] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.523374] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker2 # [ 0.523468] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker1 # [ 0.523556] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker2 # [ 0.523656] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.523571] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.523686] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.523601] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.524083] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker1 # [ 0.524217] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned worker2 # [ 0.524265] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff] worker1 # [ 0.524228] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker2 # [ 0.524544] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker1 # [ 0.524234] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker2 # [ 0.524854] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] worker1 # [ 0.524280] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker2 # [ 0.524872] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.524327] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned client # [ 0.492613] pci 0000:00:03.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint worker2 # [ 0.524902] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.492828] pci 0000:00:03.0: BAR 0 [io 0x0000-0x003f] worker1 # [ 0.524374] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned worker2 # [ 0.525388] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint client # [ 0.492846] pci 0000:00:03.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.525593] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.524420] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned client # [ 0.492876] pci 0000:00:03.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.525610] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.524466] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned client # [ 0.493365] pci 0000:00:04.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint worker2 # [ 0.525639] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.493548] pci 0000:00:04.0: BAR 0 [io 0x0000-0x001f] worker1 # [ 0.524513] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker2 # [ 0.526243] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned client # [ 0.493564] pci 0000:00:04.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.526256] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker1 # [ 0.524559] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned client # [ 0.493596] pci 0000:00:04.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.526261] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker1 # [ 0.524606] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned client # [ 0.494056] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint worker2 # [ 0.526308] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned lb # [ 0.515281] pci 0000:00:05.0: [1af4:1009] type 00 class 0x000200 conventional PCI endpoint worker1 # [ 0.524652] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned client # [ 0.494239] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.515499] pci 0000:00:05.0: BAR 0 [io 0x0000-0x001f] client # [ 0.494255] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.526356] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker1 # [ 0.524744] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned lb # [ 0.515517] pci 0000:00:05.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.494284] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.524790] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned worker2 # [ 0.526405] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned lb # [ 0.515547] pci 0000:00:05.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.524811] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned client # [ 0.494735] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint worker2 # [ 0.526452] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned lb # [ 0.516020] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 conventional PCI endpoint client # [ 0.494916] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f] worker1 # [ 0.524832] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned lb # [ 0.516209] pci 0000:00:06.0: BAR 0 [io 0x0000-0x007f] client # [ 0.494933] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.524854] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned lb # [ 0.516224] pci 0000:00:06.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.494962] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.524878] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned lb # [ 0.516254] pci 0000:00:06.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.495481] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.524900] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned client # [ 0.495673] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.516707] pci 0000:00:07.0: [1af4:1000] type 00 class 0x020000 conventional PCI endpoint worker1 # [ 0.524922] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned lb # [ 0.516893] pci 0000:00:07.0: BAR 0 [io 0x0000-0x001f] client # [ 0.495690] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.524943] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned lb # [ 0.516908] pci 0000:00:07.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.495720] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.524965] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned client # [ 0.495738] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref] lb # [ 0.516938] pci 0000:00:07.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.524987] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned lb # [ 0.516953] pci 0000:00:07.0: ROM [mem 0x00000000-0x0003ffff pref] client # [ 0.496221] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker1 # [ 0.525009] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned client # [ 0.496408] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.517468] pci 0000:00:08.0: [1af4:1050] type 00 class 0x038000 conventional PCI endpoint worker1 # [ 0.525032] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned lb # [ 0.517656] pci 0000:00:08.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.496438] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker1 # [ 0.525053] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned lb # [ 0.517686] pci 0000:00:08.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.496885] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker1 # [ 0.525075] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned client # [ 0.497083] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.518138] pci 0000:00:09.0: [1af4:1052] type 00 class 0x090000 conventional PCI endpoint worker1 # [ 0.525096] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned client # [ 0.497114] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.518326] pci 0000:00:09.0: BAR 1 [mem 0x00000000-0x00000fff] worker1 # [ 0.525117] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned worker1 # [ 0.525138] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned lb # [ 0.518355] pci 0000:00:09.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.497509] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker1 # [ 0.525159] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned client # [ 0.497692] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff] lb # [ 0.518743] pci 0000:00:0a.0: [8086:24cd] type 00 class 0x0c0320 conventional PCI endpoint worker1 # [ 0.525180] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned lb # [ 0.518945] pci 0000:00:0a.0: BAR 0 [mem 0x00000000-0x00000fff] client # [ 0.497940] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker1 # [ 0.525202] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned worker2 # [ 0.554629] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned client # [ 0.498234] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] lb # [ 0.519192] pci 0000:00:0b.0: [1af4:1003] type 00 class 0x078000 conventional PCI endpoint worker1 # [ 0.525224] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned client # [ 0.498252] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.519488] pci 0000:00:0b.0: BAR 0 [io 0x0000-0x003f] worker2 # [ 0.554716] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker1 # [ 0.525252] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] lb # [ 0.519505] pci 0000:00:0b.0: BAR 1 [mem 0x00000000-0x00000fff] client # [ 0.498281] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] worker2 # [ 0.554767] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker1 # [ 0.525262] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] lb # [ 0.519534] pci 0000:00:0b.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.498737] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker1 # [ 0.525266] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] worker2 # [ 0.554817] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned client # [ 0.498922] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] lb # [ 0.520007] pci 0000:00:0c.0: [1af4:1005] type 00 class 0x00ff00 conventional PCI endpoint worker2 # [ 0.554867] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned client # [ 0.498938] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] lb # [ 0.520191] pci 0000:00:0c.0: BAR 0 [io 0x0000-0x001f] client # [ 0.498968] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] lb # [ 0.520207] pci 0000:00:0c.0: BAR 1 [mem 0x00000000-0x00000fff] worker2 # [ 0.555025] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned lb # [ 0.520236] pci 0000:00:0c.0: BAR 4 [mem 0x00000000-0x00003fff 64bit pref] client # [ 0.499550] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned worker2 # [ 0.555076] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned lb # [ 0.520846] pci 0000:00:01.0: ROM [mem 0x10000000-0x1003ffff pref]: assigned client # [ 0.499561] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned worker2 # [ 0.555106] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker2 # [ 0.555129] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned lb # [ 0.520857] pci 0000:00:07.0: ROM [mem 0x10040000-0x1007ffff pref]: assigned client # [ 0.499567] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned worker2 # [ 0.555153] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned lb # [ 0.520863] pci 0000:00:01.0: BAR 4 [mem 0x8000000000-0x8000003fff 64bit pref]: assigned client # [ 0.499612] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned worker2 # [ 0.555181] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned lb # [ 0.520908] pci 0000:00:02.0: BAR 4 [mem 0x8000004000-0x8000007fff 64bit pref]: assigned client # [ 0.499660] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned worker2 # [ 0.555203] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned lb # [ 0.520957] pci 0000:00:03.0: BAR 4 [mem 0x8000008000-0x800000bfff 64bit pref]: assigned client # [ 0.499706] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned worker2 # [ 0.555225] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned lb # [ 0.521004] pci 0000:00:04.0: BAR 4 [mem 0x800000c000-0x800000ffff 64bit pref]: assigned client # [ 0.499753] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned worker2 # [ 0.555247] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned lb # [ 0.521051] pci 0000:00:05.0: BAR 4 [mem 0x8000010000-0x8000013fff 64bit pref]: assigned client # [ 0.499799] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned worker2 # [ 0.555270] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned worker2 # [ 0.555293] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned lb # [ 0.521124] pci 0000:00:06.0: BAR 4 [mem 0x8000014000-0x8000017fff 64bit pref]: assigned client # [ 0.499846] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker2 # [ 0.555317] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned lb # [ 0.521177] pci 0000:00:07.0: BAR 4 [mem 0x8000018000-0x800001bfff 64bit pref]: assigned worker1 # [ 0.590609] pci 0000:00:0a.0: enabling device (0000 -> 0002) client # [ 0.499893] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned worker2 # [ 0.555341] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned lb # [ 0.521224] pci 0000:00:08.0: BAR 4 [mem 0x800001c000-0x800001ffff 64bit pref]: assigned client # [ 0.499940] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned worker2 # [ 0.555362] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned lb # [ 0.521270] pci 0000:00:09.0: BAR 4 [mem 0x8000020000-0x8000023fff 64bit pref]: assigned client # [ 0.499987] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned worker2 # [ 0.555385] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned worker2 # [ 0.555407] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned lb # [ 0.521318] pci 0000:00:0b.0: BAR 4 [mem 0x8000024000-0x8000027fff 64bit pref]: assigned client # [ 0.500049] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned worker2 # [ 0.555429] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned lb # [ 0.521401] pci 0000:00:0c.0: BAR 4 [mem 0x8000028000-0x800002bfff 64bit pref]: assigned worker2 # [ 0.555450] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned lb # [ 0.521447] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned worker2 # [ 0.555472] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned lb # [ 0.521469] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned worker2 # [ 0.555494] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned lb # [ 0.521490] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned worker2 # [ 0.555516] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned worker2 # [ 0.555539] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned lb # [ 0.521512] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned worker2 # [ 0.555574] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] lb # [ 0.521537] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned worker2 # [ 0.555584] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] lb # [ 0.521560] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned worker2 # [ 0.555589] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] lb # [ 0.521582] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned worker2 # [ 0.556805] pci 0000:00:0a.0: enabling device (0000 -> 0002) lb # [ 0.521604] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned lb # [ 0.521625] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned lb # [ 0.521647] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned lb # [ 0.521670] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned lb # [ 0.521692] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned lb # [ 0.521714] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned client # [ 0.548187] pci 0000:00:01.0: BAR 1 [mem 0x10080000-0x10080fff]: assigned lb # [ 0.521736] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned client # [ 0.548281] pci 0000:00:02.0: BAR 1 [mem 0x10081000-0x10081fff]: assigned lb # [ 0.521757] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned client # [ 0.548309] pci 0000:00:03.0: BAR 1 [mem 0x10082000-0x10082fff]: assigned lb # [ 0.521778] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned client # [ 0.548333] pci 0000:00:04.0: BAR 1 [mem 0x10083000-0x10083fff]: assigned lb # [ 0.521800] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned client # [ 0.548361] pci 0000:00:05.0: BAR 1 [mem 0x10084000-0x10084fff]: assigned lb # [ 0.521821] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned client # [ 0.548385] pci 0000:00:06.0: BAR 1 [mem 0x10085000-0x10085fff]: assigned lb # [ 0.521843] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned lb # [ 0.521865] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned client # [ 0.548409] pci 0000:00:07.0: BAR 1 [mem 0x10086000-0x10086fff]: assigned lb # [ 0.521886] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned client # [ 0.548432] pci 0000:00:08.0: BAR 1 [mem 0x10087000-0x10087fff]: assigned lb # [ 0.521915] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] client # [ 0.548456] pci 0000:00:09.0: BAR 1 [mem 0x10088000-0x10088fff]: assigned worker1 # [ 0.611628] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) lb # [ 0.521924] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] client # [ 0.548481] pci 0000:00:0a.0: BAR 0 [mem 0x10089000-0x10089fff]: assigned lb # [ 0.521929] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] client # [ 0.548505] pci 0000:00:0b.0: BAR 1 [mem 0x1008a000-0x1008afff]: assigned client # [ 0.548529] pci 0000:00:0c.0: BAR 1 [mem 0x1008b000-0x1008bfff]: assigned client # [ 0.548552] pci 0000:00:06.0: BAR 0 [io 0x1000-0x107f]: assigned client # [ 0.548574] pci 0000:00:03.0: BAR 0 [io 0x1080-0x10bf]: assigned client # [ 0.548596] pci 0000:00:0b.0: BAR 0 [io 0x10c0-0x10ff]: assigned client # [ 0.548617] pci 0000:00:01.0: BAR 0 [io 0x1100-0x111f]: assigned client # [ 0.548640] pci 0000:00:02.0: BAR 0 [io 0x1120-0x113f]: assigned client # [ 0.548662] pci 0000:00:04.0: BAR 0 [io 0x1140-0x115f]: assigned client # [ 0.548683] pci 0000:00:05.0: BAR 0 [io 0x1160-0x117f]: assigned client # [ 0.548706] pci 0000:00:07.0: BAR 0 [io 0x1180-0x119f]: assigned worker2 # [ 0.601775] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) client # [ 0.548728] pci 0000:00:0c.0: BAR 0 [io 0x11a0-0x11bf]: assigned client # [ 0.548760] pci_bus 0000:00: resource 4 [io 0x0000-0xffff] lb # [ 0.583252] pci 0000:00:0a.0: enabling device (0000 -> 0002) client # [ 0.548770] pci_bus 0000:00: resource 5 [mem 0x10000000-0x3efeffff] client # [ 0.548775] pci_bus 0000:00: resource 6 [mem 0x8000000000-0xffffffffff] worker1 # [ 0.621911] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) client # [ 0.549975] pci 0000:00:0a.0: enabling device (0000 -> 0002) worker1 # [ 0.626457] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker1 # [ 0.628525] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) worker2 # [ 0.612027] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker2 # [ 0.616611] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker1 # [ 0.639663] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003) client # [ 0.586326] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) lb # [ 0.604511] virtio-pci 0000:00:01.0: enabling device (0000 -> 0003) worker2 # [ 0.622794] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) lb # [ 0.606731] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker2 # [ 0.625078] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003) worker1 # [ 0.645794] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003) worker1 # [ 0.648340] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003) lb # [ 0.617694] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker2 # [ 0.636259] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003) worker1 # [ 0.659178] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) client # [ 0.604432] virtio-pci 0000:00:02.0: enabling device (0000 -> 0003) worker2 # [ 0.642646] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003) client # [ 0.606731] virtio-pci 0000:00:03.0: enabling device (0000 -> 0003) worker2 # [ 0.645195] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) client # [ 0.608875] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) client # [ 0.611051] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003) worker1 # [ 0.665738] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker1 # [ 0.667791] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) lb # [ 0.632237] virtio-pci 0000:00:04.0: enabling device (0000 -> 0003) lb # [ 0.634428] virtio-pci 0000:00:05.0: enabling device (0000 -> 0003) lb # [ 0.638251] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003) worker2 # [ 0.655648] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) worker2 # [ 0.657779] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) client # [ 0.621637] virtio-pci 0000:00:06.0: enabling device (0000 -> 0003) client # [ 0.623742] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003) worker1 # [ 0.678280] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) lb # [ 0.641399] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003) worker2 # [ 0.669537] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) client # [ 0.635428] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker1 # [ 0.691704] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled lb # [ 0.652551] virtio-pci 0000:00:08.0: enabling device (0000 -> 0002) worker1 # [ 0.694459] msm_serial: driver initialized worker1 # [ 0.694607] SuperH (H)SCI(F) driver initialized worker1 # [ 0.694666] STM32 USART driver initialized worker2 # [ 0.679818] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled client # [ 0.642058] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) client # [ 0.643955] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) lb # [ 0.658991] virtio-pci 0000:00:09.0: enabling device (0000 -> 0002) lb # [ 0.660975] virtio-pci 0000:00:0b.0: enabling device (0000 -> 0003) worker2 # [ 0.688106] msm_serial: driver initialized worker2 # [ 0.688318] SuperH (H)SCI(F) driver initialized worker2 # [ 0.688375] STM32 USART driver initialized lb # [ 0.671212] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) client # [ 0.657857] virtio-pci 0000:00:0c.0: enabling device (0000 -> 0003) lb # [ 0.684606] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled client # [ 0.668953] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled client # [ 0.671722] msm_serial: driver initialized client # [ 0.671899] SuperH (H)SCI(F) driver initialized client # [ 0.671954] STM32 USART driver initialized lb # [ 0.687307] msm_serial: driver initialized lb # [ 0.687449] SuperH (H)SCI(F) driver initialized lb # [ 0.687505] STM32 USART driver initialized worker1 # [ 0.733849] loop: module loaded worker1 # [ 0.734084] virtio_blk virtio5: 1/0/0 default/read/poll queues worker1 # [ 0.734925] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker2 # [ 0.722805] loop: module loaded worker2 # [ 0.723051] virtio_blk virtio5: 1/0/0 default/read/poll queues worker2 # [ 0.723877] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker1 # [ 0.742398] megasas: 07.734.00.00-rc1 worker1 # [ 0.743147] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] worker1 # [ 0.745141] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.745177] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.746814] Using buffer write method worker1 # [ 0.746909] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] worker1 # [ 0.748730] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker1 # [ 0.748769] Intel/Sharp Extended Query Table at 0x0031 worker1 # [ 0.757933] Using buffer write method worker2 # [ 0.731322] megasas: 07.734.00.00-rc1 worker1 # [ 0.757974] Concatenating MTD devices: worker1 # [ 0.757978] (0): "0.flash" worker1 # [ 0.757982] (1): "0.flash" client # [ 0.703158] loop: module loaded worker2 # [ 0.732108] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] worker1 # [ 0.757985] into device "0.flash" client # [ 0.703375] virtio_blk virtio5: 1/0/0 default/read/poll queues worker2 # [ 0.734197] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.734238] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.720476] loop: module loaded worker2 # [ 0.735902] Using buffer write method lb # [ 0.720706] virtio_blk virtio5: 1/0/0 default/read/poll queues lb # [ 0.721534] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) worker2 # [ 0.735998] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] worker2 # [ 0.737755] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 worker2 # [ 0.737797] Intel/Sharp Extended Query Table at 0x0031 worker2 # [ 0.746869] Using buffer write method worker2 # [ 0.746907] Concatenating MTD devices: worker2 # [ 0.746911] (0): "0.flash" worker2 # [ 0.746916] (1): "0.flash" worker2 # [ 0.746919] into device "0.flash" client # [ 0.712285] virtio_blk virtio5: [vda] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) lb # [ 0.727665] megasas: 07.734.00.00-rc1 lb # [ 0.728431] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] client # [ 0.715862] megasas: 07.734.00.00-rc1 client # [ 0.716673] physmap-flash 0.flash: physmap platform flash device: [mem 0x00000000-0x03ffffff] lb # [ 0.748214] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.748266] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.750012] Using buffer write method lb # [ 0.750119] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] lb # [ 0.755376] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.755420] Intel/Sharp Extended Query Table at 0x0031 lb # [ 0.757098] Using buffer write method lb # [ 0.757173] Concatenating MTD devices: lb # [ 0.757177] (0): "0.flash" lb # [ 0.757181] (1): "0.flash" client # [ 0.741385] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 lb # [ 0.757185] into device "0.flash" client # [ 0.741439] Intel/Sharp Extended Query Table at 0x0031 client # [ 0.743124] Using buffer write method client # [ 0.743239] physmap-flash 0.flash: physmap platform flash device: [mem 0x04000000-0x07ffffff] client # [ 0.748323] 0.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000 client # [ 0.748363] Intel/Sharp Extended Query Table at 0x0031 client # [ 0.750053] Using buffer write method client # [ 0.750092] Concatenating MTD devices: client # [ 0.750096] (0): "0.flash" client # [ 0.750100] (1): "0.flash" client # [ 0.750103] into device "0.flash" worker1 # [ 1.017476] Freeing initrd memory: 26148K worker1 # [ 1.023750] tun: Universal TUN/TAP device driver, 1.6 worker1 # [ 1.027864] thunder_xcv, ver 1.0 worker1 # [ 1.027912] thunder_bgx, ver 1.0 worker1 # [ 1.027934] nicpf, ver 1.0 worker1 # [ 1.028512] e1000: Intel(R) PRO/1000 Network Driver worker1 # [ 1.028520] e1000: Copyright (c) 1999-2006 Intel Corporation. worker1 # [ 1.028544] e1000e: Intel(R) PRO/1000 Network Driver worker1 # [ 1.028554] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker1 # [ 1.028583] igb: Intel(R) Gigabit Ethernet Network Driver worker1 # [ 1.028589] igb: Copyright (c) 2007-2014 Intel Corporation. worker1 # [ 1.028617] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker1 # [ 1.028623] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker1 # [ 1.028756] sky2: driver version 1.30 worker1 # [ 1.037599] usbcore: registered new interface driver usb-storage worker1 # [ 1.037732] usbcore: registered new interface driver usbserial_generic worker1 # [ 1.037748] usbserial: USB Serial support registered for generic worker1 # [ 1.038367] hv_vmbus: registering driver hyperv_keyboard worker1 # [ 1.039830] ehci-pci 0000:00:0a.0: EHCI Host Controller worker1 # [ 1.039863] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 1 worker1 # [ 1.040048] ehci-pci 0000:00:0a.0: irq 15, io mem 0x10089000 worker1 # [ 1.045529] rtc-pl031 9010000.pl031: registered as rtc0 worker2 # [ 1.024196] Freeing initrd memory: 26148K worker1 # [ 1.046217] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T10:56:51 UTC (1790074611) worker1 # [ 1.046626] i2c_dev: i2c /dev entries driver worker2 # [ 1.030676] tun: Universal TUN/TAP device driver, 1.6 worker1 # [ 1.051008] sdhci: Secure Digital Host Controller Interface driver worker1 # [ 1.051021] sdhci: Copyright(c) Pierre Ossman worker1 # [ 1.051286] Synopsys Designware Multimedia Card Interface Driver worker1 # [ 1.051655] sdhci-pltfm: SDHCI platform and OF driver helper worker1 # [ 1.053413] hid: raw HID events driver (C) Jiri Kosina worker1 # [ 1.056865] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.00 worker1 # [ 1.057811] usbcore: registered new interface driver usbhid worker2 # [ 1.034843] thunder_xcv, ver 1.0 worker1 # [ 1.057820] usbhid: USB HID core driver worker2 # [ 1.034894] thunder_bgx, ver 1.0 worker2 # [ 1.034919] nicpf, ver 1.0 worker1 # [ 1.060376] hub 1-0:1.0: USB hub found worker2 # [ 1.035476] e1000: Intel(R) PRO/1000 Network Driver worker1 # [ 1.060869] hub 1-0:1.0: 6 ports detected worker2 # [ 1.035483] e1000: Copyright (c) 1999-2006 Intel Corporation. worker2 # [ 1.035507] e1000e: Intel(R) PRO/1000 Network Driver worker2 # [ 1.035516] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker2 # [ 1.035547] igb: Intel(R) Gigabit Ethernet Network Driver worker2 # [ 1.035553] igb: Copyright (c) 2007-2014 Intel Corporation. worker1 # [ 1.061847] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker2 # [ 1.035575] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker1 # [ 1.063403] drop_monitor: Initializing network drop monitor service lb # [ 1.026137] Freeing initrd memory: 26140K worker2 # [ 1.035581] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker1 # [ 1.063570] NET: Registered PF_INET6 protocol family worker2 # [ 1.035713] sky2: driver version 1.30 worker2 # [ 1.037330] usbcore: registered new interface driver usb-storage worker1 # [ 1.066651] Segment Routing with IPv6 worker1 # [ 1.066671] In-situ OAM (IOAM) with IPv6 worker2 # [ 1.037425] usbcore: registered new interface driver usbserial_generic worker1 # [ 1.066702] NET: Registered PF_PACKET protocol family worker2 # [ 1.037439] usbserial: USB Serial support registered for generic worker2 # [ 1.038051] hv_vmbus: registering driver hyperv_keyboard worker2 # [ 1.039260] ehci-pci 0000:00:0a.0: EHCI Host Controller worker1 # [ 1.068350] 9pnet: Installing 9P2000 support worker2 # [ 1.039292] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 1 worker2 # [ 1.039525] ehci-pci 0000:00:0a.0: irq 15, io mem 0x10089000 client # [ 1.014482] Freeing initrd memory: 26160K lb # [ 1.032620] tun: Universal TUN/TAP device driver, 1.6 worker2 # [ 1.051610] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.00 worker2 # [ 1.052006] hub 1-0:1.0: USB hub found worker2 # [ 1.052028] hub 1-0:1.0: 6 ports detected worker1 # [ 1.071242] Key type dns_resolver registered worker2 # [ 1.054460] rtc-pl031 9010000.pl031: registered as rtc0 client # [ 1.020935] tun: Universal TUN/TAP device driver, 1.6 worker2 # [ 1.055148] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T10:56:51 UTC (1790074611) worker2 # [ 1.055605] i2c_dev: i2c /dev entries driver lb # [ 1.036564] thunder_xcv, ver 1.0 lb # [ 1.036616] thunder_bgx, ver 1.0 lb # [ 1.036638] nicpf, ver 1.0 lb # [ 1.037219] e1000: Intel(R) PRO/1000 Network Driver lb # [ 1.037227] e1000: Copyright (c) 1999-2006 Intel Corporation. lb # [ 1.037250] e1000e: Intel(R) PRO/1000 Network Driver worker1 # [ 1.077770] registered taskstats version 1 lb # [ 1.037259] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker1 # [ 1.077939] Loading compiled-in X.509 certificates worker2 # [ 1.060148] sdhci: Secure Digital Host Controller Interface driver lb # [ 1.037287] igb: Intel(R) Gigabit Ethernet Network Driver worker2 # [ 1.060162] sdhci: Copyright(c) Pierre Ossman lb # [ 1.037292] igb: Copyright (c) 2007-2014 Intel Corporation. worker2 # [ 1.060444] Synopsys Designware Multimedia Card Interface Driver client # [ 1.024965] thunder_xcv, ver 1.0 lb # [ 1.037315] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker2 # [ 1.060821] sdhci-pltfm: SDHCI platform and OF driver helper client # [ 1.025011] thunder_bgx, ver 1.0 client # [ 1.025036] nicpf, ver 1.0 lb # [ 1.037320] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. lb # [ 1.037459] sky2: driver version 1.30 client # [ 1.025591] e1000: Intel(R) PRO/1000 Network Driver lb # [ 1.039583] ehci-pci 0000:00:0a.0: EHCI Host Controller client # [ 1.025598] e1000: Copyright (c) 1999-2006 Intel Corporation. worker2 # [ 1.065207] hid: raw HID events driver (C) Jiri Kosina client # [ 1.025622] e1000e: Intel(R) PRO/1000 Network Driver lb # [ 1.039613] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 1 worker2 # [ 1.065467] usbcore: registered new interface driver usbhid client # [ 1.025631] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. worker2 # [ 1.065474] usbhid: USB HID core driver lb # [ 1.039836] ehci-pci 0000:00:0a.0: irq 15, io mem 0x10089000 worker1 # [ 1.087125] Demotion targets for Node 0: null client # [ 1.025663] igb: Intel(R) Gigabit Ethernet Network Driver worker1 # [ 1.087259] Key type .fscrypt registered worker1 # [ 1.087265] Key type fscrypt-provisioning registered client # [ 1.025669] igb: Copyright (c) 2007-2014 Intel Corporation. worker1 # [ 1.087374] ima: No TPM chip found, activating TPM-bypass! client # [ 1.025691] igbvf: Intel(R) Gigabit Virtual Function Network Driver worker1 # [ 1.087393] ima: Allocated hash algorithm: sha1 lb # [ 1.049454] usbcore: registered new interface driver usb-storage client # [ 1.025697] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. worker1 # [ 1.087416] ima: No architecture policies found client # [ 1.025835] sky2: driver version 1.30 worker2 # [ 1.068411] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available lb # [ 1.049519] usbcore: registered new interface driver usbserial_generic client # [ 1.027480] usbcore: registered new interface driver usb-storage worker2 # [ 1.069966] drop_monitor: Initializing network drop monitor service lb # [ 1.049536] usbserial: USB Serial support registered for generic worker1 # [ 1.091597] input: gpio-keys as /devices/platform/gpio-keys/input/input0 worker2 # [ 1.070175] NET: Registered PF_INET6 protocol family client # [ 1.027574] usbcore: registered new interface driver usbserial_generic lb # [ 1.050139] hv_vmbus: registering driver hyperv_keyboard client # [ 1.027588] usbserial: USB Serial support registered for generic worker2 # [ 1.073270] Segment Routing with IPv6 client # [ 1.028765] ehci-pci 0000:00:0a.0: EHCI Host Controller worker2 # [ 1.073303] In-situ OAM (IOAM) with IPv6 lb # [ 1.053720] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.00 lb # [ 1.054110] hub 1-0:1.0: USB hub found worker2 # [ 1.073334] NET: Registered PF_PACKET protocol family client # [ 1.028793] ehci-pci 0000:00:0a.0: new USB bus registered, assigned bus number 1 lb # [ 1.054131] hub 1-0:1.0: 6 ports detected client # [ 1.028983] ehci-pci 0000:00:0a.0: irq 15, io mem 0x10089000 lb # [ 1.056364] rtc-pl031 9010000.pl031: registered as rtc0 client # [ 1.040544] ehci-pci 0000:00:0a.0: USB 2.0 started, EHCI 1.00 client # [ 1.040965] hub 1-0:1.0: USB hub found client # [ 1.041019] hub 1-0:1.0: 6 ports detected lb # [ 1.056395] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T10:56:51 UTC (1790074611) worker2 # [ 1.075151] 9pnet: Installing 9P2000 support lb # [ 1.056755] i2c_dev: i2c /dev entries driver worker2 # [ 1.077460] Key type dns_resolver registered client # [ 1.043287] hv_vmbus: registering driver hyperv_keyboard client # [ 1.044996] rtc-pl031 9010000.pl031: registered as rtc0 lb # [ 1.061855] sdhci: Secure Digital Host Controller Interface driver client # [ 1.045029] rtc-pl031 9010000.pl031: setting system clock to 2026-09-22T10:56:51 UTC (1790074611) lb # [ 1.061869] sdhci: Copyright(c) Pierre Ossman client # [ 1.045360] i2c_dev: i2c /dev entries driver lb # [ 1.062140] Synopsys Designware Multimedia Card Interface Driver lb # [ 1.062517] sdhci-pltfm: SDHCI platform and OF driver helper worker2 # [ 1.084261] registered taskstats version 1 worker2 # [ 1.084528] Loading compiled-in X.509 certificates lb # [ 1.067035] hid: raw HID events driver (C) Jiri Kosina client # [ 1.050462] sdhci: Secure Digital Host Controller Interface driver client # [ 1.050475] sdhci: Copyright(c) Pierre Ossman lb # [ 1.067300] usbcore: registered new interface driver usbhid lb # [ 1.067307] usbhid: USB HID core driver client # [ 1.050749] Synopsys Designware Multimedia Card Interface Driver client # [ 1.051129] sdhci-pltfm: SDHCI platform and OF driver helper lb # [ 1.070318] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available client # [ 1.055768] hid: raw HID events driver (C) Jiri Kosina client # [ 1.056030] usbcore: registered new interface driver usbhid client # [ 1.056037] usbhid: USB HID core driver lb # [ 1.072992] drop_monitor: Initializing network drop monitor service lb # [ 1.073241] NET: Registered PF_INET6 protocol family worker2 # [ 1.093459] Demotion targets for Node 0: null worker2 # [ 1.093599] Key type .fscrypt registered lb # [ 1.075518] Segment Routing with IPv6 worker2 # [ 1.093606] Key type fscrypt-provisioning registered lb # [ 1.075539] In-situ OAM (IOAM) with IPv6 worker2 # [ 1.093715] ima: No TPM chip found, activating TPM-bypass! lb # [ 1.075588] NET: Registered PF_PACKET protocol family client # [ 1.059023] hw perfevents: enabled with armv8_pmuv3 PMU driver, 11 (0,800003ff) counters available worker2 # [ 1.093735] ima: Allocated hash algorithm: sha1 worker1 # [ 1.114059] clk: Disabling unused clocks worker2 # [ 1.093758] ima: No architecture policies found worker1 # [ 1.114101] PM: genpd: Disabling unused power domains client # [ 1.061653] drop_monitor: Initializing network drop monitor service lb # [ 1.077204] 9pnet: Installing 9P2000 support client # [ 1.061818] NET: Registered PF_INET6 protocol family worker2 # [ 1.097929] input: gpio-keys as /devices/platform/gpio-keys/input/input0 worker1 # [ 1.118489] Freeing unused kernel memory: 4736K client # [ 1.063875] Segment Routing with IPv6 worker1 # [ 1.118718] Run /init as init process client # [ 1.063897] In-situ OAM (IOAM) with IPv6 client # [ 1.063968] NET: Registered PF_PACKET protocol family lb # [ 1.080128] Key type dns_resolver registered client # [ 1.065866] 9pnet: Installing 9P2000 support lb # [ 1.086783] registered taskstats version 1 client # [ 1.068676] Key type dns_resolver registered lb # [ 1.087497] Loading compiled-in X.509 certificates client # [ 1.075294] registered taskstats version 1 client # [ 1.075451] Loading compiled-in X.509 certificates lb # [ 1.096078] Demotion targets for Node 0: null lb # [ 1.096224] Key type .fscrypt registered worker1 # [ 1.136075] systemd[1]: Successfully made /usr/ read-only. lb # [ 1.096230] Key type fscrypt-provisioning registered lb # [ 1.096332] ima: No TPM chip found, activating TPM-bypass! lb # [ 1.096352] ima: Allocated hash algorithm: sha1 lb # [ 1.096376] ima: No architecture policies found lb # [ 1.100551] input: gpio-keys as /devices/platform/gpio-keys/input/input0 client # [ 1.084289] Demotion targets for Node 0: null client # [ 1.084434] Key type .fscrypt registered client # [ 1.084441] Key type fscrypt-provisioning registered client # [ 1.084545] ima: No TPM chip found, activating TPM-bypass! worker2 # [ 1.120246] clk: Disabling unused clocks client # [ 1.084564] ima: Allocated hash algorithm: sha1 worker2 # [ 1.120328] PM: genpd: Disabling unused power domains client # [ 1.084590] ima: No architecture policies found client # [ 1.089204] input: gpio-keys as /devices/platform/gpio-keys/input/input0 worker2 # [ 1.124675] Freeing unused kernel memory: 4736K worker2 # [ 1.124894] Run /init as init process worker2 # [ 1.141123] systemd[1]: Successfully made /usr/ read-only. lb # [ 1.123249] clk: Disabling unused clocks lb # [ 1.123287] PM: genpd: Disabling unused power domains lb # [ 1.127687] Freeing unused kernel memory: 4736K client # [ 1.111187] clk: Disabling unused clocks lb # [ 1.127931] Run /init as init process client # [ 1.111232] PM: genpd: Disabling unused power domains client # [ 1.115631] Freeing unused kernel memory: 4736K client # [ 1.115836] Run /init as init process lb # [ 1.145459] systemd[1]: Successfully made /usr/ read-only. client # [ 1.134691] systemd[1]: Successfully made /usr/ read-only. worker1 # [ 1.309634] usb 1-1: new high-speed USB device number 2 using ehci-pci worker2 # [ 1.298565] usb 1-1: new high-speed USB device number 2 using ehci-pci lb # [ 1.298943] usb 1-1: new high-speed USB device number 2 using ehci-pci client # [ 1.288175] usb 1-1: new high-speed USB device number 2 using ehci-pci worker1 # [ 1.464171] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker2 # [ 1.450536] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker1 # [ 1.471055] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker1 # [ 1.483496] systemd[1]: Detected virtualization qemu. worker1 # [ 1.483599] systemd[1]: Detected architecture arm64. worker1 # [ 1.483623] systemd[1]: Running in initrd. worker1 # [ 1.484655] systemd[1]: Initializing machine ID from random generator. worker1 # [ 1.484939] systemd[1]: Hostname set to . lb # [ 1.451331] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 client # [ 1.442533] input: QEMU QEMU USB Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-1/1-1:1.0/0003:0627:0001.0001/input/input1 worker2 # [ 1.475773] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 1.488295] systemd[1]: Detected virtualization qemu. worker2 # [ 1.490425] systemd[1]: Detected architecture arm64. worker2 # [ 1.492485] systemd[1]: Running in initrd. worker2 # [ 1.495219] systemd[1]: Initializing machine ID from random generator. worker2 # [ 1.498321] systemd[1]: Hostname set to . lb # [ 1.480533] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 1.492869] systemd[1]: Detected virtualization qemu. lb # [ 1.495039] systemd[1]: Detected architecture arm64. client # [ 1.469650] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 1.496979] systemd[1]: Running in initrd. client # [ 1.481967] systemd[1]: Detected virtualization qemu. client # [ 1.483993] systemd[1]: Detected architecture arm64. lb # [ 1.499671] systemd[1]: Initializing machine ID from random generator. client # [ 1.486073] systemd[1]: Running in initrd. lb # [ 1.502668] systemd[1]: Hostname set to . client # [ 1.488826] systemd[1]: Initializing machine ID from random generator. client # [ 1.491898] systemd[1]: Hostname set to . worker1 # [ 1.549857] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input0 worker2 # [ 1.538780] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input0 lb # [ 1.539346] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input0 client # [ 1.532527] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:00:0a.0-1/input0 worker1 # [ 1.673641] usb 1-2: new high-speed USB device number 3 using ehci-pci worker2 # [ 1.658627] usb 1-2: new high-speed USB device number 3 using ehci-pci lb # [ 1.662976] usb 1-2: new high-speed USB device number 3 using ehci-pci client # [ 1.660155] usb 1-2: new high-speed USB device number 3 using ehci-pci worker2 # [ 1.779916] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker1 # [ 1.806678] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker1 # [ 1.835985] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.817218] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.819548] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input0 worker1 # [ 1.841875] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input0 lb # [ 1.824362] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 lb # [ 1.830328] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input0 worker2 # [ 1.858841] systemd[1]: Queued start job for default target Initrd Default Target. client # [ 1.826069] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 1.842653] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker2 # [ 1.869270] systemd[1]: Created slice Slice /system/modprobe. client # [ 1.829423] input: QEMU QEMU USB Tablet as /devices/platform/4010000000.pcie/pci0000:00/0000:00:0a.0/usb1/1-2/1-2:1.0/0003:0627:0001.0002/input/input2 worker2 # [ 1.870375] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker2 # [ 1.871538] systemd[1]: Expecting device /dev/disk/by-label/nixos... worker2 # [ 1.872515] systemd[1]: Reached target Path Units. worker1 # [ 1.892000] systemd[1]: Queued start job for default target Initrd Default Target. worker2 # [ 1.873155] systemd[1]: Reached target Slice Units. worker2 # [ 1.873821] systemd[1]: Reached target Swaps. client # [ 1.829800] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Tablet] on usb-0000:00:0a.0-2/input0 worker2 # [ 1.874420] systemd[1]: Reached target Timer Units. worker2 # [ 1.875367] systemd[1]: Listening on D-Bus System Message Bus Socket. worker2 # [ 1.876369] systemd[1]: Listening on Journal Socket (/dev/log). worker2 # [ 1.877304] systemd[1]: Listening on Journal Sockets. worker2 # [ 1.878094] systemd[1]: Listening on udev Control Socket. worker2 # [ 1.878991] systemd[1]: Listening on udev Kernel Socket. worker2 # [ 1.879684] systemd[1]: Reached target Socket Units. worker1 # [ 1.899674] systemd[1]: Created slice Slice /system/modprobe. worker1 # [ 1.900733] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 1.901866] systemd[1]: Expecting device /dev/disk/by-label/nixos... worker2 # [ 1.882189] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 1.902730] systemd[1]: Reached target Path Units. worker1 # [ 1.903396] systemd[1]: Reached target Slice Units. worker1 # [ 1.904050] systemd[1]: Reached target Swaps. worker1 # [ 1.904660] systemd[1]: Reached target Timer Units. worker1 # [ 1.905504] systemd[1]: Listening on D-Bus System Message Bus Socket. worker1 # [ 1.906717] systemd[1]: Listening on Journal Socket (/dev/log). worker1 # [ 1.907639] systemd[1]: Listening on Journal Sockets. worker1 # [ 1.908472] systemd[1]: Listening on udev Control Socket. worker1 # [ 1.909295] systemd[1]: Listening on udev Kernel Socket. worker1 # [ 1.910089] systemd[1]: Reached target Socket Units. worker2 # [ 1.890781] systemd[1]: Starting Load Kernel Module 9pnet_virtio... worker2 # [ 1.890899] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 1.912273] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 1.898860] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 1.921793] systemd[1]: Starting Load Kernel Module 9pnet_virtio... worker1 # [ 1.922782] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 1.929910] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 1.931444] systemd[1]: Starting Journal Service... worker2 # [ 1.938796] systemd[1]: Starting Load Kernel Modules... worker2 # [ 1.939719] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 1.961902] systemd[1]: Starting Journal Service... worker1 # [ 1.969753] systemd[1]: Starting Load Kernel Modules... worker1 # [ 1.970625] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 1.935724] systemd[1]: Queued start job for default target Initrd Default Target. worker1 # [ 1.977911] systemd[1]: Starting Coldplug All udev Devices... worker2 # [ 1.958704] systemd[1]: Starting Coldplug All udev Devices... lb # [ 1.945421] systemd[1]: Created slice Slice /system/modprobe. client # [ 1.930055] systemd[1]: Queued start job for default target Initrd Default Target. lb # [ 1.946483] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 1.947642] systemd[1]: Expecting device /dev/disk/by-label/nixos... lb # [ 1.948495] systemd[1]: Reached target Path Units. lb # [ 1.949140] systemd[1]: Reached target Slice Units. lb # [ 1.949788] systemd[1]: Reached target Swaps. lb # [ 1.950364] systemd[1]: Reached target Timer Units. lb # [ 1.951240] systemd[1]: Listening on D-Bus System Message Bus Socket. lb # [ 1.952202] systemd[1]: Listening on Journal Socket (/dev/log). lb # [ 1.953113] systemd[1]: Listening on Journal Sockets. lb # [ 1.953895] systemd[1]: Listening on udev Control Socket. lb # [ 1.954676] systemd[1]: Listening on udev Kernel Socket. client # [ 1.938486] systemd[1]: Created slice Slice /system/modprobe. worker2 # [ 1.970900] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 1.955455] systemd[1]: Reached target Socket Units. worker2 # [ 1.972087] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. client # [ 1.939670] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker2 # [ 1.972493] systemd[1]: Finished Load Kernel Module 9pnet_virtio. worker2 # [ 1.972778] systemd[1]: Mounted Kernel Configuration File System. client # [ 1.941050] systemd[1]: Expecting device /dev/disk/by-label/nixos... client # [ 1.942186] systemd[1]: Reached target Path Units. client # [ 1.942879] systemd[1]: Reached target Slice Units. client # [ 1.943576] systemd[1]: Reached target Swaps. client # [ 1.944244] systemd[1]: Reached target Timer Units. client # [ 1.945227] systemd[1]: Listening on D-Bus System Message Bus Socket. worker1 # [ 1.997813] systemd[1]: Finished Create List of Static Device Nodes. client # [ 1.946333] systemd[1]: Listening on Journal Socket (/dev/log). worker1 # [ 1.998888] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. client # [ 1.947351] systemd[1]: Listening on Journal Sockets. worker1 # [ 1.999258] systemd[1]: Finished Load Kernel Module 9pnet_virtio. worker1 # [ 1.999522] systemd[1]: Mounted Kernel Configuration File System. client # [ 1.948273] systemd[1]: Listening on udev Control Socket. client # [ 1.949232] systemd[1]: Listening on udev Kernel Socket. lb # [ 1.957710] systemd[1]: Starting Create List of Static Device Nodes... client # [ 1.950048] systemd[1]: Reached target Socket Units. lb # [ 1.967143] systemd[1]: Starting Load Kernel Module 9pnet_virtio... lb # [ 1.968159] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 1.952498] systemd[1]: Starting Create List of Static Device Nodes... worker2 # [ 1.991066] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker1 # [ 2.014142] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 1.975280] systemd[1]: Mounting Kernel Configuration File System... client # [ 1.960355] systemd[1]: Starting Load Kernel Module 9pnet_virtio... client # [ 1.961455] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 1.968524] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 2.011697] systemd-journald[73]: Collecting audit messages is disabled. lb # [ 1.995854] systemd[1]: Starting Journal Service... worker1 # [ 2.050724] systemd-journald[73]: Collecting audit messages is disabled. client # [ 1.996506] systemd[1]: Starting Journal Service... client # [ 1.998965] systemd[1]: Starting Load Kernel Modules... lb # [ 2.015189] systemd[1]: Starting Load Kernel Modules... client # [ 1.999809] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os lb # [ 2.016066] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 2.056146] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker2 # [ 2.037312] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. lb # [ 2.028137] systemd[1]: Starting Coldplug All udev Devices... client # [ 2.014339] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 2.073650] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker2 # [ 2.054568] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev worker1 # [ 2.078385] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker2 # [ 2.063637] [drm] pci: virtio-gpu-pci detected at 0000:00:08.0 worker1 # [ 2.085424] [drm] pci: virtio-gpu-pci detected at 0000:00:08.0 worker2 # [ 2.063909] [drm] features: -virgl +edid -resource_blob -host_visible worker2 # [ 2.063921] [drm] features: -context_init lb # [ 2.047074] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 2.064737] [drm] number of scanouts: 1 worker2 # [ 2.064760] [drm] number of cap sets: 0 client # [ 2.032988] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 2.049329] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. worker2 # [ 2.070748] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 2.037386] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. lb # [ 2.055399] systemd[1]: Finished Load Kernel Module 9pnet_virtio. lb # [ 2.056629] systemd[1]: Mounted Kernel Configuration File System. worker1 # [ 2.094044] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 2.101797] [drm] features: -virgl +edid -resource_blob -host_visible worker1 # [ 2.101822] [drm] features: -context_init worker1 # [ 2.102653] [drm] number of scanouts: 1 worker1 # [ 2.102676] [drm] number of cap sets: 0 client # [ 2.048796] systemd[1]: Finished Load Kernel Module 9pnet_virtio. worker2 # [ 2.083025] systemd[1]: Starting Create Static Device Nodes in /dev... client # [ 2.049631] systemd[1]: Mounted Kernel Configuration File System. lb # [ 2.069637] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. worker2 # [ 2.088002] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic worker2 # [ 2.088034] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 0 lb # [ 2.074546] systemd-journald[73]: Collecting audit messages is disabled. worker1 # [ 2.110970] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic worker1 # [ 2.111003] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 0 client # [ 2.056714] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 2.079489] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 2.077629] device-mapper: core: CONFIG_IMA_DISABLE_HTABLE is disabled. Duplicate IMA measurements will not be recorded in the IMA log. lb # [ 2.092019] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev client # [ 2.085115] systemd-journald[73]: Collecting audit messages is disabled. lb # [ 2.109250] [drm] pci: virtio-gpu-pci detected at 0000:00:08.0 lb # [ 2.109511] [drm] features: -virgl +edid -resource_blob -host_visible lb # [ 2.109522] [drm] features: -context_init lb # [ 2.110287] [drm] number of scanouts: 1 lb # [ 2.110306] [drm] number of cap sets: 0 worker2 # [ 2.126840] Console: switching to colour frame buffer device 160x50 worker2 # [ 2.133335] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device client # [ 2.096577] device-mapper: ioctl: 4.50.0-ioctl (2025-04-28) initialised: dm-devel@lists.linux.dev client # [ 2.102927] [drm] pci: virtio-gpu-pci detected at 0000:00:08.0 worker1 # [ 2.149920] Console: switching to colour frame buffer device 160x50 client # [ 2.103187] [drm] features: -virgl +edid -resource_blob -host_visible client # [ 2.103199] [drm] features: -context_init lb # [ 2.120687] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic client # [ 2.104050] [drm] number of scanouts: 1 client # [ 2.104070] [drm] number of cap sets: 0 lb # [ 2.120720] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 0 worker2 # [ 2.155550] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 2.158143] systemd[1]: Reached target Preparation for Local File Systems. worker1 # [ 2.178507] systemd[1]: Finished Create Static Device Nodes in /dev. worker1 # [ 2.178975] systemd[1]: Reached target Preparation for Local File Systems. worker1 # [ 2.179013] systemd[1]: Reached target Local File Systems. worker1 # [ 2.183490] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker2 # [ 2.162796] systemd[1]: Reached target Local File Systems. lb # [ 2.151679] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 2.133406] virtio-pci 0000:00:08.0: [drm] Registered 1 planes with drm panic client # [ 2.133439] [drm] Initialized virtio_gpu 0.1.0 for 0000:00:08.0 on minor 0 worker2 # [ 2.175036] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker1 # [ 2.194236] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device worker2 # [ 2.178338] systemd[1]: Finished Load Kernel Modules. client # [ 2.142691] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 2.145671] systemd[1]: Starting Create Static Device Nodes in /dev... lb # [ 2.159256] Console: switching to colour frame buffer device 160x50 worker1 # [ 2.210277] systemd[1]: Finished Load Kernel Modules. lb # [ 2.170312] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 2.187840] systemd-modules-load[74]: Inserted module 'dm_mod' worker1 # [ 2.230099] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 2.197120] systemd-modules-load[74]: Module 'virtio_balloon' is built in worker2 # [ 2.198391] systemd-modules-load[74]: Module 'virtio_console' is built in worker2 # [ 2.199504] systemd-modules-load[74]: Inserted module 'virtio_gpu' client # [ 2.168407] Console: switching to colour frame buffer device 160x50 lb # [ 2.195604] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device client # [ 2.177314] virtio-pci 0000:00:08.0: [drm] fb0: virtio_gpudrmfb frame buffer device worker2 # [ 2.215154] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 2.215434] systemd[1]: Started Journal Service. worker2 # [ 2.208944] systemd-modules-load[74]: Module 'virtio_rng' is built in worker1 # [ 2.228519] systemd-modules-load[74]: Inserted module 'dm_mod' lb # [ 2.207632] systemd[1]: Finished Load Kernel Modules. worker1 # [ 2.232364] systemd-modules-load[74]: Module 'virtio_balloon' is built in worker1 # [ 2.233601] systemd-modules-load[74]: Module 'virtio_console' is built in worker1 # [ 2.234756] systemd-modules-load[74]: Inserted module 'virtio_gpu' worker1 # [ 2.235734] systemd-modules-load[74]: Module 'virtio_rng' is built in worker1 # [ 2.254490] systemd[1]: Started Journal Service. worker2 # [ 2.224206] systemd[1]: Starting Create System Files and Directories... lb # [ 2.220093] systemd[1]: Starting Apply Kernel Variables... client # [ 2.205209] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 2.205476] systemd[1]: Reached target Preparation for Local File Systems. client # [ 2.205515] systemd[1]: Reached target Local File Systems. client # [ 2.211265] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 2.247552] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 2.247934] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 2.247978] systemd[1]: Reached target Local File Systems. client # [ 2.237311] systemd[1]: Finished Load Kernel Modules. worker1 # [ 2.277335] systemd[1]: Starting Create System Files and Directories... client # [ 2.248824] systemd[1]: Starting Apply Kernel Variables... lb # [ 2.263392] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker1 # [ 2.290221] systemd[1]: Finished Apply Kernel Variables. worker2 # [ 2.274262] systemd[1]: Finished Apply Kernel Variables. lb # [ 2.283517] systemd[1]: Started Journal Service. worker1 # [ 2.310394] systemd-udevd[79]: Using default interface naming scheme 'v261'. lb # [ 2.267991] systemd-modules-load[74]: Inserted module 'dm_mod' worker2 # [ 2.300413] systemd[1]: Finished Create System Files and Directories. lb # [ 2.284500] systemd-modules-load[74]: Module 'virtio_balloon' is built in lb # [ 2.285744] systemd-modules-load[74]: Module 'virtio_console' is built in client # [ 2.272558] systemd-modules-load[74]: Inserted module 'dm_mod' worker2 # [ 2.305692] systemd-udevd[79]: Using default interface naming scheme 'v261'. lb # [ 2.286897] systemd-modules-load[74]: Inserted module 'virtio_gpu' client # [ 2.273798] systemd-modules-load[74]: Module 'virtio_balloon' is built in lb # [ 2.287952] systemd-modules-load[74]: Module 'virtio_rng' is built in client # [ 2.274925] systemd-modules-load[74]: Module 'virtio_console' is built in client # [ 2.288380] systemd[1]: Started Journal Service. worker1 # [ 2.336583] systemd[1]: Finished Create System Files and Directories. lb # [ 2.302601] systemd[1]: Starting Create System Files and Directories... client # [ 2.292719] systemd-modules-load[74]: Inserted module 'virtio_gpu' client # [ 2.296097] systemd-modules-load[74]: Module 'virtio_rng' is built in lb # [ 2.309790] systemd[1]: Finished Apply Kernel Variables. client # [ 2.300084] systemd[1]: Starting Create System Files and Directories... worker2 # [ 2.338586] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 2.357035] systemd[1]: Started Rule-based Manager for Device Events and Files. client # [ 2.320287] systemd[1]: Finished Apply Kernel Variables. client # [ 2.330607] systemd-udevd[79]: Using default interface naming scheme 'v261'. lb # [ 2.347503] systemd[1]: Finished Create System Files and Directories. lb # [ 2.360355] systemd-udevd[81]: Using default interface naming scheme 'v261'. client # [ 2.371652] systemd[1]: Finished Create System Files and Directories. worker1 # [ 2.424813] systemd[1]: Starting Virtual Console Setup... worker2 # [ 2.408836] systemd[1]: Starting Virtual Console Setup... lb # [ 2.391022] systemd[1]: Started Rule-based Manager for Device Events and Files. client # [ 2.380341] systemd[1]: Started Rule-based Manager for Device Events and Files. worker1 # [ 2.480597] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 2.484554] systemd[1]: Finished Virtual Console Setup. worker2 # [ 2.468643] systemd-vconsole-setup[104]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 2.472531] systemd[1]: Finished Virtual Console Setup. client # [ 2.441855] systemd[1]: Starting Virtual Console Setup... lb # [ 2.460942] systemd[1]: Starting Virtual Console Setup... client # [ 2.496608] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 2.516555] systemd-vconsole-setup[103]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 2.504269] systemd[1]: Finished Virtual Console Setup. lb # [ 2.520303] systemd[1]: Finished Virtual Console Setup. worker1 # [ 3.068494] systemd[1]: Finished Coldplug All udev Devices. worker1 # [ 3.070168] systemd[1]: Reached target System Initialization. worker1 # [ 3.072129] systemd[1]: Reached target Basic System. worker2 # [ 3.073569] systemd[1]: Finished Coldplug All udev Devices. worker2 # [ 3.076168] systemd[1]: Reached target System Initialization. worker2 # [ 3.080169] systemd[1]: Reached target Basic System. lb # [ 3.116497] systemd[1]: Finished Coldplug All udev Devices. client # [ 3.104472] systemd[1]: Finished Coldplug All udev Devices. lb # [ 3.118669] systemd[1]: Reached target System Initialization. client # [ 3.106274] systemd[1]: Reached target System Initialization. lb # [ 3.120156] systemd[1]: Reached target Basic System. client # [ 3.107162] systemd[1]: Reached target Basic System. worker1 # [ 3.246504] (udev-worker)[100]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 3.258326] (udev-worker)[101]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker2 # [ 3.243667] (udev-worker)[94]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 3.263288] (udev-worker)[101]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 3.269764] (udev-worker)[93]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. lb # [ 3.251881] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 3.276242] (udev-worker)[93]: Network interface NamePolicy= disabled on kernel command line. client # [ 3.248158] (udev-worker)[92]: Network interface NamePolicy= disabled on kernel command line. client # [ 3.275124] (udev-worker)[102]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 3.336650] systemd[1]: Found device /dev/disk/by-label/nixos. worker1 # [ 3.339617] systemd[1]: Reached target Initrd Root Device. client # [ 3.293451] (udev-worker)[102]: Network interface NamePolicy= disabled on kernel command line. worker1 # [ 3.344127] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... lb # [ 3.306695] (udev-worker)[107]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. lb # [ 3.311428] (udev-worker)[107]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 3.332447] systemd[1]: Found device /dev/disk/by-label/nixos. worker2 # [ 3.335722] systemd[1]: Reached target Initrd Root Device. worker2 # [ 3.339114] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... lb # [ 3.369650] systemd[1]: Found device /dev/disk/by-label/nixos. worker1 # [ 3.408397] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks lb # [ 3.372790] systemd[1]: Reached target Initrd Root Device. lb # [ 3.376605] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... client # [ 3.364518] systemd[1]: Found device /dev/disk/by-label/nixos. worker1 # [ 3.416342] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker2 # [ 3.401639] systemd-fsck[113]: nixos: clean, 12/65536 files, 13019/262144 blocks client # [ 3.371636] systemd[1]: Reached target Initrd Root Device. worker1 # [ 3.425192] systemd[1]: Mounting /sysroot... worker2 # [ 3.412069] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. client # [ 3.379023] systemd[1]: Starting File System Check on /dev/disk/by-label/nixos... worker2 # [ 3.415446] systemd[1]: Mounting /sysroot... worker1 # [ 3.483072] EXT4-fs (vda): mounted filesystem b7494a4a-0c71-4154-867d-11aa1912d093 r/w with ordered data mode. Quota mode: none. worker1 # [ 3.473103] systemd[1]: Mounted /sysroot. lb # [ 3.437415] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks worker1 # [ 3.475550] systemd[1]: Reached target Initrd Root File System. worker1 # [ 3.480354] systemd[1]: Starting Mountpoints Configured in the Real Root... worker2 # [ 3.476565] EXT4-fs (vda): mounted filesystem 1901d721-ca8c-4062-ae20-6ee1aeb92fa2 r/w with ordered data mode. Quota mode: none. lb # [ 3.448601] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker2 # [ 3.469553] systemd[1]: Mounted /sysroot. worker2 # [ 3.471968] systemd[1]: Reached target Initrd Root File System. client # [ 3.438736] systemd-fsck[112]: nixos: clean, 12/65536 files, 13019/262144 blocks lb # [ 3.455227] systemd[1]: Mounting /sysroot... client # [ 3.446135] systemd[1]: Finished File System Check on /dev/disk/by-label/nixos. worker2 # [ 3.480149] systemd[1]: Starting Mountpoints Configured in the Real Root... client # [ 3.459720] systemd[1]: Mounting /sysroot... worker1 # [ 3.513119] systemd-sysroot-fstab-check[120]: /sysroot should be mounted in the initrd, will request daemon-reload. worker1 # [ 3.516797] systemd[1]: Reload requested from client PID 120 ('systemd-sysroot') (unit initrd-parse-etc.service)... worker1 # [ 3.522287] systemd[1]: Reloading... worker2 # [ 3.511730] systemd-sysroot-fstab-check[121]: /sysroot should be mounted in the initrd, will request daemon-reload. worker2 # [ 3.516537] systemd[1]: Reload requested from client PID 121 ('systemd-sysroot') (unit initrd-parse-etc.service)... lb # [ 3.513407] EXT4-fs (vda): mounted filesystem 81debc9d-e7d6-4829-8d96-4b8ab9939e27 r/w with ordered data mode. Quota mode: none. worker2 # [ 3.521435] systemd[1]: Reloading... lb # [ 3.505022] systemd[1]: Mounted /sysroot. lb # [ 3.507594] systemd[1]: Reached target Initrd Root File System. lb # [ 3.513972] systemd[1]: Starting Mountpoints Configured in the Real Root... client # [ 3.511027] EXT4-fs (vda): mounted filesystem bbd434e5-d1a0-43bf-a370-65bcbd9a72f5 r/w with ordered data mode. Quota mode: none. client # [ 3.506941] systemd[1]: Mounted /sysroot. client # [ 3.509560] systemd[1]: Reached target Initrd Root File System. client # [ 3.513693] systemd[1]: Starting Mountpoints Configured in the Real Root... lb # [ 3.549253] systemd-sysroot-fstab-check[120]: /sysroot should be mounted in the initrd, will request daemon-reload. lb # [ 3.556102] systemd[1]: Reload requested from client PID 120 ('systemd-sysroot') (unit initrd-parse-etc.service)... lb # [ 3.561381] systemd[1]: Reloading... client # [ 3.547909] systemd-sysroot-fstab-check[120]: /sysroot should be mounted in the initrd, will request daemon-reload. client # [ 3.558283] systemd[1]: Reload requested from client PID 120 ('systemd-sysroot') (unit initrd-parse-etc.service)... client # [ 3.561616] systemd[1]: Reloading... worker1 # [ 3.755066] systemd[1]: Reloading finished in 231 ms. worker2 # [ 3.755939] systemd[1]: Reloading finished in 237 ms. worker1 # [ 3.786256] systemd-sysroot-fstab-check[120]: Requesting initrd-fs.target/start/replace... worker1 # [ 3.790492] systemd-sysroot-fstab-check[120]: Requesting swap.target/start/replace... worker1 # [ 3.797188] systemd[1]: Starting Load Kernel Module 9pnet_virtio... worker1 # [ 3.799859] systemd[1]: initrd-parse-etc.service: Deactivated successfully. worker1 # [ 3.808526] systemd[1]: Finished Mountpoints Configured in the Real Root. worker1 # [ 3.809722] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. worker2 # [ 3.793018] systemd-sysroot-fstab-check[121]: Requesting initrd-fs.target/start/replace... worker2 # [ 3.799817] systemd-sysroot-fstab-check[121]: Requesting swap.target/start/replace... worker2 # [ 3.803466] systemd[1]: Starting Load Kernel Module 9pnet_virtio... lb # [ 3.797674] systemd[1]: Reloading finished in 239 ms. worker2 # [ 3.818003] systemd[1]: initrd-parse-etc.service: Deactivated successfully. worker1 # [ 3.835291] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. client # [ 3.787482] systemd[1]: Reloading finished in 229 ms. worker2 # [ 3.822062] systemd[1]: Finished Mountpoints Configured in the Real Root. worker1 # [ 3.838944] systemd[1]: Finished Load Kernel Module 9pnet_virtio. worker2 # [ 3.825994] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. worker2 # [ 3.844740] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. worker2 # [ 3.846860] systemd[1]: Finished Load Kernel Module 9pnet_virtio. lb # [ 3.827353] systemd-sysroot-fstab-check[120]: Requesting initrd-fs.target/start/replace... lb # [ 3.832151] systemd-sysroot-fstab-check[120]: Requesting swap.target/start/replace... client # [ 3.820550] systemd-sysroot-fstab-check[120]: Requesting initrd-fs.target/start/replace... lb # [ 3.838161] systemd[1]: Starting Load Kernel Module 9pnet_virtio... client # [ 3.826631] systemd-sysroot-fstab-check[120]: Requesting swap.target/start/replace... lb # [ 3.841183] systemd[1]: initrd-parse-etc.service: Deactivated successfully. client # [ 3.829989] systemd[1]: Starting Load Kernel Module 9pnet_virtio... lb # [ 3.845148] systemd[1]: Finished Mountpoints Configured in the Real Root. client # [ 3.838650] systemd[1]: initrd-parse-etc.service: Deactivated successfully. lb # [ 3.852216] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. client # [ 3.841280] systemd[1]: Finished Mountpoints Configured in the Real Root. client # [ 3.844232] systemd[1]: initrd-parse-etc.service: Triggering OnSuccess= dependencies. lb # [ 3.869361] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. lb # [ 3.871122] systemd[1]: Finished Load Kernel Module 9pnet_virtio. client # [ 3.867350] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. client # [ 3.870525] systemd[1]: Finished Load Kernel Module 9pnet_virtio. worker2 # [ 3.949558] systemd[1]: Mounting /sysroot/nix/.ro-store... worker1 # [ 3.982477] systemd[1]: Mounting /sysroot/nix/.ro-store... worker2 # [ 3.967663] systemd[1]: Mounting /sysroot/nix/.rw-store... worker2 # [ 3.983421] systemd[1]: Mounting /sysroot/run... worker1 # [ 4.000757] systemd[1]: Mounting /sysroot/nix/.rw-store... worker1 # [ 4.008110] systemd[1]: Mounting /sysroot/run... worker2 # [ 4.004190] systemd[1]: Mounting /sysroot/tmp/shared... worker1 # [ 4.031825] systemd[1]: Mounting /sysroot/tmp/shared... worker1 # [ 4.042964] systemd[1]: Mounting /sysroot/tmp/xchg... worker2 # [ 4.037053] systemd[1]: Mounting /sysroot/tmp/xchg... worker2 # [ 4.040963] systemd[1]: Mounted /sysroot/nix/.rw-store. lb # [ 4.018821] systemd[1]: Mounting /sysroot/nix/.ro-store... lb # [ 4.031273] systemd[1]: Mounting /sysroot/nix/.rw-store... worker2 # [ 4.055078] systemd[1]: Mounted /sysroot/nix/.ro-store. client # [ 4.023392] systemd[1]: Mounting /sysroot/nix/.ro-store... lb # [ 4.043205] systemd[1]: Mounting /sysroot/run... client # [ 4.033493] systemd[1]: Mounting /sysroot/nix/.rw-store... worker1 # [ 4.089075] systemd[1]: Mounted /sysroot/nix/.ro-store. lb # [ 4.058577] systemd[1]: Mounting /sysroot/tmp/shared... client # [ 4.050339] systemd[1]: Mounting /sysroot/run... worker2 # [ 4.087328] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 4.115719] systemd[1]: Mounted /sysroot/nix/.rw-store. lb # [ 4.087442] systemd[1]: Mounting /sysroot/tmp/xchg... client # [ 4.074186] systemd[1]: Mounting /sysroot/tmp/shared... worker2 # [ 4.118520] systemd[1]: Mounted /sysroot/run. worker1 # [ 4.137559] systemd[1]: Mounted /sysroot/run. client # [ 4.090915] systemd[1]: Mounting /sysroot/tmp/xchg... worker2 # [ 4.132141] systemd[1]: Mounted /sysroot/tmp/shared. worker1 # [ 4.152492] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 4.159752] systemd[1]: Mounted /sysroot/tmp/shared. worker2 # [ 4.146057] systemd[1]: Mounted /sysroot/tmp/xchg. worker1 # [ 4.162953] systemd[1]: Mounted /sysroot/tmp/xchg. worker2 # [ 4.148455] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 4.151578] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 4.136318] systemd[1]: Mounted /sysroot/nix/.ro-store. client # [ 4.130424] systemd[1]: Mounted /sysroot/nix/.ro-store. worker1 # [ 4.189465] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker1 # [ 4.190700] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 4.155806] systemd[1]: Mounted /sysroot/nix/.rw-store. worker1 # [ 4.196377] systemd[1]: Mounting /sysroot/nix/store... lb # [ 4.170712] systemd[1]: Mounted /sysroot/run. client # [ 4.163961] systemd[1]: Mounted /sysroot/nix/.rw-store. client # [ 4.166241] systemd[1]: Mounted /sysroot/run. lb # [ 4.189188] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 4.193404] systemd[1]: Mounted /sysroot/tmp/shared. worker1 # [ 4.239206] systemd[1]: Mounted /sysroot/nix/store. worker1 # [ 4.242070] systemd[1]: Reached target Initrd File Systems. client # [ 4.192587] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 4.206529] systemd[1]: Mounted /sysroot/tmp/xchg. worker1 # [ 4.247811] systemd[1]: Starting Find NixOS closure... client # [ 4.199180] systemd[1]: Mounted /sysroot/tmp/shared. client # [ 4.203780] systemd[1]: Mounted /sysroot/tmp/xchg. lb # [ 4.225476] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 4.228841] systemd[1]: Finished rw-sysroot-nix-store.service. worker1 # [ 4.266774] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... lb # [ 4.236339] systemd[1]: Mounting /sysroot/nix/store... client # [ 4.233235] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 4.235953] systemd[1]: Finished rw-sysroot-nix-store.service. worker1 # [ 4.296444] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker1 # [ 4.318282] systemd[1]: Finished Find NixOS closure. worker1 # [ 4.320244] systemd[1]: Reached target Initrd Default Target. lb # [ 4.284549] systemd[1]: Mounted /sysroot/nix/store. lb # [ 4.286742] systemd[1]: Reached target Initrd File Systems. worker1 # [ 4.325416] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... lb # [ 4.290424] systemd[1]: Starting Find NixOS closure... lb # [ 4.307596] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker1 # [ 4.366167] systemd[1]: Stopped target Initrd Default Target. worker1 # [ 4.369731] systemd[1]: Stopped target Basic System. worker1 # [ 4.371076] systemd[1]: Stopped target Initrd Root Device. worker1 # [ 4.376718] systemd[1]: Stopped target Path Units. worker1 # [ 4.381734] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker1 # [ 4.387607] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker1 # [ 4.390840] systemd[1]: Stopped target Slice Units. lb # [ 4.356090] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker1 # [ 4.396980] systemd[1]: Stopped target Socket Units. worker1 # [ 4.399005] systemd[1]: Stopped target System Initialization. worker1 # [ 4.400714] systemd[1]: Stopped target Swaps. worker1 # [ 4.401973] systemd[1]: Stopped target Timer Units. worker1 # [ 4.403325] systemd[1]: dbus.socket: Deactivated successfully. worker1 # [ 4.406682] systemd[1]: Closed D-Bus System Message Bus Socket. lb # [ 4.372264] systemd[1]: Finished Find NixOS closure. lb # [ 4.373967] systemd[1]: Reached target Initrd Default Target. worker1 # [ 4.413172] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker1 # [ 4.415559] systemd[1]: Stopped Find NixOS closure. lb # [ 4.379623] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker1 # [ 4.417534] systemd[1]: Starting Load Kernel Module 9pnet_virtio... worker1 # [ 4.418416] systemd[1]: Starting rw-sysroot-nix-store.service... worker1 # [ 4.419270] systemd[1]: systemd-sysctl.service: Deactivated successfully. worker1 # [ 4.425609] systemd[1]: Stopped Apply Kernel Variables. worker1 # [ 4.427128] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker1 # [ 4.432767] systemd[1]: Stopped Load Kernel Modules. worker1 # [ 4.433600] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker1 # [ 4.434707] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker1 # [ 4.435772] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker1 # [ 4.445446] systemd[1]: Stopped Create System Files and Directories. lb # [ 4.418073] systemd[1]: Stopped target Initrd Default Target. worker1 # [ 4.456474] systemd[1]: Stopped target Local File Systems. lb # [ 4.421009] systemd[1]: Stopped target Basic System. worker1 # [ 4.460328] systemd[1]: Stopped target Preparation for Local File Systems. lb # [ 4.424220] systemd[1]: Stopped target Initrd Root Device. lb # [ 4.426318] systemd[1]: Stopped target Path Units. lb # [ 4.427884] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker1 # [ 4.464589] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. lb # [ 4.431801] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. lb # [ 4.434489] systemd[1]: Stopped target Slice Units. worker1 # [ 4.472454] systemd[1]: Stopped Coldplug All udev Devices. lb # [ 4.437401] systemd[1]: Stopped target Socket Units. worker1 # [ 4.474716] systemd[1]: Stopping Rule-based Manager for Device Events and Files... lb # [ 4.441360] systemd[1]: Stopped target System Initialization. worker1 # [ 4.477577] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 4.442937] systemd[1]: Stopped target Swaps. lb # [ 4.445578] systemd[1]: Stopped target Timer Units. worker1 # [ 4.484858] systemd[1]: Stopped Virtual Console Setup. lb # [ 4.448443] systemd[1]: dbus.socket: Deactivated successfully. worker1 # [ 4.485733] systemd[1]: initrd-cleanup.service: Deactivated successfully. worker1 # [ 4.486705] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker1 # [ 4.487617] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 4.452585] systemd[1]: Closed D-Bus System Message Bus Socket. worker1 # [ 4.490800] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 4.454861] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. worker1 # [ 4.493359] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. worker1 # [ 4.496214] systemd[1]: Finished Load Kernel Module 9pnet_virtio. worker1 # [ 4.497153] systemd[1]: systemd-udevd.service: Deactivated successfully. lb # [ 4.461175] systemd[1]: Stopped Find NixOS closure. lb # [ 4.462812] systemd[1]: Starting Load Kernel Module 9pnet_virtio... worker1 # [ 4.499130] systemd[1]: Stopped Rule-based Manager for Device Events and Files. lb # [ 4.464997] systemd[1]: Starting rw-sysroot-nix-store.service... lb # [ 4.468183] systemd[1]: systemd-sysctl.service: Deactivated successfully. worker1 # [ 4.504670] systemd[1]: systemd-udevd.service: Consumed 1.041s CPU time over 2.327s wall clock time, 16.7M memory peak. worker1 # [ 4.506156] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. worker1 # [ 4.507175] systemd[1]: Closed udev Control Socket. worker1 # [ 4.507882] systemd[1]: Starting Cleanup udev Database... lb # [ 4.473106] systemd[1]: Stopped Apply Kernel Variables. worker1 # [ 4.509660] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. lb # [ 4.474170] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker1 # [ 4.510802] systemd[1]: Stopped Create Static Device Nodes in /dev. worker1 # [ 4.511695] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker1 # [ 4.513105] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. worker1 # [ 4.514108] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker1 # [ 4.515328] systemd[1]: Stopped Create List of Static Device Nodes. lb # [ 4.484323] systemd[1]: Stopped Load Kernel Modules. lb # [ 4.485472] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. lb # [ 4.488192] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. lb # [ 4.491430] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. lb # [ 4.504547] systemd[1]: Stopped Create System Files and Directories. lb # [ 4.505619] systemd[1]: Stopped target Local File Systems. lb # [ 4.506596] systemd[1]: Stopped target Preparation for Local File Systems. lb # [ 4.507537] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. worker1 # [ 4.548120] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. worker1 # [ 4.552444] systemd[1]: Finished Cleanup udev Database. worker1 # [ 4.553352] systemd[1]: Reached target Switch Root. worker1 # [ 4.554058] systemd[1]: Starting NixOS Activation... lb # [ 4.522151] systemd[1]: Stopped Coldplug All udev Devices. lb # [ 4.523053] systemd[1]: Stopping Rule-based Manager for Device Events and Files... lb # [ 4.527234] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 4.533094] systemd[1]: Stopped Virtual Console Setup. lb # [ 4.533925] systemd[1]: initrd-cleanup.service: Deactivated successfully. lb # [ 4.534920] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. lb # [ 4.535973] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. lb # [ 4.537292] systemd[1]: Finished Load Kernel Module 9pnet_virtio. lb # [ 4.538137] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. lb # [ 4.539152] systemd[1]: Finished rw-sysroot-nix-store.service. lb # [ 4.544290] systemd[1]: systemd-udevd.service: Deactivated successfully. lb # [ 4.547169] systemd[1]: Stopped Rule-based Manager for Device Events and Files. lb # [ 4.548295] systemd[1]: systemd-udevd.service: Consumed 1.029s CPU time over 2.295s wall clock time, 19.1M memory peak. lb # [ 4.552531] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. lb # [ 4.553631] systemd[1]: Closed udev Control Socket. lb # [ 4.556322] systemd[1]: Starting Cleanup udev Database... lb # [ 4.557171] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. lb # [ 4.559783] systemd[1]: Stopped Create Static Device Nodes in /dev. lb # [ 4.561051] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. lb # [ 4.562171] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. lb # [ 4.563172] systemd[1]: kmod-static-nodes.service: Deactivated successfully. lb # [ 4.564235] systemd[1]: Stopped Create List of Static Device Nodes. lb # [ 4.597349] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. lb # [ 4.599124] systemd[1]: Finished Cleanup udev Database. lb # [ 4.600939] systemd[1]: Reached target Switch Root. lb # [ 4.603301] systemd[1]: Starting NixOS Activation... worker1 # [ 4.721627] initrd-nixos-activation-start[172]: booting system configuration /nix/store/hl1ynmh5snhcjq7al2fl9h42d6chq0rx-nixos-system-worker1-test worker2 # [ 4.766327] (udev-worker)[103]: mtd0ro: Failed to find and pin callout binary "/nix/store/awq6qdiyxlvdp21km39qj020fcshgkia-systemd-261.2/lib/udev/mtd_probe": No such file or directory worker2 # [ 4.774018] (udev-worker)[103]: mtd0ro: /nix/store/awq6qdiyxlvdp21km39qj020fcshgkia-systemd-261.2/lib/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory worker1 # [ 4.796781] initrd-nixos-activation-start[172]: running activation script... lb # [ 4.782104] initrd-nixos-activation-start[172]: booting system configuration /nix/store/mmnw0b9xdzgvlxb4mzaa4gjdldj887b1-nixos-system-lb-test worker2 # [ 4.804541] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 4.808080] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 4.809626] systemd[1]: Stopping Virtual Console Setup... worker2 # [ 4.812218] systemd[1]: Starting Virtual Console Setup... client # [ 4.788470] (udev-worker)[97]: mtd0ro: Failed to find and pin callout binary "/nix/store/awq6qdiyxlvdp21km39qj020fcshgkia-systemd-261.2/lib/udev/mtd_probe": No such file or directory client # [ 4.791376] (udev-worker)[97]: mtd0ro: /nix/store/awq6qdiyxlvdp21km39qj020fcshgkia-systemd-261.2/lib/udev/rules.d/75-probe_mtd.rules:5 IMPORT{program}="mtd_probe $devnode": Failed to execute "mtd_probe /dev/mtd0ro", ignoring: No such file or directory worker2 # [ 4.833378] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 4.835040] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 4.844535] systemd[1]: Starting Virtual Console Setup... client # [ 4.826832] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 4.829079] systemd[1]: Stopped Virtual Console Setup. client # [ 4.832989] systemd[1]: Stopping Virtual Console Setup... client # [ 4.834054] systemd[1]: Starting Virtual Console Setup... worker2 # [ 4.870453] systemd-vconsole-setup[157]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 4.853001] initrd-nixos-activation-start[172]: running activation script... worker2 # [ 4.873942] systemd[1]: Finished Virtual Console Setup. client # [ 4.850465] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 4.851655] systemd[1]: Stopped Virtual Console Setup. client # [ 4.856421] systemd[1]: Starting Virtual Console Setup... client # [ 4.877072] systemd-vconsole-setup[155]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 4.880949] systemd[1]: Finished Virtual Console Setup. worker2 # [ 4.948876] systemd[1]: Mounting /sysroot/nix/store... worker2 # [ 4.992614] systemd[1]: Mounted /sysroot/nix/store. worker2 # [ 4.995804] systemd[1]: Reached target Initrd File Systems. worker2 # [ 4.997854] systemd[1]: Starting Find NixOS closure... worker2 # [ 5.005279] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker2 # [ 5.047902] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker2 # [ 5.052457] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. client # [ 5.023870] systemd[1]: Mounting /sysroot/nix/store... worker2 # [ 5.064294] systemd[1]: Finished Find NixOS closure. worker2 # [ 5.066844] systemd[1]: Reached target Initrd Default Target. worker2 # [ 5.068316] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker2 # [ 5.097072] systemd[1]: Stopped target Initrd Default Target. worker2 # [ 5.100465] systemd[1]: Stopped target Basic System. worker2 # [ 5.101456] systemd[1]: Stopped target Initrd Root Device. worker2 # [ 5.104506] systemd[1]: Stopped target Path Units. worker2 # [ 5.105393] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. worker2 # [ 5.107676] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. worker2 # [ 5.111779] systemd[1]: Stopped target Slice Units. worker2 # [ 5.112806] systemd[1]: Stopped target Socket Units. worker2 # [ 5.116253] systemd[1]: Stopped target System Initialization. worker2 # [ 5.117299] systemd[1]: Stopped target Swaps. worker2 # [ 5.118017] systemd[1]: Stopped target Timer Units. client # [ 5.085643] systemd[1]: Mounted /sysroot/nix/store. worker2 # [ 5.120228] systemd[1]: dbus.socket: Deactivated successfully. worker2 # [ 5.121520] systemd[1]: Closed D-Bus System Message Bus Socket. client # [ 5.088987] systemd[1]: Reached target Initrd File Systems. worker2 # [ 5.125454] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. client # [ 5.093388] systemd[1]: Starting Find NixOS closure... worker2 # [ 5.131541] systemd[1]: Stopped Find NixOS closure. worker2 # [ 5.133739] systemd[1]: Starting Load Kernel Module 9pnet_virtio... worker2 # [ 5.134706] systemd[1]: Starting rw-sysroot-nix-store.service... client # [ 5.101490] systemd[1]: Starting Create Volatile Files and Directories in the Real Root... worker2 # [ 5.135585] systemd[1]: systemd-sysctl.service: Deactivated successfully. worker2 # [ 5.144537] systemd[1]: Stopped Apply Kernel Variables. worker2 # [ 5.145455] systemd[1]: systemd-modules-load.service: Deactivated successfully. worker2 # [ 5.146703] systemd[1]: Stopped Load Kernel Modules. worker2 # [ 5.147451] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. worker2 # [ 5.160531] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. worker2 # [ 5.167047] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. worker2 # [ 5.169890] systemd[1]: Stopped Create System Files and Directories. worker2 # [ 5.170929] systemd[1]: Stopped target Local File Systems. worker2 # [ 5.171731] systemd[1]: Stopped target Preparation for Local File Systems. client # [ 5.144722] systemd[1]: Finished Create Volatile Files and Directories in the Real Root. worker2 # [ 5.184518] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. worker2 # [ 5.185663] systemd[1]: Stopped Coldplug All udev Devices. worker2 # [ 5.186680] systemd[1]: Stopping Rule-based Manager for Device Events and Files... client # [ 5.152766] systemd[1]: sysroot-run-credentials-systemd\x2dtmpfiles\x2dsetup\x2dsysroot.service.mount: Deactivated successfully. worker2 # [ 5.187812] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 5.194062] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 5.195550] systemd[1]: initrd-cleanup.service: Deactivated successfully. client # [ 5.164406] systemd[1]: Finished Find NixOS closure. worker2 # [ 5.198886] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. worker2 # [ 5.200481] systemd[1]: systemd-udevd.service: Deactivated successfully. client # [ 5.167299] systemd[1]: Reached target Initrd Default Target. client # [ 5.169224] systemd[1]: Starting Cleaning Up and Shutting Down Daemons... worker2 # [ 5.204282] systemd[1]: Stopped Rule-based Manager for Device Events and Files. worker2 # [ 5.205460] systemd[1]: systemd-udevd.service: Consumed 1.534s CPU time over 3.016s wall clock time, 22.4M memory peak. worker2 # [ 5.208311] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. worker2 # [ 5.211570] systemd[1]: Finished rw-sysroot-nix-store.service. worker2 # [ 5.212861] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. worker2 # [ 5.213950] systemd[1]: Finished Load Kernel Module 9pnet_virtio. worker2 # [ 5.214819] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. worker2 # [ 5.215821] systemd[1]: Closed udev Control Socket. worker2 # [ 5.216686] systemd[1]: Starting Cleanup udev Database... worker2 # [ 5.217450] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. worker2 # [ 5.218730] systemd[1]: Stopped Create Static Device Nodes in /dev. worker2 # [ 5.219595] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. worker2 # [ 5.226087] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. worker2 # [ 5.228372] systemd[1]: kmod-static-nodes.service: Deactivated successfully. worker2 # [ 5.232600] systemd[1]: Stopped Create List of Static Device Nodes. client # [ 5.201958] systemd[1]: Stopped target Initrd Default Target. client # [ 5.204454] systemd[1]: Stopped target Basic System. client # [ 5.206414] systemd[1]: Stopped target Initrd Root Device. worker1 # [ 5.256373] initrd-nixos-activation-start[195]: setting up /etc... client # [ 5.209117] systemd[1]: Stopped target Path Units. client # [ 5.212110] systemd[1]: systemd-ask-password-console.path: Deactivated successfully. client # [ 5.214091] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch. client # [ 5.217477] systemd[1]: Stopped target Slice Units. client # [ 5.219743] systemd[1]: Stopped target Socket Units. client # [ 5.224325] systemd[1]: Stopped target System Initialization. client # [ 5.225514] systemd[1]: Stopped target Swaps. client # [ 5.226236] systemd[1]: Stopped target Timer Units. client # [ 5.227016] systemd[1]: dbus.socket: Deactivated successfully. client # [ 5.231519] systemd[1]: Closed D-Bus System Message Bus Socket. client # [ 5.232687] systemd[1]: initrd-find-nixos-closure.service: Deactivated successfully. client # [ 5.233972] systemd[1]: Stopped Find NixOS closure. worker2 # [ 5.266897] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. client # [ 5.235495] systemd[1]: Starting Load Kernel Module 9pnet_virtio... worker2 # [ 5.269844] systemd[1]: Finished Cleanup udev Database. worker2 # [ 5.272208] systemd[1]: Reached target Switch Root. client # [ 5.239114] systemd[1]: Starting rw-sysroot-nix-store.service... worker2 # [ 5.276406] systemd[1]: Starting NixOS Activation... client # [ 5.250350] systemd[1]: systemd-sysctl.service: Deactivated successfully. client # [ 5.251532] systemd[1]: Stopped Apply Kernel Variables. client # [ 5.256886] systemd[1]: systemd-modules-load.service: Deactivated successfully. client # [ 5.262583] systemd[1]: Stopped Load Kernel Modules. client # [ 5.267526] systemd[1]: systemd-tmpfiles-setup-sysroot.service: Deactivated successfully. client # [ 5.269039] systemd[1]: Stopped Create Volatile Files and Directories in the Real Root. client # [ 5.271617] systemd[1]: systemd-tmpfiles-setup.service: Deactivated successfully. client # [ 5.273154] systemd[1]: Stopped Create System Files and Directories. client # [ 5.274052] systemd[1]: Stopped target Local File Systems. client # [ 5.274820] systemd[1]: Stopped target Preparation for Local File Systems. client # [ 5.275759] systemd[1]: systemd-udev-trigger.service: Deactivated successfully. lb # [ 5.296639] initrd-nixos-activation-start[195]: setting up /etc... client # [ 5.284546] systemd[1]: Stopped Coldplug All udev Devices. client # [ 5.290494] systemd[1]: Stopping Rule-based Manager for Device Events and Files... client # [ 5.291812] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 5.297478] systemd[1]: Stopped Virtual Console Setup. client # [ 5.298325] systemd[1]: initrd-cleanup.service: Deactivated successfully. client # [ 5.299246] systemd[1]: Finished Cleaning Up and Shutting Down Daemons. client # [ 5.302935] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. client # [ 5.304318] systemd[1]: Finished Load Kernel Module 9pnet_virtio. client # [ 5.308298] systemd[1]: systemd-udevd.service: Deactivated successfully. client # [ 5.309372] systemd[1]: Stopped Rule-based Manager for Device Events and Files. client # [ 5.310423] systemd[1]: systemd-udevd.service: Consumed 1.524s CPU time over 3.090s wall clock time, 22.3M memory peak. client # [ 5.313527] systemd[1]: systemd-udevd-control.socket: Deactivated successfully. client # [ 5.316200] systemd[1]: Closed udev Control Socket. client # [ 5.317127] systemd[1]: Starting Cleanup udev Database... client # [ 5.318023] systemd[1]: systemd-tmpfiles-setup-dev.service: Deactivated successfully. client # [ 5.319079] systemd[1]: Stopped Create Static Device Nodes in /dev. client # [ 5.319949] systemd[1]: systemd-tmpfiles-setup-dev-early.service: Deactivated successfully. client # [ 5.321264] systemd[1]: Stopped Create Static Device Nodes in /dev gracefully. client # [ 5.322258] systemd[1]: kmod-static-nodes.service: Deactivated successfully. client # [ 5.323217] systemd[1]: Stopped Create List of Static Device Nodes. client # [ 5.328287] systemd[1]: rw-sysroot-nix-store.service: Deactivated successfully. client # [ 5.329397] systemd[1]: Finished rw-sysroot-nix-store.service. client # [ 5.367609] systemd[1]: initrd-udevadm-cleanup-db.service: Deactivated successfully. client # [ 5.369829] systemd[1]: Finished Cleanup udev Database. client # [ 5.372940] systemd[1]: Reached target Switch Root. client # [ 5.376209] systemd[1]: Starting NixOS Activation... worker2 # [ 5.448792] initrd-nixos-activation-start[182]: booting system configuration /nix/store/02j81afdj8xzihnmsy7rp3hq1zfspx7w-nixos-system-worker2-test worker1 # [ 5.532636] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. worker2 # [ 5.517608] initrd-nixos-activation-start[182]: running activation script... worker1 # [ 5.535576] systemd[1]: Finished NixOS Activation. worker1 # [ 5.537064] systemd[1]: Starting Switch Root... worker1 # [ 5.561319] systemd[1]: Switching root. lb # [ 5.570664] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. client # [ 5.556777] initrd-nixos-activation-start[180]: booting system configuration /nix/store/k6qpk00c7qa52lid1b7zqxvp2lykb521-nixos-system-client-test lb # [ 5.573672] systemd[1]: Finished NixOS Activation. lb # [ 5.575039] systemd[1]: Starting Switch Root... lb # [ 5.601644] systemd[1]: Switching root. client # [ 5.628737] initrd-nixos-activation-start[180]: running activation script... worker1 # [ 5.753128] systemd-journald[73]: Received SIGTERM from PID 1 (systemd). lb # [ 5.790994] systemd-journald[73]: Received SIGTERM from PID 1 (systemd). worker2 # [ 5.969490] initrd-nixos-activation-start[205]: setting up /etc... client # [ 6.096427] initrd-nixos-activation-start[203]: setting up /etc... worker2 # [ 6.249944] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. worker2 # [ 6.252928] systemd[1]: Finished NixOS Activation. worker2 # [ 6.254270] systemd[1]: Starting Switch Root... worker2 # [ 6.277605] systemd[1]: Switching root. worker1 # [ 6.367443] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker1 # [ 6.380112] systemd[1]: Detected virtualization qemu. worker1 # [ 6.383414] systemd[1]: Detected architecture arm64. worker1 # [ 6.387860] systemd[1]: Detected first boot. worker1 # [ 6.395255] systemd[1]: Initializing machine ID from random generator. client # [ 6.374814] systemd[1]: initrd-nixos-activation.service: Deactivated successfully. client # [ 6.377913] systemd[1]: Finished NixOS Activation. client # [ 6.379167] systemd[1]: Starting Switch Root... lb # [ 6.402041] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) lb # [ 6.415046] systemd[1]: Detected virtualization qemu. lb # [ 6.418284] systemd[1]: Detected architecture arm64. lb # [ 6.421645] systemd[1]: Detected first boot. lb # [ 6.430282] systemd[1]: Initializing machine ID from random generator. client # [ 6.405973] systemd[1]: Switching root. worker2 # [ 6.470670] systemd-journald[73]: Received SIGTERM from PID 1 (systemd). client # [ 6.591775] systemd-journald[73]: Received SIGTERM from PID 1 (systemd). worker1 # [ 6.745188] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 6.766701] systemd[1]: bpf-restrict-fs: LSM BPF program attached worker1 # [ 6.949668] systemd[1]: Applying preset policy. lb # [ 6.977541] systemd[1]: Applying preset policy. worker2 # [ 7.087301] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) worker2 # [ 7.093821] systemd[1]: Detected virtualization qemu. worker2 # [ 7.096028] systemd[1]: Detected architecture arm64. worker2 # [ 7.100540] systemd[1]: Detected first boot. worker2 # [ 7.108045] systemd[1]: Initializing machine ID from random generator. client # [ 7.212381] systemd[1]: systemd 261.2 running in system mode (+PAM +AUDIT -SELINUX +APPARMOR +IMA +IPE +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 +KMOD +LIBCRYPTSETUP +LIBCRYPTSETUP_PLUGINS +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +BPF_FRAMEWORK -BTF -XKBCOMMON +UTMP +LIBARCHIVE) client # [ 7.224690] systemd[1]: Detected virtualization qemu. client # [ 7.227965] systemd[1]: Detected architecture arm64. client # [ 7.232356] systemd[1]: Detected first boot. client # [ 7.240300] systemd[1]: Initializing machine ID from random generator. worker1 # [ 7.453754] systemd[1]: Populated /etc with preset unit settings. worker2 # [ 7.452290] systemd[1]: bpf-restrict-fs: LSM BPF program attached client # [ 7.462442] systemd[1]: bpf-restrict-fs: LSM BPF program attached lb # [ 7.497313] systemd[1]: Populated /etc with preset unit settings. worker2 # [ 7.629476] systemd[1]: Applying preset policy. client # [ 7.647954] systemd[1]: Applying preset policy. worker1 # [ 7.999979] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker1 # [ 8.003446] systemd[1]: Stopped initrd-switch-root.service. worker1 # [ 8.006531] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. worker1 # [ 8.011080] systemd[1]: Created slice Slice /system/getty. worker1 # [ 8.013144] systemd[1]: Created slice User and Session Slice. worker1 # [ 8.016152] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. worker1 # [ 8.016853] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker1 # [ 8.017282] systemd[1]: Expecting device /dev/hvc0... worker1 # [ 8.021339] systemd[1]: Expecting device /dev/ttyAMA0... worker1 # [ 8.021975] systemd[1]: Reached target Local Encrypted Volumes. worker1 # [ 8.022393] systemd[1]: Stopped target initrd-fs.target. worker1 # [ 8.022711] systemd[1]: Stopped target initrd-root-fs.target. worker1 # [ 8.023030] systemd[1]: Stopped target initrd-switch-root.target. worker1 # [ 8.023333] systemd[1]: Reached target Virtual Machines and Containers. worker1 # [ 8.023641] systemd[1]: Reached target Path Units. worker1 # [ 8.023937] systemd[1]: Reached target Remote File Systems. worker1 # [ 8.024236] systemd[1]: Reached target Slice Units. worker1 # [ 8.024518] systemd[1]: Reached target Swaps. worker1 # [ 8.035519] systemd[1]: Listening on Query the User Interactively for a Password. worker1 # [ 8.041451] systemd[1]: Listening on Process Core Dump Socket. worker1 # [ 8.046394] systemd[1]: Listening on Credential Encryption/Decryption. worker1 # [ 8.050975] systemd[1]: Listening on Factory Reset Management. worker1 # [ 8.051480] systemd[1]: Listening on Hostname Service Socket. worker1 # [ 8.059336] systemd[1]: Starting Journal Log Access Socket... lb # [ 8.021825] systemd[1]: initrd-switch-root.service: Deactivated successfully. lb # [ 8.023493] systemd[1]: Stopped initrd-switch-root.service. worker1 # [ 8.062002] systemd[1]: Listening on Journal Audit Socket. lb # [ 8.027153] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. worker1 # [ 8.069332] systemd[1]: Listening on Console Output Muting Service Socket. lb # [ 8.031442] systemd[1]: Created slice Slice /system/getty. worker1 # [ 8.072042] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. lb # [ 8.033590] systemd[1]: Created slice User and Session Slice. worker1 # [ 8.073917] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os lb # [ 8.035192] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 8.037073] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker1 # [ 8.076016] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki lb # [ 8.039283] systemd[1]: Expecting device /dev/hvc0... lb # [ 8.041915] systemd[1]: Expecting device /dev/ttyAMA0... lb # [ 8.043078] systemd[1]: Reached target Local Encrypted Volumes. lb # [ 8.045718] systemd[1]: Stopped target initrd-fs.target. lb # [ 8.046778] systemd[1]: Stopped target initrd-root-fs.target. lb # [ 8.048519] systemd[1]: Stopped target initrd-switch-root.target. worker1 # [ 8.088190] systemd[1]: Listening on Disk Repartitioning Service Socket. worker1 # [ 8.088803] systemd[1]: Listening on udev Control Socket. lb # [ 8.051320] systemd[1]: Reached target Virtual Machines and Containers. worker1 # [ 8.089250] systemd[1]: Listening on udev Varlink Socket. lb # [ 8.052588] systemd[1]: Reached target Path Units. lb # [ 8.054206] systemd[1]: Reached target Remote File Systems. lb # [ 8.056745] systemd[1]: Reached target Slice Units. worker1 # [ 8.095636] systemd[1]: Mounting Huge Pages File System... lb # [ 8.057800] systemd[1]: Reached target Swaps. lb # [ 8.062420] systemd[1]: Listening on Query the User Interactively for a Password. worker1 # [ 8.101338] systemd[1]: Mounting POSIX Message Queue File System... lb # [ 8.067190] systemd[1]: Listening on Process Core Dump Socket. lb # [ 8.071099] systemd[1]: Listening on Credential Encryption/Decryption. lb # [ 8.075186] systemd[1]: Listening on Factory Reset Management. worker1 # [ 8.111615] systemd[1]: Mounting Kernel Debug File System... lb # [ 8.076814] systemd[1]: Listening on Hostname Service Socket. lb # [ 8.082910] systemd[1]: Starting Journal Log Access Socket... lb # [ 8.085506] systemd[1]: Listening on Journal Audit Socket. worker1 # [ 8.127481] systemd[1]: Mounting Kernel Trace File System... lb # [ 8.090104] systemd[1]: Listening on Console Output Muting Service Socket. lb # [ 8.092128] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. lb # [ 8.093727] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os lb # [ 8.096663] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker1 # [ 8.142911] systemd[1]: Starting Create List of Static Device Nodes... lb # [ 8.107484] systemd[1]: Listening on Disk Repartitioning Service Socket. lb # [ 8.109052] systemd[1]: Listening on udev Control Socket. lb # [ 8.110663] systemd[1]: Listening on udev Varlink Socket. lb # [ 8.115300] systemd[1]: Mounting Huge Pages File System... worker1 # [ 8.154610] systemd[1]: Starting Load Kernel Module 9pnet_virtio... worker2 # [ 8.137330] systemd[1]: Populated /etc with preset unit settings. lb # [ 8.121213] systemd[1]: Mounting POSIX Message Queue File System... worker1 # [ 8.161854] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 8.130549] systemd[1]: Mounting Kernel Debug File System... client # [ 8.122072] systemd[1]: Populated /etc with preset unit settings. worker1 # [ 8.177680] systemd[1]: Mounting Kernel Configuration File System... worker1 # [ 8.178257] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm lb # [ 8.143164] systemd[1]: Mounting Kernel Trace File System... worker1 # [ 8.178601] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore lb # [ 8.160666] systemd[1]: Starting Create List of Static Device Nodes... worker1 # [ 8.199886] systemd[1]: Starting Load Kernel Module fuse... worker1 # [ 8.206079] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 lb # [ 8.178617] systemd[1]: Starting Load Kernel Module 9pnet_virtio... lb # [ 8.180335] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 8.193386] systemd[1]: Mounting Kernel Configuration File System... lb # [ 8.195953] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm lb # [ 8.201328] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker1 # [ 8.289324] systemd[1]: Starting Journal Service... worker1 # [ 8.313522] systemd[1]: Starting Load Kernel Modules... worker1 # [ 8.344121] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker1 # [ 8.354729] systemd[1]: Starting Remount Root and Kernel File Systems... worker1 # [ 8.357633] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 8.372448] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 8.379623] systemd[1]: Listening on Journal Log Access Socket. worker1 # [ 8.383974] systemd[1]: Mounted Huge Pages File System. worker1 # [ 8.390131] systemd[1]: Mounted POSIX Message Queue File System. worker1 # [ 8.392842] systemd[1]: Mounted Kernel Debug File System. worker1 # [ 8.393353] systemd[1]: Mounted Kernel Trace File System. worker1 # [ 8.396830] systemd[1]: Mounted Kernel Configuration File System. lb # [ 8.361792] systemd[1]: Starting Load Kernel Module fuse... lb # [ 8.377812] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker1 # [ 8.461104] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 8.424297] systemd[1]: Starting Journal Service... worker1 # [ 8.468259] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 8.474171] systemd[1]: Starting Load Kernel Modules... lb # [ 8.489767] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker1 # [ 8.530276] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. worker1 # [ 8.534201] systemd[1]: Finished Load Kernel Module 9pnet_virtio. lb # [ 8.517824] systemd[1]: Starting Remount Root and Kernel File Systems... lb # [ 8.518350] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 8.564405] systemd[1]: Finished Load Kernel Modules. lb # [ 8.528904] fuse: init (API version 7.45) worker1 # [ 8.582978] systemd[1]: Starting Firewall... lb # [ 8.544464] systemd[1]: Starting Coldplug All udev Devices... worker1 # [ 8.586165] EXT4-fs (vda): re-mounted b7494a4a-0c71-4154-867d-11aa1912d093. worker1 # [ 8.601287] fuse: init (API version 7.45) lb # [ 8.574749] systemd-journald[267]: Collecting audit messages is enabled. worker1 # [ 8.615310] systemd[1]: Starting Apply Kernel Variables... worker1 # [ 8.626896] systemd[1]: Finished Remount Root and Kernel File Systems. worker1 # [ 8.629772] systemd[1]: Listening on Disk Image Download Service Socket. worker1 # [ 8.630204] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore lb # [ 8.600961] systemd[1]: Listening on Journal Log Access Socket. worker1 # [ 8.640834] systemd[1]: Starting Load/Save OS Random Seed... lb # [ 8.590303] systemd[1]: Queued start job for default target Multi-User System. worker1 # [ 8.643632] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 8.647663] systemd-journald[267]: Collecting audit messages is enabled. lb # [ 8.595772] systemd[1]: systemd-journald.service: Deactivated successfully.[ 8.613363] systemd[1]: Started Journal Service. lb # lb # [ 8.615979] systemd[1]: Mounted Huge Pages File System. worker1 # [ 8.670035] systemd[1]: modprobe@fuse.service: Deactivated successfully. worker1 # [ 8.673254] systemd[1]: Finished Load Kernel Module fuse. lb # [ 8.624756] systemd[1]: Mounted POSIX Message Queue File System. lb # [ 8.625876] systemd[1]: Mounted Kernel Debug File System. lb # [ 8.626866] systemd[1]: Mounted Kernel Trace File System. worker1 # [ 8.682503] systemd[1]: Mounting FUSE Control File System... lb # [ 8.640908] systemd[1]: Finished Create List of Static Device Nodes. lb # [ 8.649610] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. worker1 # [ 8.706034] systemd[1]: Started Journal Service. client # [ 8.656564] systemd[1]: initrd-switch-root.service: Deactivated successfully. lb # [ 8.659169] systemd[1]: Finished Load Kernel Module 9pnet_virtio. client # [ 8.658032] systemd[1]: Stopped initrd-switch-root.service. client # [ 8.661506] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. lb # [ 8.663664] systemd[1]: Mounted Kernel Configuration File System. worker2 # [ 8.698442] systemd[1]: initrd-switch-root.service: Deactivated successfully. worker2 # [ 8.700203] systemd[1]: Stopped initrd-switch-root.service. client # [ 8.665687] systemd[1]: Created slice Slice /system/getty. worker2 # [ 8.703697] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. lb # [ 8.667034] systemd[1]: modprobe@fuse.service: Deactivated successfully.[ 8.683459] EXT4-fs (vda): re-mounted 81debc9d-e7d6-4829-8d96-4b8ab9939e27. client # [ 8.668911] systemd[1]: Created slice User and Session Slice. worker2 # [ 8.708386] systemd[1]: Created slice Slice /system/getty. client # [ 8.671466] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. client # [ 8.674350] systemd[1]: Started Forward Password Requests to Wall Directory Watch. worker2 # [ 8.710465] systemd[1]: Created slice User and Session Slice. lb # worker1 # [ 8.713190] systemd[1]: Queued start job for default target Multi-User System. worker1 # [ 8.714686] systemd[1]: systemd-journald.service: Deactivated successfully. client # [ 8.675785] systemd[1]: Expecting device /dev/hvc0... worker2 # [ 8.712278] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. lb # [ 8.680407] systemd[1]: Finished Load Kernel Module fuse. client # [ 8.678451] systemd[1]: Expecting device /dev/ttyAMA0... worker2 # [ 8.715408] systemd[1]: Started Forward Password Requests to Wall Directory Watch. lb # [ 8.682696] systemd-modules-load[268]: Module 'atkbd' is built in client # [ 8.680620] systemd[1]: Reached target Local Encrypted Volumes. worker2 # [ 8.716871] systemd[1]: Expecting device /dev/hvc0... client # [ 8.682905] systemd[1]: Stopped target initrd-fs.target. worker2 # [ 8.719549] systemd[1]: Expecting device /dev/ttyAMA0... worker2 # [ 8.720749] systemd[1]: Reached target Local Encrypted Volumes. client # [ 8.685108] systemd[1]: Stopped target initrd-root-fs.target. worker2 # [ 8.722550] systemd[1]: Stopped target initrd-fs.target. client # [ 8.687446] systemd[1]: Stopped target initrd-switch-root.target. worker2 # [ 8.724240] systemd[1]: Stopped target initrd-root-fs.target. lb # [ 8.687410] systemd-modules-load[268]: Module 'loop' is built in client # [ 8.689763] systemd[1]: Reached target Virtual Machines and Containers. worker2 # [ 8.726059] systemd[1]: Stopped target initrd-switch-root.target. lb # [ 8.692960] systemd[1]: Finished Load Kernel Modules. client # [ 8.692026] systemd[1]: Reached target Path Units. worker2 # [ 8.727935] systemd[1]: Reached target Virtual Machines and Containers. worker1 # [ 8.732674] systemd-modules-load[268]: Module 'atkbd' is built in client # [ 8.694146] systemd[1]: Reached target Remote File Systems. worker2 # [ 8.730662] systemd[1]: Reached target Path Units. worker2 # [ 8.731731] systemd[1]: Reached target Remote File Systems. client # [ 8.696411] systemd[1]: Reached target Slice Units. worker2 # [ 8.733443] systemd[1]: Reached target Slice Units. client # [ 8.698130] systemd[1]: Reached target Swaps. lb # [ 8.700597] systemd[1]: Finished Remount Root and Kernel File Systems. worker2 # [ 8.735084] systemd[1]: Reached target Swaps. client # [ 8.703336] systemd[1]: Listening on Query the User Interactively for a Password. worker2 # [ 8.740320] systemd[1]: Listening on Query the User Interactively for a Password. worker1 # [ 8.744177] systemd-modules-load[268]: Module 'loop' is built in lb # [ 8.708191] systemd[1]: Listening on Disk Image Download Service Socket. client # [ 8.708632] systemd[1]: Listening on Process Core Dump Socket. worker2 # [ 8.746144] systemd[1]: Listening on Process Core Dump Socket. client # [ 8.712987] systemd[1]: Listening on Credential Encryption/Decryption. worker2 # [ 8.750975] systemd[1]: Listening on Credential Encryption/Decryption. client # [ 8.717199] systemd[1]: Listening on Factory Reset Management. lb # [ 8.720220] systemd[1]: Mounting FUSE Control File System... client # [ 8.718523] systemd[1]: Listening on Hostname Service Socket. worker2 # [ 8.755603] systemd[1]: Listening on Factory Reset Management. worker1 # [ 8.757169] systemd[1]: Starting Flush Journal to Persistent Storage... worker2 # [ 8.757065] systemd[1]: Listening on Hostname Service Socket. client # [ 8.724979] systemd[1]: Starting Journal Log Access Socket... client # [ 8.726782] systemd[1]: Listening on Journal Audit Socket. worker2 # [ 8.764468] systemd[1]: Starting Journal Log Access Socket... client # [ 8.731503] systemd[1]: Listening on Console Output Muting Service Socket. worker2 # [ 8.767249] systemd[1]: Listening on Journal Audit Socket. lb # [ 8.733763] systemd[1]: Starting Firewall... worker1 # [ 8.769926] systemd[1]: Mounted FUSE Control File System. client # [ 8.733314] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. worker2 # [ 8.772463] systemd[1]: Listening on Console Output Muting Service Socket. client # [ 8.735513] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 8.774654] systemd[1]: Listening on Userspace Out-Of-Memory (OOM) Killer Socket. client # [ 8.738777] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker2 # [ 8.776483] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 8.778916] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki lb # [ 8.749402] systemd-oomd[270]: No swap; memory pressure usage will be degraded client # [ 8.750070] systemd[1]: Listening on Disk Repartitioning Service Socket. client # [ 8.751642] systemd[1]: Listening on udev Control Socket. client # [ 8.754171] systemd[1]: Listening on udev Varlink Socket. client # [ 8.758706] systemd[1]: Mounting Huge Pages File System... worker2 # [ 8.792788] systemd[1]: Listening on Disk Repartitioning Service Socket. worker2 # [ 8.793381] systemd[1]: Listening on udev Control Socket. worker2 # [ 8.793795] systemd[1]: Listening on udev Varlink Socket. client # [ 8.763427] systemd[1]: Mounting POSIX Message Queue File System... lb # [ 8.766737] systemd[1]: Starting Flush Journal to Persistent Storage... worker2 # [ 8.800557] systemd[1]: Mounting Huge Pages File System... lb # [ 8.768655] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 8.807338] systemd[1]: Mounting POSIX Message Queue File System... worker1 # [ 8.809990] systemd-oomd[269]: No swap; memory pressure usage will be degraded client # [ 8.774965] systemd[1]: Mounting Kernel Debug File System... lb # [ 8.775660] systemd[1]: Starting Load/Save OS Random Seed... worker2 # [ 8.817390] systemd[1]: Mounting Kernel Debug File System... client # [ 8.786936] systemd[1]: Mounting Kernel Trace File System... worker1 # [ 8.824720] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. lb # [ 8.801771] systemd[1]: Starting Apply Kernel Variables... worker2 # [ 8.836111] systemd[1]: Mounting Kernel Trace File System... client # [ 8.803678] systemd[1]: Starting Create List of Static Device Nodes... client # [ 8.817322] systemd[1]: Starting Load Kernel Module 9pnet_virtio... worker2 # [ 8.851256] systemd[1]: Starting Create List of Static Device Nodes... client # [ 8.819409] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 8.861494] systemd[1]: Finished Load/Save OS Random Seed. worker1 # [ 8.864953] systemd[1]: Reached target First Boot Complete. worker1 # [ 8.886190] systemd-journald[267]: Received client request to flush runtime journal. client # [ 8.833604] systemd[1]: Mounting Kernel Configuration File System... client # [ 8.838438] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm client # [ 8.841489] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker2 # [ 8.877710] systemd[1]: Starting Load Kernel Module 9pnet_virtio... lb # [ 8.847913] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... worker2 # [ 8.882703] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 8.854435] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 8.859574] systemd[1]: Starting Load Kernel Module fuse... lb # [ 8.864561] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. client # [ 8.862137] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker2 # [ 8.895778] systemd[1]: Mounting Kernel Configuration File System... worker2 # [ 8.896308] systemd[1]: Load Kernel Module drm skipped, unmet condition check ConditionKernelModuleLoaded=!drm worker2 # [ 8.896638] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker2 # [ 8.924639] systemd[1]: Starting Load Kernel Module fuse... worker2 # [ 8.925172] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker1 # [ 8.940366] systemd[1]: Finished Apply Kernel Variables. worker1 # [ 8.948714] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. worker1 # [ 8.953914] systemd[1]: Starting Create Static Device Nodes in /dev... worker1 # [ 8.958268] systemd[1]: Finished Flush Journal to Persistent Storage. lb # [ 8.943815] systemd-journald[267]: Received client request to flush runtime journal. client # [ 8.953367] systemd[1]: Starting Journal Service... worker2 # [ 8.999614] systemd[1]: Starting Journal Service... worker2 # [ 9.022102] systemd[1]: Starting Load Kernel Modules... client # [ 8.994038] systemd[1]: Starting Load Kernel Modules... worker2 # [ 9.041334] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker2 # [ 9.059423] systemd[1]: Starting Remount Root and Kernel File Systems... worker2 # [ 9.062275] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker1 # [ 9.068266] systemd[1]: Finished Create Static Device Nodes in /dev. lb # [ 9.036323] systemd[1]: Finished Load/Save OS Random Seed. client # [ 9.037480] systemd[1]: Starting Userspace Out-Of-Memory (OOM) Killer... worker1 # [ 9.077382] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 9.046608] systemd[1]: Reached target First Boot Complete. worker2 # [ 9.079846] systemd[1]: Starting Coldplug All udev Devices... lb # [ 9.052352] systemd[1]: Mounted FUSE Control File System. worker1 # [ 9.089046] systemd[1]: Mounting /run/wrappers... worker1 # [ 9.090198] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 9.055889] systemd[1]: Finished Apply Kernel Variables. worker2 # [ 9.090641] systemd[1]: Listening on Journal Log Access Socket. worker2 # [ 9.093452] systemd[1]: Mounted Huge Pages File System. lb # [ 9.062755] systemd[1]: Finished Flush Journal to Persistent Storage. worker2 # [ 9.096286] systemd[1]: Mounted POSIX Message Queue File System. worker2 # [ 9.103006] systemd[1]: Mounted Kernel Debug File System. worker2 # [ 9.103752] systemd[1]: Mounted Kernel Trace File System. worker2 # [ 9.104290] systemd[1]: Mounted Kernel Configuration File System. client # [ 9.071690] systemd[1]: Starting Remount Root and Kernel File Systems... client # [ 9.072218] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 9.082727] fuse: init (API version 7.45) worker2 # [ 9.133796] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 9.142920] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... client # [ 9.111504] systemd-journald[274]: Collecting audit messages is enabled. client # [ 9.118434] systemd[1]: Starting Coldplug All udev Devices... lb # [ 9.123815] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. lb # [ 9.135065] systemd[1]: Starting Create Static Device Nodes in /dev... client # [ 9.122180] systemd[1]: Queued start job for default target Multi-User System. worker1 # [ 9.173381] systemd[1]: Mounted /run/wrappers. client # [ 9.139162] systemd[1]: Started Journal Service. client # [ 9.130583] systemd[1]: systemd-journald.service: Deactivated successfully. worker1 # [ 9.181437] systemd[1]: Reached target Local File Systems. worker1 # [ 9.192124] systemd[1]: Listening on Boot Loader Control Service Socket. worker1 # [ 9.200618] systemd[1]: Starting register-nix-paths.service... client # [ 9.157973] systemd[1]: Listening on Journal Log Access Socket. worker1 # [ 9.211065] systemd[1]: Starting Create SUID/SGID Wrappers... client # [ 9.164306] systemd[1]: Mounted Huge Pages File System. client # [ 9.166672] systemd[1]: Mounted POSIX Message Queue File System. worker1 # [ 9.228323] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. client # [ 9.174857] systemd[1]: Mounted Kernel Debug File System. worker1 # [ 9.230201] systemd[1]: Starting Save Transient machine-id to Disk... client # [ 9.183678] systemd[1]: Mounted Kernel Trace File System. client # [ 9.190177] systemd-modules-load[275]: Module 'atkbd' is built in worker1 # [ 9.244544] systemd[1]: Starting Create System Files and Directories... client # [ 9.194807] systemd[1]: Finished Create List of Static Device Nodes. worker2 # [ 9.247463] EXT4-fs (vda): re-mounted 1901d721-ca8c-4062-ae20-6ee1aeb92fa2. client # [ 9.204306] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. client # [ 9.205476] systemd-modules-load[275]: Module 'loop' is built in client # [ 9.206344] systemd[1]: Finished Load Kernel Module 9pnet_virtio. client # [ 9.207221] systemd[1]: Mounted Kernel Configuration File System. client # [ 9.220301] EXT4-fs (vda): re-mounted bbd434e5-d1a0-43bf-a370-65bcbd9a72f5. worker2 # [ 9.260369] systemd[1]: Finished Remount Root and Kernel File Systems. worker2 # [ 9.261313] systemd[1]: Listening on Disk Image Download Service Socket. worker2 # [ 9.265086] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore client # [ 9.221760] systemd[1]: modprobe@fuse.service: Deactivated successfully. client # [ 9.226275] systemd[1]: Finished Load Kernel Module fuse. lb # [ 9.242025] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 9.275786] systemd[1]: Starting Load/Save OS Random Seed... worker2 # [ 9.278659] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os client # [ 9.233933] systemd[1]: Finished Load Kernel Modules. client # [ 9.234799] systemd[1]: Finished Remount Root and Kernel File Systems. worker2 # [ 9.283477] systemd[1]: modprobe@9pnet_virtio.service: Deactivated successfully. lb # [ 9.251166] systemd[1]: Reached target Preparation for Local File Systems. worker2 # [ 9.288109] systemd[1]: Finished Load Kernel Module 9pnet_virtio. lb # [ 9.256359] systemd[1]: Mounting /run/wrappers... worker2 # [ 9.293926] systemd[1]: Finished Load Kernel Modules. client # [ 9.243429] systemd[1]: Listening on Disk Image Download Service Socket. lb # [ 9.261571] systemd[1]: Starting Rule-based Manager for Device Events and Files... client # [ 9.253961] systemd[1]: Mounting FUSE Control File System... worker2 # [ 9.307888] systemd[1]: Starting Firewall... client # [ 9.270174] systemd[1]: Starting Flush Journal to Persistent Storage... client # [ 9.271392] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 9.324380] systemd[1]: Starting Apply Kernel Variables... client # [ 9.292214] systemd[1]: Starting Load/Save OS Random Seed... client # [ 9.295022] systemd-oomd[277]: No swap; memory pressure usage will be degraded worker1 # [ 9.350714] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker2 # [ 9.356874] systemd-journald[276]: Collecting audit messages is enabled. worker1 # [ 9.364673] systemd[1]: Finished Save Transient machine-id to Disk. worker2 # [ 9.369300] fuse: init (API version 7.45) client # [ 9.325029] systemd[1]: Starting Apply Kernel Variables... lb # [ 9.349916] systemd[1]: Mounted /run/wrappers. client # [ 9.339987] systemd[1]: Starting Create Static Device Nodes in /dev gracefully... lb # [ 9.357926] systemd[1]: Reached target Local File Systems. lb # [ 9.368488] systemd[1]: Listening on Boot Loader Control Service Socket. client # [ 9.357583] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 9.410289] systemd[1]: Started Journal Service. client # [ 9.364243] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. lb # [ 9.380392] systemd[1]: Starting register-nix-paths.service... worker2 # [ 9.405002] systemd[1]: Queued start job for default target Multi-User System. worker2 # [ 9.409430] systemd[1]: systemd-journald.service: Deactivated successfully. lb # [ 9.393567] systemd[1]: Starting Create SUID/SGID Wrappers... lb # [ 9.396882] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker2 # [ 9.428445] systemd-modules-load[277]: Module 'atkbd' is built in client # [ 9.410904] systemd-journald[274]: Received client request to flush runtime journal. worker2 # [ 9.442964] systemd-modules-load[277]: Module 'loop' is built in worker2 # [ 9.446663] systemd[1]: Starting Flush Journal to Persistent Storage... lb # [ 9.446304] systemd[1]: Starting Save Transient machine-id to Disk... worker2 # [ 9.472636] systemd[1]: modprobe@fuse.service: Deactivated successfully. lb # [ 9.462293] systemd[1]: Starting Create System Files and Directories... worker2 # [ 9.482603] systemd[1]: Finished Load Kernel Module fuse. worker2 # [ 9.499300] systemd-oomd[278]: No swap; memory pressure usage will be degraded worker1 # [ 9.533078] systemd[1]: Finished Create System Files and Directories. worker2 # [ 9.514173] systemd[1]: Started Userspace Out-Of-Memory (OOM) Killer. worker1 # [ 9.540615] systemd[1]: Starting Rebuild Journal Catalog... worker2 # [ 9.525383] systemd[1]: Finished Load/Save OS Random Seed. client # [ 9.501555] systemd[1]: Finished Load/Save OS Random Seed. worker1 # [ 9.552429] systemd[1]: Starting Record System Boot/Shutdown in UTMP... client # [ 9.508667] systemd[1]: Reached target First Boot Complete. worker2 # [ 9.535403] systemd[1]: Reached target First Boot Complete. client # [ 9.512975] systemd[1]: Mounted FUSE Control File System. worker2 # [ 9.550470] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 9.518926] systemd[1]: Finished Flush Journal to Persistent Storage. worker2 # [ 9.571938] systemd-journald[276]: Received client request to flush runtime journal. client # [ 9.530808] systemd[1]: Finished Apply Kernel Variables. client # [ 9.568922] systemd[1]: Finished Create Static Device Nodes in /dev gracefully. client # [ 9.576649] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 9.642069] systemd[1]: Starting Create Static Device Nodes in /dev... worker2 # [ 9.651315] systemd[1]: Finished Apply Kernel Variables. worker2 # [ 9.673096] systemd[1]: Finished Flush Journal to Persistent Storage. worker2 # [ 9.696249] systemd[1]: Mounting FUSE Control File System... lb # [ 9.679556] systemd[1]: Finished Save Transient machine-id to Disk. worker1 # [ 9.719429] systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [ 9.705120] systemd[1]: Finished Create Static Device Nodes in /dev. client # [ 9.708081] systemd[1]: Reached target Preparation for Local File Systems. worker2 # [ 9.744867] systemd[1]: Mounted FUSE Control File System. client # [ 9.715754] systemd[1]: Mounting /run/wrappers... lb # [ 9.734664] systemd[1]: Finished Create System Files and Directories. client # [ 9.724818] systemd[1]: Starting Rule-based Manager for Device Events and Files... worker2 # [ 9.767262] systemd[1]: Finished Create Static Device Nodes in /dev. worker2 # [ 9.772639] systemd[1]: Reached target Preparation for Local File Systems. lb # [ 9.754406] systemd[1]: Starting Rebuild Journal Catalog... worker2 # [ 9.777267] systemd[1]: Mounting /run/wrappers... worker2 # [ 9.785711] systemd[1]: Starting Rule-based Manager for Device Events and Files... lb # [ 9.766750] systemd[1]: Starting Record System Boot/Shutdown in UTMP... worker1 # [ 9.825265] systemd-udevd[310]: Using default interface naming scheme 'v261'. client # [ 9.798570] systemd[1]: Mounted /run/wrappers. client # [ 9.799807] systemd[1]: Reached target Local File Systems. client # [ 9.806710] systemd[1]: Listening on Boot Loader Control Service Socket. worker1 # [ 9.860218] systemd[1]: Finished Rebuild Journal Catalog. worker1 # [ 9.867289] systemd[1]: Starting Update is Completed... client # [ 9.819942] systemd[1]: Starting register-nix-paths.service... worker2 # [ 9.860803] systemd[1]: Mounted /run/wrappers. client # [ 9.830085] systemd[1]: Starting Create SUID/SGID Wrappers... worker2 # [ 9.863130] systemd[1]: Reached target Local File Systems. client # [ 9.831278] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker2 # [ 9.871655] systemd[1]: Listening on Boot Loader Control Service Socket. lb # [ 9.856844] systemd-udevd[315]: Using default interface naming scheme 'v261'. client # [ 9.843183] systemd[1]: Starting Save Transient machine-id to Disk... worker2 # [ 9.880499] systemd[1]: Starting register-nix-paths.service... worker2 # [ 9.892332] systemd[1]: Starting Create SUID/SGID Wrappers... worker2 # [ 9.903526] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. client # [ 9.880732] systemd-udevd[307]: Using default interface naming scheme 'v261'. worker2 # [ 9.917186] systemd[1]: Starting Save Transient machine-id to Disk... worker2 # [ 9.930132] systemd[1]: Starting Create System Files and Directories... client # [ 9.891290] systemd[1]: Starting Create System Files and Directories... lb # [ 9.937768] systemd[1]: Finished Record System Boot/Shutdown in UTMP. worker1 # [ 9.976698] systemd[1]: Finished Update is Completed. lb # [ 10.002926] systemd[1]: Finished Rebuild Journal Catalog. client # [ 9.990668] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. worker2 # [ 10.033644] systemd-udevd[318]: Using default interface naming scheme 'v261'. client # [ 10.003153] systemd[1]: Finished Save Transient machine-id to Disk. lb # [ 10.024588] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. lb # [ 10.028264] systemd[1]: Starting Update is Completed... worker2 # [ 10.076399] systemd[1]: Finished Save Transient machine-id to Disk. worker1 # [ 10.129578] systemd[1]: Started Rule-based Manager for Device Events and Files. client # [ 10.125178] systemd[1]: Finished Create System Files and Directories. lb # [ 10.145947] systemd[1]: Finished Update is Completed. client # [ 10.144732] systemd[1]: Starting Rebuild Journal Catalog... client # [ 10.157592] systemd[1]: Starting Record System Boot/Shutdown in UTMP... worker2 # [ 10.192293] systemd[1]: Finished Create System Files and Directories. client # [ 10.176504] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 10.205358] systemd[1]: Starting Rebuild Journal Catalog... worker2 # [ 10.217704] systemd[1]: Starting Record System Boot/Shutdown in UTMP... lb # [ 10.207659] systemd[1]: Started Rule-based Manager for Device Events and Files. client # [ 10.316836] systemd[1]: Finished Record System Boot/Shutdown in UTMP. worker2 # [ 10.362639] systemd[1]: Finished Record System Boot/Shutdown in UTMP. client # [ 10.351183] systemd[1]: Finished Rebuild Journal Catalog. client # [ 10.359708] systemd[1]: Starting Update is Completed... worker2 # [ 10.444832] systemd[1]: Started Rule-based Manager for Device Events and Files. worker2 # [ 10.457931] systemd[1]: Finished Rebuild Journal Catalog. worker2 # [ 10.469654] systemd[1]: Starting Update is Completed... client # [ 10.437774] systemd[1]: Finished Update is Completed. worker2 # [ 10.589848] systemd[1]: Finished Update is Completed. worker1 # [ 10.617564] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker1 # [ 10.621759] systemd[1]: Finished Create SUID/SGID Wrappers. worker2 # [ 10.687674] systemd[1]: etc-machine\x2did.mount: Deactivated successfully. lb # [ 10.816705] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. lb # [ 10.828192] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 10.985916] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. client # [ 10.987117] systemd[1]: Finished Create SUID/SGID Wrappers. worker2 # [ 11.128159] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker2 # [ 11.129548] systemd[1]: Finished Create SUID/SGID Wrappers. client # [ 11.118302] systemd[1]: Finished Coldplug All udev Devices. client # [ 11.187720] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 11.249272] systemd[1]: Finished register-nix-paths.service. worker1 # [ 11.325716] systemd[1]: Finished Firewall. lb # [ 11.311785] systemd[1]: Finished Coldplug All udev Devices. lb # [ 11.314713] systemd[1]: Reached target System Initialization. lb # [ 11.316857] systemd[1]: Started Discard unused filesystem blocks once a week. lb # [ 11.322842] systemd[1]: Started niks3 garbage collection timer. lb # [ 11.329954] systemd[1]: Started Daily Cleanup of Temporary Directories. lb # [ 11.330954] systemd[1]: Reached target Timer Units. lb # [ 11.331677] systemd[1]: Listening on D-Bus System Message Bus Socket. lb # [ 11.339394] systemd[1]: Listening on niks3 server socket. lb # [ 11.341059] systemd[1]: Listening on Nix Daemon Socket. lb # [ 11.345808] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. lb # [ 11.353107] systemd[1]: Reached target Socket Units. lb # [ 11.354083] systemd[1]: Reached target Basic System. lb # [ 11.364399] systemd[1]: Starting Import lastlog data into lastlog2 database... lb # [ 11.366238] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 11.395325] systemd[1]: Starting Post-Boot Actions... worker1 # [ 11.438607] systemd[1]: Finished Coldplug All udev Devices. lb # [ 11.406191] systemd[1]: Started Reset console on configuration changes. client # [ 11.408674] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. lb # [ 11.443930] systemd[1]: Starting resolvconf update... lb # [ 11.482636] systemd[1]: Started rustfs.service. client # [ 11.474875] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. lb # [ 11.514757] nsncd[477]: Sep 22 10:57:01.971 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" lb # [ 11.529184] systemd[1]: Starting rustfs-bucket.service... lb # [ 11.547216] systemd[1]: Started Name Service Cache Daemon (nsncd). lb # [ 11.617090] systemd[1]: Finished Post-Boot Actions. lb # [ 11.652166] systemd[1]: Reached target Host and Network Name Lookups. lb # [ 11.655793] systemd[1]: Reached target User and Group Name Lookups. lb # [ 11.667114] systemd[1]: Starting D-Bus System Message Bus... worker1 # [ 11.750911] systemd[1]: Finished register-nix-paths.service. worker1 # [ 11.753350] systemd[1]: Reached target System Initialization. worker1 # [ 11.755391] systemd[1]: Started Discard unused filesystem blocks once a week. lb # [ 11.719249] systemd[1]: Starting User Login Management... worker1 # [ 11.757861] systemd[1]: Started Daily Cleanup of Temporary Directories. worker1 # [ 11.760244] systemd[1]: Reached target Timer Units. worker1 # [ 11.765446] systemd[1]: Listening on D-Bus System Message Bus Socket. worker1 # [ 11.770752] systemd[1]: Listening on Nix Daemon Socket. worker1 # [ 11.779225] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 11.787888] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. client # [ 11.748339] systemd[1]: Finished register-nix-paths.service. client # [ 11.752245] systemd[1]: Reached target System Initialization. worker1 # [ 11.804180] systemd[1]: Reached target Socket Units. client # [ 11.756482] systemd[1]: Started Discard unused filesystem blocks once a week. client # [ 11.757714] systemd[1]: Started Daily Cleanup of Temporary Directories. worker1 # [ 11.812303] systemd[1]: Reached target Basic System. client # [ 11.759536] systemd[1]: Reached target Timer Units. client # [ 11.764422] systemd[1]: Listening on D-Bus System Message Bus Socket. client # [ 11.765399] systemd[1]: Listening on Nix Daemon Socket. client # [ 11.766155] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. worker1 # [ 11.816431] systemd[1]: Starting Import lastlog data into lastlog2 database... client # [ 11.767359] systemd[1]: Reached target Socket Units. lb # [ 11.789092] systemd[1]: Finished Import lastlog data into lastlog2 database. client # [ 11.776158] systemd[1]: Reached target Basic System. client # [ 11.780258] systemd[1]: Started backdoor.service. client # [ 11.782155] systemd[1]: Starting Import lastlog data into lastlog2 database... worker1 # [ 11.823428] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 11.796551] systemd[1]: Finished Firewall. client # [ 11.786285] systemd[1]: Starting nftables firewall... worker1 # [ 11.834396] systemd[1]: Starting Post-Boot Actions... worker1 # [ 11.845821] systemd[1]: Started Reset console on configuration changes. client # [ 11.800120] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 11.856784] systemd[1]: Starting resolvconf update... worker2 # [ 11.840646] systemd[1]: Finished Firewall. worker1 # [ 11.868865] systemd[1]: Starting D-Bus System Message Bus... client # [ 11.825557] (udev-worker)[337]: Network interface NamePolicy= disabled on kernel command line. client # [ 11.844604] systemd[1]: Starting Post-Boot Actions... client # [ 11.863808] systemd[1]: Started Reset console on configuration changes. worker1 # [ 11.945503] systemd[1]: Finished Post-Boot Actions. worker2 # [ 11.952437] systemd[1]: Finished Coldplug All udev Devices. client # [ 11.922495] (udev-worker)[343]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. client # [ 11.930490] (udev-worker)[343]: Network interface NamePolicy= disabled on kernel command line. client # [ 11.938336] systemd[1]: Starting resolvconf update... lb # [ 11.969454] dbus-broker-launch[514]: Looking up NSS user entry for 'systemd-timesync'... worker1 # [ 12.010147] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 11.998131] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs worker1 # [ 12.018939] systemd[1]: Reached target Host and Network Name Lookups. lb # [ 11.980591] dbus-broker-launch[514]: NSS returned no entry for 'systemd-timesync' client # connecting to host... worker1 # [ 12.030954] nsncd[489]: Sep 22 10:57:02.483 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" lb # [ 11.990097] dbus-broker-launch[514]: Invalid user-name in /nix/store/kxmx3smba321l9v2dpmh9zahhsbc77yh-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker1 # [ 12.042316] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 12.051084] systemd[1]: Starting User Login Management... client # [ 12.007191] nsncd[397]: Sep 22 10:57:02.473 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" lb # [ 12.027509] systemd[1]: Started D-Bus System Message Bus. client # [ 12.017017] systemd[1]: Finished Post-Boot Actions. worker1 # [ 12.067545] systemd[1]: Finished Import lastlog data into lastlog2 database. client # [ 12.019684] systemd[1]: Started Name Service Cache Daemon (nsncd). client: Guest shell says: b'Spawning backdoor root shell...\n' lb # [ 12.056691] systemd-logind[519]: New seat seat0. lb # [ 12.065320] systemd[1]: Started User Login Management. lb # [ 12.080513] systemd[1]: Starting linger-users.service... client # [ 12.071054] systemd[1]: Reached target Host and Network Name Lookups. client # [ 12.077916] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 12.133638] dbus-broker-launch[493]: Looking up NSS user entry for 'systemd-timesync'... client: connected to guest root shell lb # [ 12.115264] dbus-broker-launch[514]: Ready client: (connecting took 12.53 seconds) client # [ 12.097243] systemd[1]: Starting D-Bus System Message Bus... client: (finished: waiting for the VM to finish booting, in 12.53 seconds) worker1 # [ 12.159279] dbus-broker-launch[493]: NSS returned no entry for 'systemd-timesync' worker1 # [ 12.162612] dbus-broker-launch[493]: Invalid user-name in /nix/store/a9ijqx61hpssavy5d9lh27li51m8hnl7-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" client # [ 12.136329] systemd[1]: Starting User Login Management... lb # [ 12.154394] systemd[1]: Stopped target Host and Network Name Lookups. lb # [ 12.157794] systemd[1]: Stopping Host and Network Name Lookups... client # [ 12.144333] systemd[1]: Finished Import lastlog data into lastlog2 database. lb # [ 12.164669] systemd[1]: Stopped target User and Group Name Lookups. lb # [ 12.173975] systemd[1]: Stopping User and Group Name Lookups... worker1 # [ 12.211883] systemd[1]: Started D-Bus System Message Bus. lb # [ 12.181551] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker2 # [ 12.202894] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. lb # [ 12.188298] systemd[1]: nscd.service: Deactivated successfully. lb # [ 12.196000] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker1 # [ 12.244811] systemd-logind[501]: New seat seat0. worker1 # [ 12.255500] systemd[1]: Started User Login Management. lb # [ 12.219731] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 12.225262] systemd[1]: linger-users.service: Deactivated successfully. worker1 # [ 12.263933] systemd[1]: Starting linger-users.service... worker2 # [ 12.248133] systemd[1]: Finished register-nix-paths.service. lb # [ 12.232345] systemd[1]: Finished linger-users.service. worker2 # [ 12.254309] systemd[1]: Reached target System Initialization. worker2 # [ 12.256418] systemd[1]: Started Discard unused filesystem blocks once a week. worker2 # [ 12.259224] systemd[1]: Started Daily Cleanup of Temporary Directories. worker2 # [ 12.262531] systemd[1]: Reached target Timer Units. worker2 # [ 12.266506] systemd[1]: Listening on D-Bus System Message Bus Socket. worker1 # [ 12.285887] dbus-broker-launch[493]: Ready worker2 # [ 12.269249] systemd[1]: Listening on Nix Daemon Socket. worker2 # [ 12.272432] systemd[1]: Listening on Nix worker-protocol over gRPC. worker2 # [ 12.277301] systemd[1]: Listening on Virtual Machine and Container Registration Service Socket. worker1 # [ 12.297981] systemd[1]: Stopped target Host and Network Name Lookups. worker2 # [ 12.287679] systemd[1]: Reached target Socket Units. worker1 # [ 12.304687] systemd[1]: Stopping Host and Network Name Lookups... worker1 # [ 12.307815] systemd[1]: Stopped target User and Group Name Lookups. worker2 # [ 12.290278] systemd[1]: Reached target Basic System. worker2 # [ 12.300580] systemd[1]: Starting Import lastlog data into lastlog2 database... worker1 # [ 12.317521] systemd[1]: Stopping User and Group Name Lookups... worker2 # [ 12.306437] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker1 # [ 12.327605] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker2 # [ 12.315360] systemd[1]: Starting Post-Boot Actions... worker1 # [ 12.334172] systemd[1]: nscd.service: Deactivated successfully. worker1 # [ 12.339105] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker2 # [ 12.328697] systemd[1]: Started Reset console on configuration changes. worker1 # [ 12.348498] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker2 # [ 12.340157] systemd[1]: Starting resolvconf update... worker2 # [ 12.350205] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. lb # [ 12.331947] nsncd[581]: Sep 22 10:57:02.788 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 12.361274] systemd[1]: Started backdoor.service. lb # [ 12.344212] systemd[1]: Started Name Service Cache Daemon (nsncd). lb # [ 12.349941] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 12.372249] systemd[1]: Starting D-Bus System Message Bus... lb # [ 12.353980] systemd[1]: Reached target User and Group Name Lookups. lb # [ 12.359809] systemd[1]: Finished resolvconf update. lb # [ 12.365767] systemd[1]: Reached target Preparation for Network. worker1 # [ 12.402094] systemd[1]: linger-users.service: Deactivated successfully. worker1 # [ 12.406758] systemd[1]: Finished linger-users.service. lb # [ 12.373000] systemd[1]: Starting DHCP Client... lb # [ 12.382511] systemd[1]: Starting Extra networking commands.... worker1 # [ 12.432471] systemd[1]: Started Name Service Cache Daemon (nsncd). worker1 # [ 12.434397] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 12.437750] systemd[1]: Reached target User and Group Name Lookups. worker1 # [ 12.441432] nsncd[559]: Sep 22 10:57:02.908 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 12.407507] dbus-broker-launch[427]: Looking up NSS user entry for 'systemd-timesync'... worker2 # [ 12.456912] systemd[1]: Finished Post-Boot Actions. worker1 # [ 12.496573] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:09.0/virtio8/input/input3 client # [ 12.432802] systemd-logind[432]: Watching system buttons on /dev/input/event0 (gpio-keys) client # [ 12.439379] systemd-logind[432]: New seat seat0. worker1 # [ 12.491439] systemd[1]: Finished resolvconf update. worker1 # [ 12.494931] systemd[1]: Reached target Preparation for Network. client # [ 12.449061] dbus-broker-launch[427]: NSS returned no entry for 'systemd-timesync' worker1 # [ 12.500518] systemd[1]: Starting DHCP Client... worker1 # [ 12.506078] systemd[1]: Starting Extra networking commands.... client # [ 12.457141] dbus-broker-launch[427]: Invalid user-name in /nix/store/ydh4ywywm30g15n5m6x5vfz42682s2cc-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" client # [ 12.473374] systemd[1]: Started User Login Management. client # [ 12.479017] systemd[1]: Starting linger-users.service... worker2 # [ 12.527042] systemd[1]: Started Name Service Cache Daemon (nsncd). client # [ 12.499886] systemd[1]: Started D-Bus System Message Bus. worker2 # [ 12.534426] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 12.541780] nsncd[497]: Sep 22 10:57:02.987 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 12.557260] systemd[1]: Reached target User and Group Name Lookups. worker2 # connecting to host... worker2 # [ 12.565854] systemd[1]: Starting User Login Management... client # [ 12.539315] systemd[1]: Stopped target Host and Network Name Lookups. worker2 # [ 12.572294] systemd[1]: Finished Import lastlog data into lastlog2 database. client # [ 12.543834] systemd[1]: Stopping Host and Network Name Lookups... client # [ 12.552000] systemd[1]: Stopped target User and Group Name Lookups. client # [ 12.560927] systemd[1]: Stopping User and Group Name Lookups... client # [ 12.573487] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... client # [ 12.574486] systemd[1]: nscd.service: Deactivated successfully. client # [ 12.575299] systemd[1]: Stopped Name Service Cache Daemon (nsncd). client # [ 12.587524] dbus-broker-launch[427]: Ready client # [ 12.611877] systemd[1]: Starting Name Service Cache Daemon (nsncd)... worker2 # [ 12.649933] dbus-broker-launch[503]: Looking up NSS user entry for 'systemd-timesync'... client # [ 12.618096] systemd[1]: linger-users.service: Deactivated successfully. client # [ 12.623707] systemd[1]: Finished linger-users.service. worker2 # [ 12.669786] dbus-broker-launch[503]: NSS returned no entry for 'systemd-timesync' worker2 # [ 12.678804] dbus-broker-launch[503]: Invalid user-name in /nix/store/a9ijqx61hpssavy5d9lh27li51m8hnl7-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" client # [ 12.685879] mousedev: PS/2 mouse device common for all mice client # [ 12.677514] systemd[1]: Condition check resulted in Virtio network device being skipped. worker2 # [ 12.712751] systemd[1]: Started D-Bus System Message Bus. lb # [ 12.713853] dhcpcd[610]: dhcpcd-10.3.2 starting worker1 # [ 12.758708] dhcpcd[592]: dhcpcd-10.3.2 starting lb # [ 12.733831] dhcpcd[645]: dev: loaded udev client # [ 12.727005] systemd[1]: Started Name Service Cache Daemon (nsncd). worker1 # [ 12.777806] dhcpcd[629]: dev: loaded udev lb # [ 12.742920] dhcpcd[645]: no valid interfaces found client # [ 12.729643] nsncd[500]: Sep 22 10:57:03.192 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" client # [ 12.731634] systemd[1]: Reached target Host and Network Name Lookups. worker1 # [ 12.781745] dhcpcd[629]: no valid interfaces found lb # [ 12.746488] dhcpcd[645]: libudev: received NULL device worker1 # [ 12.784329] dhcpcd[629]: no valid interfaces found lb # [ 12.750337] dhcpcd[645]: no valid interfaces found worker2 # [ 12.772594] dbus-broker-launch[503]: Ready worker2 # [ 12.773630] systemd-logind[512]: New seat seat0. lb # [ 12.754360] dhcpcd[645]: libudev: received NULL device client # [ 12.739893] systemd[1]: Reached target User and Group Name Lookups. client # [ 12.745927] systemd[1]: Finished resolvconf update. worker2 # [ 12.782746] systemd[1]: Started User Login Management. worker2 # [ 12.788192] systemd[1]: Starting linger-users.service... client # [ 12.759584] systemd[1]: Starting DHCP Client... worker2 # [ 12.807620] systemd[1]: Stopped target Host and Network Name Lookups. worker1 # [ 12.831571] systemd[1]: Finished Extra networking commands.. worker2 # [ 12.815135] systemd[1]: Stopping Host and Network Name Lookups... worker1 # [ 12.834701] systemd[1]: Reached target Network. worker1 # [ 12.836769] systemd[1]: Starting Permit User Sessions... worker2 # [ 12.823479] systemd[1]: Stopped target User and Group Name Lookups. worker2 # [ 12.826000] systemd[1]: Stopping User and Group Name Lookups... worker2 # [ 12.826823] systemd[1]: Stopping Name Service Cache Daemon (nsncd)... worker2 # [ 12.827681] systemd[1]: nscd.service: Deactivated successfully. lb # [ 12.820510] systemd[1]: Finished Extra networking commands.. lb # [ 12.825500] systemd[1]: Reached target Network. worker2 # [ 12.845682] systemd[1]: Stopped Name Service Cache Daemon (nsncd). worker2 # [ 12.846628] systemd[1]: Starting Name Service Cache Daemon (nsncd)... lb # [ 12.832951] systemd[1]: Starting PostgreSQL Server... lb # [ 12.844156] systemd[1]: Starting Permit User Sessions... worker1 # [ 12.893443] systemd[1]: Finished Permit User Sessions. worker1 # [ 12.897190] systemd[1]: Started Getty on tty1. worker1 # [ 12.898735] systemd[1]: Reached target Login Prompts. worker2 # [ 12.905829] systemd[1]: linger-users.service: Deactivated successfully. worker2 # [ 12.910860] systemd[1]: Finished linger-users.service. client # [ 12.893698] systemd-logind[432]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) worker2 # [ 12.946798] nsncd[570]: Sep 22 10:57:03.402 INFO started, config: Config { ignored_request_types: {}, worker_count: 8, handoff_timeout: 10s }, path: "/var/run/nscd/socket" worker2 # [ 12.955144] systemd[1]: Started Name Service Cache Daemon (nsncd). worker2 # [ 12.958646] systemd[1]: Reached target Host and Network Name Lookups. worker2 # [ 12.961752] systemd[1]: Reached target User and Group Name Lookups. lb # [ 12.945850] systemd[1]: Finished Permit User Sessions. worker2 # [ 12.969075] (udev-worker)[470]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker2 # [ 12.971258] (udev-worker)[470]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 12.984249] (udev-worker)[387]: Network interface NamePolicy= disabled on kernel command line. lb # [ 12.965869] systemd[1]: Started Getty on tty1. lb # [ 12.968851] systemd[1]: Reached target Login Prompts. worker2 # [ 12.995590] systemd[1]: Finished resolvconf update. worker2 # [ 13.000742] systemd[1]: Reached target Preparation for Network. worker2 # [ 13.001645] systemd[1]: Starting DHCP Client... worker2 # [ 13.006635] systemd[1]: Starting Extra networking commands.... client # [ 12.999283] dhcpcd[533]: dhcpcd-10.3.2 starting client # [ 13.006949] systemd[1]: Finished nftables firewall. client # [ 13.009973] systemd[1]: Reached target Preparation for Network. client # [ 13.016687] systemd[1]: Starting Address configuration of eth1... client # [ 13.022796] dhcpcd[543]: dev: loaded udev client # [ 13.032516] systemd[1]: Starting Extra networking commands.... client # [ 13.097654] 8021q: 802.1Q VLAN Support v1.8 worker2 # [ 13.147373] systemd[1]: Condition check resulted in Virtio network device being skipped. worker1 # [ 13.166100] systemd[1]: Starting Virtual Console Setup... worker2 # [ 13.152621] systemd[1]: Starting Address configuration of eth1... worker1 # [ 13.191645] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 13.195373] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 13.198075] systemd[1]: Starting Virtual Console Setup... worker1 # [ 13.207309] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs client # [ 13.186470] 8021q: adding VLAN 0 to HW filter on device eth1 client # [ 13.216912] network-addresses-eth1-start[545]: adding address 192.168.1.1/24... done lb # [ 13.236416] postgresql-pre-start[680]: The files belonging to this database system will be owned by user "postgres". lb # [ 13.239944] postgresql-pre-start[680]: This user must also own the server process. lb # [ 13.251585] postgresql-pre-start[680]: The database cluster will be initialized with locale "en_US.UTF-8". lb # [ 13.255696] postgresql-pre-start[680]: The default database encoding has accordingly been set to "UTF8". lb # [ 13.260955] postgresql-pre-start[680]: The default text search configuration will be set to "english". lb # [ 13.264921] postgresql-pre-start[680]: Data page checksums are enabled. lb # [ 13.268439] postgresql-pre-start[680]: fixing permissions on existing directory /var/lib/postgresql/18 ... ok worker2 # [ 13.290384] dhcpcd[607]: dhcpcd-10.3.2 starting client # [ 13.259774] network-addresses-eth1-start[545]: adding address 2001:db8:1::1/64... done lb # [ 13.274088] postgresql-pre-start[680]: creating subdirectories ... ok lb # [ 13.280902] postgresql-pre-start[680]: selecting dynamic shared memory implementation ... posix client # [ 13.278457] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker2 # [ 13.313760] dhcpcd[652]: dev: loaded udev client # [ 13.296238] systemd[1]: Finished Address configuration of eth1. worker2 # [ 13.342380] network-addresses-eth1-start[610]: adding address 192.168.1.4/24... done client # [ 13.329450] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' client # [ 13.329987] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' client # [ 13.335520] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 client # [ 13.335916] cfg80211: failed to load regulatory.db worker2 # [ 13.372960] 8021q: 802.1Q VLAN Support v1.8 worker2 # [ 13.373369] 8021q: adding VLAN 0 to HW filter on device eth1 lb # [ 13.362658] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:09.0/virtio8/input/input3 worker2 # [ 13.382046] network-addresses-eth1-start[610]: adding address 2001:db8:1::4/64... done worker1 # [ 13.408746] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. worker2 # [ 13.431312] systemd[1]: Finished Address configuration of eth1. worker1 # [ 13.453925] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. worker2 # [ 13.440688] systemd[1]: Finished Extra networking commands.. worker2 # [ 13.441618] systemd[1]: Reached target Network. worker1 # [ 13.458261] systemd[1]: Started backdoor.service. client # [ 13.409580] dhcpcd[617]: /nix/store/grcr27glg0bai58hwyh3zxlj5sf30hhx-openresolv-3.17.4/sbin/.resolvconf-wrapped: line 1250: kill: (557) - Operation not permitted client # [ 13.415235] dhcpcd[617]: .resolvconf-wrapped: clearing stale lock pid 557 worker2 # [ 13.457000] systemd[1]: Starting Permit User Sessions... client # [ 13.424993] systemd[1]: Finished Extra networking commands.. client # [ 13.428822] systemd[1]: Reached target Network. client # [ 13.436296] systemd[1]: Starting Permit User Sessions... worker2 # [ 13.482266] cfg80211: Loading compiled-in X.509 certificates for regulatory database lb # [ 13.474950] postgresql-pre-start[680]: selecting default "max_connections" ... 100 worker2 # [ 13.519933] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker2 # [ 13.520503] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' client # [ 13.487204] 8021q: adding VLAN 0 to HW filter on device eth0 client # [ 13.477471] dhcpcd[543]: eth0: waiting for carrier worker2 # [ 13.525562] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 client # [ 13.480279] dhcpcd[543]: eth0: carrier acquired worker1 # [ 13.545973] 8021q: 802.1Q VLAN Support v1.8 worker2 # [ 13.525936] cfg80211: failed to load regulatory.db client # [ 13.508230] dhcpcd[543]: DUID 00:01:00:01:32:45:1b:7f:52:54:00:12:34:56 client # [ 13.511580] dhcpcd[543]: eth0: IAID 00:12:34:56 client # [ 13.516859] dhcpcd[543]: eth0: adding address fe80::5054:ff:fe12:3456 worker2 # [ 13.553476] systemd[1]: Finished Permit User Sessions. client # [ 13.523054] systemd[1]: Finished Permit User Sessions. worker2 # [ 13.557685] systemd[1]: Started Getty on tty1. worker2 # [ 13.559865] systemd[1]: Reached target Login Prompts. client # [ 13.532392] systemd[1]: Started Getty on tty1. client # [ 13.535319] systemd[1]: Reached target Login Prompts. worker2 # [ 13.563609] systemd-logind[512]: Watching system buttons on /dev/input/event0 (gpio-keys) worker1 # connecting to host... worker1 # [ 13.644936] cfg80211: Loading compiled-in X.509 certificates for regulatory database client: (finished: waiting for unit network-addresses-eth1.service, in 14.03 seconds) client: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; } worker2 # [ 13.651731] 8021q: adding VLAN 0 to HW filter on device eth0 worker2 # [ 13.640808] dhcpcd[652]: eth0: waiting for carrier worker2 # [ 13.641779] dhcpcd[652]: eth0: carrier acquired worker1 # [ 13.682360] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' worker1 # [ 13.682894] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' worker1 # [ 13.688276] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 worker1 # [ 13.688653] cfg80211: failed to load regulatory.db worker2 # [ 13.659510] dhcpcd[652]: DUID 00:01:00:01:32:45:1b:80:52:54:00:12:34:56 worker2 # [ 13.661639] dhcpcd[652]: eth0: IAID 00:12:34:56 worker2 # [ 13.662398] dhcpcd[652]: eth0: adding address fe80::5054:ff:fe12:3456 worker1 # [ 13.698728] (udev-worker)[655]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. worker1 # [ 13.703816] (udev-worker)[655]: Network interface NamePolicy= disabled on kernel command line. client: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.1/ || { ip -4 addr >&2; false; }, in 0.06 seconds) lb: waiting for unit network-addresses-eth1.service lb: waiting for the VM to finish booting worker1 # [ 13.707524] (udev-worker)[645]: Network interface NamePolicy= disabled on kernel command line. lb # [ 13.678314] postgresql-pre-start[680]: selecting default "shared_buffers" ... 128MB worker2 # [ 13.815902] mousedev: PS/2 mouse device common for all mice worker1 # [ 13.855354] systemd-vconsole-setup[660]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 13.862223] systemd[1]: Finished Virtual Console Setup. worker2 # [ 13.864446] systemd-logind[512]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) worker1 # [ 13.895652] systemd[1]: Condition check resulted in Virtio network device being skipped. worker1 # [ 13.901164] systemd[1]: Starting Address configuration of eth1... worker1 # [ 14.012764] 8021q: adding VLAN 0 to HW filter on device eth0 worker1 # [ 13.999178] dhcpcd[629]: eth0: waiting for carrier worker1 # [ 14.003095] dhcpcd[629]: eth0: carrier acquired worker1 # [ 14.016236] systemd-logind[501]: Watching system buttons on /dev/input/event0 (gpio-keys) worker1 # [ 14.023018] dhcpcd[629]: DUID 00:01:00:01:32:45:1b:80:52:54:00:12:34:56[ 14.042468] 8021q: adding VLAN 0 to HW filter on device eth1 worker1 # worker1 # [ 14.029694] dhcpcd[629]: eth0: IAID 00:12:34:56 worker1 # [ 14.030380] dhcpcd[629]: eth0: adding address fe80::5054:ff:fe12:3456 worker1 # [ 14.049847] network-addresses-eth1-start[680]: adding address 192.168.1.3/24... done worker1 # [ 14.065398] network-addresses-eth1-start[680]: adding address 2001:db8:1::3/64... done lb # [ 14.030150] systemd[1]: Starting Virtual Console Setup... worker1 # [ 14.087815] systemd[1]: Finished Address configuration of eth1. lb # [ 14.060136] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 14.064605] systemd[1]: Stopped Virtual Console Setup. lb # [ 14.069973] systemd[1]: Load Kernel Module configfs skipped, unmet condition check ConditionKernelModuleLoaded=!configfs lb # [ 14.078512] systemd[1]: Starting Virtual Console Setup... worker1 # [ 14.146951] mousedev: PS/2 mouse device common for all mice worker1 # [ 14.148561] dhcpcd[629]: eth0: soliciting a DHCP lease worker1 # [ 14.156628] dhcpcd[629]: eth0: offered 10.0.2.15 from 10.0.2.2 worker1 # [ 14.164389] dhcpcd[629]: eth0: probing address 10.0.2.15/24 worker1 # [ 14.244120] systemd-logind[501]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) client # [ 14.210885] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:09.0/virtio8/input/input3 lb # [ 14.343378] systemd[1]: Condition check resulted in /dev/hvc0 being skipped. lb # [ 14.400340] systemd[1]: Condition check resulted in /dev/ttyAMA0 being skipped. lb # [ 14.412195] systemd[1]: Started backdoor.service. worker2 # [ 14.474255] input: QEMU Virtio Keyboard as /devices/platform/4010000000.pcie/pci0000:00/0000:00:09.0/virtio8/input/input3 lb # [ 14.507191] 8021q: 802.1Q VLAN Support v1.8 lb # connecting to host... client # [ 14.546821] systemd[1]: Starting Virtual Console Setup... client # [ 14.558638] dhcpcd[543]: eth0: soliciting a DHCP lease client # [ 14.560570] dhcpcd[543]: eth0: offered 10.0.2.15 from 10.0.2.2 lb: Guest shell says: b'Spawning backdoor root shell...\n' lb: connected to guest root shell lb: (connecting took 0.90 seconds) lb: (finished: waiting for the VM to finish booting, in 0.90 seconds) client # [ 14.568444] dhcpcd[543]: eth0: probing address 10.0.2.15/24 client # [ 14.585789] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. client # [ 14.594621] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. client # [ 14.601628] systemd[1]: Stopped Virtual Console Setup. client # [ 14.609405] systemd[1]: Starting Virtual Console Setup... lb # [ 14.629301] (udev-worker)[679]: Network interface NamePolicy= disabled on kernel command line. lb # [ 14.665751] cfg80211: Loading compiled-in X.509 certificates for regulatory database worker2 # [ 14.675768] dhcpcd[652]: eth0: soliciting a DHCP lease worker2 # [ 14.680598] dhcpcd[652]: eth0: offered 10.0.2.15 from 10.0.2.2 worker2 # [ 14.688588] dhcpcd[652]: eth0: probing address 10.0.2.15/24 client # [ 14.656378] systemd-logind[432]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) lb # [ 14.685859] systemd-vconsole-setup[705]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 14.702916] systemd[1]: Finished Virtual Console Setup. lb # [ 14.721484] Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' lb # [ 14.722007] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600' lb # [ 14.727544] faux_driver regulatory: Direct firmware load for regulatory.db failed with error -2 lb # [ 14.727928] cfg80211: failed to load regulatory.db lb # [ 14.728435] (udev-worker)[675]: eth1: Config file /etc/systemd/network/40-eth1.link is applied to device based on potentially unpredictable interface name. lb # [ 14.733479] (udev-worker)[675]: Network interface NamePolicy= disabled on kernel command line. worker2 # [ 14.787741] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker2 # [ 14.808388] systemd[1]: Starting Virtual Console Setup... client # [ 14.786633] dhcpcd[543]: eth0: soliciting an IPv6 router client # [ 14.788231] dhcpcd[543]: eth0: Router Advertisement from fe80::2 client # [ 14.789304] dhcpcd[543]: eth0: adding address fec0::5054:ff:fe12:3456/64 client # [ 14.790372] dhcpcd[543]: eth0: adding route to fec0::/64 client # [ 14.791304] dhcpcd[543]: eth0: adding default route via fe80::2 worker2 # [ 14.833294] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker2 # [ 14.838814] systemd[1]: Stopped Virtual Console Setup. worker2 # [ 14.843619] systemd[1]: Starting Virtual Console Setup... worker2 # [ 14.891122] systemd-logind[512]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) lb # [ 14.987847] systemd[1]: Condition check resulted in Virtio network device being skipped. lb # [ 14.999095] systemd[1]: Starting Address configuration of eth1... lb # [ 15.073143] 8021q: adding VLAN 0 to HW filter on device eth0 lb # [ 15.060445] dhcpcd[645]: eth0: waiting for carrier lb # [ 15.065969] dhcpcd[645]: eth0: carrier acquired client # [ 15.053412] systemd-vconsole-setup[649]: Configuration of first virtual console was skipped, ignoring remaining ones. client # [ 15.057633] systemd[1]: Finished Virtual Console Setup. lb # [ 15.091745] dhcpcd[645]: DUID 00:01:00:01:32:45:1b:81:52:54:00:12:34:56 lb # [ 15.095690] dhcpcd[645]: eth0: IAID 00:12:34:56 lb # [ 15.098979] dhcpcd[645]: eth0: adding address fe80::5054:ff:fe12:3456 lb # [ 15.140102] 8021q: adding VLAN 0 to HW filter on device eth1 lb # [ 15.150004] network-addresses-eth1-start[736]: adding address 192.168.1.2/24... done lb # [ 15.167231] network-addresses-eth1-start[736]: adding address 2001:db8:1::2/64... done worker1 # [ 15.213486] dhcpcd[629]: eth0: soliciting an IPv6 router worker1 # [ 15.215049] dhcpcd[629]: eth0: Router Advertisement from fe80::2 worker1 # [ 15.215931] dhcpcd[629]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker1 # [ 15.219434] dhcpcd[629]: eth0: adding route to fec0::/64 worker1 # [ 15.221714] dhcpcd[629]: eth0: adding default route via fe80::2 lb # [ 15.192722] systemd[1]: Finished Address configuration of eth1. lb # [ 15.197309] systemd-logind[519]: Watching system buttons on /dev/input/event0 (gpio-keys) worker1 # [ 15.274258] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. worker1 # [ 15.280290] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. worker1 # [ 15.281455] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 15.282162] systemd[1]: Stopping Virtual Console Setup... worker1 # [ 15.290494] systemd[1]: Starting Virtual Console Setup... worker2 # [ 15.275695] systemd-vconsole-setup[702]: Configuration of first virtual console was skipped, ignoring remaining ones. worker2 # [ 15.282552] systemd[1]: Finished Virtual Console Setup. worker1 # [ 15.313588] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 15.294534] mousedev: PS/2 mouse device common for all mice worker1 # [ 15.319035] systemd[1]: Stopped Virtual Console Setup. worker1 # [ 15.323044] systemd[1]: Starting Virtual Console Setup... worker1 # [ 15.354654] systemd-logind[501]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) lb # [ 15.467718] systemd-logind[519]: Watching system buttons on /dev/input/event1 (QEMU QEMU USB Keyboard) worker1 # [ 15.652645] systemd-vconsole-setup[703]: Configuration of first virtual console was skipped, ignoring remaining ones. worker1 # [ 15.657036] systemd[1]: Finished Virtual Console Setup. worker2 # [ 15.754815] dhcpcd[652]: eth0: soliciting an IPv6 router worker2 # [ 15.758609] dhcpcd[652]: eth0: Router Advertisement from fe80::2 worker2 # [ 15.761541] dhcpcd[652]: eth0: adding address fec0::5054:ff:fe12:3456/64 worker2 # [ 15.764428] dhcpcd[652]: eth0: adding route to fec0::/64 worker2 # [ 15.766762] dhcpcd[652]: eth0: adding default route via fe80::2 lb: (finished: waiting for unit network-addresses-eth1.service, in 2.25 seconds) lb: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; } lb: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.2/ || { ip -4 addr >&2; false; }, in 0.05 seconds) worker1: waiting for unit network-addresses-eth1.service worker1: waiting for the VM to finish booting worker1: Guest shell says: b'Spawning backdoor root shell...\n' worker1: connected to guest root shell worker1: (connecting took 0.00 seconds) worker1: (finished: waiting for the VM to finish booting, in 0.00 seconds) worker1: (finished: waiting for unit network-addresses-eth1.service, in 0.13 seconds) worker1: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; } worker1: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.3/ || { ip -4 addr >&2; false; }, in 0.05 seconds) worker2: waiting for unit network-addresses-eth1.service worker2: waiting for the VM to finish booting worker2: Guest shell says: b'Spawning backdoor root shell...\n' worker2: connected to guest root shell worker2: (connecting took 0.00 seconds) worker2: (finished: waiting for the VM to finish booting, in 0.00 seconds) worker2: (finished: waiting for unit network-addresses-eth1.service, in 0.11 seconds) worker2: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; } worker2: (finished: must succeed: ip -4 addr show | grep -qF 192.168.1.4/ || { ip -4 addr >&2; false; }, in 0.05 seconds) lb: waiting for unit niks3.service lb # [ 16.450149] dhcpcd[645]: eth0: soliciting an IPv6 router lb # [ 16.452375] dhcpcd[645]: eth0: Router Advertisement from fe80::2 lb # [ 16.456568] dhcpcd[645]: eth0: adding address fec0::5054:ff:fe12:3456/64 lb # [ 16.460559] dhcpcd[645]: eth0: adding route to fec0::/64 lb # [ 16.464304] dhcpcd[645]: eth0: adding default route via fe80::2 lb # [ 16.755806] postgresql-pre-start[680]: selecting default time zone ... UTC lb # [ 16.762389] postgresql-pre-start[680]: creating configuration files ... ok lb # [ 16.978420] dhcpcd[645]: eth0: soliciting a DHCP lease lb # [ 16.984568] dhcpcd[645]: eth0: offered 10.0.2.15 from 10.0.2.2 lb # [ 16.988307] dhcpcd[645]: eth0: probing address 10.0.2.15/24 lb # [ 17.324265] postgresql-pre-start[680]: running bootstrap script ... ok lb # [ 17.856436] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 17.857588] systemd[1]: Stopped Virtual Console Setup. lb # [ 17.858354] systemd[1]: Stopping Virtual Console Setup... lb # [ 17.867390] systemd[1]: Starting Virtual Console Setup... lb # [ 17.941445] systemd[1]: Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. lb # [ 17.956833] systemd[1]: systemd-vconsole-setup.service: Deactivated successfully. lb # [ 17.960149] systemd[1]: Stopped Virtual Console Setup. lb # [ 17.977380] systemd[1]: Starting Virtual Console Setup... lb # [ 18.029874] systemd-logind[519]: Watching system buttons on /dev/input/event3 (QEMU Virtio Keyboard) lb # [ 18.456750] systemd-vconsole-setup[805]: Configuration of first virtual console was skipped, ignoring remaining ones. lb # [ 18.466344] systemd[1]: Finished Virtual Console Setup. lb # [ 18.574869] postgresql-pre-start[680]: performing post-bootstrap initialization ... ok lb # [ 18.729138] postgresql-pre-start[680]: syncing data to disk ... ok lb # [ 18.730289] postgresql-pre-start[680]: initdb: warning: enabling "trust" authentication for local connections lb # [ 18.731632] postgresql-pre-start[680]: initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. lb # [ 18.734019] postgresql-pre-start[680]: Success. You can now start the database server using: lb # [ 18.735411] postgresql-pre-start[680]: pg_ctl -D /var/lib/postgresql/18 -l logfile start client # [ 18.852553] dhcpcd[543]: eth0: leased 10.0.2.15 for 86400 seconds client # [ 18.857159] dhcpcd[543]: eth0: adding route to 10.0.2.0/24 client # [ 18.859752] dhcpcd[543]: eth0: adding default route via 10.0.2.2 lb # [ 18.928772] postgres[811]: [811] LOG: starting PostgreSQL 18.6 on aarch64-unknown-linux-gnu, compiled by clang version 21.1.8, 64-bit lb # [ 18.933601] postgres[811]: [811] LOG: listening on IPv6 address "::1", port 5432 lb # [ 18.935016] postgres[811]: [811] LOG: listening on IPv4 address "127.0.0.1", port 5432 lb # [ 18.936929] postgres[811]: [811] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432" lb # [ 18.954692] postgres[829]: [829] LOG: database system was shut down at 2026-09-22 10:57:09 GMT lb # [ 18.961428] postgres[811]: [811] LOG: database system is ready to accept connections lb # [ 18.966811] systemd[1]: Started PostgreSQL Server. lb # [ 18.979396] systemd[1]: Starting PostgreSQL Setup Scripts... client # [ 19.031885] systemd[1]: Started DHCP Client. client # [ 19.035076] systemd[1]: Reached target Multi-User System. client # [ 19.037081] systemd[1]: Startup finished in 1.108s (kernel) + 5.607s (initrd) + 12.320s (userspace) = 19.036s. lb # [ 19.252858] postgresql-setup-start[840]: CREATE DATABASE lb # [ 19.315177] postgresql-setup-start[845]: CREATE ROLE lb # [ 19.346661] postgresql-setup-start[847]: ALTER DATABASE lb # [ 19.357064] systemd[1]: Finished PostgreSQL Setup Scripts. lb # [ 19.359387] systemd[1]: Reached target PostgreSQL. worker1 # [ 19.465188] dhcpcd[629]: eth0: leased 10.0.2.15 for 86400 seconds worker1 # [ 19.465531] dhcpcd[629]: eth0: adding route to 10.0.2.0/24 worker1 # [ 19.471254] dhcpcd[629]: eth0: adding default route via 10.0.2.2 worker2 # [ 19.575752] dhcpcd[652]: eth0: leased 10.0.2.15 for 86400 seconds worker2 # [ 19.580829] dhcpcd[652]: eth0: adding route to 10.0.2.0/24 worker2 # [ 19.584227] dhcpcd[652]: eth0: adding default route via 10.0.2.2 worker1 # [ 19.656496] systemd[1]: Started DHCP Client. worker1 # [ 19.658799] systemd[1]: Reached target Multi-User System. worker1 # [ 19.659746] systemd[1]: Startup finished in 1.106s (kernel) + 4.758s (initrd) + 13.793s (userspace) = 19.658s. worker2 # [ 19.760694] systemd[1]: Started DHCP Client. worker2 # [ 19.762691] systemd[1]: Reached target Multi-User System. worker2 # [ 19.764537] systemd[1]: Startup finished in 1.115s (kernel) + 5.472s (initrd) + 13.176s (userspace) = 19.763s. lb # [ 22.709297] dhcpcd[645]: eth0: leased 10.0.2.15 for 86400 seconds lb # [ 22.709669] dhcpcd[645]: eth0: adding route to 10.0.2.0/24 lb # [ 22.709811] dhcpcd[645]: eth0: adding default route via 10.0.2.2 lb # [ 22.898519] systemd[1]: Started DHCP Client. lb # [ 22.901748] systemd[1]: Reached target Network is Online. lb # [ 22.907968] systemd[1]: Started Envoy reverse proxy. lb # [ 22.916169] systemd[1]: Started mock-oidc.service. lb # [ 23.116870] mock-oidc-server[934]: Mock OIDC Server running lb # [ 23.117807] mock-oidc-server[934]: OIDC Address: 192.168.1.2:8080 lb # [ 23.118698] mock-oidc-server[934]: Issue Address: 0.0.0.0:8081 lb # [ 23.119490] mock-oidc-server[934]: Issuer: http://192.168.1.2:8080/oidc lb # [ 23.120624] mock-oidc-server[934]: JWKS: http://192.168.1.2:8080/oidc/.well-known/jwks.json lb # [ 23.122164] mock-oidc-server[934]: Discovery: http://192.168.1.2:8080/oidc/.well-known/openid-configuration lb # [ 23.123956] mock-oidc-server[934]: Issue tokens: http://0.0.0.0:8081/issue?sub=... lb # [ 23.705872] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:467] initializing epoch 0 (base id=0, hot restart version=11.120) lb # [ 23.707825] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:469] statically linked extensions: lb # [ 23.709464] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.matching.stat_matching_data.input: stat_full_name_match_input lb # [ 23.711499] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case lb # [ 23.714376] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.envelope, envoy.http.stateful_session.header lb # [ 23.717316] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.quic.connection_id_generator: envoy.quic.connection_id_generator.quic_lb, envoy.quic.deterministic_connection_id_generator lb # [ 23.719954] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme lb # [ 23.723727] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.dynamic_metadata, envoy.matching.inputs.dynamic_module_data_input, envoy.matching.inputs.filter_state, envoy.matching.inputs.local_reply, envoy.matching.inputs.network_namespace, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params lb # [ 23.747333] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.string_matcher: envoy.string_matcher.lua lb # [ 23.750267] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.access_loggers: envoy.access_loggers.dynamic_modules, envoy.access_loggers.file, envoy.access_loggers.fluentd, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stats, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.fluentd_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log lb # [ 23.757970] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.thrift_proxy.transports: auto, framed, header, unframed lb # [ 23.759943] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.quic.connection_debug_visitor: envoy.quic.connection_debug_visitor.basic, envoy.quic.connection_debug_visitor.quic_stats lb # [ 23.762935] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls lb # [ 23.768393] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor lb # [ 23.771187] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.upstream_options: envoy.extensions.upstreams.http.reverse_tunnel.v3.ReverseTunnelUpstreamCodecOptions, envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options lb # [ 23.776140] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.http.cache_v2: envoy.extensions.http.cache_v2.file_system_http_cache, envoy.extensions.http.cache_v2.simple lb # [ 23.780044] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.dynamic_modules, envoy.filters.udp_listener.udp_proxy lb # [ 23.782841] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.samplers: envoy.tracers.opentelemetry.samplers.always_on, envoy.tracers.opentelemetry.samplers.cel, envoy.tracers.opentelemetry.samplers.dynatrace, envoy.tracers.opentelemetry.samplers.parent_based, envoy.tracers.opentelemetry.samplers.trace_id_ratio_based lb # [ 23.787682] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.retry_priorities: envoy.retry_priorities.previous_priorities lb # [ 23.789908] envoy[933]: [2026-09-22 10:57:14.162][933][info][main] [source/server/server.cc:471] envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.dynamic_modules, envoy.tls.cert_validator.spiffe lb # [ 23.792815] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port lb # [ 23.794970] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.dynamic_modules, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd lb # [ 23.801097] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.dynamic_modules, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.ext_proc, envoy.filters.network.generic_proxy, envoy.filters.network.geoip, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.reverse_tunnel, envoy.filters.network.reverse_tunnel_drain_aware_http_connection_manager, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_bandwidth_limit, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy lb # [ 23.818931] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.rbac.principals: envoy.rbac.principals.mtls_authenticated lb # [ 23.821298] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor lb # [ 23.823963] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router lb # [ 23.827330] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation lb # [ 23.831140] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter lb # [ 23.833312] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable lb # [ 23.835447] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.content_parsers: envoy.content_parsers.json lb # [ 23.837371] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo, envoy.network.dns_resolver.hickory lb # [ 23.841344] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.http.header_validators: envoy.http.header_validators.envoy_default lb # [ 23.843434] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.http.forward_client_cert_details: forward_client_cert lb # [ 23.845588] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.srds_factory: envoy.srds_factory.default lb # [ 23.848106] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.generic_proxy.filters: envoy.filters.generic.router lb # [ 23.849941] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.tls.certificate_mappers: envoy.tls.certificate_mappers.sni, envoy.tls.certificate_mappers.static_name lb # [ 23.858060] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.request_id: envoy.request_id.uuid lb # [ 23.859891] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator lb # [ 23.862757] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] network.connection.client: default, envoy_internal lb # [ 23.865117] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher lb # [ 23.867853] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.upstreams: envoy.filters.connection_pools.tcp.generic lb # [ 23.869966] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.tls.certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret lb # [ 23.874204] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.resolvers: envoy.ip, envoy.resolvers.reverse_connection lb # [ 23.876374] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_mappers: envoy.tls.certificate_mappers.static_name, envoy.tls.upstream_certificate_mappers.filter_state_override lb # [ 23.879397] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.hashable_string, envoy.network.application_protocols, envoy.network.ip, envoy.network.network_namespace, envoy.network.transport_socket.http_11_proxy.info, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.ratelimit.hits_addend, envoy.router.debug_config, envoy.string, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.tcp_proxy.per_connection_idle_timeout_ms, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port lb # [ 23.889149] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.generic_proxy.codecs: envoy.generic_proxy.codecs.dubbo, envoy.generic_proxy.codecs.http1 lb # [ 23.891298] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.filters.upstream_network: envoy.filters.upstream_network.reverse_tunnel_lifecycle lb # [ 23.893494] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.common.key_value: envoy.key_value.file_based lb # [ 23.895307] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.domain_matcher, envoy.matching.custom_matchers.ip_range_matcher lb # [ 23.898250] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts lb # [ 23.901658] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff lb # [ 23.904358] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.formatter: envoy.formatter.cel, envoy.formatter.dynamic_modules, envoy.formatter.file_content, envoy.formatter.generic_secret, envoy.formatter.metadata, envoy.formatter.req_without_query lb # [ 23.907708] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.router.cluster_specifier_plugin: envoy.router.cluster_specifier_plugin.lua, envoy.router.cluster_specifier_plugin.matcher lb # [ 23.910995] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.matching.input_matchers: envoy.matching.input_matchers.generic_request_matcher, envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.dynamic_modules, envoy.matching.matchers.ip, envoy.matching.matchers.metadata_matcher, envoy.matching.matchers.runtime_fraction lb # [ 23.915841] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action lb # [ 23.918253] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.tls.upstream_certificate_selectors: envoy.tls.certificate_selectors.on_demand_secret lb # [ 23.921006] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.connection_handler: envoy.connection_handler.default lb # [ 23.922918] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche lb # [ 23.925087] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation lb # [ 23.927976] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel, envoy.access_loggers.extension_filters.process_ratelimit lb # [ 23.931239] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.tracers: envoy.tracers.datadog, envoy.tracers.dynamic_modules, envoy.tracers.fluentd, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin lb # [ 23.934988] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.http.ext_proc.processing_request_modifiers: envoy.extensions.http.ext_proc.mapped_attribute_builder lb # [ 23.938287] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.health_checkers: envoy.health_checkers.dynamic_modules, envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift lb # [ 23.941871] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.matching.transport_socket.input: envoy.matching.inputs.endpoint_metadata, envoy.matching.inputs.locality_metadata, envoy.matching.inputs.transport_socket_filter_state lb # [ 23.945099] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.quic.proof_source: envoy.quic.proof_source.filter_chain lb # [ 23.947094] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.network_namespace, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san lb # [ 23.955016] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.protocols: dubbo lb # [ 23.956867] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.bootstrap: envoy.bootstrap.dynamic_modules, envoy.bootstrap.internal_listener, envoy.bootstrap.reverse_tunnel.downstream_socket_interface, envoy.bootstrap.reverse_tunnel.upstream_socket_interface, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface, envoy.extensions.network.socket_interface.sockmap lb # [ 23.961911] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] quic.http_server_connection: quic.http_server_connection.default lb # [ 23.964107] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.matching.generic_proxy_request_input.input: envoy.matching.generic_proxy.input.host, envoy.matching.generic_proxy.input.method, envoy.matching.generic_proxy.input.path, envoy.matching.generic_proxy.input.property, envoy.matching.generic_proxy.input.request, envoy.matching.generic_proxy.input.service lb # [ 23.968715] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.clusters: envoy.cluster.dns, envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.composite, envoy.clusters.dynamic_forward_proxy, envoy.clusters.dynamic_modules, envoy.clusters.mcp_multicluster, envoy.clusters.redis, envoy.clusters.reverse_connection lb # [ 23.974089] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.fault, envoy.filters.http.a2a, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.ai_protocol_manager, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.api_key_auth, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.bandwidth_share, envoy.filters.http.basic_auth, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cache_v2, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.credential_injector, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_server, envoy.filters.http.file_system_buffer, envoy.filters.http.filter_chain, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_reverse_transcoder, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.mcp, envoy.filters.http.mcp_json_rest_bridge, envoy.filters.http.mcp_router, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.proto_api_scrubber, envoy.filters.http.proto_message_extraction, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.sse_to_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.thrift_to_metadata, envoy.filters.http.transform, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router lb # [ 24.007121] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.filters: envoy.filters.dubbo.router lb # [ 24.009284] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.regex_engines: envoy.regex_engines.google_re2 lb # [ 24.011240] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8 lb # [ 24.013840] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.tracers.opentelemetry.resource_detectors: envoy.tracers.opentelemetry.resource_detectors.dynatrace, envoy.tracers.opentelemetry.resource_detectors.environment, envoy.tracers.opentelemetry.resource_detectors.static_config lb # [ 24.017798] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.path.match: envoy.path.match.uri_template.uri_template_matcher lb # [ 24.019845] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy lb # [ 24.023272] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.matching.stat_tag_matching_data.input: stat_tag_value_input lb # [ 24.025436] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter lb # [ 24.027578] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.matching.action: envoy.matching.action.transport_socket.name, envoy.matching.actions.format_string, filter-chain-name lb # [ 24.030863] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.load_balancing_policies: envoy.load_balancing_policies.client_side_weighted_round_robin, envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.dynamic_modules, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.load_aware_locality, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.override_host, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset, envoy.load_balancing_policies.wrr_locality lb # [ 24.038963] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.filters.http.upstream: envoy.buffer, envoy.ext_proc, envoy.extensions.filters.http.dynamic_modules, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.composite, envoy.filters.http.credential_injector, envoy.filters.http.ext_proc, envoy.filters.http.header_mutation, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.upstream_codec, envoy.filters.http.upstream_rbac, envoy.filters.http.wasm lb # [ 24.047088] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.ext_authz, envoy.filters.udp.session.http_capsule lb # [ 24.051282] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.http.ext_proc.response_processors: envoy.extensions.http.ext_proc.save_processing_response lb # [ 24.053852] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple lb # [ 24.056734] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.grpc_credentials: envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata lb # [ 24.059098] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory lb # [ 24.062491] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.http.injected_credentials: envoy.http.injected_credentials.generic, envoy.http.injected_credentials.oauth2 lb # [ 24.065481] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.health_check.event_sinks: envoy.health_check.event_sink.file lb # [ 24.067529] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.geoip_providers: envoy.geoip_providers.maxmind lb # [ 24.069554] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.rds_factory: envoy.rds_factory.default lb # [ 24.071423] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.route.early_data_policy: envoy.route.early_data_policy.default lb # [ 24.074174] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector, envoy.upstream.local_address_selector.filter_state_override lb # [ 24.077463] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest lb # [ 24.083290] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.quic.client_packet_writer: envoy.quic.packet_writer.default lb # [ 24.085766] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.dynamic_modules, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls lb # [ 24.092450] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr lb # [ 24.094897] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.quic.server_preferred_address: quic.server_preferred_address.datasource, quic.server_preferred_address.fixed lb # [ 24.097616] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.ssl.server_context_factory: envoy.ssl.server_context_factory.default lb # [ 24.099735] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso lb # [ 24.102627] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.resource_monitors: envoy.resource_monitors.cgroup_memory, envoy.resource_monitors.cpu_utilization, envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource lb # [ 24.106274] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.filters.listener: envoy.filters.listener.dynamic_modules, envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.set_filter_state, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector lb # [ 24.113331] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.dubbo_proxy.serializers: dubbo.hessian2 lb # [ 24.115331] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.route_config_update_requester: envoy.route_config_update_requester.default lb # [ 24.117749] envoy[933]: [2026-09-22 10:57:14.163][933][info][main] [source/server/server.cc:471] envoy.built_in_formatters: envoy.built_in_formatters.cel, envoy.built_in_formatters.http.default, envoy.built_in_formatters.metadata, envoy.built_in_formatters.stream_info.default, envoy.built_in_formatters.xfcc_value lb # [ 24.121714] envoy[933]: [2026-09-22 10:57:14.202][933][info][main] [source/server/server.cc:531] HTTP header map info: lb # [ 24.123499] envoy[933]: [2026-09-22 10:57:14.204][933][info][main] [source/server/server.cc:534] request header map: 664 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-host,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-request-id lb # [ 24.140002] envoy[933]: [2026-09-22 10:57:14.204][933][info][main] [source/server/server.cc:534] request trailer map: 120 bytes: lb # [ 24.141748] envoy[933]: [2026-09-22 10:57:14.204][933][info][main] [source/server/server.cc:534] response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id lb # [ 24.151627] envoy[933]: [2026-09-22 10:57:14.204][933][info][main] [source/server/server.cc:534] response trailer map: 144 bytes: grpc-message,grpc-status lb # [ 24.154093] envoy[933]: [2026-09-22 10:57:14.481][933][info][main] [source/server/server.cc:968] runtime: {} lb # [ 24.155670] envoy[933]: [2026-09-22 10:57:14.509][933][info][main] [source/server/server.cc:420] Raised file descriptor limits from 1024 to 524288. lb # [ 24.158434] envoy[933]: [2026-09-22 10:57:14.509][933][info][main] [source/server/server.cc:796] Starting admin HTTP server at 127.0.0.1:9901 lb # [ 24.160447] envoy[933]: [2026-09-22 10:57:14.509][933][info][admin] [source/server/admin/admin.cc:66] admin address: 127.0.0.1:9901 lb # [ 24.162082] envoy[933]: [2026-09-22 10:57:14.514][933][info][config] [source/server/configuration_impl.cc:181] loading tracing configuration lb # [ 24.163927] envoy[933]: [2026-09-22 10:57:14.514][933][info][config] [source/server/configuration_impl.cc:132] loading 0 static secret(s) lb # [ 24.165901] envoy[933]: [2026-09-22 10:57:14.514][933][info][config] [source/server/configuration_impl.cc:138] loading 2 cluster(s) lb # [ 24.195424] envoy[933]: [2026-09-22 10:57:14.653][933][info][config] [source/server/configuration_impl.cc:148] loading 1 listener(s) worker2 # [ 24.219306] systemd[1]: Reached target Network is Online. lb # [ 24.205045] envoy[933]: [2026-09-22 10:57:14.663][933][info][config] [source/server/configuration_impl.cc:164] loading stats configuration lb # [ 24.208105] envoy[933]: [2026-09-22 10:57:14.666][933][warning][main] [source/server/server.cc:1038] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.global_downstream_max_connections` resource monitor. lb # [ 24.211588] envoy[933]: [2026-09-22 10:57:14.666][933][info][main] [source/server/server.cc:1077] starting main dispatch loop worker1 # [ 24.248251] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 24.233079] systemd[1]: Starting scheduler-token.service... worker2 # [ 24.364529] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 24.368586] systemd[1]: Finished scheduler-token.service. worker2 # [ 24.370027] systemd[1]: scheduler-token.service: Consumed 58ms CPU time over 134ms wall clock time, 2.9M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 24.380757] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 24.513415] nix-grpc-daemon[778]: ts=2026-09-22T10:57:14Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 24.526558] nix-grpc-daemon[778]: ts=2026-09-22T10:57:14Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 24.529245] nix-grpc-daemon[778]: ts=2026-09-22T10:57:14Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 24.567898] nix-grpc-daemon[778]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 24.569827] nix-grpc-daemon[778]: E0000 00:00:1790074635.038998 778 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 24.585167] nix-grpc-daemon[778]: E0000 00:00:1790074635.056244 778 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 24.589279] nix-grpc-daemon[778]: ts=2026-09-22T10:57:15Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 24.591184] systemd[1]: Started Nix worker-protocol over gRPC. lb # [ 24.590852] envoy[933]: [2026-09-22 10:57:15.049][933][info][runtime] [source/common/runtime/runtime_impl.cc:567] RTDS has finished initialization worker2 # [ 24.614135] nix-grpc-daemon[780]: ts=2026-09-22T10:57:15Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 24.631038] nix-grpc-daemon[778]: E0000 00:00:1790074635.100737 820 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 24.595599] envoy[933]: [2026-09-22 10:57:15.051][933][info][upstream] [source/common/upstream/cluster_manager_impl.cc:250] cm init: all clusters initialized lb # [ 24.603682] envoy[933]: [2026-09-22 10:57:15.051][933][info][main] [source/server/server.cc:1057] all clusters initialized. initializing init manager worker2 # [ 24.626735] nix-grpc-daemon[780]: ts=2026-09-22T10:57:15Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 24.629253] nix-grpc-daemon[780]: ts=2026-09-22T10:57:15Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 lb # [ 24.609214] envoy[933]: [2026-09-22 10:57:15.060][933][info][config] [source/common/listener_manager/listener_manager_impl.cc:1067] all dependencies initialized. starting workers worker1 # [ 24.650225] nix-grpc-daemon[778]: E0000 00:00:1790074635.121183 820 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 24.656100] nix-grpc-daemon[778]: E0000 00:00:1790074635.124844 820 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 24.674769] nix-grpc-daemon[778]: E0000 00:00:1790074635.143895 820 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 24.664421] nix-grpc-daemon[780]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 24.684366] nix-grpc-daemon[778]: ts=2026-09-22T10:57:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.666334] nix-grpc-daemon[780]: E0000 00:00:1790074635.123329 780 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 24.688699] nix-grpc-daemon[778]: ts=2026-09-22T10:57:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.681401] nix-grpc-daemon[780]: E0000 00:00:1790074635.140466 780 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 24.685502] nix-grpc-daemon[780]: ts=2026-09-22T10:57:15Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 24.687327] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 24.723004] nix-grpc-daemon[780]: E0000 00:00:1790074635.180728 822 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 24.742965] nix-grpc-daemon[780]: E0000 00:00:1790074635.201899 822 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 24.746736] nix-grpc-daemon[780]: E0000 00:00:1790074635.205482 822 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 24.766304] nix-grpc-daemon[780]: E0000 00:00:1790074635.224204 822 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 24.782751] nix-grpc-daemon[780]: ts=2026-09-22T10:57:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.784801] nix-grpc-daemon[780]: ts=2026-09-22T10:57:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 24.890125] nix-grpc-daemon[778]: ts=2026-09-22T10:57:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 24.895596] nix-grpc-daemon[778]: ts=2026-09-22T10:57:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 24.986568] nix-grpc-daemon[780]: ts=2026-09-22T10:57:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 24.992391] nix-grpc-daemon[780]: ts=2026-09-22T10:57:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.096945] nix-grpc-daemon[778]: ts=2026-09-22T10:57:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.102694] nix-grpc-daemon[778]: ts=2026-09-22T10:57:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.194063] nix-grpc-daemon[780]: ts=2026-09-22T10:57:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.199860] nix-grpc-daemon[780]: ts=2026-09-22T10:57:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.303837] nix-grpc-daemon[778]: ts=2026-09-22T10:57:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.309702] nix-grpc-daemon[778]: ts=2026-09-22T10:57:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.401210] nix-grpc-daemon[780]: ts=2026-09-22T10:57:15Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.406862] nix-grpc-daemon[780]: ts=2026-09-22T10:57:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.510805] nix-grpc-daemon[778]: ts=2026-09-22T10:57:15Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.517984] nix-grpc-daemon[778]: ts=2026-09-22T10:57:15Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.608333] nix-grpc-daemon[780]: ts=2026-09-22T10:57:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.615605] nix-grpc-daemon[780]: ts=2026-09-22T10:57:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.717808] nix-grpc-daemon[778]: ts=2026-09-22T10:57:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.723419] nix-grpc-daemon[778]: ts=2026-09-22T10:57:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 25.815726] nix-grpc-daemon[780]: ts=2026-09-22T10:57:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 25.821464] nix-grpc-daemon[780]: ts=2026-09-22T10:57:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 25.924295] nix-grpc-daemon[778]: ts=2026-09-22T10:57:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 25.929849] nix-grpc-daemon[778]: ts=2026-09-22T10:57:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.022467] nix-grpc-daemon[780]: ts=2026-09-22T10:57:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.027928] nix-grpc-daemon[780]: ts=2026-09-22T10:57:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.130967] nix-grpc-daemon[778]: ts=2026-09-22T10:57:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.136807] nix-grpc-daemon[778]: ts=2026-09-22T10:57:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.229572] nix-grpc-daemon[780]: ts=2026-09-22T10:57:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.235214] nix-grpc-daemon[780]: ts=2026-09-22T10:57:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.337854] nix-grpc-daemon[778]: ts=2026-09-22T10:57:16Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.343561] nix-grpc-daemon[778]: ts=2026-09-22T10:57:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.436732] nix-grpc-daemon[780]: ts=2026-09-22T10:57:16Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.442255] nix-grpc-daemon[780]: ts=2026-09-22T10:57:16Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.546916] nix-grpc-daemon[778]: ts=2026-09-22T10:57:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.551783] nix-grpc-daemon[778]: ts=2026-09-22T10:57:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.646322] nix-grpc-daemon[780]: ts=2026-09-22T10:57:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.650765] nix-grpc-daemon[780]: ts=2026-09-22T10:57:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.747684] nix-grpc-daemon[778]: ts=2026-09-22T10:57:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.753373] nix-grpc-daemon[778]: ts=2026-09-22T10:57:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 26.847172] nix-grpc-daemon[780]: ts=2026-09-22T10:57:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 26.855654] nix-grpc-daemon[780]: ts=2026-09-22T10:57:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 26.954021] nix-grpc-daemon[778]: ts=2026-09-22T10:57:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 26.958538] nix-grpc-daemon[778]: ts=2026-09-22T10:57:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.055165] nix-grpc-daemon[780]: ts=2026-09-22T10:57:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.060974] nix-grpc-daemon[780]: ts=2026-09-22T10:57:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.159430] nix-grpc-daemon[778]: ts=2026-09-22T10:57:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.165185] nix-grpc-daemon[778]: ts=2026-09-22T10:57:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.262280] nix-grpc-daemon[780]: ts=2026-09-22T10:57:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.267998] nix-grpc-daemon[780]: ts=2026-09-22T10:57:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.366084] nix-grpc-daemon[778]: ts=2026-09-22T10:57:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.371556] nix-grpc-daemon[778]: ts=2026-09-22T10:57:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.469238] nix-grpc-daemon[780]: ts=2026-09-22T10:57:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.474707] nix-grpc-daemon[780]: ts=2026-09-22T10:57:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.574928] nix-grpc-daemon[778]: ts=2026-09-22T10:57:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.579309] nix-grpc-daemon[778]: ts=2026-09-22T10:57:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.678090] nix-grpc-daemon[780]: ts=2026-09-22T10:57:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.682502] nix-grpc-daemon[780]: ts=2026-09-22T10:57:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.775851] nix-grpc-daemon[778]: ts=2026-09-22T10:57:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.781652] nix-grpc-daemon[778]: ts=2026-09-22T10:57:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 27.879069] nix-grpc-daemon[780]: ts=2026-09-22T10:57:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 27.884679] nix-grpc-daemon[780]: ts=2026-09-22T10:57:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 27.982411] nix-grpc-daemon[778]: ts=2026-09-22T10:57:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 27.987945] nix-grpc-daemon[778]: ts=2026-09-22T10:57:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.085846] nix-grpc-daemon[780]: ts=2026-09-22T10:57:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.091406] nix-grpc-daemon[780]: ts=2026-09-22T10:57:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.189091] nix-grpc-daemon[778]: ts=2026-09-22T10:57:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.194674] nix-grpc-daemon[778]: ts=2026-09-22T10:57:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.292644] nix-grpc-daemon[780]: ts=2026-09-22T10:57:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.298103] nix-grpc-daemon[780]: ts=2026-09-22T10:57:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.395833] nix-grpc-daemon[778]: ts=2026-09-22T10:57:18Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.401446] nix-grpc-daemon[778]: ts=2026-09-22T10:57:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.499431] nix-grpc-daemon[780]: ts=2026-09-22T10:57:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.505154] nix-grpc-daemon[780]: ts=2026-09-22T10:57:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.604561] nix-grpc-daemon[778]: ts=2026-09-22T10:57:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.608899] nix-grpc-daemon[778]: ts=2026-09-22T10:57:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.708373] nix-grpc-daemon[780]: ts=2026-09-22T10:57:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.712665] nix-grpc-daemon[780]: ts=2026-09-22T10:57:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 28.805225] nix-grpc-daemon[778]: ts=2026-09-22T10:57:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 28.810794] nix-grpc-daemon[778]: ts=2026-09-22T10:57:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 28.909436] nix-grpc-daemon[780]: ts=2026-09-22T10:57:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 28.914870] nix-grpc-daemon[780]: ts=2026-09-22T10:57:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.011944] nix-grpc-daemon[778]: ts=2026-09-22T10:57:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.017708] nix-grpc-daemon[778]: ts=2026-09-22T10:57:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.116525] nix-grpc-daemon[780]: ts=2026-09-22T10:57:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.122384] nix-grpc-daemon[780]: ts=2026-09-22T10:57:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.218766] nix-grpc-daemon[778]: ts=2026-09-22T10:57:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.224471] nix-grpc-daemon[778]: ts=2026-09-22T10:57:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.323913] nix-grpc-daemon[780]: ts=2026-09-22T10:57:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.329416] nix-grpc-daemon[780]: ts=2026-09-22T10:57:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.425361] nix-grpc-daemon[778]: ts=2026-09-22T10:57:19Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.430893] nix-grpc-daemon[778]: ts=2026-09-22T10:57:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.531023] nix-grpc-daemon[780]: ts=2026-09-22T10:57:19Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.536608] nix-grpc-daemon[780]: ts=2026-09-22T10:57:19Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.633986] nix-grpc-daemon[778]: ts=2026-09-22T10:57:20Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.638309] nix-grpc-daemon[778]: ts=2026-09-22T10:57:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.739017] nix-grpc-daemon[780]: ts=2026-09-22T10:57:20Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.744393] nix-grpc-daemon[780]: ts=2026-09-22T10:57:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 29.834728] nix-grpc-daemon[778]: ts=2026-09-22T10:57:20Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 29.840506] nix-grpc-daemon[778]: ts=2026-09-22T10:57:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 29.941742] nix-grpc-daemon[780]: ts=2026-09-22T10:57:20Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 29.947292] nix-grpc-daemon[780]: ts=2026-09-22T10:57:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.041343] nix-grpc-daemon[778]: ts=2026-09-22T10:57:20Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.046836] nix-grpc-daemon[778]: ts=2026-09-22T10:57:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.149012] nix-grpc-daemon[780]: ts=2026-09-22T10:57:20Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.154694] nix-grpc-daemon[780]: ts=2026-09-22T10:57:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.247865] nix-grpc-daemon[778]: ts=2026-09-22T10:57:20Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.258386] nix-grpc-daemon[778]: ts=2026-09-22T10:57:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.356176] nix-grpc-daemon[780]: ts=2026-09-22T10:57:20Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.361670] nix-grpc-daemon[780]: ts=2026-09-22T10:57:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.454535] nix-grpc-daemon[778]: ts=2026-09-22T10:57:20Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.460184] nix-grpc-daemon[778]: ts=2026-09-22T10:57:20Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.562918] nix-grpc-daemon[780]: ts=2026-09-22T10:57:21Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.568379] nix-grpc-daemon[780]: ts=2026-09-22T10:57:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.663046] nix-grpc-daemon[778]: ts=2026-09-22T10:57:21Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.667388] nix-grpc-daemon[778]: ts=2026-09-22T10:57:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.771810] nix-grpc-daemon[780]: ts=2026-09-22T10:57:21Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.776567] nix-grpc-daemon[780]: ts=2026-09-22T10:57:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 30.863735] nix-grpc-daemon[778]: ts=2026-09-22T10:57:21Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 30.867970] nix-grpc-daemon[778]: ts=2026-09-22T10:57:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 30.972816] nix-grpc-daemon[780]: ts=2026-09-22T10:57:21Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 30.978287] nix-grpc-daemon[780]: ts=2026-09-22T10:57:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.069210] nix-grpc-daemon[778]: ts=2026-09-22T10:57:21Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.074873] nix-grpc-daemon[778]: ts=2026-09-22T10:57:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.179534] nix-grpc-daemon[780]: ts=2026-09-22T10:57:21Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.185289] nix-grpc-daemon[780]: ts=2026-09-22T10:57:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.275895] nix-grpc-daemon[778]: ts=2026-09-22T10:57:21Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.281775] nix-grpc-daemon[778]: ts=2026-09-22T10:57:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.386387] nix-grpc-daemon[780]: ts=2026-09-22T10:57:21Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.391960] nix-grpc-daemon[780]: ts=2026-09-22T10:57:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.482941] nix-grpc-daemon[778]: ts=2026-09-22T10:57:21Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.489086] nix-grpc-daemon[778]: ts=2026-09-22T10:57:21Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.593298] nix-grpc-daemon[780]: ts=2026-09-22T10:57:22Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.598869] nix-grpc-daemon[780]: ts=2026-09-22T10:57:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.691819] nix-grpc-daemon[778]: ts=2026-09-22T10:57:22Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.696657] nix-grpc-daemon[778]: ts=2026-09-22T10:57:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 31.801935] nix-grpc-daemon[780]: ts=2026-09-22T10:57:22Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 31.806193] nix-grpc-daemon[780]: ts=2026-09-22T10:57:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 31.892823] nix-grpc-daemon[778]: ts=2026-09-22T10:57:22Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 31.898536] nix-grpc-daemon[778]: ts=2026-09-22T10:57:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.002955] nix-grpc-daemon[780]: ts=2026-09-22T10:57:22Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.008891] nix-grpc-daemon[780]: ts=2026-09-22T10:57:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.099499] nix-grpc-daemon[778]: ts=2026-09-22T10:57:22Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.105238] nix-grpc-daemon[778]: ts=2026-09-22T10:57:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 32.210133] nix-grpc-daemon[780]: ts=2026-09-22T10:57:22Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.215787] nix-grpc-daemon[780]: ts=2026-09-22T10:57:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.306347] nix-grpc-daemon[778]: ts=2026-09-22T10:57:22Z level=info event=scheduler_connected addr=lb:50051 lb # [ 32.274997] rustfs-bucket-start[1025]: mb s3://farm worker1 # [ 32.312332] nix-grpc-daemon[778]: ts=2026-09-22T10:57:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 32.287781] systemd[1]: Finished rustfs-bucket.service. lb # [ 32.301531] systemd[1]: Starting niks3 server... worker2 # [ 32.417244] nix-grpc-daemon[780]: ts=2026-09-22T10:57:22Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.422907] nix-grpc-daemon[780]: ts=2026-09-22T10:57:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.513447] nix-grpc-daemon[778]: ts=2026-09-22T10:57:22Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.519282] nix-grpc-daemon[778]: ts=2026-09-22T10:57:22Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 32.599934] postgres[1044]: [1044] ERROR: relation "goose_db_version" does not exist at character 36 lb # [ 32.604384] postgres[1044]: [1044] STATEMENT: SELECT version_id, is_applied from goose_db_version ORDER BY id DESC worker2 # [ 32.624390] nix-grpc-daemon[780]: ts=2026-09-22T10:57:23Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.629918] nix-grpc-daemon[780]: ts=2026-09-22T10:57:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 32.644001] niks3-server[1035]: 2026/09/22 10:57:23 OK 20241026095416_initial_model.sql (24.21ms) lb # [ 32.648342] niks3-server[1035]: 2026/09/22 10:57:23 OK 20251210153512_drop_unused_gin_index.sql (4.72ms) lb # [ 32.655715] niks3-server[1035]: 2026/09/22 10:57:23 OK 20251218171726_add_pins.sql (7.29ms) lb # [ 32.661804] niks3-server[1035]: 2026/09/22 10:57:23 OK 20260628120000_add_object_size_and_stats.sql (6.03ms) lb # [ 32.667669] niks3-server[1035]: 2026/09/22 10:57:23 OK 20260905000000_add_claims.sql (5.81ms) lb # [ 32.673361] niks3-server[1035]: 2026/09/22 10:57:23 OK 20260920000000_drop_claims.sql (5.51ms) lb # [ 32.674756] niks3-server[1035]: 2026/09/22 10:57:23 goose: successfully migrated database to version: 20260920000000 lb # [ 32.680918] niks3-server[1035]: 2026/09/22 10:57:23 OK 1_commit_pending_closure.sql (7.51ms) lb # [ 32.685011] niks3-server[1035]: 2026/09/22 10:57:23 OK 2_object_stats_trigger.sql (3.89ms) lb # [ 32.686312] niks3-server[1035]: 2026/09/22 10:57:23 goose: up to current file version: 2 worker1 # [ 32.722287] nix-grpc-daemon[778]: ts=2026-09-22T10:57:23Z level=info event=scheduler_connected addr=lb:50051 lb # [ 32.691015] niks3-server[1035]: 2026/09/22 10:57:23 INFO Loaded signing key name=farm-test-1 path=/nix/store/8lzz3wcf17v05vfdp1zcfaa9hipw3f70-key worker1 # [ 32.726643] nix-grpc-daemon[778]: ts=2026-09-22T10:57:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 32.741400] niks3-server[1035]: 2026/09/22 10:57:23 INFO Read proxy enabled — serving cache objects from S3 lb # [ 32.742877] niks3-server[1035]: 2026/09/22 10:57:23 INFO Using socket-activated listener address=0.0.0.0:5751 lb # [ 32.745851] niks3-server[1035]: 2026/09/22 10:57:23 INFO systemd watchdog enabled interval=15s lb # [ 32.751221] systemd[1]: Started niks3 server. lb # [ 32.752337] niks3-server[1035]: 2026/09/22 10:57:23 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 32.754286] systemd[1]: Reached target Multi-User System. lb # [ 32.755848] systemd[1]: Startup finished in 1.117s (kernel) + 4.790s (initrd) + 26.842s (userspace) = 32.750s. worker2 # [ 32.833293] nix-grpc-daemon[780]: ts=2026-09-22T10:57:23Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 32.837651] nix-grpc-daemon[780]: ts=2026-09-22T10:57:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 32.922596] nix-grpc-daemon[778]: ts=2026-09-22T10:57:23Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 32.928248] nix-grpc-daemon[778]: ts=2026-09-22T10:57:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.034238] nix-grpc-daemon[780]: ts=2026-09-22T10:57:23Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.039902] nix-grpc-daemon[780]: ts=2026-09-22T10:57:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.129152] nix-grpc-daemon[778]: ts=2026-09-22T10:57:23Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.134709] nix-grpc-daemon[778]: ts=2026-09-22T10:57:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.241728] nix-grpc-daemon[780]: ts=2026-09-22T10:57:23Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.247176] nix-grpc-daemon[780]: ts=2026-09-22T10:57:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.335617] nix-grpc-daemon[778]: ts=2026-09-22T10:57:23Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.341214] nix-grpc-daemon[778]: ts=2026-09-22T10:57:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 33.448536] nix-grpc-daemon[780]: ts=2026-09-22T10:57:23Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.454083] nix-grpc-daemon[780]: ts=2026-09-22T10:57:23Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.543067] nix-grpc-daemon[778]: ts=2026-09-22T10:57:24Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.551634] nix-grpc-daemon[778]: ts=2026-09-22T10:57:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: waiting for unit niks3.service, in 17.28 seconds) lb: waiting for TCP port 5751 on localhost worker2 # [ 33.655421] nix-grpc-daemon[780]: ts=2026-09-22T10:57:24Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.659557] nix-grpc-daemon[780]: ts=2026-09-22T10:57:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # Connection to localhost (127.0.0.1) 5751 port [tcp/*] succeeded! lb: (finished: waiting for TCP port 5751 on localhost, in 0.10 seconds) lb: waiting for unit envoy.service worker1 # [ 33.752933] nix-grpc-daemon[778]: ts=2026-09-22T10:57:24Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.758599] nix-grpc-daemon[778]: ts=2026-09-22T10:57:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: waiting for unit envoy.service, in 0.07 seconds) lb: waiting for TCP port 50051 on localhost lb # Connection to localhost (::1) 50051 port [tcp/*] succeeded! lb # [ 33.801014] envoy[933]: conn peer=[::1]:37022 tls=TLSv1.2 subject="-" sni=- flags=- tls_fail="-" rx=0 tx=0 ms=3 lb: (finished: waiting for TCP port 50051 on localhost, in 0.05 seconds) worker2 # [ 33.863117] nix-grpc-daemon[780]: ts=2026-09-22T10:57:24Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 33.867443] nix-grpc-daemon[780]: ts=2026-09-22T10:57:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 33.953442] nix-grpc-daemon[778]: ts=2026-09-22T10:57:24Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 33.958180] nix-grpc-daemon[778]: ts=2026-09-22T10:57:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.010190] systemd[1]: Starting scheduler-token.service... worker2 # [ 34.063357] nix-grpc-daemon[780]: ts=2026-09-22T10:57:24Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.068177] nix-grpc-daemon[780]: ts=2026-09-22T10:57:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.119779] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 34.125997] systemd[1]: Finished scheduler-token.service. worker2 # [ 34.127972] systemd[1]: scheduler-token.service: Consumed 41ms CPU time over 114ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 34.159369] nix-grpc-daemon[778]: ts=2026-09-22T10:57:24Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.165434] nix-grpc-daemon[778]: ts=2026-09-22T10:57:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.11 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 0.11 seconds) subtest: one node schedules and both builders hold a WorkerSession on it lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 34.268905] nix-grpc-daemon[780]: ts=2026-09-22T10:57:24Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.274479] nix-grpc-daemon[780]: ts=2026-09-22T10:57:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) worker1 # [ 34.366359] nix-grpc-daemon[778]: ts=2026-09-22T10:57:24Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.372336] nix-grpc-daemon[778]: ts=2026-09-22T10:57:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.475658] nix-grpc-daemon[780]: ts=2026-09-22T10:57:24Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.481245] nix-grpc-daemon[780]: ts=2026-09-22T10:57:24Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.573517] nix-grpc-daemon[778]: ts=2026-09-22T10:57:25Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.580420] nix-grpc-daemon[778]: ts=2026-09-22T10:57:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.682391] nix-grpc-daemon[780]: ts=2026-09-22T10:57:25Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.688190] nix-grpc-daemon[780]: ts=2026-09-22T10:57:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.782953] nix-grpc-daemon[778]: ts=2026-09-22T10:57:25Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.787322] nix-grpc-daemon[778]: ts=2026-09-22T10:57:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 34.889692] nix-grpc-daemon[780]: ts=2026-09-22T10:57:25Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 34.892914] nix-grpc-daemon[780]: ts=2026-09-22T10:57:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 34.983916] nix-grpc-daemon[778]: ts=2026-09-22T10:57:25Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 34.989523] nix-grpc-daemon[778]: ts=2026-09-22T10:57:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 35.094535] nix-grpc-daemon[780]: ts=2026-09-22T10:57:25Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.100406] nix-grpc-daemon[780]: ts=2026-09-22T10:57:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 35.190757] nix-grpc-daemon[778]: ts=2026-09-22T10:57:25Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 35.196581] nix-grpc-daemon[778]: ts=2026-09-22T10:57:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 35.301632] nix-grpc-daemon[780]: ts=2026-09-22T10:57:25Z level=info event=scheduler_connected addr=lb:50051 lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 35.307175] nix-grpc-daemon[780]: ts=2026-09-22T10:57:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 35.397470] nix-grpc-daemon[778]: ts=2026-09-22T10:57:25Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 35.403685] nix-grpc-daemon[778]: ts=2026-09-22T10:57:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds) worker2 # [ 35.508644] nix-grpc-daemon[780]: ts=2026-09-22T10:57:25Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.514195] nix-grpc-daemon[780]: ts=2026-09-22T10:57:25Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 35.604759] nix-grpc-daemon[778]: ts=2026-09-22T10:57:26Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 35.610448] nix-grpc-daemon[778]: ts=2026-09-22T10:57:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 35.715550] nix-grpc-daemon[780]: ts=2026-09-22T10:57:26Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.721603] nix-grpc-daemon[780]: ts=2026-09-22T10:57:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 35.813288] nix-grpc-daemon[778]: ts=2026-09-22T10:57:26Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 35.818419] nix-grpc-daemon[778]: ts=2026-09-22T10:57:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 35.924967] nix-grpc-daemon[780]: ts=2026-09-22T10:57:26Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 35.929214] nix-grpc-daemon[780]: ts=2026-09-22T10:57:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 36.019462] nix-grpc-daemon[778]: ts=2026-09-22T10:57:26Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 36.025186] nix-grpc-daemon[778]: ts=2026-09-22T10:57:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 36.126177] nix-grpc-daemon[780]: ts=2026-09-22T10:57:26Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.131704] nix-grpc-daemon[780]: ts=2026-09-22T10:57:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 36.225921] nix-grpc-daemon[778]: ts=2026-09-22T10:57:26Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 36.231410] nix-grpc-daemon[778]: ts=2026-09-22T10:57:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 36.332948] nix-grpc-daemon[780]: ts=2026-09-22T10:57:26Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.338294] nix-grpc-daemon[780]: ts=2026-09-22T10:57:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 36.432420] nix-grpc-daemon[778]: ts=2026-09-22T10:57:26Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 36.437678] nix-grpc-daemon[778]: ts=2026-09-22T10:57:26Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds) worker2 # [ 36.539649] nix-grpc-daemon[780]: ts=2026-09-22T10:57:26Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.545144] nix-grpc-daemon[780]: ts=2026-09-22T10:57:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 36.638724] nix-grpc-daemon[778]: ts=2026-09-22T10:57:27Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 36.644194] nix-grpc-daemon[778]: ts=2026-09-22T10:57:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 36.746387] nix-grpc-daemon[780]: ts=2026-09-22T10:57:27Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.753675] nix-grpc-daemon[780]: ts=2026-09-22T10:57:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 36.852845] nix-grpc-daemon[778]: ts=2026-09-22T10:57:27Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 36.859174] nix-grpc-daemon[778]: ts=2026-09-22T10:57:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 36.954996] nix-grpc-daemon[780]: ts=2026-09-22T10:57:27Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 36.958310] nix-grpc-daemon[780]: ts=2026-09-22T10:57:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 37.049226] nix-grpc-daemon[778]: ts=2026-09-22T10:57:27Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 37.055072] nix-grpc-daemon[778]: ts=2026-09-22T10:57:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.159738] nix-grpc-daemon[780]: ts=2026-09-22T10:57:27Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.165481] nix-grpc-daemon[780]: ts=2026-09-22T10:57:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 37.256418] nix-grpc-daemon[778]: ts=2026-09-22T10:57:27Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 37.261832] nix-grpc-daemon[778]: ts=2026-09-22T10:57:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.366678] nix-grpc-daemon[780]: ts=2026-09-22T10:57:27Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.372662] nix-grpc-daemon[780]: ts=2026-09-22T10:57:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 37.462771] nix-grpc-daemon[778]: ts=2026-09-22T10:57:27Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 37.468432] nix-grpc-daemon[778]: ts=2026-09-22T10:57:27Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 37.574367] nix-grpc-daemon[780]: ts=2026-09-22T10:57:28Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.580256] nix-grpc-daemon[780]: ts=2026-09-22T10:57:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds) worker1 # [ 37.669028] nix-grpc-daemon[778]: ts=2026-09-22T10:57:28Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 37.674549] nix-grpc-daemon[778]: ts=2026-09-22T10:57:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.781625] nix-grpc-daemon[780]: ts=2026-09-22T10:57:28Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.787237] nix-grpc-daemon[780]: ts=2026-09-22T10:57:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 37.877781] nix-grpc-daemon[778]: ts=2026-09-22T10:57:28Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 37.882079] nix-grpc-daemon[778]: ts=2026-09-22T10:57:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 37.991020] nix-grpc-daemon[780]: ts=2026-09-22T10:57:28Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 37.995454] nix-grpc-daemon[780]: ts=2026-09-22T10:57:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 38.078242] nix-grpc-daemon[778]: ts=2026-09-22T10:57:28Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 38.084001] nix-grpc-daemon[778]: ts=2026-09-22T10:57:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 38.192551] nix-grpc-daemon[780]: ts=2026-09-22T10:57:28Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.198283] nix-grpc-daemon[780]: ts=2026-09-22T10:57:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 38.285294] nix-grpc-daemon[778]: ts=2026-09-22T10:57:28Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 38.290935] nix-grpc-daemon[778]: ts=2026-09-22T10:57:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 38.399784] nix-grpc-daemon[780]: ts=2026-09-22T10:57:28Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.405417] nix-grpc-daemon[780]: ts=2026-09-22T10:57:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 38.491983] nix-grpc-daemon[778]: ts=2026-09-22T10:57:28Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 38.497630] nix-grpc-daemon[778]: ts=2026-09-22T10:57:28Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 38.606518] nix-grpc-daemon[780]: ts=2026-09-22T10:57:29Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.612157] nix-grpc-daemon[780]: ts=2026-09-22T10:57:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 38.698634] nix-grpc-daemon[778]: ts=2026-09-22T10:57:29Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 38.704212] nix-grpc-daemon[778]: ts=2026-09-22T10:57:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds) worker2 # [ 38.813560] nix-grpc-daemon[780]: ts=2026-09-22T10:57:29Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 38.819222] nix-grpc-daemon[780]: ts=2026-09-22T10:57:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 38.907170] nix-grpc-daemon[778]: ts=2026-09-22T10:57:29Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 38.911430] nix-grpc-daemon[778]: ts=2026-09-22T10:57:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 39.022739] nix-grpc-daemon[780]: ts=2026-09-22T10:57:29Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.027072] nix-grpc-daemon[780]: ts=2026-09-22T10:57:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 39.107754] nix-grpc-daemon[778]: ts=2026-09-22T10:57:29Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 39.113339] nix-grpc-daemon[778]: ts=2026-09-22T10:57:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 39.223829] nix-grpc-daemon[780]: ts=2026-09-22T10:57:29Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.229433] nix-grpc-daemon[780]: ts=2026-09-22T10:57:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 39.314312] nix-grpc-daemon[778]: ts=2026-09-22T10:57:29Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 39.319908] nix-grpc-daemon[778]: ts=2026-09-22T10:57:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 39.430843] nix-grpc-daemon[780]: ts=2026-09-22T10:57:29Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.436670] nix-grpc-daemon[780]: ts=2026-09-22T10:57:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 39.520486] nix-grpc-daemon[778]: ts=2026-09-22T10:57:29Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 39.525983] nix-grpc-daemon[778]: ts=2026-09-22T10:57:29Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 39.638157] nix-grpc-daemon[780]: ts=2026-09-22T10:57:30Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.644341] nix-grpc-daemon[780]: ts=2026-09-22T10:57:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 39.727127] nix-grpc-daemon[778]: ts=2026-09-22T10:57:30Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 39.732687] nix-grpc-daemon[778]: ts=2026-09-22T10:57:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds) worker2 # [ 39.845350] nix-grpc-daemon[780]: ts=2026-09-22T10:57:30Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 39.850854] nix-grpc-daemon[780]: ts=2026-09-22T10:57:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 39.935890] nix-grpc-daemon[778]: ts=2026-09-22T10:57:30Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 39.940285] nix-grpc-daemon[778]: ts=2026-09-22T10:57:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 40.054663] nix-grpc-daemon[780]: ts=2026-09-22T10:57:30Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 40.059097] nix-grpc-daemon[780]: ts=2026-09-22T10:57:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 40.136080] nix-grpc-daemon[778]: ts=2026-09-22T10:57:30Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 40.139033] nix-grpc-daemon[778]: ts=2026-09-22T10:57:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 40.254930] nix-grpc-daemon[780]: ts=2026-09-22T10:57:30Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 40.262715] nix-grpc-daemon[780]: ts=2026-09-22T10:57:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 40.339887] nix-grpc-daemon[778]: ts=2026-09-22T10:57:30Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 40.344172] nix-grpc-daemon[778]: ts=2026-09-22T10:57:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 40.461585] nix-grpc-daemon[780]: ts=2026-09-22T10:57:30Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 40.467491] nix-grpc-daemon[780]: ts=2026-09-22T10:57:30Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 40.545089] nix-grpc-daemon[778]: ts=2026-09-22T10:57:31Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 40.550731] nix-grpc-daemon[778]: ts=2026-09-22T10:57:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 40.668712] nix-grpc-daemon[780]: ts=2026-09-22T10:57:31Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 40.675865] nix-grpc-daemon[780]: ts=2026-09-22T10:57:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 40.751407] nix-grpc-daemon[778]: ts=2026-09-22T10:57:31Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 40.756966] nix-grpc-daemon[778]: ts=2026-09-22T10:57:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds) worker2 # [ 40.876963] nix-grpc-daemon[780]: ts=2026-09-22T10:57:31Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 40.882906] nix-grpc-daemon[780]: ts=2026-09-22T10:57:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 40.960630] nix-grpc-daemon[778]: ts=2026-09-22T10:57:31Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 40.964936] nix-grpc-daemon[778]: ts=2026-09-22T10:57:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 41.086621] nix-grpc-daemon[780]: ts=2026-09-22T10:57:31Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 41.092543] nix-grpc-daemon[780]: ts=2026-09-22T10:57:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 41.161459] nix-grpc-daemon[778]: ts=2026-09-22T10:57:31Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 41.167376] nix-grpc-daemon[778]: ts=2026-09-22T10:57:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 41.287774] nix-grpc-daemon[780]: ts=2026-09-22T10:57:31Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 41.294364] nix-grpc-daemon[780]: ts=2026-09-22T10:57:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 41.368243] nix-grpc-daemon[778]: ts=2026-09-22T10:57:31Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 41.373887] nix-grpc-daemon[778]: ts=2026-09-22T10:57:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 41.494778] nix-grpc-daemon[780]: ts=2026-09-22T10:57:31Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 41.500762] nix-grpc-daemon[780]: ts=2026-09-22T10:57:31Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 41.575130] nix-grpc-daemon[778]: ts=2026-09-22T10:57:32Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 41.581062] nix-grpc-daemon[778]: ts=2026-09-22T10:57:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 41.702087] nix-grpc-daemon[780]: ts=2026-09-22T10:57:32Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 41.707671] nix-grpc-daemon[780]: ts=2026-09-22T10:57:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 41.781917] nix-grpc-daemon[778]: ts=2026-09-22T10:57:32Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 41.787556] nix-grpc-daemon[778]: ts=2026-09-22T10:57:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 41.908880] nix-grpc-daemon[780]: ts=2026-09-22T10:57:32Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 41.914635] nix-grpc-daemon[780]: ts=2026-09-22T10:57:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker1 # [ 41.990825] nix-grpc-daemon[778]: ts=2026-09-22T10:57:32Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 41.995064] nix-grpc-daemon[778]: ts=2026-09-22T10:57:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 42.117360] nix-grpc-daemon[780]: ts=2026-09-22T10:57:32Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 42.122261] nix-grpc-daemon[780]: ts=2026-09-22T10:57:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 42.191949] nix-grpc-daemon[778]: ts=2026-09-22T10:57:32Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 42.197676] nix-grpc-daemon[778]: ts=2026-09-22T10:57:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 42.323941] nix-grpc-daemon[780]: ts=2026-09-22T10:57:32Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 42.329593] nix-grpc-daemon[780]: ts=2026-09-22T10:57:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 42.398352] nix-grpc-daemon[778]: ts=2026-09-22T10:57:32Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 42.401476] nix-grpc-daemon[778]: ts=2026-09-22T10:57:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 42.530712] nix-grpc-daemon[780]: ts=2026-09-22T10:57:32Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 42.536345] nix-grpc-daemon[780]: ts=2026-09-22T10:57:32Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 42.602724] nix-grpc-daemon[778]: ts=2026-09-22T10:57:33Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 42.609308] nix-grpc-daemon[778]: ts=2026-09-22T10:57:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 42.737406] nix-grpc-daemon[780]: ts=2026-09-22T10:57:33Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 42.743163] nix-grpc-daemon[780]: ts=2026-09-22T10:57:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 42.784903] nix-grpc-daemon[780]: ts=2026-09-22T10:57:33Z level=info event=scheduler_take_over lb # [ 42.766522] niks3-server[1035]: 2026/09/22 10:57:33 INFO lead: acquired remote=192.168.1.4:35020 worker1 # [ 42.809957] nix-grpc-daemon[778]: ts=2026-09-22T10:57:33Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 42.815278] nix-grpc-daemon[778]: ts=2026-09-22T10:57:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 42.944517] nix-grpc-daemon[780]: ts=2026-09-22T10:57:33Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 42.949967] nix-grpc-daemon[780]: ts=2026-09-22T10:57:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 43.019028] nix-grpc-daemon[778]: ts=2026-09-22T10:57:33Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 43.023345] nix-grpc-daemon[778]: ts=2026-09-22T10:57:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds) worker2 # [ 43.153299] nix-grpc-daemon[780]: ts=2026-09-22T10:57:33Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 43.157659] nix-grpc-daemon[780]: ts=2026-09-22T10:57:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 43.220072] nix-grpc-daemon[778]: ts=2026-09-22T10:57:33Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 43.225564] nix-grpc-daemon[778]: ts=2026-09-22T10:57:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 43.354480] nix-grpc-daemon[780]: ts=2026-09-22T10:57:33Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 43.360313] nix-grpc-daemon[780]: ts=2026-09-22T10:57:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 43.426447] nix-grpc-daemon[778]: ts=2026-09-22T10:57:33Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 43.432154] nix-grpc-daemon[778]: ts=2026-09-22T10:57:33Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 43.562064] nix-grpc-daemon[780]: ts=2026-09-22T10:57:34Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 43.571648] nix-grpc-daemon[780]: ts=2026-09-22T10:57:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 43.632786] nix-grpc-daemon[778]: ts=2026-09-22T10:57:34Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 43.636643] nix-grpc-daemon[778]: ts=2026-09-22T10:57:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 43.769222] nix-grpc-daemon[780]: ts=2026-09-22T10:57:34Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 43.776926] nix-grpc-daemon[780]: ts=2026-09-22T10:57:34Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 43.837363] nix-grpc-daemon[778]: ts=2026-09-22T10:57:34Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 43.832602] nix-grpc-daemon[780]: ts=2026-09-22T10:57:34Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:40124 worker2 # [ 43.838434] nix-grpc-daemon[780]: ts=2026-09-22T10:57:34Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker2 # [ 43.978077] nix-grpc-daemon[780]: ts=2026-09-22T10:57:34Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 43.999501] nix-grpc-daemon[780]: ts=2026-09-22T10:57:34Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 44.004766] nix-grpc-daemon[780]: ts=2026-09-22T10:57:34Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:40124 worker2 # [ 44.010207] nix-grpc-daemon[780]: ts=2026-09-22T10:57:34Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.09 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open' worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'event=worker_session_open', in 0.07 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.10 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.06 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds) (finished: subtest: one node schedules and both builders hold a WorkerSession on it, in 10.76 seconds) subtest: balancer auth: client cert, bearer token, nothing, unknown CN worker1 # [ 45.429368] nix-grpc-daemon[778]: ts=2026-09-22T10:57:35Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 45.449862] systemd[1]: Started Nix Daemon. worker1 # [ 45.735027] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 45.759607] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 17 ms (attempt 1/5) worker1 # [ 45.776855] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 74 ms (attempt 2/5) worker1 # [ 45.851129] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 300 ms (attempt 3/5) worker1 # [ 46.151657] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 379 ms (attempt 4/5) worker1 # [ 46.534342] nix-grpc-daemon[778]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 46.540712] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 46.559883] nix-grpc-daemon[778]: ts=2026-09-22T10:57:37Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 46.574697] nix-grpc-daemon[778]: ts=2026-09-22T10:57:37Z level=debug event=path_info_miss path=k6qpk00c7qa52lid1b7zqxvp2lykb521-nixos-system-client-test error="error: path '/nix/store/k6qpk00c7qa52lid1b7zqxvp2lykb521-nixos-system-client-test' is required, but there is no substituter that can build it" worker2 # [ 46.585389] nix-grpc-daemon[780]: ts=2026-09-22T10:57:37Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 46.606669] systemd[1]: Started Nix Daemon. worker2 # [ 46.971453] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) worker2 # [ 46.998122] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 0 ms (attempt 1/5) worker2 # [ 47.001392] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 60 ms (attempt 2/5) worker2 # [ 47.061385] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 298 ms (attempt 3/5) worker2 # [ 47.359955] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 93 ms (attempt 4/5) worker2 # [ 47.455579] nix-grpc-daemon[780]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 47.461163] nix-grpc-daemon[780]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 47.474280] nix-grpc-daemon[780]: ts=2026-09-22T10:57:37Z level=debug event=path_info_miss path=k6qpk00c7qa52lid1b7zqxvp2lykb521-nixos-system-client-test error="error: path '/nix/store/k6qpk00c7qa52lid1b7zqxvp2lykb521-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt client: (finished: must succeed: curl -sfG http://lb:8081/issue --data-urlencode 'aud=grpc://lb:50051' --data-urlencode sub=dev:alice > /root/dev.jwt && test -s /root/dev.jwt, in 0.10 seconds) client: must succeed: install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key client: (finished: must succeed: install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.pem /var/lib/nix-grpc-store/client.crt && install -D /nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/foreign.key /var/lib/nix-grpc-store/client.key, in 0.05 seconds) worker2 # [ 47.939794] nix-grpc-daemon[780]: ts=2026-09-22T10:57:38Z level=debug event=rpc_start method=QueryMissing cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:40132 worker1 # [ 47.987277] nix-grpc-daemon[778]: ts=2026-09-22T10:57:38Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 47.993165] nix-grpc-daemon[778]: ts=2026-09-22T10:57:38Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:41338 worker1 # [ 48.003685] nix-grpc-daemon[778]: ts=2026-09-22T10:57:38Z level=debug event=path_info_miss path=k6qpk00c7qa52lid1b7zqxvp2lykb521-nixos-system-client-test error="error: path '/nix/store/k6qpk00c7qa52lid1b7zqxvp2lykb521-nixos-system-client-test' is required, but there is no substituter that can build it" worker1 # [ 48.019188] nix-grpc-daemon[778]: ts=2026-09-22T10:57:38Z level=debug event=rpc_start method=QueryPathInfos cn=oidc:mock:dev:alice peer=ipv4:192.168.1.2:41338 worker1 # [ 48.028083] nix-grpc-daemon[778]: ts=2026-09-22T10:57:38Z level=debug event=path_info_miss path=k6qpk00c7qa52lid1b7zqxvp2lykb521-nixos-system-client-test error="error: path '/nix/store/k6qpk00c7qa52lid1b7zqxvp2lykb521-nixos-system-client-test' is required, but there is no substituter that can build it" client: must succeed: rm -r /var/lib/nix-grpc-store client: (finished: must succeed: rm -r /var/lib/nix-grpc-store, in 0.04 seconds) worker2 # [ 50.326722] nix-grpc-daemon[780]: ts=2026-09-22T10:57:40Z level=info event=denied method=QueryMissing cn=- role=none worker2 # [ 50.595239] nix-grpc-daemon[780]: ts=2026-09-22T10:57:41Z level=info event=denied method=QueryMissing cn=stranger role=none (finished: subtest: balancer auth: client cert, bearer token, nothing, unknown CN, in 5.61 seconds) subtest: DAG build is scheduled across workers and lands in the cache client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2 worker2 # [ 50.858328] nix-grpc-daemon[780]: ts=2026-09-22T10:57:41Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:40132 client # don't know how to build these paths: client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker2 # [ 50.871501] nix-grpc-daemon[780]: ts=2026-09-22T10:57:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 50.910734] nix-grpc-daemon[780]: ts=2026-09-22T10:57:41Z level=debug event=want client=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv cached=0 worker2 # [ 50.916991] nix-grpc-daemon[780]: ts=2026-09-22T10:57:41Z level=debug event=assigned drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 50.983453] nix-grpc-daemon[778]: ts=2026-09-22T10:57:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 51.001371] nix-grpc-daemon[778]: ts=2026-09-22T10:57:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 client # copying 1 paths to 'lb:50051'... worker1 # [ 51.018524] nix-grpc-daemon[778]: ts=2026-09-22T10:57:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 client # copying path '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' to 'lb:50051'... worker1 # [ 51.049230] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 51.080665] nix-grpc-daemon[778]: ts=2026-09-22T10:57:41Z level=info event=niks3_push_started pid=857 worker1 # [ 51.389256] nix-grpc-daemon[857]: time=2026-09-22T10:57:41.858Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca="" worker1 # [ 51.587858] nix-daemon[844]: accepted connection from pid 870, user nix-grpc-daemon (trusted) worker1 # [ 51.607508] nix-daemon[844]: reaped child process 873, status = succeeded worker1 # [ 51.788933] nix-daemon[844]: accepted connection from pid 875, user nix-grpc-daemon (trusted) worker1 # [ 51.806713] nix-daemon[844]: reaped child process 878, status = succeeded lb # [ 51.782935] niks3-server[1035]: 2026/09/22 10:57:42 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 51.845204] nix-grpc-daemon[857]: time=2026-09-22T10:57:42.316Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 51.848414] nix-grpc-daemon[857]: time=2026-09-22T10:57:42.318Z level=INFO msg="Uploading 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv (504B)" lb # [ 51.852308] niks3-server[1035]: 2026/09/22 10:57:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/1/sign lb # [ 51.872498] niks3-server[1035]: 2026/09/22 10:57:42 INFO Signed narinfos id=1 count=1 worker1 # [ 51.909794] nix-grpc-daemon[857]: time=2026-09-22T10:57:42.380Z level=INFO msg="Uploading 1 narinfos" lb # [ 51.919344] niks3-server[1035]: 2026/09/22 10:57:42 INFO Received complete upload request method=POST path=/api/pending_closures/1/complete lb # [ 51.926397] niks3-server[1035]: 2026/09/22 10:57:42 INFO Registered completed upload object_key=nar/1il50j4vra11xfnqgjhjgrjn1753frlhxnss6mh1dmp5ww3is34w.nar.zst lb # [ 51.936479] niks3-server[1035]: 2026/09/22 10:57:42 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.ls lb # [ 51.943990] niks3-server[1035]: 2026/09/22 10:57:42 INFO Registered completed upload object_key=619ccx60pfv52kv4hhgqpj1ivdmfcr5r.narinfo lb # [ 51.951807] niks3-server[1035]: 2026/09/22 10:57:42 INFO Completed upload id=1 worker1 # [ 51.988797] nix-grpc-daemon[857]: time=2026-09-22T10:57:42.459Z level=INFO msg="Upload complete. (591ms)" worker1 # [ 52.000933] nix-grpc-daemon[778]: ts=2026-09-22T10:57:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_in=504 worker1 # [ 52.016836] nix-grpc-daemon[778]: ts=2026-09-22T10:57:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 52.032936] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 52.036607] nix-daemon[844]: reaped child process 855, status = succeeded client # building '/nix/store/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv' on lb:50051... client # farm-a> node-a: building 619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv worker1 # [ 52.053068] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-t1 worker1 # [ 53.485669] nix-daemon[844]: reaped child process 882, status = succeeded worker1 # [ 53.654997] nix-daemon[844]: accepted connection from pid 898, user nix-grpc-daemon (trusted) worker1 # [ 53.673185] nix-daemon[844]: reaped child process 901, status = succeeded lb # [ 53.649901] niks3-server[1035]: 2026/09/22 10:57:44 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 53.698586] nix-grpc-daemon[857]: time=2026-09-22T10:57:44.168Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 53.700513] nix-grpc-daemon[857]: time=2026-09-22T10:57:44.168Z level=INFO msg="Uploading dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a (128B)" lb # [ 53.695555] niks3-server[1035]: 2026/09/22 10:57:44 INFO Registered completed upload object_key=log/619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv lb # [ 53.718036] niks3-server[1035]: 2026/09/22 10:57:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/2/sign worker1 # [ 53.763408] nix-grpc-daemon[857]: time=2026-09-22T10:57:44.233Z level=INFO msg="Uploading 1 narinfos" lb # [ 53.727076] niks3-server[1035]: 2026/09/22 10:57:44 INFO Registered completed upload object_key=nar/1rmil4ja9jzfwn6l5hwbir70b6plx99sgk0pnjjy377ngbc3vz28.nar.zst lb # [ 53.742262] niks3-server[1035]: 2026/09/22 10:57:44 INFO Signed narinfos id=2 count=1 lb # [ 53.750461] niks3-server[1035]: 2026/09/22 10:57:44 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.ls lb # [ 53.766520] niks3-server[1035]: 2026/09/22 10:57:44 INFO Received complete upload request method=POST path=/api/pending_closures/2/complete lb # [ 53.775338] niks3-server[1035]: 2026/09/22 10:57:44 INFO Registered completed upload object_key=dg404zlvxfrspnxl6cc1qkc6i83m5a3a.narinfo lb # [ 53.780332] niks3-server[1035]: 2026/09/22 10:57:44 INFO Completed upload id=2 worker1 # [ 53.817143] nix-grpc-daemon[857]: time=2026-09-22T10:57:44.288Z level=INFO msg="Upload complete. (346ms)" worker1 # [ 53.829311] nix-grpc-daemon[778]: ts=2026-09-22T10:57:44Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=1 drv=619ccx60pfv52kv4hhgqpj1ivdmfcr5r-farm-a.drv assign_id=1 outputs=1 worker2 # [ 53.823008] nix-grpc-daemon[780]: ts=2026-09-22T10:57:44Z level=debug event=want client=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv cached=0 worker2 # [ 53.828737] nix-grpc-daemon[780]: ts=2026-09-22T10:57:44Z level=debug event=assigned drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 53.851225] nix-daemon[844]: reaped child process 880, status = succeeded worker1 # [ 53.891379] nix-grpc-daemon[778]: ts=2026-09-22T10:57:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 53.906772] nix-grpc-daemon[778]: ts=2026-09-22T10:57:44Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 53.921761] nix-grpc-daemon[857]: time=2026-09-22T10:57:44.392Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker1 # [ 53.932851] nix-grpc-daemon[778]: ts=2026-09-22T10:57:44Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 client # copying path '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' to 'lb:50051'... worker1 # [ 53.956253] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 54.163106] nix-daemon[844]: accepted connection from pid 905, user nix-grpc-daemon (trusted) worker1 # [ 54.181236] nix-daemon[844]: reaped child process 908, status = succeeded lb # [ 54.193824] envoy[933]: conn peer=[2001:db8:1::1]:57918 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=749 tx=423 ms=2122 lb # [ 54.198911] envoy[933]: conn peer=[2001:db8:1::1]:57920 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=2596 tx=364 ms=96 lb # [ 54.203836] envoy[933]: conn peer=[2001:db8:1::1]:57924 tls=TLSv1.3 subject="-" sni=- flags=- tls_fail="-" rx=423 tx=198 ms=12 lb # [ 54.209037] envoy[933]: conn peer=[2001:db8:1::1]:57934 tls=TLSv1.3 subject="CN=stranger" sni=- flags=- tls_fail="-" rx=423 tx=181 ms=13 worker1 # [ 54.358566] nix-daemon[844]: accepted connection from pid 910, user nix-grpc-daemon (trusted) worker1 # [ 54.376888] nix-daemon[844]: reaped child process 913, status = succeeded lb # [ 54.351760] niks3-server[1035]: 2026/09/22 10:57:44 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 54.400870] nix-grpc-daemon[857]: time=2026-09-22T10:57:44.871Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 54.402411] nix-grpc-daemon[857]: time=2026-09-22T10:57:44.871Z level=INFO msg="Uploading 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv (624B)" lb # [ 54.382966] niks3-server[1035]: 2026/09/22 10:57:44 INFO Registered completed upload object_key=nar/1xk4kbz8l387hfrqq2xi6as79has336h62f2ra95472yqjclksjg.nar.zst worker1 # [ 54.433787] nix-grpc-daemon[857]: time=2026-09-22T10:57:44.904Z level=INFO msg="Uploading 1 narinfos" lb # [ 54.397414] niks3-server[1035]: 2026/09/22 10:57:44 INFO Received sign narinfos request method=POST path=/api/pending_closures/3/sign lb # [ 54.408985] niks3-server[1035]: 2026/09/22 10:57:44 INFO Signed narinfos id=3 count=1 lb # [ 54.416389] niks3-server[1035]: 2026/09/22 10:57:44 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.ls lb # [ 54.427519] niks3-server[1035]: 2026/09/22 10:57:44 INFO Received complete upload request method=POST path=/api/pending_closures/3/complete worker1 # [ 54.468360] nix-grpc-daemon[857]: time=2026-09-22T10:57:44.939Z level=INFO msg="Upload complete. (492ms)" lb # [ 54.432354] niks3-server[1035]: 2026/09/22 10:57:44 INFO Completed upload id=3 lb # [ 54.435751] niks3-server[1035]: 2026/09/22 10:57:44 INFO Registered completed upload object_key=79z38hfi82zcmb9yq1gcmfviimgf6z2l.narinfo worker1 # [ 54.476162] nix-grpc-daemon[778]: ts=2026-09-22T10:57:44Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 54.484839] nix-grpc-daemon[778]: ts=2026-09-22T10:57:44Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 54.496815] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 54.502624] nix-daemon[844]: reaped child process 903, status = succeeded client # building '/nix/store/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv' on lb:50051... worker1 # [ 54.513938] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) client # farm-b> node-a: building 79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-t1 worker1 # [ 55.936397] nix-daemon[844]: reaped child process 917, status = succeeded worker1 # [ 56.093643] nix-daemon[844]: accepted connection from pid 933, user nix-grpc-daemon (trusted) worker1 # [ 56.110306] nix-daemon[844]: reaped child process 936, status = succeeded lb # [ 56.090759] niks3-server[1035]: 2026/09/22 10:57:46 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 56.138427] nix-grpc-daemon[857]: time=2026-09-22T10:57:46.608Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 56.139869] nix-grpc-daemon[857]: time=2026-09-22T10:57:46.608Z level=INFO msg="Uploading 8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b (176B)" lb # [ 56.125093] niks3-server[1035]: 2026/09/22 10:57:46 INFO Registered completed upload object_key=log/79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv lb # [ 56.139049] niks3-server[1035]: 2026/09/22 10:57:46 INFO Registered completed upload object_key=nar/1mx8h9irf9ykfzxs6q3lwxbzxwv5d2s9mh2x0sky7448mlc1qxrx.nar.zst lb # [ 56.151632] niks3-server[1035]: 2026/09/22 10:57:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/4/sign worker1 # [ 56.194146] nix-grpc-daemon[857]: time=2026-09-22T10:57:46.664Z level=INFO msg="Uploading 1 narinfos" lb # [ 56.162376] niks3-server[1035]: 2026/09/22 10:57:46 INFO Signed narinfos id=4 count=1 lb # [ 56.168062] niks3-server[1035]: 2026/09/22 10:57:46 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.ls lb # [ 56.177519] niks3-server[1035]: 2026/09/22 10:57:46 INFO Received complete upload request method=POST path=/api/pending_closures/4/complete lb # [ 56.182497] niks3-server[1035]: 2026/09/22 10:57:46 INFO Completed upload id=4 lb # [ 56.185505] niks3-server[1035]: 2026/09/22 10:57:46 INFO Registered completed upload object_key=8n9g36vw9957r150a7wg1k6yj9fbqm8r.narinfo worker1 # [ 56.218562] nix-grpc-daemon[857]: time=2026-09-22T10:57:46.689Z level=INFO msg="Upload complete. (292ms)" worker1 # [ 56.229271] nix-grpc-daemon[778]: ts=2026-09-22T10:57:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=1 drv=79z38hfi82zcmb9yq1gcmfviimgf6z2l-farm-b.drv assign_id=2 outputs=1 worker2 # [ 56.221414] nix-grpc-daemon[780]: ts=2026-09-22T10:57:46Z level=debug event=want client=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=0 worker1 # [ 56.247897] nix-daemon[844]: reaped child process 915, status = succeeded worker2 # [ 56.227496] nix-grpc-daemon[780]: ts=2026-09-22T10:57:46Z level=debug event=assigned drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 56.285424] nix-grpc-daemon[778]: ts=2026-09-22T10:57:46Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 56.302370] nix-grpc-daemon[778]: ts=2026-09-22T10:57:46Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 56.317908] nix-grpc-daemon[857]: time=2026-09-22T10:57:46.789Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... worker1 # [ 56.329688] nix-grpc-daemon[778]: ts=2026-09-22T10:57:46Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 client # copying path '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' to 'lb:50051'... worker1 # [ 56.353608] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 56.563742] nix-daemon[844]: accepted connection from pid 940, user nix-grpc-daemon (trusted) worker1 # [ 56.581800] nix-daemon[844]: reaped child process 943, status = succeeded worker1 # [ 56.755888] nix-daemon[844]: accepted connection from pid 945, user nix-grpc-daemon (trusted) worker1 # [ 56.773733] nix-daemon[844]: reaped child process 948, status = succeeded lb # [ 56.750014] niks3-server[1035]: 2026/09/22 10:57:47 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 56.797362] nix-grpc-daemon[857]: time=2026-09-22T10:57:47.268Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 56.798895] nix-grpc-daemon[857]: time=2026-09-22T10:57:47.268Z level=INFO msg="Uploading czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv (752B)" lb # [ 56.781203] niks3-server[1035]: 2026/09/22 10:57:47 INFO Registered completed upload object_key=nar/02rh0zwkcxgd7rc74r5h676rr0m8dbayk3glb776b2mdlirxh9lv.nar.zst lb # [ 56.796214] niks3-server[1035]: 2026/09/22 10:57:47 INFO Received sign narinfos request method=POST path=/api/pending_closures/5/sign worker1 # [ 56.840574] nix-grpc-daemon[857]: time=2026-09-22T10:57:47.311Z level=INFO msg="Uploading 1 narinfos" lb # [ 56.804090] niks3-server[1035]: 2026/09/22 10:57:47 INFO Signed narinfos id=5 count=1 lb # [ 56.813547] niks3-server[1035]: 2026/09/22 10:57:47 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.ls lb # [ 56.824395] niks3-server[1035]: 2026/09/22 10:57:47 INFO Received complete upload request method=POST path=/api/pending_closures/5/complete lb # [ 56.831418] niks3-server[1035]: 2026/09/22 10:57:47 INFO Completed upload id=5 worker1 # [ 56.868160] nix-grpc-daemon[857]: time=2026-09-22T10:57:47.339Z level=INFO msg="Upload complete. (497ms)" lb # [ 56.834605] niks3-server[1035]: 2026/09/22 10:57:47 INFO Registered completed upload object_key=czd6ljynjaw3bl3w8iz1nx68whkdis8f.narinfo worker1 # [ 56.876351] nix-grpc-daemon[778]: ts=2026-09-22T10:57:47Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 56.891209] nix-grpc-daemon[778]: ts=2026-09-22T10:57:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 56.898701] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 56.903703] nix-daemon[844]: reaped child process 938, status = succeeded client # building '/nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv' on lb:50051... client # farm-top> node-a: building czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker1 # [ 56.918641] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-t1 worker1 # [ 58.165932] nix-daemon[844]: reaped child process 952, status = succeeded worker1 # [ 58.345575] nix-daemon[844]: accepted connection from pid 968, user nix-grpc-daemon (trusted) worker1 # [ 58.363257] nix-daemon[844]: reaped child process 971, status = succeeded lb # [ 58.338629] niks3-server[1035]: 2026/09/22 10:57:48 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 58.387276] nix-grpc-daemon[857]: time=2026-09-22T10:57:48.856Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 58.389047] nix-grpc-daemon[857]: time=2026-09-22T10:57:48.856Z level=INFO msg="Uploading nblg066ygaz2avzlmix3rnngr62hld8f-farm-top (232B)" lb # [ 58.375704] niks3-server[1035]: 2026/09/22 10:57:48 INFO Registered completed upload object_key=log/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv lb # [ 58.394196] niks3-server[1035]: 2026/09/22 10:57:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/6/sign worker1 # [ 58.435841] nix-grpc-daemon[857]: time=2026-09-22T10:57:48.906Z level=INFO msg="Uploading 1 narinfos" lb # [ 58.401154] niks3-server[1035]: 2026/09/22 10:57:48 INFO Signed narinfos id=6 count=1 lb # [ 58.408351] niks3-server[1035]: 2026/09/22 10:57:48 INFO Registered completed upload object_key=nar/1wplyx8g7zja504cjvqmxis1q6laml9bidgiq4ya0i9p2pvqb9qp.nar.zst lb # [ 58.416713] niks3-server[1035]: 2026/09/22 10:57:48 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.ls lb # [ 58.426876] niks3-server[1035]: 2026/09/22 10:57:48 INFO Received complete upload request method=POST path=/api/pending_closures/6/complete worker1 # [ 58.463332] nix-grpc-daemon[857]: time=2026-09-22T10:57:48.934Z level=INFO msg="Upload complete. (333ms)" lb # [ 58.431457] niks3-server[1035]: 2026/09/22 10:57:48 INFO Completed upload id=6 lb # [ 58.435103] niks3-server[1035]: 2026/09/22 10:57:48 INFO Registered completed upload object_key=nblg066ygaz2avzlmix3rnngr62hld8f.narinfo worker1 # [ 58.477225] nix-grpc-daemon[778]: ts=2026-09-22T10:57:48Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=1 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv assign_id=3 outputs=1 worker1 # [ 58.493869] nix-daemon[844]: reaped child process 950, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2, in 7.90 seconds) client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath, in 0.23 seconds) client: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client: (finished: must fail: test -e /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.04 seconds) client: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top client # copying 3 paths... client # copying path '/nix/store/dg404zlvxfrspnxl6cc1qkc6i83m5a3a-farm-a' from 'http://lb:5751'... client # copying path '/nix/store/8n9g36vw9957r150a7wg1k6yj9fbqm8r-farm-b' from 'http://lb:5751'... client # copying path '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix copy --from http://lb:5751 --no-check-sigs /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top && grep farm-top-t1 /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.32 seconds) (finished: subtest: DAG build is scheduled across workers and lands in the cache, in 8.60 seconds) subtest: repeat is answered Cached by the scheduler without building worker1: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker1 # finding garbage collector roots... worker1 # deleting specified paths... worker1 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker1 # deleting unused links... worker1 # note: hard linking is currently saving 0.0 KiB worker1: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.28 seconds) worker2: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top worker2 # finding garbage collector roots... worker2 # deleting specified paths... worker2 # deleting '/nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top' worker2 # deleting unused links... worker2 # note: hard linking is currently saving 0.0 KiB worker2: (finished: must succeed: nix-store --delete /nix/store/nblg066ygaz2avzlmix3rnngr62hld8f-farm-top, in 0.30 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2 client # don't know how to build these paths: client # /nix/store/czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv worker2 # [ 60.066144] nix-grpc-daemon[780]: ts=2026-09-22T10:57:50Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:40132 worker1 # [ 60.088968] nix-grpc-daemon[778]: ts=2026-09-22T10:57:50Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 60.094010] nix-grpc-daemon[857]: time=2026-09-22T10:57:50.564Z level=INFO msg="All 2 paths already cached" worker2 # [ 60.094960] nix-grpc-daemon[780]: ts=2026-09-22T10:57:50Z level=debug event=want client=2 drv=czd6ljynjaw3bl3w8iz1nx68whkdis8f-farm-top.drv cached=1 client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 >&2, in 0.33 seconds) client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 outPath, in 0.21 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) (finished: subtest: repeat is answered Cached by the scheduler without building, in 1.45 seconds) subtest: inputs already in the cache are not sent to the scheduler lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.06 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b >&2 client # don't know how to build these paths: client # /nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker2 # [ 61.138460] nix-grpc-daemon[780]: ts=2026-09-22T10:57:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:40132 worker1 # [ 61.160494] nix-grpc-daemon[778]: ts=2026-09-22T10:57:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 61.174351] nix-grpc-daemon[857]: time=2026-09-22T10:57:51.642Z level=INFO msg="All 2 paths already cached" worker2 # [ 61.167693] nix-grpc-daemon[780]: ts=2026-09-22T10:57:51Z level=debug event=want client=3 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv cached=0 worker2 # [ 61.173925] nix-grpc-daemon[780]: ts=2026-09-22T10:57:51Z level=debug event=assigned drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 61.232390] nix-grpc-daemon[778]: ts=2026-09-22T10:57:51Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 61.249157] nix-grpc-daemon[778]: ts=2026-09-22T10:57:51Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 61.264073] nix-grpc-daemon[857]: time=2026-09-22T10:57:51.735Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' to 'lb:50051'... worker1 # [ 61.275408] nix-grpc-daemon[778]: ts=2026-09-22T10:57:51Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 61.290420] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 61.493119] nix-daemon[844]: accepted connection from pid 988, user nix-grpc-daemon (trusted) worker1 # [ 61.514095] nix-daemon[844]: reaped child process 991, status = succeeded worker1 # [ 61.682968] nix-daemon[844]: accepted connection from pid 993, user nix-grpc-daemon (trusted) worker1 # [ 61.702702] nix-daemon[844]: reaped child process 996, status = succeeded lb # [ 61.679713] niks3-server[1035]: 2026/09/22 10:57:52 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 61.726632] nix-grpc-daemon[857]: time=2026-09-22T10:57:52.197Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 61.728195] nix-grpc-daemon[857]: time=2026-09-22T10:57:52.197Z level=INFO msg="Uploading blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv (752B)" lb # [ 61.710680] niks3-server[1035]: 2026/09/22 10:57:52 INFO Registered completed upload object_key=nar/1vy3ds565pni9mnkldcs6alimlb6bg39cimm60s4swkk5yjhwlqf.nar.zst lb # [ 61.726844] niks3-server[1035]: 2026/09/22 10:57:52 INFO Received sign narinfos request method=POST path=/api/pending_closures/7/sign worker1 # [ 61.769276] nix-grpc-daemon[857]: time=2026-09-22T10:57:52.240Z level=INFO msg="Uploading 1 narinfos" lb # [ 61.739867] niks3-server[1035]: 2026/09/22 10:57:52 INFO Signed narinfos id=7 count=1 lb # [ 61.746620] niks3-server[1035]: 2026/09/22 10:57:52 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.ls worker1 # [ 61.793819] nix-grpc-daemon[857]: time=2026-09-22T10:57:52.265Z level=INFO msg="Upload complete. (485ms)" lb # [ 61.757795] niks3-server[1035]: 2026/09/22 10:57:52 INFO Received complete upload request method=POST path=/api/pending_closures/7/complete lb # [ 61.761241] niks3-server[1035]: 2026/09/22 10:57:52 INFO Completed upload id=7 lb # [ 61.764428] niks3-server[1035]: 2026/09/22 10:57:52 INFO Registered completed upload object_key=blgxcsidls7na1m5da3m0bja1prb87vl.narinfo worker1 # [ 61.803651] nix-grpc-daemon[778]: ts=2026-09-22T10:57:52Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 61.817960] nix-grpc-daemon[778]: ts=2026-09-22T10:57:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 61.824305] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 61.829899] nix-daemon[844]: reaped child process 986, status = succeeded client # building '/nix/store/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv' on lb:50051... worker1 # [ 61.842369] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) client # farm-top> node-a: building blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-t1b worker1 # [ 63.490127] nix-daemon[844]: reaped child process 1000, status = succeeded worker1 # [ 63.661066] nix-daemon[844]: accepted connection from pid 1016, user nix-grpc-daemon (trusted) worker1 # [ 63.679276] nix-daemon[844]: reaped child process 1019, status = succeeded lb # [ 63.657092] niks3-server[1035]: 2026/09/22 10:57:54 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 63.708427] nix-grpc-daemon[857]: time=2026-09-22T10:57:54.177Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 63.710177] nix-grpc-daemon[857]: time=2026-09-22T10:57:54.177Z level=INFO msg="Uploading 0xnysbbdbmapgf23wb2dbsfhgh18yxvp-farm-top (232B)" lb # [ 63.693952] niks3-server[1035]: 2026/09/22 10:57:54 INFO Registered completed upload object_key=log/blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv lb # [ 63.711102] niks3-server[1035]: 2026/09/22 10:57:54 INFO Registered completed upload object_key=nar/1f6pjd04v1xmbxqqjhczkkfk4ns9qg6ssnspdalp41q9hkxf9yfr.nar.zst lb # [ 63.721950] niks3-server[1035]: 2026/09/22 10:57:54 INFO Received sign narinfos request method=POST path=/api/pending_closures/8/sign worker1 # [ 63.762660] nix-grpc-daemon[857]: time=2026-09-22T10:57:54.233Z level=INFO msg="Uploading 1 narinfos" lb # [ 63.729700] niks3-server[1035]: 2026/09/22 10:57:54 INFO Signed narinfos id=8 count=1 lb # [ 63.731411] niks3-server[1035]: 2026/09/22 10:57:54 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.ls lb # [ 63.739016] niks3-server[1035]: 2026/09/22 10:57:54 INFO Received complete upload request method=POST path=/api/pending_closures/8/complete lb # [ 63.744341] niks3-server[1035]: 2026/09/22 10:57:54 INFO Completed upload id=8 worker1 # [ 63.781123] nix-grpc-daemon[857]: time=2026-09-22T10:57:54.252Z level=INFO msg="Upload complete. (324ms)" lb # [ 63.747275] niks3-server[1035]: 2026/09/22 10:57:54 INFO Registered completed upload object_key=0xnysbbdbmapgf23wb2dbsfhgh18yxvp.narinfo worker1 # [ 63.793085] nix-grpc-daemon[778]: ts=2026-09-22T10:57:54Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=1 drv=blgxcsidls7na1m5da3m0bja1prb87vl-farm-top.drv assign_id=4 outputs=1 worker1 # [ 63.808360] nix-daemon[844]: reaped child process 998, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b >&2, in 2.95 seconds) client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b outPath client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag t1 --argstr top t1b outPath, in 0.23 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true lb # [ 64.193636] envoy[933]: conn peer=[2001:db8:1::1]:57940 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9536 tx=4410 ms=7627 lb # [ 64.196692] envoy[933]: conn peer=[2001:db8:1::1]:59212 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=993 tx=480 ms=51 lb # [ 64.199643] envoy[933]: conn peer=[2001:db8:1::1]:60148 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4215 tx=1935 ms=2652 worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep -c 'event=want ' || true, in 0.07 seconds) (finished: subtest: inputs already in the cache are not sent to the scheduler, in 3.58 seconds) subtest: outputs read back through any worker carry the cache signature client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag sig >&2 worker2 # [ 64.521880] nix-grpc-daemon[780]: ts=2026-09-22T10:57:54Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:40132 client # don't know how to build these paths: client # /nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker2 # [ 64.531707] nix-grpc-daemon[780]: ts=2026-09-22T10:57:54Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 64.558267] nix-grpc-daemon[780]: ts=2026-09-22T10:57:55Z level=debug event=want client=4 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv cached=0 worker2 # [ 64.566230] nix-grpc-daemon[780]: ts=2026-09-22T10:57:55Z level=debug event=assigned drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 64.574001] nix-grpc-daemon[780]: ts=2026-09-22T10:57:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 64.594762] nix-grpc-daemon[780]: ts=2026-09-22T10:57:55Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40132 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' to 'lb:50051'... worker2 # [ 64.611115] nix-grpc-daemon[780]: ts=2026-09-22T10:57:55Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 64.627812] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) worker2 # [ 64.655327] nix-grpc-daemon[780]: ts=2026-09-22T10:57:55Z level=info event=niks3_push_started pid=917 worker2 # [ 64.928347] nix-grpc-daemon[917]: time=2026-09-22T10:57:55.385Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca="" worker2 # [ 65.151654] nix-daemon[866]: accepted connection from pid 930, user nix-grpc-daemon (trusted) worker2 # [ 65.175400] nix-daemon[866]: reaped child process 933, status = succeeded worker2 # [ 65.357339] nix-daemon[866]: accepted connection from pid 935, user nix-grpc-daemon (trusted) worker2 # [ 65.377549] nix-daemon[866]: reaped child process 938, status = succeeded lb # [ 65.374027] niks3-server[1035]: 2026/09/22 10:57:55 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 65.402989] nix-grpc-daemon[917]: time=2026-09-22T10:57:55.862Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 65.406332] nix-grpc-daemon[917]: time=2026-09-22T10:57:55.864Z level=INFO msg="Uploading 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv (512B)" lb # [ 65.411178] niks3-server[1035]: 2026/09/22 10:57:55 INFO Registered completed upload object_key=nar/13mhmmy2qay13imbbdi287vhryz1hwnyg0w6agd9s0ph37wnsqll.nar.zst worker2 # [ 65.443440] nix-grpc-daemon[917]: time=2026-09-22T10:57:55.902Z level=INFO msg="Uploading 1 narinfos" lb # [ 65.423450] niks3-server[1035]: 2026/09/22 10:57:55 INFO Received sign narinfos request method=POST path=/api/pending_closures/9/sign lb # [ 65.432934] niks3-server[1035]: 2026/09/22 10:57:55 INFO Signed narinfos id=9 count=1 lb # [ 65.441169] niks3-server[1035]: 2026/09/22 10:57:55 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.ls lb # [ 65.452517] niks3-server[1035]: 2026/09/22 10:57:55 INFO Received complete upload request method=POST path=/api/pending_closures/9/complete worker2 # [ 65.478279] nix-grpc-daemon[917]: time=2026-09-22T10:57:55.937Z level=INFO msg="Upload complete. (542ms)" lb # [ 65.460430] niks3-server[1035]: 2026/09/22 10:57:55 INFO Completed upload id=9 lb # [ 65.464731] niks3-server[1035]: 2026/09/22 10:57:55 INFO Registered completed upload object_key=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y.narinfo worker2 # [ 65.488304] nix-grpc-daemon[780]: ts=2026-09-22T10:57:55Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=0 paths=1 nar_bytes_in=512 worker2 # [ 65.500204] nix-grpc-daemon[780]: ts=2026-09-22T10:57:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 65.507579] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) worker2 # [ 65.513636] nix-daemon[866]: reaped child process 915, status = succeeded client # building '/nix/store/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv' on lb:50051... client # farm-a> worker2: building 3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv worker2 # [ 65.527789] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-sig worker2 # [ 66.570419] nix-daemon[866]: reaped child process 942, status = succeeded worker2 # [ 66.748619] nix-daemon[866]: accepted connection from pid 958, user nix-grpc-daemon (trusted) worker2 # [ 66.767638] nix-daemon[866]: reaped child process 961, status = succeeded lb # [ 66.760758] niks3-server[1035]: 2026/09/22 10:57:57 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 66.791475] nix-grpc-daemon[917]: time=2026-09-22T10:57:57.249Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 66.795023] nix-grpc-daemon[917]: time=2026-09-22T10:57:57.249Z level=INFO msg="Uploading 2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a (128B)" lb # [ 66.804696] niks3-server[1035]: 2026/09/22 10:57:57 INFO Registered completed upload object_key=log/3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv lb # [ 66.816828] niks3-server[1035]: 2026/09/22 10:57:57 INFO Registered completed upload object_key=nar/05mvm4xykg9y2d7fm3g502ld2ajkd821jwng7v5g8vd73ynsbnah.nar.zst lb # [ 66.833370] niks3-server[1035]: 2026/09/22 10:57:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/10/sign worker2 # [ 66.860197] nix-grpc-daemon[917]: time=2026-09-22T10:57:57.317Z level=INFO msg="Uploading 1 narinfos" lb # [ 66.843289] niks3-server[1035]: 2026/09/22 10:57:57 INFO Signed narinfos id=10 count=1 lb # [ 66.852387] niks3-server[1035]: 2026/09/22 10:57:57 INFO Received complete upload request method=POST path=/api/pending_closures/10/complete lb # [ 66.888748] niks3-server[1035]: 2026/09/22 10:57:57 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.ls lb # [ 66.895801] niks3-server[1035]: 2026/09/22 10:57:57 INFO Completed upload id=10 worker2 # [ 66.914752] nix-grpc-daemon[917]: time=2026-09-22T10:57:57.373Z level=INFO msg="Upload complete. (375ms)" lb # [ 66.899963] niks3-server[1035]: 2026/09/22 10:57:57 INFO Registered completed upload object_key=2ykckkv44ccpw95ad66fm21ggp6krwj9.narinfo worker2 # [ 66.938541] nix-grpc-daemon[780]: ts=2026-09-22T10:57:57Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=1 drv=3m2hhq3bz9jazpcvgg4g03aa5wnlj12y-farm-a.drv assign_id=5 outputs=1 worker2 # [ 66.955234] nix-grpc-daemon[780]: ts=2026-09-22T10:57:57Z level=debug event=want client=4 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv cached=0 worker2 # [ 66.959624] nix-grpc-daemon[780]: ts=2026-09-22T10:57:57Z level=debug event=assigned drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 66.963816] nix-daemon[866]: reaped child process 940, status = succeeded worker2 # [ 66.985451] nix-grpc-daemon[780]: ts=2026-09-22T10:57:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 67.001496] nix-grpc-daemon[780]: ts=2026-09-22T10:57:57Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 67.017348] nix-grpc-daemon[917]: time=2026-09-22T10:57:57.476Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' to 'lb:50051'... worker2 # [ 67.029003] nix-grpc-daemon[780]: ts=2026-09-22T10:57:57Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 67.043982] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) worker2 # [ 67.252159] nix-daemon[866]: accepted connection from pid 967, user nix-grpc-daemon (trusted) worker2 # [ 67.274079] nix-daemon[866]: reaped child process 970, status = succeeded worker2 # [ 67.451690] nix-daemon[866]: accepted connection from pid 972, user nix-grpc-daemon (trusted) worker2 # [ 67.472980] nix-daemon[866]: reaped child process 975, status = succeeded lb # [ 67.464718] niks3-server[1035]: 2026/09/22 10:57:57 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 67.497674] nix-grpc-daemon[917]: time=2026-09-22T10:57:57.956Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 67.499180] nix-grpc-daemon[917]: time=2026-09-22T10:57:57.956Z level=INFO msg="Uploading z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv (624B)" lb # [ 67.497198] niks3-server[1035]: 2026/09/22 10:57:57 INFO Registered completed upload object_key=nar/0cnhkgsbmxv5hj7zw3fa59qfxikw6251mlhz2gz1v6pgn8c8shwp.nar.zst lb # [ 67.511337] niks3-server[1035]: 2026/09/22 10:57:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/11/sign worker2 # [ 67.539573] nix-grpc-daemon[917]: time=2026-09-22T10:57:57.998Z level=INFO msg="Uploading 1 narinfos" lb # [ 67.519677] niks3-server[1035]: 2026/09/22 10:57:57 INFO Signed narinfos id=11 count=1 lb # [ 67.530000] niks3-server[1035]: 2026/09/22 10:57:57 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.ls lb # [ 67.539384] niks3-server[1035]: 2026/09/22 10:57:57 INFO Received complete upload request method=POST path=/api/pending_closures/11/complete lb # [ 67.549705] niks3-server[1035]: 2026/09/22 10:57:58 INFO Registered completed upload object_key=z8kadp39h0myij9ln0hpxb61ldn71h4i.narinfo lb # [ 67.554501] niks3-server[1035]: 2026/09/22 10:57:58 INFO Completed upload id=11 worker2 # [ 67.574469] nix-grpc-daemon[917]: time=2026-09-22T10:57:58.033Z level=INFO msg="Upload complete. (513ms)" worker2 # [ 67.583840] nix-grpc-daemon[780]: ts=2026-09-22T10:57:58Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=0 paths=1 nar_bytes_in=624 worker2 # [ 67.599278] nix-grpc-daemon[780]: ts=2026-09-22T10:57:58Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 67.606471] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) worker2 # [ 67.612354] nix-daemon[866]: reaped child process 965, status = succeeded client # building '/nix/store/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv' on lb:50051... client # farm-b> worker2: building z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv worker2 # [ 67.627209] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-sig worker2 # [ 68.455353] nix-daemon[866]: reaped child process 979, status = succeeded worker2 # [ 68.627443] nix-daemon[866]: accepted connection from pid 995, user nix-grpc-daemon (trusted) worker2 # [ 68.647288] nix-daemon[866]: reaped child process 998, status = succeeded lb # [ 68.640813] niks3-server[1035]: 2026/09/22 10:57:59 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 68.671539] nix-grpc-daemon[917]: time=2026-09-22T10:57:59.129Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 68.673189] nix-grpc-daemon[917]: time=2026-09-22T10:57:59.129Z level=INFO msg="Uploading h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b (176B)" lb # [ 68.680982] niks3-server[1035]: 2026/09/22 10:57:59 INFO Registered completed upload object_key=nar/0f75gpfkjj0g9xi2113yvvpdw0l16spc2f1xik1ifnzfbrvx4pfk.nar.zst worker2 # [ 68.714987] nix-grpc-daemon[917]: time=2026-09-22T10:57:59.173Z level=INFO msg="Uploading 1 narinfos" lb # [ 68.696576] niks3-server[1035]: 2026/09/22 10:57:59 INFO Registered completed upload object_key=log/z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv lb # [ 68.706305] niks3-server[1035]: 2026/09/22 10:57:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/12/sign lb # [ 68.712525] niks3-server[1035]: 2026/09/22 10:57:59 INFO Signed narinfos id=12 count=1 lb # [ 68.719254] niks3-server[1035]: 2026/09/22 10:57:59 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.ls lb # [ 68.722727] niks3-server[1035]: 2026/09/22 10:57:59 INFO Received complete upload request method=POST path=/api/pending_closures/12/complete lb # [ 68.727299] niks3-server[1035]: 2026/09/22 10:57:59 INFO Completed upload id=12 worker2 # [ 68.746544] nix-grpc-daemon[917]: time=2026-09-22T10:57:59.205Z level=INFO msg="Upload complete. (309ms)" lb # [ 68.730167] niks3-server[1035]: 2026/09/22 10:57:59 INFO Registered completed upload object_key=h1r17djqpb9gp4y7v8nzx65ns9vdr50c.narinfo worker2 # [ 68.755964] nix-grpc-daemon[780]: ts=2026-09-22T10:57:59Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=1 drv=z8kadp39h0myij9ln0hpxb61ldn71h4i-farm-b.drv assign_id=6 outputs=1 worker2 # [ 68.777995] nix-grpc-daemon[780]: ts=2026-09-22T10:57:59Z level=debug event=want client=4 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv cached=0 worker2 # [ 68.781846] nix-grpc-daemon[780]: ts=2026-09-22T10:57:59Z level=debug event=assigned drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 68.786150] nix-daemon[866]: reaped child process 977, status = succeeded worker2 # [ 68.816055] nix-grpc-daemon[780]: ts=2026-09-22T10:57:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 68.833195] nix-grpc-daemon[780]: ts=2026-09-22T10:57:59Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 68.848501] nix-grpc-daemon[917]: time=2026-09-22T10:57:59.307Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' to 'lb:50051'... worker2 # [ 68.860472] nix-grpc-daemon[780]: ts=2026-09-22T10:57:59Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 68.878323] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) worker2 # [ 69.078944] nix-daemon[866]: accepted connection from pid 1004, user nix-grpc-daemon (trusted) worker2 # [ 69.101039] nix-daemon[866]: reaped child process 1007, status = succeeded worker2 # [ 69.268093] nix-daemon[866]: accepted connection from pid 1009, user nix-grpc-daemon (trusted) worker2 # [ 69.287109] nix-daemon[866]: reaped child process 1012, status = succeeded lb # [ 69.278142] niks3-server[1035]: 2026/09/22 10:57:59 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 69.305675] nix-grpc-daemon[917]: time=2026-09-22T10:57:59.764Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 69.307152] nix-grpc-daemon[917]: time=2026-09-22T10:57:59.764Z level=INFO msg="Uploading knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv (752B)" lb # [ 69.300350] niks3-server[1035]: 2026/09/22 10:57:59 INFO Registered completed upload object_key=nar/0lpagyrngav67x80d1hq0njjlkmqdkvqawraz8dsp36ag50v6iwf.nar.zst lb # [ 69.306850] niks3-server[1035]: 2026/09/22 10:57:59 INFO Received sign narinfos request method=POST path=/api/pending_closures/13/sign worker2 # [ 69.330603] nix-grpc-daemon[917]: time=2026-09-22T10:57:59.789Z level=INFO msg="Uploading 1 narinfos" lb # [ 69.313251] niks3-server[1035]: 2026/09/22 10:57:59 INFO Signed narinfos id=13 count=1 lb # [ 69.314891] niks3-server[1035]: 2026/09/22 10:57:59 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.ls lb # [ 69.322806] niks3-server[1035]: 2026/09/22 10:57:59 INFO Received complete upload request method=POST path=/api/pending_closures/13/complete lb # [ 69.327716] niks3-server[1035]: 2026/09/22 10:57:59 INFO Registered completed upload object_key=knvx2sz45j0sdqm3fnbdrv7m887k7a6x.narinfo lb # [ 69.330644] niks3-server[1035]: 2026/09/22 10:57:59 INFO Completed upload id=13 worker2 # [ 69.350497] nix-grpc-daemon[917]: time=2026-09-22T10:57:59.809Z level=INFO msg="Upload complete. (454ms)" worker2 # [ 69.356238] nix-grpc-daemon[780]: ts=2026-09-22T10:57:59Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=0 paths=1 nar_bytes_in=752 worker2 # [ 69.367439] nix-grpc-daemon[780]: ts=2026-09-22T10:57:59Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 69.373457] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) worker2 # [ 69.377418] nix-daemon[866]: reaped child process 1002, status = succeeded client # building '/nix/store/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv' on lb:50051... worker2 # [ 69.388900] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) client # farm-top> worker2: building knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-sig worker2 # [ 70.813070] nix-daemon[866]: reaped child process 1016, status = succeeded worker2 # [ 70.982736] nix-daemon[866]: accepted connection from pid 1032, user nix-grpc-daemon (trusted) worker2 # [ 71.004368] nix-daemon[866]: reaped child process 1035, status = succeeded lb # [ 70.997289] niks3-server[1035]: 2026/09/22 10:58:01 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 71.029937] nix-grpc-daemon[917]: time=2026-09-22T10:58:01.487Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 71.031437] nix-grpc-daemon[917]: time=2026-09-22T10:58:01.487Z level=INFO msg="Uploading 4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top (232B)" lb # [ 71.040331] niks3-server[1035]: 2026/09/22 10:58:01 INFO Registered completed upload object_key=log/knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv worker2 # [ 71.078923] nix-grpc-daemon[917]: time=2026-09-22T10:58:01.537Z level=INFO msg="Uploading 1 narinfos" lb # [ 71.051844] niks3-server[1035]: 2026/09/22 10:58:01 INFO Registered completed upload object_key=nar/07ar364rwqqniwi2nkdpqlgv7ws7702n5lqz703ackfmssrh40v8.nar.zst lb # [ 71.063921] niks3-server[1035]: 2026/09/22 10:58:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/14/sign lb # [ 71.071932] niks3-server[1035]: 2026/09/22 10:58:01 INFO Signed narinfos id=14 count=1 worker2 # [ 71.098924] nix-grpc-daemon[917]: time=2026-09-22T10:58:01.558Z level=INFO msg="Upload complete. (301ms)" lb # [ 71.079707] niks3-server[1035]: 2026/09/22 10:58:01 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.ls worker2 # [ 71.105575] nix-grpc-daemon[780]: ts=2026-09-22T10:58:01Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=1 drv=knvx2sz45j0sdqm3fnbdrv7m887k7a6x-farm-top.drv assign_id=7 outputs=1 lb # [ 71.085284] niks3-server[1035]: 2026/09/22 10:58:01 INFO Received complete upload request method=POST path=/api/pending_closures/14/complete lb # [ 71.089785] niks3-server[1035]: 2026/09/22 10:58:01 INFO Completed upload id=14 lb # [ 71.092791] niks3-server[1035]: 2026/09/22 10:58:01 INFO Registered completed upload object_key=4s2wbz2hg55lpb7s5vn3ynd342bs09lw.narinfo worker2 # [ 71.115649] nix-daemon[866]: reaped child process 1014, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag sig >&2, in 6.92 seconds) client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag sig outPath client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag sig outPath, in 0.24 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&ca-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker1 # [ 71.654196] nix-grpc-daemon[778]: ts=2026-09-22T10:58:02Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:52252 client # these 3 paths will be fetched (0.0 KiB download, 0.5 KiB unpacked): client # /nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a client # /nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b client # /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker1 # [ 71.697063] nix-grpc-daemon[778]: ts=2026-09-22T10:58:02Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:52252 worker1 # [ 71.710418] nix-grpc-daemon[778]: copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'http://lb:5751'... worker1 # [ 71.729269] nix-grpc-daemon[778]: copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'http://lb:5751'... worker1 # [ 71.744163] nix-grpc-daemon[778]: copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'http://lb:5751'... client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker1:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&ca-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.36 seconds) client: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&ca-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top worker2 # [ 72.015683] nix-grpc-daemon[780]: ts=2026-09-22T10:58:02Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:43874 worker2 # [ 72.025122] nix-grpc-daemon[780]: ts=2026-09-22T10:58:02Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv6:%5B2001:db8:1::1%5D:43874 client: (finished: must succeed: nix path-info --sigs --store 'grpc://worker2:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&ca-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ca.pem' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.32 seconds) client: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top client # finding garbage collector roots... client # removing stale temporary roots file "/nix/var/nix/temproots/877" client # removing stale temporary roots file "/nix/var/nix/temproots/856" client # removing stale temporary roots file "/nix/var/nix/temproots/844" client # removing stale temporary roots file "/nix/var/nix/temproots/823" client # removing stale temporary roots file "/nix/var/nix/temproots/885" client # removing stale temporary roots file "/nix/var/nix/temproots/900" client # removing stale temporary roots file "/nix/var/nix/temproots/864" client # removing stale temporary roots file "/nix/var/nix/temproots/808" client # deleting specified paths... client # deleting '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' client # deleting unused links... client # note: hard linking is currently saving 0.0 KiB worker1 # [ 72.581550] nix-grpc-daemon[778]: ts=2026-09-22T10:58:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:41338 worker2 # [ 72.579210] nix-grpc-daemon[780]: ts=2026-09-22T10:58:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 72.591357] nix-grpc-daemon[917]: time=2026-09-22T10:58:03.049Z level=INFO msg="All 1 paths already cached" worker1 # [ 72.618420] nix-grpc-daemon[778]: ts=2026-09-22T10:58:03Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 72.628303] nix-grpc-daemon[778]: ts=2026-09-22T10:58:03Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:41338 client # copying 3 paths... worker1 # [ 72.643479] nix-grpc-daemon[778]: ts=2026-09-22T10:58:03Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:41338 client # copying path '/nix/store/2ykckkv44ccpw95ad66fm21ggp6krwj9-farm-a' from 'grpc://lb:50051'... worker1 # [ 72.682041] nix-grpc-daemon[778]: ts=2026-09-22T10:58:03Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 72.690344] nix-grpc-daemon[778]: ts=2026-09-22T10:58:03Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_out=232 worker1 # [ 72.698243] nix-grpc-daemon[778]: ts=2026-09-22T10:58:03Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 72.705113] nix-grpc-daemon[778]: ts=2026-09-22T10:58:03Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_out=176 worker1 # [ 72.714741] nix-grpc-daemon[778]: ts=2026-09-22T10:58:03Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 72.721972] nix-grpc-daemon[778]: ts=2026-09-22T10:58:03Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_out=128 client # copying path '/nix/store/h1r17djqpb9gp4y7v8nzx65ns9vdr50c-farm-b' from 'grpc://lb:50051'... client # copying path '/nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top' from 'grpc://lb:50051'... client: (finished: must succeed: nix-store --delete /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top && nix copy --from 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --option trusted-public-keys 'farm-test-1:RkClDwvfixdOwourBI4UD9hudE3xfU5EBQcMFUVuRV8=' /nix/store/4s2wbz2hg55lpb7s5vn3ynd342bs09lw-farm-top, in 0.71 seconds) (finished: subtest: outputs read back through any worker carry the cache signature, in 8.55 seconds) subtest: two clients wanting the same drv build it once client: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup client # [ 72.831018] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup. client # Running as unit: dedup1.service; invocation ID: eb3f46277fbf44c4aa764ebb416a456e client: (finished: must succeed: systemd-run --unit dedup1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup, in 0.07 seconds) client: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup worker1 # [ 73.136140] nix-grpc-daemon[778]: ts=2026-09-22T10:58:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:41338 client # [ 73.094348] nix[953]: don't know how to build these paths: client # [ 73.095328] nix[953]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker2 # [ 73.137445] nix-grpc-daemon[780]: ts=2026-09-22T10:58:03Z level=debug event=want client=5 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 worker2 # [ 73.143882] nix-grpc-daemon[780]: ts=2026-09-22T10:58:03Z level=debug event=assigned drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 73.201643] nix-grpc-daemon[778]: ts=2026-09-22T10:58:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 73.217312] nix-grpc-daemon[778]: ts=2026-09-22T10:58:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 client # [ 73.180417] nix[953]: copying 1 paths to 'lb:50051'... worker1 # [ 73.230568] nix-grpc-daemon[778]: ts=2026-09-22T10:58:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 client # [ 73.184194] nix[953]: copying path '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' to 'lb:50051'... worker1 # [ 73.249624] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 73.469228] nix-daemon[844]: accepted connection from pid 1026, user nix-grpc-daemon (trusted) worker1 # [ 73.489643] nix-daemon[844]: reaped child process 1029, status = succeeded worker1 # [ 73.673386] nix-daemon[844]: accepted connection from pid 1031, user nix-grpc-daemon (trusted) worker1 # [ 73.692473] nix-daemon[844]: reaped child process 1034, status = succeeded lb # [ 73.669020] niks3-server[1035]: 2026/09/22 10:58:04 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 73.719587] nix-grpc-daemon[857]: time=2026-09-22T10:58:04.190Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 73.721385] nix-grpc-daemon[857]: time=2026-09-22T10:58:04.190Z level=INFO msg="Uploading cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv (432B)" lb # [ 73.702619] niks3-server[1035]: 2026/09/22 10:58:04 INFO Registered completed upload object_key=nar/0l7vpaziqxpw0srrsgimqvaa0dyxa9glicn3ilkwk06l64hn837w.nar.zst lb # [ 73.720116] niks3-server[1035]: 2026/09/22 10:58:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/15/sign worker1 # [ 73.762893] nix-grpc-daemon[857]: time=2026-09-22T10:58:04.233Z level=INFO msg="Uploading 1 narinfos" lb # [ 73.733520] niks3-server[1035]: 2026/09/22 10:58:04 INFO Signed narinfos id=15 count=1 lb # [ 73.740511] niks3-server[1035]: 2026/09/22 10:58:04 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.ls lb # [ 73.750362] niks3-server[1035]: 2026/09/22 10:58:04 INFO Received complete upload request method=POST path=/api/pending_closures/15/complete lb # [ 73.755178] niks3-server[1035]: 2026/09/22 10:58:04 INFO Completed upload id=15 worker1 # [ 73.791130] nix-grpc-daemon[857]: time=2026-09-22T10:58:04.262Z level=INFO msg="Upload complete. (523ms)" lb # [ 73.758147] niks3-server[1035]: 2026/09/22 10:58:04 INFO Registered completed upload object_key=cp23830c44gs2dw4xy80s5wqyjm7vw84.narinfo worker1 # [ 73.801112] nix-grpc-daemon[778]: ts=2026-09-22T10:58:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_in=432 worker1 # [ 73.820009] nix-grpc-daemon[778]: ts=2026-09-22T10:58:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 73.827313] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 73.835705] nix-daemon[844]: reaped child process 1024, status = succeeded client # [ 73.790534] nix[953]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... worker1 # [ 73.844181] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) client # [ 73.937905] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup. client # Running as unit: dedup2.service; invocation ID: 66ec79723e234ad58d1631bf24ca3422 client: (finished: must succeed: sleep 1; systemd-run --unit dedup2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag dedup, in 1.13 seconds) lb # [ 74.196993] envoy[933]: conn peer=[2001:db8:1::1]:60154 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9545 tx=4389 ms=6608 lb # [ 74.199666] envoy[933]: conn peer=[2001:db8:1::1]:44412 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1051 tx=2885 ms=212 lb # [ 74.202298] envoy[933]: conn peer=[2001:db8:1::1]:44438 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=330 ms=81 lb # [ 74.204707] envoy[933]: conn peer=[2001:db8:1::1]:44432 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=382 tx=376 ms=99 lb # [ 74.207064] envoy[933]: conn peer=[2001:db8:1::1]:44418 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=410 ms=120 worker2 # [ 74.307165] nix-grpc-daemon[780]: ts=2026-09-22T10:58:04Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:40132 client # [ 74.282082] nix[969]: don't know how to build these paths: client # [ 74.284384] nix[969]: /nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker2 # [ 74.327289] nix-grpc-daemon[780]: ts=2026-09-22T10:58:04Z level=debug event=want client=6 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv cached=0 worker1 # [ 74.350869] nix-grpc-daemon[778]: ts=2026-09-22T10:58:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 client # [ 74.309370] nix[969]: building '/nix/store/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv' on lb:50051... client: must succeed: sleep 2 client: (finished: must succeed: sleep 2, in 2.07 seconds) client: waiting for success: ! systemctl is-active dedup1 dedup2 worker1 # [ 77.691680] nix-daemon[844]: reaped child process 1038, status = succeeded worker1 # [ 77.876872] nix-daemon[844]: accepted connection from pid 1068, user nix-grpc-daemon (trusted) worker1 # [ 77.900187] nix-daemon[844]: reaped child process 1071, status = succeeded lb # [ 77.875401] niks3-server[1035]: 2026/09/22 10:58:08 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 77.928260] nix-grpc-daemon[857]: time=2026-09-22T10:58:08.395Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 77.929820] nix-grpc-daemon[857]: time=2026-09-22T10:58:08.395Z level=INFO msg="Uploading vamz940my4dma9a2daklr9pw9x9pp9hp-slow-dedup (120B)" lb # [ 77.924467] niks3-server[1035]: 2026/09/22 10:58:08 INFO Registered completed upload object_key=nar/02xhr0h0sj1ji8m0s8995xzbbb8mhbqdpdl1ggx94yqqaykihvz7.nar.zst lb # [ 77.937981] niks3-server[1035]: 2026/09/22 10:58:08 INFO Registered completed upload object_key=log/cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv worker1 # [ 77.989109] nix-grpc-daemon[857]: time=2026-09-22T10:58:08.460Z level=INFO msg="Uploading 1 narinfos" lb # [ 77.947341] niks3-server[1035]: 2026/09/22 10:58:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/16/sign lb # [ 77.957321] niks3-server[1035]: 2026/09/22 10:58:08 INFO Signed narinfos id=16 count=1 lb # [ 77.959479] niks3-server[1035]: 2026/09/22 10:58:08 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.ls lb # [ 77.972158] niks3-server[1035]: 2026/09/22 10:58:08 INFO Received complete upload request method=POST path=/api/pending_closures/16/complete worker1 # [ 78.013485] nix-grpc-daemon[857]: time=2026-09-22T10:58:08.484Z level=INFO msg="Upload complete. (355ms)" lb # [ 77.977730] niks3-server[1035]: 2026/09/22 10:58:08 INFO Completed upload id=16 lb # [ 77.980399] niks3-server[1035]: 2026/09/22 10:58:08 INFO Registered completed upload object_key=vamz940my4dma9a2daklr9pw9x9pp9hp.narinfo worker1 # [ 78.020605] nix-grpc-daemon[778]: ts=2026-09-22T10:58:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=4 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 worker1 # [ 78.032209] nix-grpc-daemon[778]: ts=2026-09-22T10:58:08Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=3 drv=cp23830c44gs2dw4xy80s5wqyjm7vw84-slow-dedup.drv assign_id=8 outputs=1 worker1 # [ 78.036087] nix-daemon[844]: reaped child process 1036, status = succeeded client # [ 78.032883] systemd[1]: dedup1.service: Deactivated successfully. client # [ 78.040259] systemd[1]: dedup1.service: Consumed 174ms CPU time over 5.207s wall clock time, 39.3M memory peak, 7.2K incoming IP traffic, 8.4K outgoing IP traffic. client # [ 78.077819] systemd[1]: dedup2.service: Deactivated successfully. client # [ 78.080618] systemd[1]: dedup2.service: Consumed 204ms CPU time over 4.143s wall clock time, 33.2M memory peak, 6.1K incoming IP traffic, 5.6K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active dedup1 dedup2, in 1.17 seconds) client: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; } client: (finished: must succeed: systemctl show -p Result dedup1 dedup2 | grep -c success | grep -qx 2 || { journalctl -u dedup1 -u dedup2 >&2; false; }, in 0.07 seconds) worker1: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker1: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.09 seconds) worker2: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true worker2: (finished: must succeed: journalctl -u nix-grpc-daemon -o cat | grep 'method=BuildDerivation.*slow-dedup' | grep -o 'assign_id=[0-9]*' || true, in 0.09 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.10 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.11 seconds) (finished: subtest: two clients wanting the same drv build it once, in 6.38 seconds) subtest: --timeout and --max-silent-time reach the builder worker1 # [ 79.558486] nix-grpc-daemon[778]: ts=2026-09-22T10:58:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:41338 worker2 # [ 79.563749] nix-grpc-daemon[780]: ts=2026-09-22T10:58:10Z level=debug event=want client=7 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv cached=0 worker2 # [ 79.569855] nix-grpc-daemon[780]: ts=2026-09-22T10:58:10Z level=debug event=assigned drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 79.610385] nix-grpc-daemon[780]: ts=2026-09-22T10:58:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 79.633619] nix-grpc-daemon[780]: ts=2026-09-22T10:58:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 79.651751] nix-grpc-daemon[780]: ts=2026-09-22T10:58:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 79.673280] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) worker2 # [ 79.942605] nix-daemon[866]: accepted connection from pid 1066, user nix-grpc-daemon (trusted) worker2 # [ 79.964192] nix-daemon[866]: reaped child process 1069, status = succeeded worker2 # [ 80.169714] nix-daemon[866]: accepted connection from pid 1071, user nix-grpc-daemon (trusted) worker2 # [ 80.192911] nix-daemon[866]: reaped child process 1074, status = succeeded lb # [ 80.194557] niks3-server[1035]: 2026/09/22 10:58:10 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 80.224838] nix-grpc-daemon[917]: time=2026-09-22T10:58:10.683Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 80.226374] nix-grpc-daemon[917]: time=2026-09-22T10:58:10.683Z level=INFO msg="Uploading sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv (424B)" lb # [ 80.234808] niks3-server[1035]: 2026/09/22 10:58:10 INFO Registered completed upload object_key=nar/1m12xmj6yzmb9y2j8ly0yvh5fyzy1nqk7ajhqwr4wn3b5h0bsch5.nar.zst worker2 # [ 80.282009] nix-grpc-daemon[917]: time=2026-09-22T10:58:10.740Z level=INFO msg="Uploading 1 narinfos" lb # [ 80.255545] niks3-server[1035]: 2026/09/22 10:58:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/17/sign lb # [ 80.270359] niks3-server[1035]: 2026/09/22 10:58:10 INFO Signed narinfos id=17 count=1 lb # [ 80.275959] niks3-server[1035]: 2026/09/22 10:58:10 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.ls lb # [ 80.286849] niks3-server[1035]: 2026/09/22 10:58:10 INFO Received complete upload request method=POST path=/api/pending_closures/17/complete worker2 # [ 80.312875] nix-grpc-daemon[917]: time=2026-09-22T10:58:10.772Z level=INFO msg="Upload complete. (617ms)" lb # [ 80.296263] niks3-server[1035]: 2026/09/22 10:58:10 INFO Completed upload id=17 lb # [ 80.299183] niks3-server[1035]: 2026/09/22 10:58:10 INFO Registered completed upload object_key=sfkglm39i7dsf4k91pn9ij8xz5mkgirx.narinfo worker2 # [ 80.324473] nix-grpc-daemon[780]: ts=2026-09-22T10:58:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=0 paths=1 nar_bytes_in=424 worker2 # [ 80.340835] nix-grpc-daemon[780]: ts=2026-09-22T10:58:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 80.360769] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) worker2 # [ 80.368882] nix-daemon[866]: reaped child process 1064, status = succeeded worker2 # [ 80.390693] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) lb # [ 84.203542] envoy[933]: conn peer=[2001:db8:1::1]:44452 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3018 tx=1562 ms=4903 lb # [ 84.209407] envoy[933]: conn peer=[2001:db8:1::1]:44462 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1430 tx=1194 ms=3770 worker2 # [ 86.418425] nix-grpc-daemon[780]: ts=2026-09-22T10:58:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=6 drv=sfkglm39i7dsf4k91pn9ij8xz5mkgirx-slow-tmot.drv assign_id=9 outputs=0 worker2 # [ 86.449580] nix-daemon[866]: reaped child process 1076, status = succeeded worker2 # [ 86.451758] nix-daemon[866]: reaped child process 1078, status = succeeded worker2 # [ 86.847879] nix-grpc-daemon[780]: ts=2026-09-22T10:58:17Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 86.868926] nix-grpc-daemon[780]: ts=2026-09-22T10:58:17Z level=debug event=want client=8 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv cached=0 worker2 # [ 86.875881] nix-grpc-daemon[780]: ts=2026-09-22T10:58:17Z level=debug event=assigned drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 86.905158] nix-grpc-daemon[778]: ts=2026-09-22T10:58:17Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 86.931179] nix-grpc-daemon[778]: ts=2026-09-22T10:58:17Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 86.953470] nix-grpc-daemon[778]: ts=2026-09-22T10:58:17Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 86.982696] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 87.248873] nix-daemon[844]: accepted connection from pid 1084, user nix-grpc-daemon (trusted) worker1 # [ 87.273321] nix-daemon[844]: reaped child process 1087, status = succeeded worker1 # [ 87.478574] nix-daemon[844]: accepted connection from pid 1089, user nix-grpc-daemon (trusted) worker1 # [ 87.498779] nix-daemon[844]: reaped child process 1092, status = succeeded lb # [ 87.482466] niks3-server[1035]: 2026/09/22 10:58:17 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 87.533168] nix-grpc-daemon[857]: time=2026-09-22T10:58:18.003Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 87.534694] nix-grpc-daemon[857]: time=2026-09-22T10:58:18.003Z level=INFO msg="Uploading pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv (424B)" lb # [ 87.535742] niks3-server[1035]: 2026/09/22 10:58:17 INFO Registered completed upload object_key=nar/14f5yrnllfav3svmw4v4svw15r4zlf5yv1p972jy14mmq6gdw943.nar.zst lb # [ 87.550973] niks3-server[1035]: 2026/09/22 10:58:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/18/sign worker1 # [ 87.593921] nix-grpc-daemon[857]: time=2026-09-22T10:58:18.064Z level=INFO msg="Uploading 1 narinfos" lb # [ 87.561213] niks3-server[1035]: 2026/09/22 10:58:18 INFO Signed narinfos id=18 count=1 lb # [ 87.563402] niks3-server[1035]: 2026/09/22 10:58:18 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.ls lb # [ 87.574623] niks3-server[1035]: 2026/09/22 10:58:18 INFO Received complete upload request method=POST path=/api/pending_closures/18/complete lb # [ 87.585267] niks3-server[1035]: 2026/09/22 10:58:18 INFO Completed upload id=18 worker1 # [ 87.622235] nix-grpc-daemon[857]: time=2026-09-22T10:58:18.093Z level=INFO msg="Upload complete. (611ms)" lb # [ 87.589700] niks3-server[1035]: 2026/09/22 10:58:18 INFO Registered completed upload object_key=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s.narinfo worker1 # [ 87.637123] nix-grpc-daemon[778]: ts=2026-09-22T10:58:18Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_in=424 worker1 # [ 87.656993] nix-grpc-daemon[778]: ts=2026-09-22T10:58:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 87.666055] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 87.673958] nix-daemon[844]: reaped child process 1082, status = succeeded worker1 # [ 87.691998] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 93.912838] nix-grpc-daemon[778]: ts=2026-09-22T10:58:24Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=6 drv=pqqkii3gblyblsbkdbaa4hlzm7wlfq9s-slow-tmom.drv assign_id=10 outputs=0 worker1 # [ 93.955733] nix-daemon[844]: reaped child process 1094, status = succeeded worker1 # [ 93.958693] nix-daemon[844]: reaped child process 1096, status = succeeded (finished: subtest: --timeout and --max-silent-time reach the builder, in 14.91 seconds) subtest: the first client leaving does not fail the second client: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share client # [ 94.151382] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share. client # Running as unit: share1.service; invocation ID: 04ecdfc3afc2448b8f6f40fe5ea3fd51 client: (finished: must succeed: systemd-run --unit share1 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share, in 0.11 seconds) client: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share lb # [ 94.204607] envoy[933]: conn peer=[2001:db8:1::1]:44468 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3018 tx=995 ms=6947 client # [ 94.431405] nix[1042]: don't know how to build these paths: client # [ 94.433145] nix[1042]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker2 # [ 94.462454] nix-grpc-daemon[780]: ts=2026-09-22T10:58:24Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 94.481512] nix-grpc-daemon[780]: ts=2026-09-22T10:58:24Z level=debug event=want client=9 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 worker2 # [ 94.490156] nix-grpc-daemon[780]: ts=2026-09-22T10:58:24Z level=debug event=assigned drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 94.501769] nix-grpc-daemon[780]: ts=2026-09-22T10:58:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 94.514975] nix-grpc-daemon[780]: ts=2026-09-22T10:58:24Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40132 client # [ 94.496160] nix[1042]: copying 1 paths to 'lb:50051'... client # [ 94.497045] nix[1042]: copying path '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' to 'lb:50051'... worker2 # [ 94.530549] nix-grpc-daemon[780]: ts=2026-09-22T10:58:24Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 94.549648] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) worker2 # [ 94.772826] nix-daemon[866]: accepted connection from pid 1105, user nix-grpc-daemon (trusted) worker2 # [ 94.796447] nix-daemon[866]: reaped child process 1108, status = succeeded lb # [ 94.210195] envoy[933]: conn peer=[2001:db8:1::1]:56592 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2976 tx=953 ms=7078 worker2 # [ 94.975638] nix-daemon[866]: accepted connection from pid 1110, user nix-grpc-daemon (trusted) worker2 # [ 94.995580] nix-daemon[866]: reaped child process 1113, status = succeeded lb # [ 94.993170] niks3-server[1035]: 2026/09/22 10:58:25 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 95.022939] nix-grpc-daemon[917]: time=2026-09-22T10:58:25.481Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 95.024626] nix-grpc-daemon[917]: time=2026-09-22T10:58:25.481Z level=INFO msg="Uploading rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv (432B)" lb # [ 95.026351] niks3-server[1035]: 2026/09/22 10:58:25 INFO Registered completed upload object_key=nar/0chdjpc9an2ckdblh08lnyrmzvw3h8qs76cn5ydwjwvgwkrm731l.nar.zst worker2 # [ 95.061413] nix-grpc-daemon[917]: time=2026-09-22T10:58:25.520Z level=INFO msg="Uploading 1 narinfos" lb # [ 95.041724] niks3-server[1035]: 2026/09/22 10:58:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/19/sign lb # [ 95.053452] niks3-server[1035]: 2026/09/22 10:58:25 INFO Signed narinfos id=19 count=1 lb # [ 95.060643] niks3-server[1035]: 2026/09/22 10:58:25 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.ls lb # [ 95.069745] niks3-server[1035]: 2026/09/22 10:58:25 INFO Received complete upload request method=POST path=/api/pending_closures/19/complete lb # [ 95.076647] niks3-server[1035]: 2026/09/22 10:58:25 INFO Registered completed upload object_key=rxafkggp1xca5h7p0r6gwpq3qqp2kn63.narinfo lb # [ 95.082609] niks3-server[1035]: 2026/09/22 10:58:25 INFO Completed upload id=19 worker2 # [ 95.102481] nix-grpc-daemon[917]: time=2026-09-22T10:58:25.561Z level=INFO msg="Upload complete. (533ms)" worker2 # [ 95.108306] nix-grpc-daemon[780]: ts=2026-09-22T10:58:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=0 paths=1 nar_bytes_in=432 worker2 # [ 95.116651] nix-grpc-daemon[780]: ts=2026-09-22T10:58:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 95.125933] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) worker2 # [ 95.130280] nix-daemon[866]: reaped child process 1103, status = succeeded worker2 # [ 95.139761] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) client # [ 95.106438] nix[1042]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... client # [ 95.271975] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share. client # Running as unit: share2.service; invocation ID: a05c60e8f9b84ff8910216599d78a818 client: (finished: must succeed: sleep 1; systemd-run --unit share2 nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag share, in 1.13 seconds) worker1 # [ 95.610571] nix-grpc-daemon[778]: ts=2026-09-22T10:58:26Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:41338 client # [ 95.569057] nix[1058]: don't know how to build these paths: client # [ 95.569979] nix[1058]: /nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv worker2 # [ 95.615159] nix-grpc-daemon[780]: ts=2026-09-22T10:58:26Z level=debug event=want client=10 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv cached=0 client # [ 95.591531] nix[1058]: building '/nix/store/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv' on lb:50051... worker2 # [ 95.623862] nix-grpc-daemon[780]: ts=2026-09-22T10:58:26Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 client: must succeed: sleep 2; systemctl kill -s INT share1 client # [ 98.722610] systemd[1]: share1.service: Sent signal SIGINT to main process 1042 (nix) on client request. client: (finished: must succeed: sleep 2; systemctl kill -s INT share1, in 2.10 seconds) client: waiting for success: ! systemctl is-active share1 client # [ 98.734908] nix[1042]: shutting downrxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client # [ 98.739503] nix[1042]: error: interrupted by the user client # [ 98.794247] systemd[1]: share1.service: Main process exited, code=exited, status=1/FAILURE client # [ 98.802640] systemd[1]: share1.service: Failed with result 'exit-code'. client # [ 98.804775] systemd[1]: share1.service: Consumed 183ms CPU time over 4.643s wall clock time, 41.2M memory peak, 6.1K incoming IP traffic, 8.2K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active share1, in 0.09 seconds) client: waiting for success: ! systemctl is-active share2 worker2 # [ 99.370815] nix-daemon[866]: reaped child process 1117, status = succeeded worker2 # [ 99.544917] nix-daemon[866]: accepted connection from pid 1153, user nix-grpc-daemon (trusted) worker2 # [ 99.566984] nix-daemon[866]: reaped child process 1156, status = succeeded lb # [ 99.561024] niks3-server[1035]: 2026/09/22 10:58:30 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 99.595638] nix-grpc-daemon[917]: time=2026-09-22T10:58:30.051Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 99.597701] nix-grpc-daemon[917]: time=2026-09-22T10:58:30.051Z level=INFO msg="Uploading fjww0fd4rwk1s35jav3cyal62l24n356-slow-share (120B)" lb # [ 99.602280] niks3-server[1035]: 2026/09/22 10:58:30 INFO Registered completed upload object_key=log/rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv lb # [ 99.618040] niks3-server[1035]: 2026/09/22 10:58:30 INFO Registered completed upload object_key=nar/172qq270pqbv9ipsik19pmld0m7fj8r5nyyxq3slfx9n7zcaa9s0.nar.zst lb # [ 99.631237] niks3-server[1035]: 2026/09/22 10:58:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/20/sign worker2 # [ 99.657497] nix-grpc-daemon[917]: time=2026-09-22T10:58:30.115Z level=INFO msg="Uploading 1 narinfos" lb # [ 99.637779] niks3-server[1035]: 2026/09/22 10:58:30 INFO Signed narinfos id=20 count=1 lb # [ 99.643043] niks3-server[1035]: 2026/09/22 10:58:30 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.ls lb # [ 99.655215] niks3-server[1035]: 2026/09/22 10:58:30 INFO Received complete upload request method=POST path=/api/pending_closures/20/complete worker2 # [ 99.680640] nix-grpc-daemon[917]: time=2026-09-22T10:58:30.139Z level=INFO msg="Upload complete. (327ms)" lb # [ 99.662243] niks3-server[1035]: 2026/09/22 10:58:30 INFO Completed upload id=20 lb # [ 99.665157] niks3-server[1035]: 2026/09/22 10:58:30 INFO Registered completed upload object_key=fjww0fd4rwk1s35jav3cyal62l24n356.narinfo worker2 # [ 99.697344] nix-grpc-daemon[780]: ts=2026-09-22T10:58:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=4 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=1 worker2 # [ 99.712514] nix-grpc-daemon[780]: ts=2026-09-22T10:58:30Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=4 drv=rxafkggp1xca5h7p0r6gwpq3qqp2kn63-slow-share.drv assign_id=11 outputs=1 worker2 # [ 99.721064] nix-daemon[866]: reaped child process 1115, status = succeeded client # [ 99.748564] systemd[1]: share2.service: Deactivated successfully. client # [ 99.752932] systemd[1]: share2.service: Consumed 189ms CPU time over 4.480s wall clock time, 35.2M memory peak, 6.1K incoming IP traffic, 5.7K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active share2, in 1.12 seconds) client: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; } client: (finished: must succeed: systemctl show -p Result share2 | grep -q success || { journalctl -u share2 >&2; false; }, in 0.04 seconds) (finished: subtest: the first client leaving does not fail the second, in 6.20 seconds) subtest: build hook: nix-daemon with builders = grpc://lb client: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag hook 2>/tmp/hook.log client # [ 100.460519] systemd[1]: Started Nix Daemon. client # [ 100.650282] nix-daemon[1105]: accepted connection from pid 1102, user root (trusted) worker2 # [ 100.898247] nix-grpc-daemon[780]: ts=2026-09-22T10:58:31Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:40132 worker1 # [ 100.926865] nix-grpc-daemon[778]: ts=2026-09-22T10:58:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 worker2 # [ 100.933231] nix-grpc-daemon[780]: ts=2026-09-22T10:58:31Z level=debug event=want client=11 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv cached=0 worker2 # [ 100.939863] nix-grpc-daemon[780]: ts=2026-09-22T10:58:31Z level=debug event=assigned drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 100.960507] nix-grpc-daemon[778]: ts=2026-09-22T10:58:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 100.978508] nix-grpc-daemon[778]: ts=2026-09-22T10:58:31Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 100.994810] nix-grpc-daemon[778]: ts=2026-09-22T10:58:31Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 101.013202] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 101.220386] nix-daemon[844]: accepted connection from pid 1135, user nix-grpc-daemon (trusted) worker1 # [ 101.241686] nix-daemon[844]: reaped child process 1138, status = succeeded worker1 # [ 101.414448] nix-daemon[844]: accepted connection from pid 1140, user nix-grpc-daemon (trusted) worker1 # [ 101.433148] nix-daemon[844]: reaped child process 1143, status = succeeded lb # [ 101.407575] niks3-server[1035]: 2026/09/22 10:58:31 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 101.450567] nix-grpc-daemon[857]: time=2026-09-22T10:58:31.921Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 101.452253] nix-grpc-daemon[857]: time=2026-09-22T10:58:31.921Z level=INFO msg="Uploading xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv (512B)" lb # [ 101.429138] niks3-server[1035]: 2026/09/22 10:58:31 INFO Received sign narinfos request method=POST path=/api/pending_closures/21/sign worker1 # [ 101.470561] nix-grpc-daemon[857]: time=2026-09-22T10:58:31.941Z level=INFO msg="Uploading 1 narinfos" lb # [ 101.436259] niks3-server[1035]: 2026/09/22 10:58:31 INFO Signed narinfos id=21 count=1 lb # [ 101.439913] niks3-server[1035]: 2026/09/22 10:58:31 INFO Registered completed upload object_key=nar/1f1i5kab3qhx7pg1k0ma5r1m7ipfazkfkfb944dm7qcjr466pwij.nar.zst lb # [ 101.444733] niks3-server[1035]: 2026/09/22 10:58:31 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.ls lb # [ 101.451262] niks3-server[1035]: 2026/09/22 10:58:31 INFO Received complete upload request method=POST path=/api/pending_closures/21/complete lb # [ 101.455514] niks3-server[1035]: 2026/09/22 10:58:31 INFO Completed upload id=21 worker1 # [ 101.491820] nix-grpc-daemon[857]: time=2026-09-22T10:58:31.963Z level=INFO msg="Upload complete. (460ms)" lb # [ 101.457516] niks3-server[1035]: 2026/09/22 10:58:31 INFO Registered completed upload object_key=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv.narinfo worker1 # [ 101.498135] nix-grpc-daemon[778]: ts=2026-09-22T10:58:31Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 101.504365] nix-grpc-daemon[778]: ts=2026-09-22T10:58:31Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 101.514928] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 101.516639] nix-daemon[844]: reaped child process 1133, status = succeeded worker1 # [ 101.527272] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 102.968209] nix-daemon[844]: reaped child process 1147, status = succeeded worker1 # [ 103.152377] nix-daemon[844]: accepted connection from pid 1163, user nix-grpc-daemon (trusted) worker1 # [ 103.170409] nix-daemon[844]: reaped child process 1166, status = succeeded lb # [ 103.146848] niks3-server[1035]: 2026/09/22 10:58:33 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 103.192420] nix-grpc-daemon[857]: time=2026-09-22T10:58:33.661Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 103.193957] nix-grpc-daemon[857]: time=2026-09-22T10:58:33.661Z level=INFO msg="Uploading yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a (128B)" lb # [ 103.178070] niks3-server[1035]: 2026/09/22 10:58:33 INFO Registered completed upload object_key=log/xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv lb # [ 103.185249] niks3-server[1035]: 2026/09/22 10:58:33 INFO Registered completed upload object_key=nar/0pbbxl4kxy7zl7x4lp43ywljwn82mxzna2wgpb1bz1z765nwp9ki.nar.zst lb # [ 103.191925] niks3-server[1035]: 2026/09/22 10:58:33 INFO Received sign narinfos request method=POST path=/api/pending_closures/22/sign worker1 # [ 103.232564] nix-grpc-daemon[857]: time=2026-09-22T10:58:33.703Z level=INFO msg="Uploading 1 narinfos" lb # [ 103.197970] niks3-server[1035]: 2026/09/22 10:58:33 INFO Signed narinfos id=22 count=1 lb # [ 103.201348] niks3-server[1035]: 2026/09/22 10:58:33 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.ls lb # [ 103.208980] niks3-server[1035]: 2026/09/22 10:58:33 INFO Received complete upload request method=POST path=/api/pending_closures/22/complete lb # [ 103.212988] niks3-server[1035]: 2026/09/22 10:58:33 INFO Completed upload id=22 worker1 # [ 103.249232] nix-grpc-daemon[857]: time=2026-09-22T10:58:33.720Z level=INFO msg="Upload complete. (312ms)" lb # [ 103.215225] niks3-server[1035]: 2026/09/22 10:58:33 INFO Registered completed upload object_key=yfs64sxmsg77fnzczdvc3g3yfdwh9fks.narinfo worker1 # [ 103.260845] nix-grpc-daemon[778]: ts=2026-09-22T10:58:33Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=1 drv=xs7jpxrx1nnx7zbhs6p2v76pjq3964kv-farm-a.drv assign_id=12 outputs=1 worker1 # [ 103.274388] nix-grpc-daemon[778]: ts=2026-09-22T10:58:33Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 103.277985] nix-daemon[844]: reaped child process 1145, status = succeeded worker1 # [ 103.291226] nix-grpc-daemon[778]: ts=2026-09-22T10:58:33Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 103.296364] nix-grpc-daemon[778]: ts=2026-09-22T10:58:33Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_out=128 worker2 # [ 103.579122] nix-grpc-daemon[780]: ts=2026-09-22T10:58:34Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 103.586346] nix-grpc-daemon[780]: ts=2026-09-22T10:58:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 103.614664] nix-grpc-daemon[780]: ts=2026-09-22T10:58:34Z level=debug event=want client=12 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv cached=0 worker1 # [ 103.634306] nix-grpc-daemon[778]: ts=2026-09-22T10:58:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker2 # [ 103.620716] nix-grpc-daemon[780]: ts=2026-09-22T10:58:34Z level=debug event=assigned drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 103.645319] nix-grpc-daemon[778]: ts=2026-09-22T10:58:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 103.656187] nix-grpc-daemon[857]: time=2026-09-22T10:58:34.127Z level=INFO msg="All 1 paths already cached" worker1 # [ 103.664067] nix-grpc-daemon[778]: ts=2026-09-22T10:58:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 103.678283] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 103.897230] nix-daemon[844]: accepted connection from pid 1170, user nix-grpc-daemon (trusted) worker1 # [ 103.916544] nix-daemon[844]: reaped child process 1173, status = succeeded worker1 # [ 104.095588] nix-daemon[844]: accepted connection from pid 1175, user nix-grpc-daemon (trusted) worker1 # [ 104.113566] nix-daemon[844]: reaped child process 1178, status = succeeded lb # [ 104.091402] niks3-server[1035]: 2026/09/22 10:58:34 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 104.139115] nix-grpc-daemon[857]: time=2026-09-22T10:58:34.609Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 104.142080] nix-grpc-daemon[857]: time=2026-09-22T10:58:34.610Z level=INFO msg="Uploading r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv (624B)" lb # [ 104.120668] niks3-server[1035]: 2026/09/22 10:58:34 INFO Registered completed upload object_key=nar/15k66v3mh3v6p5vymv4pv3v3sfl56i2b1ia2rf093c37w6bhc04m.nar.zst lb # [ 104.135757] niks3-server[1035]: 2026/09/22 10:58:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/23/sign worker1 # [ 104.178457] nix-grpc-daemon[857]: time=2026-09-22T10:58:34.649Z level=INFO msg="Uploading 1 narinfos" lb # [ 104.144974] niks3-server[1035]: 2026/09/22 10:58:34 INFO Signed narinfos id=23 count=1 lb # [ 104.157107] niks3-server[1035]: 2026/09/22 10:58:34 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.ls lb # [ 104.167681] niks3-server[1035]: 2026/09/22 10:58:34 INFO Received complete upload request method=POST path=/api/pending_closures/23/complete lb # [ 104.176183] niks3-server[1035]: 2026/09/22 10:58:34 INFO Completed upload id=23 worker1 # [ 104.213581] nix-grpc-daemon[857]: time=2026-09-22T10:58:34.684Z level=INFO msg="Upload complete. (517ms)" lb # [ 104.178941] niks3-server[1035]: 2026/09/22 10:58:34 INFO Registered completed upload object_key=r1jsr4b2g1qwly395z6idx3z2362ih8i.narinfo worker1 # [ 104.221889] nix-grpc-daemon[778]: ts=2026-09-22T10:58:34Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 104.235996] nix-grpc-daemon[778]: ts=2026-09-22T10:58:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 lb # [ 104.203261] envoy[933]: conn peer=[2001:db8:1::1]:42174 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2990 tx=814 ms=4330 worker1 # [ 104.243328] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) lb # [ 104.208420] envoy[933]: conn peer=[2001:db8:1::1]:42180 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1443 tx=1198 ms=4161 worker1 # [ 104.248768] nix-daemon[844]: reaped child process 1168, status = succeeded lb # [ 104.213507] envoy[933]: conn peer=[2001:db8:1::1]:39802 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3414 tx=1942 ms=2441 lb # [ 104.219260] envoy[933]: conn peer=[2001:db8:1::1]:39812 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=410 tx=333 ms=63 worker1 # [ 104.263445] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 105.686255] nix-daemon[844]: reaped child process 1182, status = succeeded worker1 # [ 105.845947] nix-daemon[844]: accepted connection from pid 1198, user nix-grpc-daemon (trusted) worker1 # [ 105.866536] nix-daemon[844]: reaped child process 1201, status = succeeded lb # [ 105.843254] niks3-server[1035]: 2026/09/22 10:58:36 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 105.891287] nix-grpc-daemon[857]: time=2026-09-22T10:58:36.359Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 105.893060] nix-grpc-daemon[857]: time=2026-09-22T10:58:36.359Z level=INFO msg="Uploading la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b (176B)" lb # [ 105.886722] niks3-server[1035]: 2026/09/22 10:58:36 INFO Registered completed upload object_key=log/r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv lb # [ 105.898849] niks3-server[1035]: 2026/09/22 10:58:36 INFO Registered completed upload object_key=nar/069d0xnyw2q345iqzx6agxc5fxifc6q36bkkklw2aa4qwanxdd1g.nar.zst lb # [ 105.907483] niks3-server[1035]: 2026/09/22 10:58:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/24/sign worker1 # [ 105.949368] nix-grpc-daemon[857]: time=2026-09-22T10:58:36.420Z level=INFO msg="Uploading 1 narinfos" lb # [ 105.915421] niks3-server[1035]: 2026/09/22 10:58:36 INFO Signed narinfos id=24 count=1 lb # [ 105.920666] niks3-server[1035]: 2026/09/22 10:58:36 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.ls lb # [ 105.931314] niks3-server[1035]: 2026/09/22 10:58:36 INFO Received complete upload request method=POST path=/api/pending_closures/24/complete lb # [ 105.937156] niks3-server[1035]: 2026/09/22 10:58:36 INFO Completed upload id=24 worker1 # [ 105.972929] nix-grpc-daemon[857]: time=2026-09-22T10:58:36.444Z level=INFO msg="Upload complete. (299ms)" lb # [ 105.939745] niks3-server[1035]: 2026/09/22 10:58:36 INFO Registered completed upload object_key=la6ryzsihag4zpxgxls54fsg34dvlalk.narinfo worker1 # [ 105.985111] nix-grpc-daemon[778]: ts=2026-09-22T10:58:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=1 drv=r1jsr4b2g1qwly395z6idx3z2362ih8i-farm-b.drv assign_id=13 outputs=1 worker1 # [ 106.002653] nix-grpc-daemon[778]: ts=2026-09-22T10:58:36Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 106.008162] nix-daemon[844]: reaped child process 1180, status = succeeded worker2 # [ 106.013362] nix-grpc-daemon[780]: ts=2026-09-22T10:58:36Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 106.029535] nix-grpc-daemon[780]: copying path '/nix/store/yfs64sxmsg77fnzczdvc3g3yfdwh9fks-farm-a' from 'http://lb:5751'... worker2 # [ 106.049145] nix-grpc-daemon[780]: copying path '/nix/store/la6ryzsihag4zpxgxls54fsg34dvlalk-farm-b' from 'http://lb:5751'... worker2 # [ 106.067803] nix-grpc-daemon[780]: ts=2026-09-22T10:58:36Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=0 paths=1 nar_bytes_out=176 worker2 # [ 106.376982] nix-grpc-daemon[780]: ts=2026-09-22T10:58:36Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:40132 worker1 # [ 106.398632] nix-grpc-daemon[778]: ts=2026-09-22T10:58:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 106.409082] nix-grpc-daemon[857]: time=2026-09-22T10:58:36.880Z level=INFO msg="All 2 paths already cached" worker2 # [ 106.408859] nix-grpc-daemon[780]: ts=2026-09-22T10:58:36Z level=debug event=want client=13 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv cached=0 worker2 # [ 106.411892] nix-grpc-daemon[780]: ts=2026-09-22T10:58:36Z level=debug event=assigned drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 106.473520] nix-grpc-daemon[778]: ts=2026-09-22T10:58:36Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 106.489348] nix-grpc-daemon[778]: ts=2026-09-22T10:58:36Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 106.506707] nix-grpc-daemon[857]: time=2026-09-22T10:58:36.977Z level=INFO msg="All 2 paths already cached" worker1 # [ 106.516719] nix-grpc-daemon[778]: ts=2026-09-22T10:58:36Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 106.531446] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 106.732591] nix-daemon[844]: accepted connection from pid 1205, user nix-grpc-daemon (trusted) worker1 # [ 106.753295] nix-daemon[844]: reaped child process 1208, status = succeeded worker1 # [ 106.931877] nix-daemon[844]: accepted connection from pid 1210, user nix-grpc-daemon (trusted) worker1 # [ 106.949924] nix-daemon[844]: reaped child process 1213, status = succeeded lb # [ 106.927002] niks3-server[1035]: 2026/09/22 10:58:37 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 106.970348] nix-grpc-daemon[857]: time=2026-09-22T10:58:37.440Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 106.972057] nix-grpc-daemon[857]: time=2026-09-22T10:58:37.440Z level=INFO msg="Uploading aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv (752B)" lb # [ 106.946365] niks3-server[1035]: 2026/09/22 10:58:37 INFO Registered completed upload object_key=nar/1ahq7sp1sl2scxqzfjd8gqimpqfkjbyi9ivxdhdwx0jzh2gbv74b.nar.zst lb # [ 106.955336] niks3-server[1035]: 2026/09/22 10:58:37 INFO Received sign narinfos request method=POST path=/api/pending_closures/25/sign worker1 # [ 106.995488] nix-grpc-daemon[857]: time=2026-09-22T10:58:37.466Z level=INFO msg="Uploading 1 narinfos" lb # [ 106.962592] niks3-server[1035]: 2026/09/22 10:58:37 INFO Signed narinfos id=25 count=1 lb # [ 106.966350] niks3-server[1035]: 2026/09/22 10:58:37 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.ls lb # [ 106.972384] niks3-server[1035]: 2026/09/22 10:58:37 INFO Received complete upload request method=POST path=/api/pending_closures/25/complete worker1 # [ 107.014904] nix-grpc-daemon[857]: time=2026-09-22T10:58:37.486Z level=INFO msg="Upload complete. (465ms)" lb # [ 106.978994] niks3-server[1035]: 2026/09/22 10:58:37 INFO Completed upload id=25 lb # [ 106.983489] niks3-server[1035]: 2026/09/22 10:58:37 INFO Registered completed upload object_key=aa4abikjj4w30bfx1p8hxmq0916qbz1c.narinfo worker1 # [ 107.024491] nix-grpc-daemon[778]: ts=2026-09-22T10:58:37Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 107.027984] nix-grpc-daemon[778]: ts=2026-09-22T10:58:37Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 107.039281] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 107.040971] nix-daemon[844]: reaped child process 1203, status = succeeded worker1 # [ 107.051571] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 108.288549] nix-daemon[844]: reaped child process 1217, status = succeeded worker1 # [ 108.452808] nix-daemon[844]: accepted connection from pid 1233, user nix-grpc-daemon (trusted) worker1 # [ 108.473145] nix-daemon[844]: reaped child process 1236, status = succeeded lb # [ 108.448993] niks3-server[1035]: 2026/09/22 10:58:38 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 108.498738] nix-grpc-daemon[857]: time=2026-09-22T10:58:38.967Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 108.500385] nix-grpc-daemon[857]: time=2026-09-22T10:58:38.967Z level=INFO msg="Uploading 39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top (232B)" lb # [ 108.489237] niks3-server[1035]: 2026/09/22 10:58:38 INFO Registered completed upload object_key=nar/0vq3rhfhhbqyx5pbvaa9ddj9wbw9arph334y3bp3idfrikwnjc92.nar.zst lb # [ 108.502635] niks3-server[1035]: 2026/09/22 10:58:38 INFO Registered completed upload object_key=log/aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv worker1 # [ 108.548243] nix-grpc-daemon[857]: time=2026-09-22T10:58:39.019Z level=INFO msg="Uploading 1 narinfos" lb # [ 108.513155] niks3-server[1035]: 2026/09/22 10:58:38 INFO Received sign narinfos request method=POST path=/api/pending_closures/26/sign lb # [ 108.521623] niks3-server[1035]: 2026/09/22 10:58:38 INFO Signed narinfos id=26 count=1 lb # [ 108.526460] niks3-server[1035]: 2026/09/22 10:58:38 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.ls lb # [ 108.534488] niks3-server[1035]: 2026/09/22 10:58:38 INFO Received complete upload request method=POST path=/api/pending_closures/26/complete worker1 # [ 108.571035] nix-grpc-daemon[857]: time=2026-09-22T10:58:39.042Z level=INFO msg="Upload complete. (310ms)" lb # [ 108.538074] niks3-server[1035]: 2026/09/22 10:58:38 INFO Completed upload id=26 lb # [ 108.539894] niks3-server[1035]: 2026/09/22 10:58:38 INFO Registered completed upload object_key=39qxwzy2zkij70y9zvbr59pb0df38w5h.narinfo worker1 # [ 108.581035] nix-grpc-daemon[778]: ts=2026-09-22T10:58:39Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=1 drv=aa4abikjj4w30bfx1p8hxmq0916qbz1c-farm-top.drv assign_id=14 outputs=1 worker1 # [ 108.596893] nix-daemon[844]: reaped child process 1215, status = succeeded worker2 # [ 108.613780] nix-grpc-daemon[780]: ts=2026-09-22T10:58:39Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 108.628668] nix-grpc-daemon[780]: copying path '/nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top' from 'http://lb:5751'... worker1 # [ 108.690206] nix-grpc-daemon[778]: ts=2026-09-22T10:58:39Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 108.696363] nix-grpc-daemon[778]: ts=2026-09-22T10:58:39Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_out=232 client # [ 108.731343] nix-daemon[1105]: reaped child process 1107, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build --log-format internal-json --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --print-out-paths --no-link -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag hook 2>/tmp/hook.log, in 8.48 seconds) client: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top client: (finished: must succeed: grep farm-top-hook /nix/store/39qxwzy2zkij70y9zvbr59pb0df38w5h-farm-top, in 0.03 seconds) client: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F 'LOG-farm-top-hook' /tmp/hook.log | grep -qF '"type":101' && grep -F 'farmPhase' /tmp/hook.log | grep -qF '"type":104' || { cat /tmp/hook.log >&2; false; }, in 0.04 seconds) client: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; } client: (finished: must succeed: grep -F '"type":105' /tmp/hook.log | grep -F lb:50051 | grep -qF '"fields":["/nix/store/' || { grep -F '"type":105' /tmp/hook.log >&2; false; }, in 0.03 seconds) (finished: subtest: build hook: nix-daemon with builders = grpc://lb, in 8.59 seconds) subtest: an input only one worker has still reaches the builder client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag w1only client # this derivation will be built: client # /nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv client # building '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag w1only, in 0.44 seconds) client: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure client: (finished: must succeed: nix-store --export /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only > /tmp/shared/inp.closure, in 0.20 seconds) worker1: must succeed: nix-store --import < /tmp/shared/inp.closure worker1: (finished: must succeed: nix-store --import < /tmp/shared/inp.closure, in 0.25 seconds) worker2: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only worker2: (finished: must fail: test -e /nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only, in 0.04 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt a >&2 client # [ 110.032465] nix-daemon[1105]: accepted connection from pid 1188, user root (trusted) client # this derivation will be built: client # /nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'... worker1 # [ 110.301854] nix-grpc-daemon[778]: ts=2026-09-22T10:58:40Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 110.308774] nix-grpc-daemon[778]: ts=2026-09-22T10:58:40Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 110.519415] nix-daemon[844]: accepted connection from pid 1243, user nix-grpc-daemon (trusted) worker1 # [ 110.536407] nix-daemon[844]: reaped child process 1246, status = succeeded lb # [ 110.514840] niks3-server[1035]: 2026/09/22 10:58:40 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 110.561284] nix-grpc-daemon[857]: time=2026-09-22T10:58:41.031Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 110.562734] nix-grpc-daemon[857]: time=2026-09-22T10:58:41.031Z level=INFO msg="Uploading dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only (120B)" lb # [ 110.543260] niks3-server[1035]: 2026/09/22 10:58:41 INFO Registered completed upload object_key=nar/08ghl87f009pwqz85nbqv9hm0ln6gai6ggzilfmw3qnkaalfi6ss.nar.zst worker1 # [ 110.592938] nix-grpc-daemon[857]: time=2026-09-22T10:58:41.064Z level=INFO msg="Uploading 1 narinfos" lb # [ 110.558108] niks3-server[1035]: 2026/09/22 10:58:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/27/sign lb # [ 110.568934] niks3-server[1035]: 2026/09/22 10:58:41 INFO Signed narinfos id=27 count=1 lb # [ 110.574882] niks3-server[1035]: 2026/09/22 10:58:41 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.ls worker1 # [ 110.619022] nix-grpc-daemon[857]: time=2026-09-22T10:58:41.090Z level=INFO msg="Upload complete. (305ms)" lb # [ 110.583925] niks3-server[1035]: 2026/09/22 10:58:41 INFO Received complete upload request method=POST path=/api/pending_closures/27/complete lb # [ 110.589795] niks3-server[1035]: 2026/09/22 10:58:41 INFO Completed upload id=27 lb # [ 110.592941] niks3-server[1035]: 2026/09/22 10:58:41 INFO Registered completed upload object_key=dymj0d1vj8q0ns6p096qk7fii766dap6.narinfo worker2 # [ 110.630431] nix-grpc-daemon[780]: ts=2026-09-22T10:58:41Z level=debug event=want client=14 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv cached=0 worker2 # [ 110.638041] nix-grpc-daemon[780]: ts=2026-09-22T10:58:41Z level=debug event=assigned drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 110.645946] nix-grpc-daemon[780]: ts=2026-09-22T10:58:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 110.657926] nix-grpc-daemon[780]: ts=2026-09-22T10:58:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40132 client # copying 2 paths to 'lb:50051'... client # copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' to 'lb:50051'... worker2 # [ 110.678589] nix-grpc-daemon[780]: ts=2026-09-22T10:58:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40132 client # copying path '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' to 'lb:50051'... worker2 # [ 110.694757] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) worker2 # [ 110.900415] nix-daemon[866]: accepted connection from pid 1174, user nix-grpc-daemon (trusted) worker2 # [ 110.919858] nix-daemon[866]: reaped child process 1177, status = succeeded worker2 # [ 111.098353] nix-daemon[866]: accepted connection from pid 1179, user nix-grpc-daemon (trusted) worker2 # [ 111.115505] nix-daemon[866]: reaped child process 1182, status = succeeded lb # [ 111.111129] niks3-server[1035]: 2026/09/22 10:58:41 INFO Received uploads request method=POST path=/api/pending_closures lb # [ 111.120945] niks3-server[1035]: 2026/09/22 10:58:41 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 111.149790] nix-grpc-daemon[917]: time=2026-09-22T10:58:41.608Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker2 # [ 111.151489] nix-grpc-daemon[917]: time=2026-09-22T10:58:41.610Z level=INFO msg="Uploading 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv (568B)" worker2 # [ 111.154200] nix-grpc-daemon[917]: time=2026-09-22T10:58:41.612Z level=INFO msg="Uploading 4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv (416B)" lb # [ 111.163248] niks3-server[1035]: 2026/09/22 10:58:41 INFO Registered completed upload object_key=nar/0p6zgq1gdngshl62q0hv3lslqwv90s7fmflwc7wgnycf4ka6ik69.nar.zst lb # [ 111.174784] niks3-server[1035]: 2026/09/22 10:58:41 INFO Registered completed upload object_key=nar/0i2vbprwp0wxa61nkss61gzivqmmjh6klkgv0rlfnxk46rcaba1v.nar.zst lb # [ 111.187051] niks3-server[1035]: 2026/09/22 10:58:41 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.ls lb # [ 111.195038] niks3-server[1035]: 2026/09/22 10:58:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/29/sign worker2 # [ 111.215446] nix-grpc-daemon[917]: time=2026-09-22T10:58:41.673Z level=INFO msg="Uploading 3 narinfos" lb # [ 111.200198] niks3-server[1035]: 2026/09/22 10:58:41 INFO Signed narinfos id=29 count=2 lb # [ 111.202535] niks3-server[1035]: 2026/09/22 10:58:41 INFO Received sign narinfos request method=POST path=/api/pending_closures/28/sign lb # [ 111.211044] niks3-server[1035]: 2026/09/22 10:58:41 INFO Signed narinfos id=28 count=1 lb # [ 111.215440] niks3-server[1035]: 2026/09/22 10:58:41 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.ls lb # [ 111.230107] niks3-server[1035]: 2026/09/22 10:58:41 INFO Registered completed upload object_key=71l2x9mybp0i90r6pdnglwjnayj4xpl8.narinfo lb # [ 111.236209] niks3-server[1035]: 2026/09/22 10:58:41 INFO Received complete upload request method=POST path=/api/pending_closures/28/complete lb # [ 111.241769] niks3-server[1035]: 2026/09/22 10:58:41 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo lb # [ 111.245362] niks3-server[1035]: 2026/09/22 10:58:41 INFO Registered completed upload object_key=4llc9br53475v1z1bdqrjyfi2sjvwnvr.narinfo lb # [ 111.248515] niks3-server[1035]: 2026/09/22 10:58:41 INFO Completed upload id=28 lb # [ 111.251189] niks3-server[1035]: 2026/09/22 10:58:41 INFO Received complete upload request method=POST path=/api/pending_closures/29/complete lb # [ 111.254198] niks3-server[1035]: 2026/09/22 10:58:41 INFO Completed upload id=29 worker2 # [ 111.274045] nix-grpc-daemon[917]: time=2026-09-22T10:58:41.733Z level=INFO msg="Upload complete. (560ms)" worker2 # [ 111.285543] nix-grpc-daemon[780]: ts=2026-09-22T10:58:41Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=0 paths=2 nar_bytes_in=984 worker2 # [ 111.300801] nix-grpc-daemon[780]: ts=2026-09-22T10:58:41Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 111.313405] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) worker2 # [ 111.320396] nix-daemon[866]: reaped child process 1172, status = succeeded worker2 # [ 111.339764] nix-grpc-daemon[780]: copying path '/nix/store/dymj0d1vj8q0ns6p096qk7fii766dap6-local-input-w1only' from 'http://lb:5751'... client # building '/nix/store/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv' on lb:50051... client # uses-input-w1onlya> worker2: building 71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv worker2 # [ 111.363084] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) client # uses-input-w1onlya> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 112.388498] nix-daemon[866]: reaped child process 1187, status = succeeded worker2 # [ 112.543672] nix-daemon[866]: accepted connection from pid 1203, user nix-grpc-daemon (trusted) worker2 # [ 112.562114] nix-daemon[866]: reaped child process 1206, status = succeeded lb # [ 112.554731] niks3-server[1035]: 2026/09/22 10:58:43 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 112.585720] nix-grpc-daemon[917]: time=2026-09-22T10:58:43.043Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 112.584671] niks3-server[1035]: 2026/09/22 10:58:43 INFO Received sign narinfos request method=POST path=/api/pending_closures/30/sign worker2 # [ 112.615406] nix-grpc-daemon[917]: time=2026-09-22T10:58:43.074Z level=INFO msg="Uploading 1 narinfos" lb # [ 112.600691] niks3-server[1035]: 2026/09/22 10:58:43 INFO Signed narinfos id=30 count=1 lb # [ 112.603666] niks3-server[1035]: 2026/09/22 10:58:43 INFO Registered completed upload object_key=log/71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv lb # [ 112.614514] niks3-server[1035]: 2026/09/22 10:58:43 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.ls lb # [ 112.621998] niks3-server[1035]: 2026/09/22 10:58:43 INFO Received complete upload request method=POST path=/api/pending_closures/30/complete lb # [ 112.627014] niks3-server[1035]: 2026/09/22 10:58:43 INFO Completed upload id=30 worker2 # [ 112.646979] nix-grpc-daemon[917]: time=2026-09-22T10:58:43.106Z level=INFO msg="Upload complete. (270ms)" lb # [ 112.629934] niks3-server[1035]: 2026/09/22 10:58:43 INFO Registered completed upload object_key=2i7aa0cdpp60vpdm180r56gqfh25m7hc.narinfo worker2 # [ 112.655046] nix-grpc-daemon[780]: ts=2026-09-22T10:58:43Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=1 drv=71l2x9mybp0i90r6pdnglwjnayj4xpl8-uses-input-w1onlya.drv assign_id=15 outputs=1 worker2 # [ 112.670641] nix-daemon[866]: reaped child process 1184, status = succeeded client # copying 1 paths... worker2 # [ 112.705455] nix-grpc-daemon[780]: ts=2026-09-22T10:58:43Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:40132 client # copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'grpc://lb:50051'... worker1 # [ 112.758386] nix-grpc-daemon[778]: ts=2026-09-22T10:58:43Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 112.772138] nix-grpc-daemon[778]: copying path '/nix/store/2i7aa0cdpp60vpdm180r56gqfh25m7hc-uses-input-w1onlya' from 'http://lb:5751'... worker1 # [ 112.791264] nix-grpc-daemon[778]: ts=2026-09-22T10:58:43Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_out=120 client # [ 112.876433] nix-daemon[1105]: reaped child process 1191, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt a >&2, in 3.09 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt b >&2 client # [ 113.095281] nix-daemon[1105]: accepted connection from pid 1210, user root (trusted) client # this derivation will be built: client # /nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker2 # [ 113.346629] nix-grpc-daemon[780]: ts=2026-09-22T10:58:43Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:40132 client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'... worker1 # [ 113.368352] nix-grpc-daemon[778]: ts=2026-09-22T10:58:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 113.381064] nix-grpc-daemon[857]: time=2026-09-22T10:58:43.851Z level=INFO msg="All 1 paths already cached" worker2 # [ 113.382862] nix-grpc-daemon[780]: ts=2026-09-22T10:58:43Z level=debug event=want client=15 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv cached=0 worker2 # [ 113.389377] nix-grpc-daemon[780]: ts=2026-09-22T10:58:43Z level=debug event=assigned drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 113.445038] nix-grpc-daemon[778]: ts=2026-09-22T10:58:43Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 113.460676] nix-grpc-daemon[778]: ts=2026-09-22T10:58:43Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' to 'lb:50051'... worker1 # [ 113.482917] nix-grpc-daemon[778]: ts=2026-09-22T10:58:43Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 113.498642] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 113.522297] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 66 ms (attempt 1/5) worker1 # [ 113.596507] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 138 ms (attempt 2/5) worker1 # [ 113.742216] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 282 ms (attempt 3/5) worker1 # [ 114.032883] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 722 ms (attempt 4/5) lb # [ 114.204870] envoy[933]: conn peer=[2001:db8:1::1]:39828 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3888 tx=2176 ms=2548 lb # [ 114.205261] envoy[933]: conn peer=[2001:db8:1::1]:39844 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=436 tx=376 ms=115 lb # [ 114.205398] envoy[933]: conn peer=[2001:db8:1::1]:39848 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=4356 tx=2427 ms=2319 lb # [ 114.205531] envoy[933]: conn peer=[2001:db8:1::1]:39850 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=412 tx=406 ms=27 lb # [ 114.224866] envoy[933]: conn peer=[2001:db8:1::1]:59866 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=329 ms=96 lb # [ 114.229887] envoy[933]: conn peer=[2001:db8:1::1]:39858 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3892 tx=2086 ms=2553 worker1 # [ 114.754741] nix-grpc-daemon[778]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 114.755151] nix-grpc-daemon[778]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 114.767009] nix-grpc-daemon[778]: copying path '/nix/store/4llc9br53475v1z1bdqrjyfi2sjvwnvr-local-input-w1only.drv' from 'http://lb:5751'... worker1 # [ 114.991174] nix-daemon[844]: accepted connection from pid 1256, user nix-grpc-daemon (trusted) worker1 # [ 115.010615] nix-daemon[844]: reaped child process 1259, status = succeeded worker1 # [ 115.182611] nix-daemon[844]: accepted connection from pid 1261, user nix-grpc-daemon (trusted) worker1 # [ 115.199184] nix-daemon[844]: reaped child process 1264, status = succeeded lb # [ 115.174488] niks3-server[1035]: 2026/09/22 10:58:45 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 115.222024] nix-grpc-daemon[857]: time=2026-09-22T10:58:45.692Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 115.223444] nix-grpc-daemon[857]: time=2026-09-22T10:58:45.692Z level=INFO msg="Uploading 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv (568B)" lb # [ 115.209777] niks3-server[1035]: 2026/09/22 10:58:45 INFO Registered completed upload object_key=nar/14jznd749vfl63s1gv7i29r5z6jcai2byp2nz83dr5gib6z2cyaz.nar.zst lb # [ 115.223704] niks3-server[1035]: 2026/09/22 10:58:45 INFO Received sign narinfos request method=POST path=/api/pending_closures/31/sign worker1 # [ 115.265614] nix-grpc-daemon[857]: time=2026-09-22T10:58:45.736Z level=INFO msg="Uploading 1 narinfos" lb # [ 115.233407] niks3-server[1035]: 2026/09/22 10:58:45 INFO Signed narinfos id=31 count=1 lb # [ 115.235834] niks3-server[1035]: 2026/09/22 10:58:45 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.ls worker1 # [ 115.284076] nix-grpc-daemon[857]: time=2026-09-22T10:58:45.755Z level=INFO msg="Upload complete. (492ms)" lb # [ 115.248767] niks3-server[1035]: 2026/09/22 10:58:45 INFO Received complete upload request method=POST path=/api/pending_closures/31/complete lb # [ 115.253487] niks3-server[1035]: 2026/09/22 10:58:45 INFO Completed upload id=31 lb # [ 115.256492] niks3-server[1035]: 2026/09/22 10:58:45 INFO Registered completed upload object_key=12ialcyr3l4m98rzfgk6apzimy55z3j8.narinfo worker1 # [ 115.293675] nix-grpc-daemon[778]: ts=2026-09-22T10:58:45Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=1 paths=1 nar_bytes_in=568 worker1 # [ 115.302480] nix-grpc-daemon[778]: ts=2026-09-22T10:58:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 115.312358] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) worker1 # [ 115.316491] nix-daemon[844]: reaped child process 1249, status = succeeded client # building '/nix/store/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv' on lb:50051... worker1 # [ 115.327783] nix-daemon[844]: accepted connection from pid 778, user nix-grpc-daemon (trusted) client # uses-input-w1onlyb> node-a: building 12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv client # uses-input-w1onlyb> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 116.163639] nix-daemon[844]: reaped child process 1268, status = succeeded worker1 # [ 116.335518] nix-daemon[844]: accepted connection from pid 1284, user nix-grpc-daemon (trusted) worker1 # [ 116.354949] nix-daemon[844]: reaped child process 1287, status = succeeded lb # [ 116.330220] niks3-server[1035]: 2026/09/22 10:58:46 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 116.377635] nix-grpc-daemon[857]: time=2026-09-22T10:58:46.846Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 116.360336] niks3-server[1035]: 2026/09/22 10:58:46 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.ls worker1 # [ 116.411159] nix-grpc-daemon[857]: time=2026-09-22T10:58:46.882Z level=INFO msg="Uploading 1 narinfos" lb # [ 116.368690] niks3-server[1035]: 2026/09/22 10:58:46 INFO Received sign narinfos request method=POST path=/api/pending_closures/32/sign lb # [ 116.381122] niks3-server[1035]: 2026/09/22 10:58:46 INFO Signed narinfos id=32 count=1 lb # [ 116.387041] niks3-server[1035]: 2026/09/22 10:58:46 INFO Registered completed upload object_key=log/12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv lb # [ 116.398440] niks3-server[1035]: 2026/09/22 10:58:46 INFO Received complete upload request method=POST path=/api/pending_closures/32/complete lb # [ 116.406580] niks3-server[1035]: 2026/09/22 10:58:46 INFO Completed upload id=32 worker1 # [ 116.443296] nix-grpc-daemon[857]: time=2026-09-22T10:58:46.914Z level=INFO msg="Upload complete. (306ms)" lb # [ 116.410249] niks3-server[1035]: 2026/09/22 10:58:46 INFO Registered completed upload object_key=jfy62nvxcd0b9rkhly152q21ddbrcn28.narinfo client # copying 1 paths... worker1 # [ 116.452989] nix-grpc-daemon[778]: ts=2026-09-22T10:58:46Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=1 drv=12ialcyr3l4m98rzfgk6apzimy55z3j8-uses-input-w1onlyb.drv assign_id=16 outputs=1 worker2 # [ 116.447617] nix-grpc-daemon[780]: ts=2026-09-22T10:58:46Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:40132 worker1 # [ 116.470450] nix-daemon[844]: reaped child process 1266, status = succeeded worker2 # [ 116.455553] nix-grpc-daemon[780]: copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'http://lb:5751'... client # copying path '/nix/store/jfy62nvxcd0b9rkhly152q21ddbrcn28-uses-input-w1onlyb' from 'grpc://lb:50051'... worker1 # [ 116.514639] nix-grpc-daemon[778]: ts=2026-09-22T10:58:46Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 116.521909] nix-grpc-daemon[778]: ts=2026-09-22T10:58:46Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_out=120 client # [ 116.558777] nix-daemon[1105]: reaped child process 1213, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt b >&2, in 3.68 seconds) client: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt c >&2 client # [ 116.776135] nix-daemon[1105]: accepted connection from pid 1232, user root (trusted) client # this derivation will be built: client # /nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker2 # [ 117.021846] nix-grpc-daemon[780]: ts=2026-09-22T10:58:47Z level=debug event=rpc_start method=StoreInfo cn=ci-1 peer=ipv4:192.168.1.2:40132 client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux'... worker1 # [ 117.042679] nix-grpc-daemon[778]: ts=2026-09-22T10:58:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 117.052744] nix-grpc-daemon[857]: time=2026-09-22T10:58:47.523Z level=INFO msg="All 1 paths already cached" worker2 # [ 117.054721] nix-grpc-daemon[780]: ts=2026-09-22T10:58:47Z level=debug event=want client=16 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv cached=0 worker2 # [ 117.064530] nix-grpc-daemon[780]: ts=2026-09-22T10:58:47Z level=debug event=assigned drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 117.073405] nix-grpc-daemon[780]: ts=2026-09-22T10:58:47Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 117.080955] nix-grpc-daemon[780]: ts=2026-09-22T10:58:47Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:40132 client # copying 1 paths to 'lb:50051'... worker2 # [ 117.093563] nix-grpc-daemon[917]: time=2026-09-22T10:58:47.550Z level=INFO msg="All 1 paths already cached" client # copying path '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' to 'lb:50051'... worker2 # [ 117.099018] nix-grpc-daemon[780]: ts=2026-09-22T10:58:47Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 117.114120] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) worker2 # [ 117.312969] nix-daemon[866]: accepted connection from pid 1215, user nix-grpc-daemon (trusted) worker2 # [ 117.331374] nix-daemon[866]: reaped child process 1218, status = succeeded worker2 # [ 117.502334] nix-daemon[866]: accepted connection from pid 1220, user nix-grpc-daemon (trusted) worker2 # [ 117.519191] nix-daemon[866]: reaped child process 1223, status = succeeded lb # [ 117.513735] niks3-server[1035]: 2026/09/22 10:58:47 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 117.541553] nix-grpc-daemon[917]: time=2026-09-22T10:58:48.000Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 117.543027] nix-grpc-daemon[917]: time=2026-09-22T10:58:48.000Z level=INFO msg="Uploading n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv (568B)" lb # [ 117.541828] niks3-server[1035]: 2026/09/22 10:58:47 INFO Registered completed upload object_key=nar/1qd3ps964395j2cc3rlqd58m21s7f6na9s0k2z5kclx0c3l4sl40.nar.zst worker2 # [ 117.575349] nix-grpc-daemon[917]: time=2026-09-22T10:58:48.034Z level=INFO msg="Uploading 1 narinfos" lb # [ 117.557844] niks3-server[1035]: 2026/09/22 10:58:48 INFO Received sign narinfos request method=POST path=/api/pending_closures/33/sign lb # [ 117.568120] niks3-server[1035]: 2026/09/22 10:58:48 INFO Signed narinfos id=33 count=1 lb # [ 117.570734] niks3-server[1035]: 2026/09/22 10:58:48 INFO Registered completed upload object_key=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr.ls lb # [ 117.576781] niks3-server[1035]: 2026/09/22 10:58:48 INFO Received complete upload request method=POST path=/api/pending_closures/33/complete lb # [ 117.581829] niks3-server[1035]: 2026/09/22 10:58:48 INFO Completed upload id=33 worker2 # [ 117.600876] nix-grpc-daemon[917]: time=2026-09-22T10:58:48.060Z level=INFO msg="Upload complete. (471ms)" worker2 # [ 117.612205] nix-grpc-daemon[780]: ts=2026-09-22T10:58:48Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=0 paths=1 nar_bytes_in=568 worker2 # [ 117.622101] nix-grpc-daemon[780]: ts=2026-09-22T10:58:48Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 worker2 # [ 117.627695] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) worker2 # [ 117.631988] nix-daemon[866]: reaped child process 1213, status = succeeded client # building '/nix/store/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv' on lb:50051... worker2 # [ 117.644336] nix-daemon[866]: accepted connection from pid 780, user nix-grpc-daemon (trusted) client # uses-input-w1onlyc> worker2: building n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv client # uses-input-w1onlyc> disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 118.891625] nix-daemon[866]: reaped child process 1227, status = succeeded worker2 # [ 119.081809] nix-daemon[866]: accepted connection from pid 1243, user nix-grpc-daemon (trusted) worker2 # [ 119.101096] nix-daemon[866]: reaped child process 1246, status = succeeded lb # [ 119.095752] niks3-server[1035]: 2026/09/22 10:58:49 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 119.128471] nix-grpc-daemon[917]: time=2026-09-22T10:58:49.586Z level=INFO msg="Uploading 0 paths to lb (1 already cached)" lb # [ 119.140470] niks3-server[1035]: 2026/09/22 10:58:49 INFO Received sign narinfos request method=POST path=/api/pending_closures/34/sign worker2 # [ 119.164812] nix-grpc-daemon[917]: time=2026-09-22T10:58:49.623Z level=INFO msg="Uploading 1 narinfos" lb # [ 119.152211] niks3-server[1035]: 2026/09/22 10:58:49 INFO Signed narinfos id=34 count=1 lb # [ 119.162034] niks3-server[1035]: 2026/09/22 10:58:49 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.ls lb # [ 119.171173] niks3-server[1035]: 2026/09/22 10:58:49 INFO Registered completed upload object_key=log/n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv lb # [ 119.176916] niks3-server[1035]: 2026/09/22 10:58:49 INFO Received complete upload request method=POST path=/api/pending_closures/34/complete lb # [ 119.184441] niks3-server[1035]: 2026/09/22 10:58:49 INFO Completed upload id=34 worker2 # [ 119.203808] nix-grpc-daemon[917]: time=2026-09-22T10:58:49.662Z level=INFO msg="Upload complete. (348ms)" lb # [ 119.187534] niks3-server[1035]: 2026/09/22 10:58:49 INFO Registered completed upload object_key=13r5ipj38sk1irdi724wm1mbl9lw720b.narinfo client # copying 1 paths... worker2 # [ 119.216689] nix-grpc-daemon[780]: ts=2026-09-22T10:58:49Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:40132 duration_s=1 drv=n7pcsnwv1a8b1wwcq715iprr5f6c4mcr-uses-input-w1onlyc.drv assign_id=17 outputs=1 worker1 # [ 119.242402] nix-grpc-daemon[778]: ts=2026-09-22T10:58:49Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:41338 worker2 # [ 119.237070] nix-daemon[866]: reaped child process 1225, status = succeeded client # copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'grpc://lb:50051'... worker1 # [ 119.250545] nix-grpc-daemon[778]: copying path '/nix/store/13r5ipj38sk1irdi724wm1mbl9lw720b-uses-input-w1onlyc' from 'http://lb:5751'... worker1 # [ 119.286076] nix-grpc-daemon[778]: ts=2026-09-22T10:58:49Z level=debug event=rpc_start method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 119.297880] nix-grpc-daemon[778]: ts=2026-09-22T10:58:49Z level=info event=rpc method=FetchNars cn=ci-1 peer=ipv4:192.168.1.2:41338 duration_s=0 paths=1 nar_bytes_out=120 client # [ 119.375620] nix-daemon[1105]: reaped child process 1235, status = succeeded client: (finished: must succeed: NIX_REMOTE=daemon nix build -L --max-jobs 0 --builders 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&system=aarch64-linux aarch64-linux - 4' --no-link -f /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix job --argstr tag w1only --argstr salt c >&2, in 2.82 seconds) (finished: subtest: an input only one worker has still reaches the builder, in 10.53 seconds) subtest: upload whose reference the worker lacks is completed from the cache client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag viacache client # this derivation will be built: client # /nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv client # building '/nix/store/x1rrnjihpa1xvpyxxxdrjzp3c8z8nayq-local-input-viacache.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A input --argstr tag viacache, in 0.45 seconds) client: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A referrer --argstr tag viacache client # this derivation will be built: client # /nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv client # building '/nix/store/1m587dk4visgyryf6n0mgp2rlp7pbjd1-refers-to-input-viacache.drv'... client: (finished: must succeed: nix-build --no-out-link /nix/store/d2v6gfscfnwmpkifqgfkvs9vxxfraw1i-dep.nix -A referrer --argstr tag viacache, in 0.43 seconds) client: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure client: (finished: must succeed: nix-store --export /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache > /tmp/shared/ref.closure, in 0.20 seconds) worker1: must succeed: nix-store --import < /tmp/shared/ref.closure worker1: (finished: must succeed: nix-store --import < /tmp/shared/ref.closure, in 0.24 seconds) worker2 # [ 120.823785] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 121.741739] nix-grpc-daemon[780]: ts=2026-09-22T10:58:52Z level=info event=signal_exit lb # [ 122.002608] niks3-server[1035]: 2026/09/22 10:58:52 INFO lead: released remote=192.168.1.4:35020 lb # [ 122.759671] niks3-server[1035]: 2026/09/22 10:58:53 INFO lead: acquired remote=192.168.1.3:43480 worker1 # [ 122.798887] nix-grpc-daemon[778]: ts=2026-09-22T10:58:53Z level=info event=scheduler_take_over worker2 # [ 123.014724] nix-grpc-daemon[780]: ts=2026-09-22T10:58:53Z level=info event=scheduler_restarting worker1 # [ 123.040817] nix-grpc-daemon[778]: ts=2026-09-22T10:58:53Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 123.023627] nix-grpc-daemon[780]: ts=2026-09-22T10:58:53Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:40124 worker2 # [ 123.034813] nix-grpc-daemon[780]: ts=2026-09-22T10:58:53Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 123.040294] nix-grpc-daemon[780]: ts=2026-09-22T10:58:53Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 123.044677] nix-grpc-daemon[780]: ts=2026-09-22T10:58:53Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:40124 worker2 # [ 123.049939] nix-grpc-daemon[780]: ts=2026-09-22T10:58:53Z level=info event=worker_gone addr=192.168.1.3:50051 worker2 # [ 123.079926] nix-daemon[866]: reaped child process 869, status = succeeded worker1 # [ 123.242998] nix-grpc-daemon[778]: ts=2026-09-22T10:58:53Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 123.248297] nix-grpc-daemon[778]: ts=2026-09-22T10:58:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 123.254625] nix-grpc-daemon[780]: ts=2026-09-22T10:58:53Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker1 # [ 123.449191] nix-grpc-daemon[778]: ts=2026-09-22T10:58:53Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 123.454703] nix-grpc-daemon[778]: ts=2026-09-22T10:58:53Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 123.655520] nix-grpc-daemon[778]: ts=2026-09-22T10:58:54Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 123.661095] nix-grpc-daemon[778]: ts=2026-09-22T10:58:54Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 123.863011] nix-grpc-daemon[778]: ts=2026-09-22T10:58:54Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 123.885213] nix-grpc-daemon[778]: ts=2026-09-22T10:58:54Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:50294 worker1 # [ 123.890507] nix-grpc-daemon[778]: ts=2026-09-22T10:58:54Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker2 # [ 123.902924] nix-grpc-daemon[780]: E0000 00:00:1790074734.361712 825 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 18, started true, shutdown true worker2 # [ 123.909860] nix-grpc-daemon[780]: E0000 00:00:1790074734.362049 825 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true lb # [ 124.204917] envoy[933]: conn peer=[2001:db8:1::1]:59886 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=422 tx=328 ms=26 lb # [ 124.209857] envoy[933]: conn peer=[2001:db8:1::1]:59872 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3752 tx=2125 ms=3175 lb # [ 124.209998] envoy[933]: conn peer=[2001:db8:1::1]:59888 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3753 tx=2102 ms=2318 lb # [ 124.210212] envoy[933]: conn peer=[2001:db8:1::1]:59896 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=448 tx=329 ms=73 lb # [ 124.210321] envoy[933]: conn peer=[2001:db8:1::4]:33630 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=73274 tx=3902 ms=98466 worker2 # [ 125.285411] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 125.293063] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 125.295429] systemd[1]: nix-grpc-daemon.service: Consumed 3.578s CPU time over 1min 40.900s wall clock time, 61.1M memory peak, 93.5K incoming IP traffic, 245.2K outgoing IP traffic. worker2 # [ 125.310579] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker2 # [ 125.313621] systemd[1]: Closed Nix worker-protocol over gRPC. worker2 # [ 125.315198] systemd[1]: nix-grpc-daemon.socket: Consumed 784us CPU time over 1min 53.055s wall clock time, 256K memory peak, 120.1K incoming IP traffic, 100.6K outgoing IP traffic. lb: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds) lb: (finished: waiting until exactly ['worker2'] endpoints of aarch64-linux are unhealthy, in 0.10 seconds) client: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2 worker1 # [ 125.707892] nix-grpc-daemon[778]: ts=2026-09-22T10:58:56Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:41338 worker1 # [ 125.716277] nix-grpc-daemon[778]: ts=2026-09-22T10:58:56Z level=debug event=rpc_start method=QueryPathInfos cn=ci-1 peer=ipv4:192.168.1.2:41338 client # /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache client: (finished: must succeed: nix path-info --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache >&2, in 0.31 seconds) worker2: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache worker2: (finished: must fail: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache, in 0.04 seconds) worker2 # [ 125.841775] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 125.939541] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2 # [ 126.272189] systemd[1]: Starting scheduler-token.service... worker2 # [ 126.390191] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 126.397091] systemd[1]: Finished scheduler-token.service. worker2 # [ 126.398904] systemd[1]: scheduler-token.service: Consumed 57ms CPU time over 121ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 126.408968] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 126.594398] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:57Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 126.596956] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:57Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 126.601307] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:57Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 126.608062] nix-grpc-daemon[1270]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 126.609526] nix-grpc-daemon[1270]: E0000 00:00:1790074737.067088 1270 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 126.624166] nix-grpc-daemon[1270]: E0000 00:00:1790074737.073751 1270 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 126.628479] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:57Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 126.632814] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 126.640068] nix-grpc-daemon[1270]: E0000 00:00:1790074737.095528 1309 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 126.667976] nix-grpc-daemon[778]: ts=2026-09-22T10:58:57Z level=info event=signal_exit worker2 # [ 126.664055] nix-grpc-daemon[1270]: E0000 00:00:1790074737.113004 1309 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 126.666398] nix-grpc-daemon[1270]: E0000 00:00:1790074737.113178 1309 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 126.673923] nix-grpc-daemon[1270]: E0000 00:00:1790074737.132341 1309 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 126.686289] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:57Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 126.705653] nix-grpc-daemon[778]: ts=2026-09-22T10:58:57Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:50294 worker1 # [ 126.712246] nix-grpc-daemon[778]: ts=2026-09-22T10:58:57Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb # [ 126.881342] niks3-server[1035]: 2026/09/22 10:58:57 INFO lead: released remote=192.168.1.3:43480 worker1 # [ 127.917263] nix-grpc-daemon[778]: ts=2026-09-22T10:58:58Z level=info event=scheduler_restarting worker2 # [ 127.902802] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:58Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 127.930242] nix-grpc-daemon[778]: ts=2026-09-22T10:58:58Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:50294 worker1 # [ 127.935278] nix-grpc-daemon[778]: ts=2026-09-22T10:58:58Z level=info event=worker_gone addr=192.168.1.4:50051 worker1 # [ 127.942042] nix-grpc-daemon[778]: ts=2026-09-22T10:58:58Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 127.946225] nix-grpc-daemon[778]: ts=2026-09-22T10:58:58Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:50294 worker1 # [ 127.950966] nix-grpc-daemon[778]: ts=2026-09-22T10:58:58Z level=info event=worker_gone addr=192.168.1.3:50051 worker1 # [ 127.975794] nix-daemon[844]: reaped child process 846, status = succeeded worker2 # [ 128.103077] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:58Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 128.108355] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 128.151001] nix-grpc-daemon[778]: ts=2026-09-22T10:58:58Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 128.309594] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:58Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 128.314842] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 128.515803] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:58Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 128.521062] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:58Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 128.722071] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 128.727181] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 128.807357] nix-grpc-daemon[778]: E0000 00:00:1790074739.278296 823 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true worker1 # [ 128.852431] nix-grpc-daemon[778]: E0000 00:00:1790074739.323517 823 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 7, started true, shutdown true worker2 # [ 128.930061] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 128.934317] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 129.131005] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 129.136403] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 129.196694] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 129.204894] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 129.207372] systemd[1]: nix-grpc-daemon.service: Consumed 4.704s CPU time over 1min 44.953s wall clock time, 62.9M memory peak, 109.9K incoming IP traffic, 198.6K outgoing IP traffic. worker1 # [ 129.225646] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker1 # [ 129.227526] systemd[1]: Closed Nix worker-protocol over gRPC. worker1 # [ 129.229167] systemd[1]: nix-grpc-daemon.socket: Consumed 799us CPU time over 1min 57.469s wall clock time, 256K memory peak, 118.5K incoming IP traffic, 103.4K outgoing IP traffic. lb: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds) lb: (finished: waiting until exactly ['worker1'] endpoints of aarch64-linux are unhealthy, in 0.09 seconds) client: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2 worker2 # [ 129.337457] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:59Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 129.342554] nix-grpc-daemon[1270]: ts=2026-09-22T10:58:59Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 129.544003] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 129.549295] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 129.573090] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:00Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 129.588879] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 129.620315] nix-grpc-daemon[1270]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 58 ms (attempt 1/5) worker2 # [ 129.685892] nix-grpc-daemon[1270]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 117 ms (attempt 2/5) worker2 # [ 129.750465] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 129.755672] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 129.811065] nix-grpc-daemon[1270]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 270 ms (attempt 3/5) worker2 # [ 129.958881] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 129.963087] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 130.088898] nix-grpc-daemon[1270]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 258 ms (attempt 4/5) worker2 # [ 130.159836] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 130.165089] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 130.347720] nix-grpc-daemon[1270]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 130.353946] nix-grpc-daemon[1270]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 130.366160] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:00Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 130.373211] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:00Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" client # copying 2 paths... client # copying path '/nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache' to 'grpc://lb:50051'... worker2 # [ 130.385518] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:00Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 client # copying path '/nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache' to 'grpc://lb:50051'... worker2 # [ 130.404962] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 130.435391] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:00Z level=info event=niks3_push_started pid=1330 worker2 # [ 130.573974] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 130.576302] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 130.655499] nix-grpc-daemon[1330]: time=2026-09-22T10:59:01.114Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca="" worker2 # [ 130.777163] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 130.779676] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 130.838054] nix-daemon[866]: accepted connection from pid 1343, user nix-grpc-daemon (trusted) worker2 # [ 130.858189] nix-daemon[866]: reaped child process 1346, status = succeeded lb # [ 130.851731] niks3-server[1035]: 2026/09/22 10:59:01 INFO Received uploads request method=POST path=/api/pending_closures lb # [ 130.861293] niks3-server[1035]: 2026/09/22 10:59:01 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 130.890025] nix-grpc-daemon[1330]: time=2026-09-22T10:59:01.349Z level=INFO msg="Uploading 2 paths to lb (0 already cached)" worker2 # [ 130.894800] nix-grpc-daemon[1330]: time=2026-09-22T10:59:01.350Z level=INFO msg="Uploading syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache (128B)" worker2 # [ 130.897077] nix-grpc-daemon[1330]: time=2026-09-22T10:59:01.351Z level=INFO msg="Uploading 4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache (184B)" lb # [ 130.906750] niks3-server[1035]: 2026/09/22 10:59:01 INFO Registered completed upload object_key=nar/1kmnh5f2l7jjcgaq9b9988bjix6z8zjf6v350xllq5hlzgnb1f75.nar.zst lb # [ 130.922552] niks3-server[1035]: 2026/09/22 10:59:01 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.ls lb # [ 130.933447] niks3-server[1035]: 2026/09/22 10:59:01 INFO Registered completed upload object_key=nar/1rf1rg73pvnaa9kk6sy9r1ydj24b4g0kb0wnx0z5hxl1nvkmlavq.nar.zst lb # [ 130.941507] niks3-server[1035]: 2026/09/22 10:59:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/35/sign lb # [ 130.952109] niks3-server[1035]: 2026/09/22 10:59:01 INFO Signed narinfos id=35 count=1 worker2 # [ 130.972351] nix-grpc-daemon[1330]: time=2026-09-22T10:59:01.430Z level=INFO msg="Uploading 3 narinfos" lb # [ 130.955546] niks3-server[1035]: 2026/09/22 10:59:01 INFO Received sign narinfos request method=POST path=/api/pending_closures/36/sign lb # [ 130.964199] niks3-server[1035]: 2026/09/22 10:59:01 INFO Signed narinfos id=36 count=2 worker2 # [ 130.984851] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:01Z level=info event=scheduler_connected addr=lb:50051 lb # [ 130.966811] niks3-server[1035]: 2026/09/22 10:59:01 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.ls worker2 # [ 130.989163] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb # [ 130.986881] niks3-server[1035]: 2026/09/22 10:59:01 INFO Registered completed upload object_key=4rig83zwv4xj1zajs08ima13w0yh676v.narinfo lb # [ 130.994225] niks3-server[1035]: 2026/09/22 10:59:01 INFO Received complete upload request method=POST path=/api/pending_closures/35/complete lb # [ 130.998793] niks3-server[1035]: 2026/09/22 10:59:01 INFO Completed upload id=35 lb # [ 131.004794] niks3-server[1035]: 2026/09/22 10:59:01 INFO Received complete upload request method=POST path=/api/pending_closures/36/complete lb # [ 131.007539] niks3-server[1035]: 2026/09/22 10:59:01 INFO Registered completed upload object_key=syvivri0wisb7fd33rxcg9kz63kc2bv7.narinfo worker2 # [ 131.033599] nix-grpc-daemon[1330]: time=2026-09-22T10:59:01.492Z level=INFO msg="Upload complete. (373ms)" lb # [ 131.019073] niks3-server[1035]: 2026/09/22 10:59:01 INFO Completed upload id=36 worker2 # [ 131.044609] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:01Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 duration_s=0 paths=2 nar_bytes_in=312 worker2 # [ 131.063239] nix-daemon[866]: reaped child process 1328, status = succeeded client: (finished: must succeed: nix copy --no-check-sigs --to 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache >&2, in 1.78 seconds) worker2: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache worker2: (finished: must succeed: test -e /nix/store/syvivri0wisb7fd33rxcg9kz63kc2bv7-local-input-viacache && test -e /nix/store/4rig83zwv4xj1zajs08ima13w0yh676v-refers-to-input-viacache, in 0.03 seconds) worker2 # [ 131.183835] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 131.189080] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 131.224967] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 131.232748] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 131.389835] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:01Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 131.395015] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 131.430919] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:01Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 131.433444] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:01Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 131.437622] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:01Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 131.446747] nix-grpc-daemon[1311]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 131.455561] nix-grpc-daemon[1311]: E0000 00:00:1790074741.915333 1311 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. lb # [ 131.421238] niks3-server[1035]: 2026/09/22 10:59:01 INFO lead: acquired remote=192.168.1.3:34446 worker1 # [ 131.460163] nix-grpc-daemon[1311]: E0000 00:00:1790074741.917789 1311 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 131.463228] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:01Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 131.466122] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:01Z level=info event=scheduler_take_over worker1 # [ 131.467492] systemd[1]: Started Nix worker-protocol over gRPC. worker1 # [ 131.481467] nix-grpc-daemon[1311]: E0000 00:00:1790074741.939995 1350 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 131.502019] nix-grpc-daemon[1311]: E0000 00:00:1790074741.957855 1350 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 131.504899] nix-grpc-daemon[1311]: E0000 00:00:1790074741.958040 1350 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 131.507344] nix-grpc-daemon[1311]: E0000 00:00:1790074741.972334 1350 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 131.516277] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:01Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 131.518156] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:01Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds) worker2 # [ 131.596242] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:02Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 131.601408] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 131.719062] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:02Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 131.724007] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 131.802652] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:02Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 131.807925] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 131.924799] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:02Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 131.929769] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 132.010929] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:02Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 132.015111] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 132.130509] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:02Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 132.135530] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:02Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 132.211481] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:02Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 132.240232] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:02Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 132.245137] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:02Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:44006 worker1 # [ 132.250461] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:02Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 132.336338] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:02Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 132.341971] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:02Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:44006 worker1 # [ 132.346825] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:02Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.17 seconds) (finished: subtest: upload whose reference the worker lacks is completed from the cache, in 13.20 seconds) subtest: a niks3 restart does not cost the leader its role lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb # [ 132.753409] niks3-server[1035]: 2026/09/22 10:59:03 INFO Shutdown signal received, draining in-flight requests timeout=10s lb # [ 132.758696] systemd[1]: Stopping niks3 server... worker2 # [ 132.779021] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:03Z level=info event=scheduler_lock_lost status=200 worker1 # [ 132.796355] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:03Z level=info event=scheduler_lock_lost status=200 lb # [ 132.763406] niks3-server[1035]: 2026/09/22 10:59:03 INFO lead: released remote=192.168.1.3:34446 lb # [ 134.208607] envoy[933]: conn peer=[2001:db8:1::1]:46820 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=591 tx=522 ms=29 lb # [ 134.213511] envoy[933]: conn peer=[2001:db8:1::3]:35386 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=15540 tx=4302 ms=103451 lb # [ 134.218541] envoy[933]: conn peer=[2001:db8:1::1]:46822 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=1006 tx=304 ms=1487 lb # [ 136.462701] systemd[1]: niks3.service: Deactivated successfully. lb # [ 136.469238] systemd[1]: Stopped niks3 server. lb # [ 136.472912] systemd[1]: niks3.service: Consumed 714ms CPU time over 1min 44.164s wall clock time, 23.8M memory peak, 95.3K incoming IP traffic, 137.4K outgoing IP traffic. lb # [ 136.499089] systemd[1]: Starting niks3 server... lb # [ 136.624501] niks3-server[1182]: 2026/09/22 10:59:07 goose: no migrations to run. current version: 20260920000000 lb # [ 136.633623] niks3-server[1182]: 2026/09/22 10:59:07 OK 1_commit_pending_closure.sql (9.99ms) lb # [ 136.637962] niks3-server[1182]: 2026/09/22 10:59:07 OK 2_object_stats_trigger.sql (3.66ms) lb # [ 136.639839] niks3-server[1182]: 2026/09/22 10:59:07 goose: up to current file version: 2 lb # [ 136.642914] niks3-server[1182]: 2026/09/22 10:59:07 INFO Loaded signing key name=farm-test-1 path=/nix/store/8lzz3wcf17v05vfdp1zcfaa9hipw3f70-key lb # [ 136.658038] niks3-server[1182]: 2026/09/22 10:59:07 INFO Read proxy enabled — serving cache objects from S3 lb # [ 136.659766] niks3-server[1182]: 2026/09/22 10:59:07 INFO Using socket-activated listener address=0.0.0.0:5751 lb # [ 136.666545] niks3-server[1182]: 2026/09/22 10:59:07 INFO systemd watchdog enabled interval=15s lb # [ 136.669286] niks3-server[1182]: 2026/09/22 10:59:07 INFO Starting HTTP server address=0.0.0.0:5751 lb # [ 136.670847] niks3-server[1182]: 2026/09/22 10:59:07 INFO lead: acquired remote=192.168.1.3:34454 lb # [ 136.673823] systemd[1]: Started niks3 server. lb: waiting for unit niks3.service lb: (finished: waiting for unit niks3.service, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.07 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield worker1: (finished: must fail: journalctl -u nix-grpc-daemon | grep -q event=scheduler_yield, in 0.07 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag after-niks3-restart >&2 worker2 # [ 137.322920] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:07Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55550 client # don't know how to build these paths: client # /nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker2 # [ 137.334403] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55550 worker1 # [ 137.395994] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:07Z level=debug event=want client=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv cached=0 worker1 # [ 137.397489] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:07Z level=debug event=assigned drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 137.424117] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 137.439482] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55550 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' to 'lb:50051'... worker2 # [ 137.455474] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 137.471283] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 137.685173] nix-daemon[866]: accepted connection from pid 1355, user nix-grpc-daemon (trusted) worker2 # [ 137.703587] nix-daemon[866]: reaped child process 1358, status = succeeded worker2 # [ 137.878876] nix-daemon[866]: accepted connection from pid 1360, user nix-grpc-daemon (trusted) worker2 # [ 137.896527] nix-daemon[866]: reaped child process 1363, status = succeeded lb # [ 137.889742] niks3-server[1182]: 2026/09/22 10:59:08 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 137.925352] nix-grpc-daemon[1330]: time=2026-09-22T10:59:08.383Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 137.926862] nix-grpc-daemon[1330]: time=2026-09-22T10:59:08.384Z level=INFO msg="Uploading mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv (544B)" lb # [ 137.935733] niks3-server[1182]: 2026/09/22 10:59:08 INFO Registered completed upload object_key=nar/0hn539w6achi7azzyhpihsi61g85x9f535dxv438yq5y50ksrp1d.nar.zst lb # [ 137.952864] niks3-server[1182]: 2026/09/22 10:59:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/37/sign worker2 # [ 137.983721] nix-grpc-daemon[1330]: time=2026-09-22T10:59:08.442Z level=INFO msg="Uploading 1 narinfos" lb # [ 137.969710] niks3-server[1182]: 2026/09/22 10:59:08 INFO Signed narinfos id=37 count=1 lb # [ 137.982633] niks3-server[1182]: 2026/09/22 10:59:08 INFO Received complete upload request method=POST path=/api/pending_closures/37/complete lb # [ 137.996196] niks3-server[1182]: 2026/09/22 10:59:08 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.ls lb # [ 137.998818] niks3-server[1182]: 2026/09/22 10:59:08 INFO Registered completed upload object_key=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z.narinfo lb # [ 138.004048] niks3-server[1182]: 2026/09/22 10:59:08 INFO Completed upload id=37 worker2 # [ 138.024039] nix-grpc-daemon[1330]: time=2026-09-22T10:59:08.483Z level=INFO msg="Upload complete. (534ms)" worker2 # [ 138.036161] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 duration_s=0 paths=1 nar_bytes_in=544 worker2 # [ 138.052633] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 138.060981] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 138.067420] nix-daemon[866]: reaped child process 1353, status = succeeded client # building '/nix/store/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv' on lb:50051... client # farm-a> worker2: building mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv worker2 # [ 138.080507] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-after-niks3-restart worker2 # [ 139.522800] nix-daemon[866]: reaped child process 1367, status = succeeded worker2 # [ 139.708848] nix-daemon[866]: accepted connection from pid 1383, user nix-grpc-daemon (trusted) worker2 # [ 139.727357] nix-daemon[866]: reaped child process 1386, status = succeeded lb # [ 139.721607] niks3-server[1182]: 2026/09/22 10:59:10 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 139.754325] nix-grpc-daemon[1330]: time=2026-09-22T10:59:10.211Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 139.755838] nix-grpc-daemon[1330]: time=2026-09-22T10:59:10.211Z level=INFO msg="Uploading j3iswx3ynilvcycpyw0z667966c9kpzx-farm-a (144B)" lb # [ 139.763392] niks3-server[1182]: 2026/09/22 10:59:10 INFO Registered completed upload object_key=log/mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv lb # [ 139.776766] niks3-server[1182]: 2026/09/22 10:59:10 INFO Registered completed upload object_key=nar/1diw08qzlsh0zkrx4kvj1mvbc57g7r65j1f459ankpg9yc5rgcpx.nar.zst lb # [ 139.787836] niks3-server[1182]: 2026/09/22 10:59:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/38/sign lb # [ 139.794541] niks3-server[1182]: 2026/09/22 10:59:10 INFO Signed narinfos id=38 count=1 worker2 # [ 139.815027] nix-grpc-daemon[1330]: time=2026-09-22T10:59:10.273Z level=INFO msg="Uploading 1 narinfos" lb # [ 139.799116] niks3-server[1182]: 2026/09/22 10:59:10 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.ls lb # [ 139.807127] niks3-server[1182]: 2026/09/22 10:59:10 INFO Received complete upload request method=POST path=/api/pending_closures/38/complete lb # [ 139.818982] niks3-server[1182]: 2026/09/22 10:59:10 INFO Completed upload id=38 lb # [ 139.821900] niks3-server[1182]: 2026/09/22 10:59:10 INFO Registered completed upload object_key=j3iswx3ynilvcycpyw0z667966c9kpzx.narinfo worker2 # [ 139.838943] nix-grpc-daemon[1330]: time=2026-09-22T10:59:10.298Z level=INFO msg="Upload complete. (347ms)" worker1 # [ 139.878430] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:10Z level=debug event=want client=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv cached=0 worker2 # [ 139.853445] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 duration_s=1 drv=mxnzwd2qgmj6r8zhjk6jlcxv8j46h04z-farm-a.drv assign_id=1 outputs=1 worker1 # [ 139.884174] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:10Z level=debug event=assigned drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 139.872932] nix-daemon[866]: reaped child process 1365, status = succeeded worker2 # [ 139.907821] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 139.923852] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 139.938325] nix-grpc-daemon[1330]: time=2026-09-22T10:59:10.397Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' to 'lb:50051'... worker2 # [ 139.948779] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 139.962108] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 140.162354] nix-daemon[866]: accepted connection from pid 1390, user nix-grpc-daemon (trusted) worker2 # [ 140.182059] nix-daemon[866]: reaped child process 1393, status = succeeded worker2 # [ 140.357567] nix-daemon[866]: accepted connection from pid 1395, user nix-grpc-daemon (trusted) worker2 # [ 140.375933] nix-daemon[866]: reaped child process 1398, status = succeeded lb # [ 140.372601] niks3-server[1182]: 2026/09/22 10:59:10 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 140.402386] nix-grpc-daemon[1330]: time=2026-09-22T10:59:10.861Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 140.403875] nix-grpc-daemon[1330]: time=2026-09-22T10:59:10.861Z level=INFO msg="Uploading 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv (656B)" lb # [ 140.400407] niks3-server[1182]: 2026/09/22 10:59:10 INFO Registered completed upload object_key=nar/1xpcisa10g5by3yknnrs1xwhxhf4s243h7gwzjii0799pwbhfjfa.nar.zst lb # [ 140.414115] niks3-server[1182]: 2026/09/22 10:59:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/39/sign worker2 # [ 140.440520] nix-grpc-daemon[1330]: time=2026-09-22T10:59:10.899Z level=INFO msg="Uploading 1 narinfos" lb # [ 140.427064] niks3-server[1182]: 2026/09/22 10:59:10 INFO Signed narinfos id=39 count=1 lb # [ 140.432443] niks3-server[1182]: 2026/09/22 10:59:10 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.ls lb # [ 140.441133] niks3-server[1182]: 2026/09/22 10:59:10 INFO Received complete upload request method=POST path=/api/pending_closures/39/complete worker2 # [ 140.465215] nix-grpc-daemon[1330]: time=2026-09-22T10:59:10.924Z level=INFO msg="Upload complete. (487ms)" lb # [ 140.445974] niks3-server[1182]: 2026/09/22 10:59:10 INFO Completed upload id=39 lb # [ 140.449019] niks3-server[1182]: 2026/09/22 10:59:10 INFO Registered completed upload object_key=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch.narinfo worker2 # [ 140.472258] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:10Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 duration_s=0 paths=1 nar_bytes_in=656 worker2 # [ 140.486085] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 140.492397] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 140.497268] nix-daemon[866]: reaped child process 1388, status = succeeded client # building '/nix/store/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv' on lb:50051... client # farm-b> worker2: building 5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv worker2 # [ 140.510490] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-after-niks3-restart worker2 # [ 141.742570] nix-daemon[866]: reaped child process 1402, status = succeeded worker2 # [ 141.911816] nix-daemon[866]: accepted connection from pid 1418, user nix-grpc-daemon (trusted) worker2 # [ 141.932206] nix-daemon[866]: reaped child process 1421, status = succeeded lb # [ 141.924871] niks3-server[1182]: 2026/09/22 10:59:12 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 141.954585] nix-grpc-daemon[1330]: time=2026-09-22T10:59:12.412Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 141.956118] nix-grpc-daemon[1330]: time=2026-09-22T10:59:12.412Z level=INFO msg="Uploading myar0ljvsqm9h7fxyh6qrqkg0175r3xn-farm-b (192B)" lb # [ 141.958685] niks3-server[1182]: 2026/09/22 10:59:12 INFO Registered completed upload object_key=log/5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv lb # [ 141.972683] niks3-server[1182]: 2026/09/22 10:59:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/40/sign worker2 # [ 142.001301] nix-grpc-daemon[1330]: time=2026-09-22T10:59:12.460Z level=INFO msg="Uploading 1 narinfos" lb # [ 141.985000] niks3-server[1182]: 2026/09/22 10:59:12 INFO Signed narinfos id=40 count=1 lb # [ 141.987914] niks3-server[1182]: 2026/09/22 10:59:12 INFO Registered completed upload object_key=nar/0zvv5rxmyk9afay83j52m7ipyhrqivhg6h6a0ksjd658pxzwi78w.nar.zst lb # [ 142.007454] niks3-server[1182]: 2026/09/22 10:59:12 INFO Received complete upload request method=POST path=/api/pending_closures/40/complete worker2 # [ 142.031252] nix-grpc-daemon[1330]: time=2026-09-22T10:59:12.490Z level=INFO msg="Upload complete. (310ms)" lb # [ 142.015103] niks3-server[1182]: 2026/09/22 10:59:12 INFO Registered completed upload object_key=myar0ljvsqm9h7fxyh6qrqkg0175r3xn.ls lb # [ 142.023167] niks3-server[1182]: 2026/09/22 10:59:12 INFO Completed upload id=40 worker2 # [ 142.041066] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 duration_s=1 drv=5ml8wmsr73y0fh0dflaimhsh2xbcd3ch-farm-b.drv assign_id=2 outputs=1 worker1 # [ 142.066158] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:12Z level=debug event=want client=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv cached=0 worker1 # [ 142.072144] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:12Z level=debug event=assigned drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 142.060635] nix-daemon[866]: reaped child process 1400, status = succeeded worker2 # [ 142.096175] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 142.111988] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 142.128361] nix-grpc-daemon[1330]: time=2026-09-22T10:59:12.587Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' to 'lb:50051'... worker2 # [ 142.139693] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 142.153871] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 142.351391] nix-daemon[866]: accepted connection from pid 1425, user nix-grpc-daemon (trusted) worker2 # [ 142.372725] nix-daemon[866]: reaped child process 1428, status = succeeded worker2 # [ 142.547273] nix-daemon[866]: accepted connection from pid 1430, user nix-grpc-daemon (trusted) worker2 # [ 142.564312] nix-daemon[866]: reaped child process 1433, status = succeeded lb # [ 142.558837] niks3-server[1182]: 2026/09/22 10:59:13 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 142.618620] nix-grpc-daemon[1330]: time=2026-09-22T10:59:13.077Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 142.620583] nix-grpc-daemon[1330]: time=2026-09-22T10:59:13.077Z level=INFO msg="Uploading 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv (784B)" worker2 # [ 142.657551] nix-grpc-daemon[1330]: time=2026-09-22T10:59:13.116Z level=INFO msg="Uploading 1 narinfos" lb # [ 142.630982] niks3-server[1182]: 2026/09/22 10:59:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/41/sign lb # [ 142.645396] niks3-server[1182]: 2026/09/22 10:59:13 INFO Registered completed upload object_key=nar/1fnvw3c0v6n1j60nd5iagl3bggzcvixsw103w98mwhnpmnmgb248.nar.zst lb # [ 142.654606] niks3-server[1182]: 2026/09/22 10:59:13 INFO Signed narinfos id=41 count=1 lb # [ 142.664266] niks3-server[1182]: 2026/09/22 10:59:13 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.ls lb # [ 142.671301] niks3-server[1182]: 2026/09/22 10:59:13 INFO Received complete upload request method=POST path=/api/pending_closures/41/complete lb # [ 142.676390] niks3-server[1182]: 2026/09/22 10:59:13 INFO Registered completed upload object_key=6j7gr603xfslr5lxcbh72430f5q1r4b0.narinfo worker2 # [ 142.696424] nix-grpc-daemon[1330]: time=2026-09-22T10:59:13.155Z level=INFO msg="Upload complete. (525ms)" lb # [ 142.680833] niks3-server[1182]: 2026/09/22 10:59:13 INFO Completed upload id=41 worker2 # [ 142.707410] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 duration_s=0 paths=1 nar_bytes_in=784 worker2 # [ 142.720854] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 142.733113] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 142.739758] nix-daemon[866]: reaped child process 1423, status = succeeded client # building '/nix/store/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv' on lb:50051... client # farm-top> worker2: building 6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv worker2 # [ 142.753440] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-after-niks3-restart worker2 # [ 143.987287] nix-daemon[866]: reaped child process 1437, status = succeeded worker2 # [ 144.150498] nix-daemon[866]: accepted connection from pid 1453, user nix-grpc-daemon (trusted) worker2 # [ 144.168123] nix-daemon[866]: reaped child process 1456, status = succeeded lb # [ 144.160862] niks3-server[1182]: 2026/09/22 10:59:14 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 144.191933] nix-grpc-daemon[1330]: time=2026-09-22T10:59:14.649Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 144.193771] nix-grpc-daemon[1330]: time=2026-09-22T10:59:14.649Z level=INFO msg="Uploading z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc-farm-top (248B)" lb # [ 144.206698] niks3-server[1182]: 2026/09/22 10:59:14 INFO Registered completed upload object_key=nar/0cypkbh15calmx90jgh2waxpq661jalf4c99dgb7j606z61xh0ld.nar.zst worker2 # [ 144.238866] nix-grpc-daemon[1330]: time=2026-09-22T10:59:14.697Z level=INFO msg="Uploading 1 narinfos" lb # [ 144.221171] niks3-server[1182]: 2026/09/22 10:59:14 INFO Registered completed upload object_key=log/6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv lb # [ 144.229711] niks3-server[1182]: 2026/09/22 10:59:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/42/sign lb # [ 144.233862] niks3-server[1182]: 2026/09/22 10:59:14 INFO Signed narinfos id=42 count=1 lb # [ 144.239927] niks3-server[1182]: 2026/09/22 10:59:14 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.ls lb # [ 144.244457] niks3-server[1182]: 2026/09/22 10:59:14 INFO Received complete upload request method=POST path=/api/pending_closures/42/complete worker2 # [ 144.263868] nix-grpc-daemon[1330]: time=2026-09-22T10:59:14.723Z level=INFO msg="Upload complete. (296ms)" lb # [ 144.248037] niks3-server[1182]: 2026/09/22 10:59:14 INFO Completed upload id=42 lb # [ 144.249789] niks3-server[1182]: 2026/09/22 10:59:14 INFO Registered completed upload object_key=z8mx9hni3m7cymmz0ddfhg3jhqvi9mlc.narinfo worker2 # [ 144.277168] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 duration_s=1 drv=6j7gr603xfslr5lxcbh72430f5q1r4b0-farm-top.drv assign_id=3 outputs=1 worker2 # [ 144.295293] nix-daemon[866]: reaped child process 1435, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag after-niks3-restart >&2, in 7.28 seconds) client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag after-niks3-restart outPath client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag after-niks3-restart outPath, in 0.24 seconds) (finished: subtest: a niks3 restart does not cost the leader its role, in 11.96 seconds) subtest: scheduler down: the other node takes the lock and keeps it lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1 # [ 144.894174] systemd[1]: Stopping Nix worker-protocol over gRPC... worker1 # [ 145.455413] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:15Z level=info event=signal_exit lb # [ 145.981971] niks3-server[1182]: 2026/09/22 10:59:16 INFO lead: released remote=192.168.1.3:34454 lb # [ 146.664161] niks3-server[1182]: 2026/09/22 10:59:17 INFO lead: acquired remote=192.168.1.4:47756 worker2 # [ 146.689572] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:17Z level=info event=scheduler_take_over worker2 # [ 147.001485] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:17Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 147.022427] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:17Z level=info event=scheduler_restarting worker1 # [ 147.022783] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:17Z level=info event=scheduler_restarting addr=lb:50051 worker1 # [ 147.022962] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:17Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:44006 worker1 # [ 147.023092] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:17Z level=info event=worker_gone addr=192.168.1.4:50051 worker1 # [ 147.023250] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:17Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:44006 worker1 # [ 147.023362] nix-grpc-daemon[1311]: ts=2026-09-22T10:59:17Z level=info event=worker_gone addr=192.168.1.3:50051 worker2 # [ 147.202796] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:17Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 147.222021] nix-grpc-daemon[1311]: E0000 00:00:1790074757.693102 1356 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 15, started true, shutdown true worker2 # [ 147.208410] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 147.238212] nix-grpc-daemon[1311]: E0000 00:00:1790074757.701071 1356 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 9, started true, shutdown true worker2 # [ 147.409543] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:17Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 147.414826] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:17Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 147.615970] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 147.621416] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:18Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 147.824227] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:18Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 147.852198] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:18Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 147.856966] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:18Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:37456 worker2 # [ 147.862313] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:18Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 148.933019] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker1 # [ 148.938595] systemd[1]: Stopped Nix worker-protocol over gRPC. worker1 # [ 148.944267] systemd[1]: nix-grpc-daemon.service: Consumed 291ms CPU time over 17.698s wall clock time, 14.7M memory peak, 10.8K incoming IP traffic, 14K outgoing IP traffic. worker1 # [ 148.957677] systemd[1]: nix-grpc-daemon.socket: Deactivated successfully. worker1 # [ 148.959864] systemd[1]: Closed Nix worker-protocol over gRPC. worker1 # [ 148.965093] systemd[1]: nix-grpc-daemon.socket: Consumed 977us CPU time over 17.744s wall clock time, 256K memory peak, 24.2K incoming IP traffic, 20.2K outgoing IP traffic. worker2: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over worker2: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q event=scheduler_take_over, in 0.09 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.05 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby >&2 worker2 # [ 149.362347] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:19Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55550 client # don't know how to build these paths: client # /nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker2 # [ 149.373609] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 149.401065] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:19Z level=debug event=want client=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv cached=0 worker2 # [ 149.408557] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:19Z level=debug event=assigned drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 149.448212] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 149.463684] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:19Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55550 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' to 'lb:50051'... worker2 # [ 149.479543] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:19Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 149.495392] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 149.703388] nix-daemon[866]: accepted connection from pid 1471, user nix-grpc-daemon (trusted) worker2 # [ 149.722734] nix-daemon[866]: reaped child process 1474, status = succeeded worker2 # [ 149.904935] nix-daemon[866]: accepted connection from pid 1476, user nix-grpc-daemon (trusted) worker2 # [ 149.922571] nix-daemon[866]: reaped child process 1479, status = succeeded lb # [ 149.917161] niks3-server[1182]: 2026/09/22 10:59:20 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 149.949139] nix-grpc-daemon[1330]: time=2026-09-22T10:59:20.407Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 149.950626] nix-grpc-daemon[1330]: time=2026-09-22T10:59:20.407Z level=INFO msg="Uploading adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv (520B)" lb # [ 149.950981] niks3-server[1182]: 2026/09/22 10:59:20 INFO Registered completed upload object_key=nar/0scd0jwin78ibwq92wkajw2zj7adk8bih7yhx8bxz1706hh53507.nar.zst lb # [ 149.965846] niks3-server[1182]: 2026/09/22 10:59:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/43/sign worker2 # [ 149.991243] nix-grpc-daemon[1330]: time=2026-09-22T10:59:20.450Z level=INFO msg="Uploading 1 narinfos" lb # [ 149.975989] niks3-server[1182]: 2026/09/22 10:59:20 INFO Signed narinfos id=43 count=1 lb # [ 149.978407] niks3-server[1182]: 2026/09/22 10:59:20 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.ls lb # [ 149.991309] niks3-server[1182]: 2026/09/22 10:59:20 INFO Received complete upload request method=POST path=/api/pending_closures/43/complete lb # [ 149.996983] niks3-server[1182]: 2026/09/22 10:59:20 INFO Completed upload id=43 worker2 # [ 150.016897] nix-grpc-daemon[1330]: time=2026-09-22T10:59:20.476Z level=INFO msg="Upload complete. (504ms)" lb # [ 149.999754] niks3-server[1182]: 2026/09/22 10:59:20 INFO Registered completed upload object_key=adj3g5w8jh83wl959p7daq6igl0gaay3.narinfo worker2 # [ 150.026466] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:20Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 duration_s=0 paths=1 nar_bytes_in=520 worker2 # [ 150.036701] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:20Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 150.048651] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 150.051226] nix-daemon[866]: reaped child process 1469, status = succeeded client # building '/nix/store/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv' on lb:50051... client # farm-a> worker2: building adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv worker2 # [ 150.063331] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-on-standby worker2 # [ 151.109377] nix-daemon[866]: reaped child process 1483, status = succeeded worker2 # [ 151.277302] nix-daemon[866]: accepted connection from pid 1499, user nix-grpc-daemon (trusted) worker2 # [ 151.295705] nix-daemon[866]: reaped child process 1502, status = succeeded lb # [ 151.289002] niks3-server[1182]: 2026/09/22 10:59:21 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 151.323901] nix-grpc-daemon[1330]: time=2026-09-22T10:59:21.779Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 151.325999] nix-grpc-daemon[1330]: time=2026-09-22T10:59:21.779Z level=INFO msg="Uploading 7qyaw17jjsjxkkgb1kfm3wxvczk56aan-farm-a (136B)" lb # [ 151.326452] niks3-server[1182]: 2026/09/22 10:59:21 INFO Registered completed upload object_key=nar/0x76wzgdd1nk8rm930z8l69v9961jpc8qhzmgganv0nhagq8lv9c.nar.zst lb # [ 151.343286] niks3-server[1182]: 2026/09/22 10:59:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/44/sign worker2 # [ 151.368162] nix-grpc-daemon[1330]: time=2026-09-22T10:59:21.827Z level=INFO msg="Uploading 1 narinfos" lb # [ 151.354023] niks3-server[1182]: 2026/09/22 10:59:21 INFO Signed narinfos id=44 count=1 lb # [ 151.359016] niks3-server[1182]: 2026/09/22 10:59:21 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.ls lb # [ 151.364801] niks3-server[1182]: 2026/09/22 10:59:21 INFO Registered completed upload object_key=log/adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv lb # [ 151.372302] niks3-server[1182]: 2026/09/22 10:59:21 INFO Received complete upload request method=POST path=/api/pending_closures/44/complete lb # [ 151.376151] niks3-server[1182]: 2026/09/22 10:59:21 INFO Completed upload id=44 worker2 # [ 151.395762] nix-grpc-daemon[1330]: time=2026-09-22T10:59:21.854Z level=INFO msg="Upload complete. (323ms)" lb # [ 151.378757] niks3-server[1182]: 2026/09/22 10:59:21 INFO Registered completed upload object_key=7qyaw17jjsjxkkgb1kfm3wxvczk56aan.narinfo worker2 # [ 151.417146] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:21Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 duration_s=1 drv=adj3g5w8jh83wl959p7daq6igl0gaay3-farm-a.drv assign_id=1 outputs=1 worker2 # [ 151.432368] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:21Z level=debug event=want client=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv cached=0 worker2 # [ 151.435844] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:21Z level=debug event=assigned drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 151.442251] nix-daemon[866]: reaped child process 1481, status = succeeded worker2 # [ 151.459967] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 151.475247] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 151.489622] nix-grpc-daemon[1330]: time=2026-09-22T10:59:21.948Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' to 'lb:50051'... worker2 # [ 151.500521] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 151.515349] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 151.720156] nix-daemon[866]: accepted connection from pid 1508, user nix-grpc-daemon (trusted) worker2 # [ 151.737874] nix-daemon[866]: reaped child process 1511, status = succeeded worker2 # [ 151.912931] nix-daemon[866]: accepted connection from pid 1513, user nix-grpc-daemon (trusted) worker2 # [ 151.930716] nix-daemon[866]: reaped child process 1516, status = succeeded lb # [ 151.922616] niks3-server[1182]: 2026/09/22 10:59:22 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 151.951925] nix-grpc-daemon[1330]: time=2026-09-22T10:59:22.410Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 151.953682] nix-grpc-daemon[1330]: time=2026-09-22T10:59:22.410Z level=INFO msg="Uploading s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv (640B)" lb # [ 151.947971] niks3-server[1182]: 2026/09/22 10:59:22 INFO Registered completed upload object_key=nar/0g5yl4v9pymang94vyp5ib5wi5jwf0bpp184y8qjavkpag12wmlv.nar.zst lb # [ 151.963605] niks3-server[1182]: 2026/09/22 10:59:22 INFO Received sign narinfos request method=POST path=/api/pending_closures/45/sign worker2 # [ 151.989021] nix-grpc-daemon[1330]: time=2026-09-22T10:59:22.448Z level=INFO msg="Uploading 1 narinfos" lb # [ 151.978830] niks3-server[1182]: 2026/09/22 10:59:22 INFO Signed narinfos id=45 count=1 lb # [ 151.985609] niks3-server[1182]: 2026/09/22 10:59:22 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.ls lb # [ 151.992285] niks3-server[1182]: 2026/09/22 10:59:22 INFO Received complete upload request method=POST path=/api/pending_closures/45/complete worker2 # [ 152.018560] nix-grpc-daemon[1330]: time=2026-09-22T10:59:22.477Z level=INFO msg="Upload complete. (487ms)" lb # [ 151.999506] niks3-server[1182]: 2026/09/22 10:59:22 INFO Completed upload id=45 lb # [ 152.002425] niks3-server[1182]: 2026/09/22 10:59:22 INFO Registered completed upload object_key=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g.narinfo worker2 # [ 152.028270] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:22Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 duration_s=0 paths=1 nar_bytes_in=640 worker2 # [ 152.039522] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:22Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 152.046904] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 152.051957] nix-daemon[866]: reaped child process 1506, status = succeeded client # building '/nix/store/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv' on lb:50051... client # farm-b> worker2: building s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker2 # [ 152.066432] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-on-standby worker2 # [ 153.115656] nix-daemon[866]: reaped child process 1520, status = succeeded worker2 # [ 153.292129] nix-daemon[866]: accepted connection from pid 1536, user nix-grpc-daemon (trusted) worker2 # [ 153.312588] nix-daemon[866]: reaped child process 1539, status = succeeded lb # [ 153.305192] niks3-server[1182]: 2026/09/22 10:59:23 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 153.334744] nix-grpc-daemon[1330]: time=2026-09-22T10:59:23.793Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 153.339844] nix-grpc-daemon[1330]: time=2026-09-22T10:59:23.795Z level=INFO msg="Uploading 6hsxxv65wr2jz8cgv2v66iq16niq1gsv-farm-b (184B)" lb # [ 153.338253] niks3-server[1182]: 2026/09/22 10:59:23 INFO Registered completed upload object_key=log/s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv worker2 # [ 153.378285] nix-grpc-daemon[1330]: time=2026-09-22T10:59:23.837Z level=INFO msg="Uploading 1 narinfos" lb # [ 153.358133] niks3-server[1182]: 2026/09/22 10:59:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/46/sign lb # [ 153.370297] niks3-server[1182]: 2026/09/22 10:59:23 INFO Signed narinfos id=46 count=1 lb # [ 153.377656] niks3-server[1182]: 2026/09/22 10:59:23 INFO Registered completed upload object_key=nar/1grn64wcl2mc78k1jvrz5mcrk1ld70lhf8b9jpiqy8mvs6gfzq0v.nar.zst lb # [ 153.384140] niks3-server[1182]: 2026/09/22 10:59:23 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.ls lb # [ 153.392867] niks3-server[1182]: 2026/09/22 10:59:23 INFO Received complete upload request method=POST path=/api/pending_closures/46/complete lb # [ 153.398038] niks3-server[1182]: 2026/09/22 10:59:23 INFO Completed upload id=46 worker2 # [ 153.417345] nix-grpc-daemon[1330]: time=2026-09-22T10:59:23.876Z level=INFO msg="Upload complete. (339ms)" lb # [ 153.401271] niks3-server[1182]: 2026/09/22 10:59:23 INFO Registered completed upload object_key=6hsxxv65wr2jz8cgv2v66iq16niq1gsv.narinfo worker2 # [ 153.436736] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 duration_s=1 drv=s5xf4hr9bjv2jdhhqy1ih0c1ld822d2g-farm-b.drv assign_id=2 outputs=1 worker2 # [ 153.452335] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:23Z level=debug event=want client=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv cached=0 worker2 # [ 153.457751] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:23Z level=debug event=assigned drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 153.461952] nix-daemon[866]: reaped child process 1518, status = succeeded worker2 # [ 153.480221] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 153.495326] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 153.510917] nix-grpc-daemon[1330]: time=2026-09-22T10:59:23.970Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' to 'lb:50051'... worker2 # [ 153.522493] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 153.536550] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 153.740130] nix-daemon[866]: accepted connection from pid 1545, user nix-grpc-daemon (trusted) worker2 # [ 153.759683] nix-daemon[866]: reaped child process 1548, status = succeeded worker2 # [ 153.930053] nix-daemon[866]: accepted connection from pid 1550, user nix-grpc-daemon (trusted) worker2 # [ 153.949369] nix-daemon[866]: reaped child process 1553, status = succeeded lb # [ 153.940736] niks3-server[1182]: 2026/09/22 10:59:24 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 153.970710] nix-grpc-daemon[1330]: time=2026-09-22T10:59:24.429Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 153.972280] nix-grpc-daemon[1330]: time=2026-09-22T10:59:24.429Z level=INFO msg="Uploading 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv (768B)" lb # [ 153.967245] niks3-server[1182]: 2026/09/22 10:59:24 INFO Registered completed upload object_key=nar/1zyx46mv29cfygdhc5ai40qr4vl88lkr1lilm0h72j23w6wakgmc.nar.zst lb # [ 153.980738] niks3-server[1182]: 2026/09/22 10:59:24 INFO Received sign narinfos request method=POST path=/api/pending_closures/47/sign worker2 # [ 154.008781] nix-grpc-daemon[1330]: time=2026-09-22T10:59:24.467Z level=INFO msg="Uploading 1 narinfos" lb # [ 153.989131] niks3-server[1182]: 2026/09/22 10:59:24 INFO Signed narinfos id=47 count=1 lb # [ 153.997482] niks3-server[1182]: 2026/09/22 10:59:24 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.ls lb # [ 154.006116] niks3-server[1182]: 2026/09/22 10:59:24 INFO Received complete upload request method=POST path=/api/pending_closures/47/complete lb # [ 154.012610] niks3-server[1182]: 2026/09/22 10:59:24 INFO Registered completed upload object_key=7s5szig4ffga53df6rls98r9lhxsrsc5.narinfo worker2 # [ 154.035879] nix-grpc-daemon[1330]: time=2026-09-22T10:59:24.495Z level=INFO msg="Upload complete. (482ms)" lb # [ 154.018057] niks3-server[1182]: 2026/09/22 10:59:24 INFO Completed upload id=47 worker2 # [ 154.043373] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 duration_s=0 paths=1 nar_bytes_in=768 worker2 # [ 154.055112] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 154.060970] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 154.065690] nix-daemon[866]: reaped child process 1543, status = succeeded client # building '/nix/store/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv' on lb:50051... client # farm-top> worker2: building 7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv worker2 # [ 154.078364] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) lb # [ 154.217166] envoy[933]: conn peer=[2001:db8:1::1]:43708 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9766 tx=4444 ms=6970 lb # [ 154.222184] envoy[933]: conn peer=[2001:db8:1::3]:39560 tls=TLSv1.3 subject="CN=worker-1" sni=- flags=- tls_fail="-" rx=1028 tx=372 ms=15716 client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-on-standby worker2 # [ 155.527319] nix-daemon[866]: reaped child process 1557, status = succeeded worker2 # [ 155.714932] nix-daemon[866]: accepted connection from pid 1573, user nix-grpc-daemon (trusted) worker2 # [ 155.733991] nix-daemon[866]: reaped child process 1576, status = succeeded lb # [ 155.728680] niks3-server[1182]: 2026/09/22 10:59:26 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 155.757070] nix-grpc-daemon[1330]: time=2026-09-22T10:59:26.214Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 155.758578] nix-grpc-daemon[1330]: time=2026-09-22T10:59:26.214Z level=INFO msg="Uploading 1ini8mfd1rzqyp8jgqihp4k889588mpx-farm-top (240B)" lb # [ 155.762702] niks3-server[1182]: 2026/09/22 10:59:26 INFO Registered completed upload object_key=nar/1mncazpqn39nifc2fwbx80la5i264r8xc1lj8fnlbx006znc1925.nar.zst lb # [ 155.776362] niks3-server[1182]: 2026/09/22 10:59:26 INFO Registered completed upload object_key=log/7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv lb # [ 155.783578] niks3-server[1182]: 2026/09/22 10:59:26 INFO Received sign narinfos request method=POST path=/api/pending_closures/48/sign worker2 # [ 155.809096] nix-grpc-daemon[1330]: time=2026-09-22T10:59:26.267Z level=INFO msg="Uploading 1 narinfos" lb # [ 155.793168] niks3-server[1182]: 2026/09/22 10:59:26 INFO Signed narinfos id=48 count=1 lb # [ 155.799555] niks3-server[1182]: 2026/09/22 10:59:26 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.ls lb # [ 155.807281] niks3-server[1182]: 2026/09/22 10:59:26 INFO Received complete upload request method=POST path=/api/pending_closures/48/complete lb # [ 155.815106] niks3-server[1182]: 2026/09/22 10:59:26 INFO Completed upload id=48 worker2 # [ 155.835065] nix-grpc-daemon[1330]: time=2026-09-22T10:59:26.294Z level=INFO msg="Upload complete. (345ms)" lb # [ 155.818280] niks3-server[1182]: 2026/09/22 10:59:26 INFO Registered completed upload object_key=1ini8mfd1rzqyp8jgqihp4k889588mpx.narinfo worker2 # [ 155.849213] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:26Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 duration_s=1 drv=7s5szig4ffga53df6rls98r9lhxsrsc5-farm-top.drv assign_id=3 outputs=1 worker2 # [ 155.866224] nix-daemon[866]: reaped child process 1555, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby >&2, in 6.80 seconds) client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby outPath client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag on-standby outPath, in 0.23 seconds) worker1 # [ 156.242766] systemd[1]: Listening on Nix worker-protocol over gRPC. worker1 # [ 156.253402] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 156.452600] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:26Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker1 # [ 156.454615] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:26Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker1 # [ 156.458718] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:26Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.3:50051 niks3=http://lb:5751 max_jobs=1 worker1 # [ 156.467028] nix-grpc-daemon[1394]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker1 # [ 156.475663] nix-grpc-daemon[1394]: E0000 00:00:1790074766.936115 1394 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 156.480476] nix-grpc-daemon[1394]: E0000 00:00:1790074766.938133 1394 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 156.483583] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:26Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker1 # [ 156.485958] systemd[1]: Started Nix worker-protocol over gRPC. lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 156.496288] nix-grpc-daemon[1394]: E0000 00:00:1790074766.958995 1433 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 156.520783] nix-grpc-daemon[1394]: E0000 00:00:1790074766.977263 1433 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 156.523143] nix-grpc-daemon[1394]: E0000 00:00:1790074766.977465 1433 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker1 # [ 156.526619] nix-grpc-daemon[1394]: E0000 00:00:1790074766.991978 1433 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker1 # [ 156.543519] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:27Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 156.527577] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:26Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:37456 worker2 # [ 156.533913] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:26Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) (finished: subtest: scheduler down: the other node takes the lock and keeps it, in 12.28 seconds) subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise client: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs client # [ 156.888723] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs. client # Running as unit: rs.service; invocation ID: 367018a80363475093cc772703a3d2dc client: (finished: must succeed: systemd-run --unit rs nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag rs, in 0.09 seconds) client # [ 157.150268] nix[1359]: don't know how to build these paths: client # [ 157.152105] nix[1359]: /nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker2 # [ 157.182945] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:27Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 157.195847] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:27Z level=debug event=want client=2 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv cached=0 worker2 # [ 157.202655] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:27Z level=debug event=assigned drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 157.245059] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 157.261247] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:55550 client # [ 157.244172] nix[1359]: copying 1 paths to 'lb:50051'... worker2 # [ 157.277198] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 client # [ 157.246562] nix[1359]: copying path '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' to 'lb:50051'... worker2 # [ 157.294968] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 157.499377] nix-daemon[866]: accepted connection from pid 1589, user nix-grpc-daemon (trusted) worker2 # [ 157.521167] nix-daemon[866]: reaped child process 1592, status = succeeded worker2 # [ 157.697155] nix-daemon[866]: accepted connection from pid 1594, user nix-grpc-daemon (trusted) worker2 # [ 157.717353] nix-daemon[866]: reaped child process 1597, status = succeeded lb # [ 157.711689] niks3-server[1182]: 2026/09/22 10:59:28 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 157.741757] nix-grpc-daemon[1330]: time=2026-09-22T10:59:28.200Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 157.743292] nix-grpc-daemon[1330]: time=2026-09-22T10:59:28.200Z level=INFO msg="Uploading ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv (416B)" lb # [ 157.740479] niks3-server[1182]: 2026/09/22 10:59:28 INFO Registered completed upload object_key=nar/0lynldv3n40qnhi1516wgiry3i3pja3sanwyyd8q4nslr915d7r1.nar.zst worker2 # [ 157.786350] nix-grpc-daemon[1330]: time=2026-09-22T10:59:28.245Z level=INFO msg="Uploading 1 narinfos" lb # [ 157.758742] niks3-server[1182]: 2026/09/22 10:59:28 INFO Received sign narinfos request method=POST path=/api/pending_closures/49/sign lb # [ 157.770928] niks3-server[1182]: 2026/09/22 10:59:28 INFO Signed narinfos id=49 count=1 lb # [ 157.776081] niks3-server[1182]: 2026/09/22 10:59:28 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.ls lb # [ 157.784432] niks3-server[1182]: 2026/09/22 10:59:28 INFO Received complete upload request method=POST path=/api/pending_closures/49/complete lb # [ 157.790308] niks3-server[1182]: 2026/09/22 10:59:28 INFO Completed upload id=49 worker2 # [ 157.809770] nix-grpc-daemon[1330]: time=2026-09-22T10:59:28.268Z level=INFO msg="Upload complete. (497ms)" lb # [ 157.792802] niks3-server[1182]: 2026/09/22 10:59:28 INFO Registered completed upload object_key=ajji8m1zd5q5wi64gp7vclb5dd8hr19m.narinfo worker2 # [ 157.819772] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:28Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:55550 duration_s=0 paths=1 nar_bytes_in=416 worker2 # [ 157.828270] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:28Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 worker2 # [ 157.840998] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) worker2 # [ 157.846288] nix-daemon[866]: reaped child process 1587, status = succeeded client # [ 157.823442] nix[1359]: building '/nix/store/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv' on lb:50051... worker2 # [ 157.860456] nix-daemon[866]: accepted connection from pid 1270, user nix-grpc-daemon (trusted) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.09 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true worker1 # [ 159.932888] systemd-journald[267]: Received client request to rotate journal, rotating. worker1 # [ 159.954808] systemd-journald[267]: Vacuuming done, freed 0B of archived journals from /var/log/journal/3457c29705e74d53b4ef05a587961acb. worker1: (finished: must succeed: journalctl --rotate --vacuum-time=1s -u nix-grpc-daemon >/dev/null 2>&1 || true, in 0.10 seconds) worker2: must succeed: systemctl restart --no-block nix-grpc-daemon.service worker2 # [ 159.994711] systemd[1]: Stopping Nix worker-protocol over gRPC... worker2: (finished: must succeed: systemctl restart --no-block nix-grpc-daemon.service, in 0.06 seconds) client: must succeed: sleep 5.0 worker2 # [ 160.635969] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:31Z level=info event=draining lb # [ 164.216572] envoy[933]: conn peer=[2001:db8:1::1]:34140 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9623 tx=4454 ms=6497 client: (finished: must succeed: sleep 5.0, in 5.06 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=' worker2 # [ 165.495487] nix-daemon[866]: reaped child process 1601, status = succeeded worker2 # [ 165.666370] nix-daemon[866]: accepted connection from pid 1642, user nix-grpc-daemon (trusted) worker2 # [ 165.685189] nix-daemon[866]: reaped child process 1645, status = succeeded lb # [ 165.676967] niks3-server[1182]: 2026/09/22 10:59:36 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 165.711882] nix-grpc-daemon[1330]: time=2026-09-22T10:59:36.167Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 165.713548] nix-grpc-daemon[1330]: time=2026-09-22T10:59:36.167Z level=INFO msg="Uploading k48d52qj0czpjmzj15yq3fd14zfw5iq6-slow-rs (120B)" lb # [ 165.723584] niks3-server[1182]: 2026/09/22 10:59:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/50/sign worker2 # [ 165.754843] nix-grpc-daemon[1330]: time=2026-09-22T10:59:36.213Z level=INFO msg="Uploading 1 narinfos" lb # [ 165.741118] niks3-server[1182]: 2026/09/22 10:59:36 INFO Signed narinfos id=50 count=1 lb # [ 165.762571] niks3-server[1182]: 2026/09/22 10:59:36 INFO Registered completed upload object_key=log/ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv lb # [ 165.774421] niks3-server[1182]: 2026/09/22 10:59:36 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.ls lb # [ 165.782898] niks3-server[1182]: 2026/09/22 10:59:36 INFO Received complete upload request method=POST path=/api/pending_closures/50/complete lb # [ 165.791002] niks3-server[1182]: 2026/09/22 10:59:36 INFO Registered completed upload object_key=k48d52qj0czpjmzj15yq3fd14zfw5iq6.narinfo lb # [ 165.797185] niks3-server[1182]: 2026/09/22 10:59:36 INFO Registered completed upload object_key=nar/19z3qdigvc4gggg31dgc0wi5w85pgaq9dk7bvngh0yhckpy55jgq.nar.zst lb # [ 165.803236] niks3-server[1182]: 2026/09/22 10:59:36 INFO Completed upload id=50 worker2 # [ 165.823219] nix-grpc-daemon[1330]: time=2026-09-22T10:59:36.282Z level=INFO msg="Upload complete. (346ms)" worker2 # [ 165.837515] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:55550 duration_s=8 drv=ajji8m1zd5q5wi64gp7vclb5dd8hr19m-slow-rs.drv assign_id=4 outputs=1 worker2 # [ 165.857632] nix-daemon[866]: reaped child process 1599, status = succeeded client # [ 165.845766] systemd[1]: rs.service: Deactivated successfully. client # [ 165.850143] systemd[1]: rs.service: Consumed 184ms CPU time over 8.960s wall clock time, 41.2M memory peak, 7.2K incoming IP traffic, 8.3K outgoing IP traffic. worker2 # [ 166.641230] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:37Z level=info event=signal_exit lb # [ 166.668215] niks3-server[1182]: 2026/09/22 10:59:37 INFO lead: released remote=192.168.1.4:47756 worker1 # [ 167.706036] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:38Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 167.693578] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:38Z level=info event=scheduler_restarting worker2 # [ 167.698792] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:38Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 167.699012] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:38Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:37456 worker2 # [ 167.699147] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:38Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 167.699260] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:38Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:37456 worker2 # [ 167.699368] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:38Z level=info event=worker_gone addr=192.168.1.3:50051 worker2 # [ 167.725105] nix-daemon[866]: reaped child process 1322, status = succeeded worker2 # [ 167.749210] nix-grpc-daemon[1270]: E0000 00:00:1790074778.208116 1315 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 17, started true, shutdown true worker1 # [ 167.905748] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:38Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 167.911171] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 167.914239] nix-grpc-daemon[1270]: ts=2026-09-22T10:59:38Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker1 # [ 168.111881] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:38Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 168.117046] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 168.317820] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:38Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 168.322933] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 168.523874] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:38Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 168.529034] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:38Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon | grep -q 'event=scheduler_restarting addr=', in 3.30 seconds) client: waiting for success: ! systemctl is-active rs worker2 # [ 168.692293] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 168.695498] systemd[1]: Stopped Nix worker-protocol over gRPC. worker2 # [ 168.702690] systemd[1]: nix-grpc-daemon.service: Consumed 3.087s CPU time over 42.277s wall clock time, 44.2M memory peak, 83.5K incoming IP traffic, 174.4K outgoing IP traffic. worker2 # [ 168.716282] systemd[1]: Starting scheduler-token.service... worker1 # [ 168.731762] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:39Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 168.736169] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" client: (finished: waiting for success: ! systemctl is-active rs, in 0.07 seconds) client: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; } client: (finished: must succeed: systemctl show -p Result --value rs | grep -qx success || { journalctl -u rs >&2; false; }, in 0.05 seconds) client: must fail: journalctl -u rs | grep -q 'reconnecting' worker2 # [ 168.814366] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 168.815694] systemd[1]: Finished scheduler-token.service. worker2 # [ 168.817136] systemd[1]: scheduler-token.service: Consumed 39ms CPU time over 95ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 168.821167] systemd[1]: Starting Nix worker-protocol over gRPC... client: (finished: must fail: journalctl -u rs | grep -q 'reconnecting', in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters worker1 # [ 168.932508] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:39Z level=info event=scheduler_connected addr=lb:50051 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker1 # [ 168.937767] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 168.994872] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:39Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 168.997550] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:39Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 169.001601] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:39Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 169.008057] nix-grpc-daemon[1655]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 169.017794] nix-grpc-daemon[1655]: E0000 00:00:1790074779.467182 1655 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 169.023678] nix-grpc-daemon[1655]: E0000 00:00:1790074779.470765 1655 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 169.026721] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:39Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 169.029561] systemd[1]: Started Nix worker-protocol over gRPC. lb # [ 169.007046] niks3-server[1182]: 2026/09/22 10:59:39 INFO lead: acquired remote=192.168.1.4:42384 worker2 # [ 169.030509] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:39Z level=info event=scheduler_take_over worker2 # [ 169.036068] nix-grpc-daemon[1655]: E0000 00:00:1790074779.492488 1695 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 169.060055] nix-grpc-daemon[1655]: E0000 00:00:1790074779.509910 1695 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 169.062398] nix-grpc-daemon[1655]: E0000 00:00:1790074779.510056 1695 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 169.070949] nix-grpc-daemon[1655]: E0000 00:00:1790074779.529456 1695 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 169.083391] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 169.085046] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 169.138488] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:39Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 169.143730] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 169.286628] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 169.291569] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 169.344585] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:39Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 169.349766] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 169.492853] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:39Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 169.498010] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:39Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker1 # [ 169.550498] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:40Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 169.555609] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:40Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 169.698888] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:40Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 169.714587] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:40Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 169.719134] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:40Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:48502 worker2 # [ 169.724652] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:40Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker1 # [ 169.757861] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:40Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 169.741476] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:40Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:48502 worker2 # [ 169.746395] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:40Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) worker2: must succeed: curl -sf http://127.0.0.1:9464/metrics worker2: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) (finished: subtest: clean scheduler restart: peers are told, running builds survive, no reconnect noise, in 13.24 seconds) subtest: scheduler restart mid-build: build finishes, no double build client: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid client # [ 170.124963] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid. client # Running as unit: mid.service; invocation ID: f7587752f37d42bbb55ae726def577a2 client: (finished: must succeed: systemd-run --unit mid nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag mid, in 0.09 seconds) worker2 # [ 170.418153] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:40Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:48514 worker2 # [ 170.431340] nix-daemon[866]: accepted connection from pid 1655, user nix-grpc-daemon (trusted) client # [ 170.415916] nix[1395]: don't know how to build these paths: client # [ 170.417093] nix[1395]: /nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker2 # [ 170.463509] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:40Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=0 worker2 # [ 170.470736] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:40Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 170.508542] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:40Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48514 worker2 # [ 170.525702] nix-grpc-daemon[1655]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 20 ms (attempt 1/5) worker2 # [ 170.553212] nix-grpc-daemon[1655]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 23 ms (attempt 2/5) worker2 # [ 170.583779] nix-grpc-daemon[1655]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 364 ms (attempt 3/5) worker2 # [ 170.948298] nix-grpc-daemon[1655]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 124 ms (attempt 4/5) worker2 # [ 171.073903] nix-grpc-daemon[1655]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 171.080612] nix-grpc-daemon[1655]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 171.098592] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:41Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:48514 client # [ 171.078388] nix[1395]: copying 1 paths to 'lb:50051'... worker2 # [ 171.117755] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:41Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48514 worker2 # [ 171.132337] nix-daemon[866]: accepted connection from pid 1655, user nix-grpc-daemon (trusted) worker2 # [ 171.161310] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:41Z level=info event=niks3_push_started pid=1724 worker2 # [ 171.376377] nix-grpc-daemon[1724]: time=2026-09-22T10:59:41.834Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca="" worker2 # [ 171.563934] nix-daemon[866]: accepted connection from pid 1737, user nix-grpc-daemon (trusted) worker2 # [ 171.579700] nix-daemon[866]: reaped child process 1744, status = succeeded client # [ 171.084227] nix[1395]: copying path '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' to 'lb:50051'... worker2 # [ 171.755790] nix-daemon[866]: accepted connection from pid 1746, user nix-grpc-daemon (trusted) worker2 # [ 171.771134] nix-daemon[866]: reaped child process 1749, status = succeeded lb # [ 171.767515] niks3-server[1182]: 2026/09/22 10:59:42 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 171.804067] nix-grpc-daemon[1724]: time=2026-09-22T10:59:42.262Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 171.805561] nix-grpc-daemon[1724]: time=2026-09-22T10:59:42.262Z level=INFO msg="Uploading chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv (424B)" lb # [ 171.813948] niks3-server[1182]: 2026/09/22 10:59:42 INFO Registered completed upload object_key=nar/14lxjrdcxwbhc1kgg39y29gnbx5cz80j4852d1i4w3rvq7g4b04g.nar.zst worker2 # [ 171.844064] nix-grpc-daemon[1724]: time=2026-09-22T10:59:42.302Z level=INFO msg="Uploading 1 narinfos" lb # [ 171.829373] niks3-server[1182]: 2026/09/22 10:59:42 INFO Received sign narinfos request method=POST path=/api/pending_closures/51/sign lb # [ 171.839893] niks3-server[1182]: 2026/09/22 10:59:42 INFO Signed narinfos id=51 count=1 lb # [ 171.850162] niks3-server[1182]: 2026/09/22 10:59:42 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.ls lb # [ 171.855188] niks3-server[1182]: 2026/09/22 10:59:42 INFO Received complete upload request method=POST path=/api/pending_closures/51/complete lb # [ 171.860043] niks3-server[1182]: 2026/09/22 10:59:42 INFO Completed upload id=51 worker2 # [ 171.879242] nix-grpc-daemon[1724]: time=2026-09-22T10:59:42.338Z level=INFO msg="Upload complete. (499ms)" lb # [ 171.862179] niks3-server[1182]: 2026/09/22 10:59:42 INFO Registered completed upload object_key=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc.narinfo worker2 # [ 171.890335] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:42Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:48514 duration_s=0 paths=1 nar_bytes_in=424 worker2 # [ 171.903706] nix-grpc-daemon[1655]: ts=2026-09-22T10:59:42Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:48514 worker2 # [ 171.911967] nix-daemon[866]: accepted connection from pid 1655, user nix-grpc-daemon (trusted) worker2 # [ 171.914070] nix-daemon[866]: reaped child process 1722, status = succeeded client # [ 171.891624] nix[1395]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051... worker2 # [ 171.927185] nix-daemon[866]: accepted connection from pid 1655, user nix-grpc-daemon (trusted) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) worker2: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service worker2 # [ 174.165452] systemd[1]: nix-grpc-daemon.service: Sent signal SIGKILL to main process 1655 (nix-grpc-daemon) on client request. worker2 # [ 174.167025] systemd[1]: nix-grpc-daemon.service: Killed unit cgroup '/system.slice/nix-grpc-daemon.service' with SIGKILL on client request. lb # [ 174.154058] niks3-server[1182]: 2026/09/22 10:59:44 INFO lead: released remote=192.168.1.4:42384 worker2 # [ 174.190809] systemd[1]: Starting scheduler-token.service... client # [ 174.156662] nix[1395]: chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), asking the scheduler again worker1 # [ 174.206203] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:44Z level=info event=scheduler_disconnected addr=lb:50051 error="upstream connect error or disconnect/reset before headers. reset reason: connection termination" client # [ 174.163142] nix[1395]: scheduler at lb:50051: Stream removed (RST_STREAM (Received RST_STREAM with error code 2)), reconnecting worker2 # [ 174.207044] systemd[1]: nix-grpc-daemon.service: Main process exited, code=killed, status=9/KILL worker2 # [ 174.212509] systemd[1]: nix-grpc-daemon.service: Failed with result 'signal'. worker2 # [ 174.213783] systemd[1]: nix-grpc-daemon.service: Consumed 515ms CPU time over 5.381s wall clock time, 34.1M memory peak, 13.9K incoming IP traffic, 21.1K outgoing IP traffic. worker2 # [ 174.230470] nix-daemon[866]: reaped child process 1714, status = succeeded worker2 # [ 174.231913] nix-daemon[866]: reaped child process 1751, status = succeeded lb # [ 174.214261] envoy[933]: conn peer=[2001:db8:1::1]:57544 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2966 tx=1518 ms=8665 worker2 # [ 174.240564] nix-daemon[1753]: unexpected Nix daemon error: error: interrupted by the user lb # [ 174.219392] envoy[933]: conn peer=[2001:db8:1::4]:49774 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=21617 tx=1696 ms=41213 lb # [ 174.224627] envoy[933]: conn peer=[2001:db8:1::4]:46340 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=3384 tx=383 ms=5112 worker2 # [ 174.248132] nix-daemon[866]: reaped child process 1753, status = failed with exit code 1 worker2 # [ 174.295693] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 174.297752] systemd[1]: Finished scheduler-token.service. worker2 # [ 174.298509] systemd[1]: scheduler-token.service: Consumed 24ms CPU time over 105ms wall clock time, 2.2M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 174.302777] systemd[1]: nix-grpc-daemon.service: Scheduled restart job immediately on client request, restart counter is at 1. worker2 # [ 174.306368] systemd[1]: Starting scheduler-token.service... worker2 # [ 174.378125] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 174.381777] systemd[1]: Finished scheduler-token.service. worker2 # [ 174.382565] systemd[1]: scheduler-token.service: Consumed 22ms CPU time over 72ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. worker2 # [ 174.386276] systemd[1]: Starting Nix worker-protocol over gRPC... worker1 # [ 174.406091] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:44Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 174.554335] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:45Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 174.556960] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:45Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 174.561039] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:45Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 174.568064] nix-grpc-daemon[1796]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 174.580175] nix-grpc-daemon[1796]: E0000 00:00:1790074785.026761 1796 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 174.586416] nix-grpc-daemon[1796]: E0000 00:00:1790074785.030405 1796 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. lb # [ 174.568330] niks3-server[1182]: 2026/09/22 10:59:45 INFO lead: acquired remote=192.168.1.4:47354 worker2 # [ 174.589964] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:45Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2: (finished: must succeed: systemctl kill -s KILL nix-grpc-daemon.service; systemctl start nix-grpc-daemon.service, in 0.48 seconds) worker1 # [ 174.604789] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:45Z level=info event=scheduler_disconnected addr=lb:50051 error="scheduler passive, another node serves" client: must succeed: sleep 5.0 worker2 # [ 174.591962] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 174.593840] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:45Z level=info event=scheduler_take_over worker2 # [ 174.596102] nix-grpc-daemon[1796]: E0000 00:00:1790074785.054932 1836 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 174.624061] nix-grpc-daemon[1796]: E0000 00:00:1790074785.072377 1836 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 174.626401] nix-grpc-daemon[1796]: E0000 00:00:1790074785.072524 1836 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 174.633343] nix-grpc-daemon[1796]: E0000 00:00:1790074785.091786 1836 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 174.645788] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 174.651616] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:45Z level=info event=oidc_jwks provider=mock keys=1 worker2 # [ 174.653194] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:45Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:53560 worker2 # [ 174.654859] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:45Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 worker2 # [ 174.697200] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:45Z level=debug event=want client=1 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv cached=0 worker2 # [ 174.699416] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:45Z level=debug event=assigned drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 174.756055] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:45Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53576 worker2 # [ 174.767608] nix-daemon[866]: accepted connection from pid 1796, user nix-grpc-daemon (trusted) worker2 # [ 174.784195] nix-daemon[866]: accepted connection from pid 1796, user nix-grpc-daemon (trusted) worker1 # [ 174.805794] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:45Z level=info event=scheduler_connected addr=lb:50051 worker2 # [ 174.790579] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:45Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:53560 worker2 # [ 174.797460] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:45Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 client # [ 174.765516] nix[1395]: building '/nix/store/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv' on lb:50051... worker2 # [ 174.805268] nix-daemon[866]: accepted connection from pid 1796, user nix-grpc-daemon (trusted) client: (finished: must succeed: sleep 5.0, in 5.06 seconds) client: waiting for success: ! systemctl is-active mid worker2 # [ 180.022879] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:50Z level=info event=niks3_push_started pid=1879 worker2 # [ 180.041429] nix-daemon[866]: reaped child process 1854, status = succeeded worker2 # [ 180.224075] nix-grpc-daemon[1879]: time=2026-09-22T10:59:50.682Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca="" worker2 # [ 180.399810] nix-daemon[866]: accepted connection from pid 1892, user nix-grpc-daemon (trusted) worker2 # [ 180.417523] nix-daemon[866]: reaped child process 1895, status = succeeded lb # [ 180.411066] niks3-server[1182]: 2026/09/22 10:59:50 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 180.442681] nix-grpc-daemon[1879]: time=2026-09-22T10:59:50.900Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 180.445484] nix-grpc-daemon[1879]: time=2026-09-22T10:59:50.900Z level=INFO msg="Uploading fk2vdwmn2msb3ja19s2jwak35ignz889-slow-mid (120B)" lb # [ 180.450787] niks3-server[1182]: 2026/09/22 10:59:50 INFO Registered completed upload object_key=log/chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv lb # [ 180.463770] niks3-server[1182]: 2026/09/22 10:59:50 INFO Registered completed upload object_key=nar/0462jb7zwy2d73k3q6v09xnk3bx318v8ik5ckmp32w2hzvwj4k2d.nar.zst lb # [ 180.478669] niks3-server[1182]: 2026/09/22 10:59:50 INFO Received sign narinfos request method=POST path=/api/pending_closures/52/sign worker2 # [ 180.506149] nix-grpc-daemon[1879]: time=2026-09-22T10:59:50.963Z level=INFO msg="Uploading 1 narinfos" lb # [ 180.488569] niks3-server[1182]: 2026/09/22 10:59:50 INFO Signed narinfos id=52 count=1 lb # [ 180.497469] niks3-server[1182]: 2026/09/22 10:59:50 INFO Registered completed upload object_key=fk2vdwmn2msb3ja19s2jwak35ignz889.ls lb # [ 180.507157] niks3-server[1182]: 2026/09/22 10:59:50 INFO Received complete upload request method=POST path=/api/pending_closures/52/complete lb # [ 180.516579] niks3-server[1182]: 2026/09/22 10:59:50 INFO Completed upload id=52 worker2 # [ 180.536517] nix-grpc-daemon[1879]: time=2026-09-22T10:59:50.995Z level=INFO msg="Upload complete. (308ms)" worker2 # [ 180.552079] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:51Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53576 duration_s=5 drv=chbb3scb8b4b0sd5wk0a0vcrl8x07pdc-slow-mid.drv assign_id=1 outputs=1 worker2 # [ 180.567216] nix-daemon[866]: reaped child process 1852, status = succeeded client # [ 180.557632] systemd[1]: mid.service: Deactivated successfully. client # [ 180.561892] systemd[1]: mid.service: Consumed 191ms CPU time over 10.437s wall clock time, 37.2M memory peak, 8.7K incoming IP traffic, 10.4K outgoing IP traffic. client: (finished: waiting for success: ! systemctl is-active mid, in 1.14 seconds) client: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; } client: (finished: must succeed: systemctl show -p Result --value mid | grep -qx success || { journalctl -u mid >&2; false; }, in 0.05 seconds) (finished: subtest: scheduler restart mid-build: build finishes, no double build, in 11.07 seconds) subtest: interrupting the client stops the build on the worker client: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr client # [ 181.168289] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr. client # Running as unit: intr.service; invocation ID: 0248c014c0c342a0afffb908baca19ce client: (finished: must succeed: systemd-run --unit intr nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag intr, in 0.05 seconds) worker1 # [ 181.469581] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:51Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 181.485295] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) client # [ 181.454997] nix[1431]: don't know how to build these paths: client # [ 181.456841] nix[1431]: /nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker2 # [ 181.498605] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:51Z level=debug event=want client=2 drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv cached=0 worker2 # [ 181.504628] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:51Z level=debug event=assigned drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 181.561065] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:52Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 181.584220] nix-grpc-daemon[1394]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 33 ms (attempt 1/5) worker1 # [ 181.624687] nix-grpc-daemon[1394]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 112 ms (attempt 2/5) worker1 # [ 181.744312] nix-grpc-daemon[1394]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 388 ms (attempt 3/5) worker1 # [ 182.132716] nix-grpc-daemon[1394]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 376 ms (attempt 4/5) worker1 # [ 182.510574] nix-grpc-daemon[1394]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker1 # [ 182.517181] nix-grpc-daemon[1394]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 182.536299] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:53Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:60204 client # [ 182.501233] nix[1431]: copying 1 paths to 'lb:50051'... worker1 # [ 182.551717] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:53Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 client # [ 182.506605] nix[1431]: copying path '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' to 'lb:50051'... worker1 # [ 182.567762] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 182.595033] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:53Z level=info event=niks3_push_started pid=1546 worker1 # [ 182.803238] nix-grpc-daemon[1546]: time=2026-09-22T10:59:53.274Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker1.key ca="" worker1 # [ 182.978273] nix-daemon[844]: accepted connection from pid 1559, user nix-grpc-daemon (trusted) worker1 # [ 182.995419] nix-daemon[844]: reaped child process 1562, status = succeeded worker1 # [ 183.171725] nix-daemon[844]: accepted connection from pid 1564, user nix-grpc-daemon (trusted) worker1 # [ 183.187548] nix-daemon[844]: reaped child process 1567, status = succeeded lb # [ 183.166843] niks3-server[1182]: 2026/09/22 10:59:53 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 183.219891] nix-grpc-daemon[1546]: time=2026-09-22T10:59:53.689Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 183.221673] nix-grpc-daemon[1546]: time=2026-09-22T10:59:53.689Z level=INFO msg="Uploading 85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv (424B)" lb # [ 183.228597] niks3-server[1182]: 2026/09/22 10:59:53 INFO Registered completed upload object_key=nar/0h460h76prjyvhqhyxlhz1xrmh3s520jbd19gaagy4hp4vc92r1p.nar.zst worker1 # [ 183.278460] nix-grpc-daemon[1546]: time=2026-09-22T10:59:53.749Z level=INFO msg="Uploading 1 narinfos" lb # [ 183.241762] niks3-server[1182]: 2026/09/22 10:59:53 INFO Received sign narinfos request method=POST path=/api/pending_closures/53/sign lb # [ 183.250117] niks3-server[1182]: 2026/09/22 10:59:53 INFO Signed narinfos id=53 count=1 lb # [ 183.255197] niks3-server[1182]: 2026/09/22 10:59:53 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.ls lb # [ 183.268198] niks3-server[1182]: 2026/09/22 10:59:53 INFO Received complete upload request method=POST path=/api/pending_closures/53/complete lb # [ 183.272471] niks3-server[1182]: 2026/09/22 10:59:53 INFO Completed upload id=53 worker1 # [ 183.309337] nix-grpc-daemon[1546]: time=2026-09-22T10:59:53.780Z level=INFO msg="Upload complete. (501ms)" lb # [ 183.276856] niks3-server[1182]: 2026/09/22 10:59:53 INFO Registered completed upload object_key=85wa5czf6m5ng93xyq44r1affb74csh8.narinfo worker1 # [ 183.320365] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:53Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=0 paths=1 nar_bytes_in=424 worker1 # [ 183.337020] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:53Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 183.349863] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 183.353726] nix-daemon[844]: reaped child process 1544, status = succeeded client # [ 183.315214] nix[1431]: building '/nix/store/85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv' on lb:50051... worker1 # [ 183.369847] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) lb # [ 184.215929] envoy[933]: conn peer=[2001:db8:1::1]:50556 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3940 tx=1942 ms=10150 client: must succeed: systemctl kill -s INT intr worker1 # [ 184.933141] nix-grpc-daemon[1394]: ts=2026-09-22T10:59:55Z level=info event=revoked_running drv=85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv client # [ 184.881256] systemd[1]: intr.service: Sent signal SIGINT to main process 1431 (nix) on client request. client # [ 184.888847] nix[1431]: shutting down85wa5czf6m5ng93xyq44r1affb74csh8-slow-intr.drv: error: gRPC BuildDerivation on 'lb:50051' failed: Stream removed (CANCELLED) client # [ 184.900639] nix[1431]: error: interrupted by the user client: (finished: must succeed: systemctl kill -s INT intr, in 0.08 seconds) client # [ 184.933608] systemd[1]: intr.service: Main process exited, code=exited, status=1/FAILURE client # [ 184.937666] systemd[1]: intr.service: Failed with result 'exit-code'. client # [ 184.939636] systemd[1]: intr.service: Consumed 159ms CPU time over 3.768s wall clock time, 39.2M memory peak, 6.5K incoming IP traffic, 8.4K outgoing IP traffic. worker1 # [ 185.012676] nix-daemon[844]: reaped child process 1569, status = succeeded worker1 # [ 185.018239] nix-daemon[1571]: unexpected Nix daemon error: error: interrupted by the user worker1 # [ 185.026617] nix-daemon[844]: reaped child process 1571, status = failed with exit code 1 (finished: subtest: interrupting the client stops the build on the worker, in 3.94 seconds) subtest: a deploy mid-build drains: the build finishes, the new generation takes over client: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw client # [ 185.129597] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key" --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw. client # Running as unit: sw.service; invocation ID: f28d13f892e84d8f976b8fd05b8404ce client: (finished: must succeed: systemd-run --unit sw nix build --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/mpvagzcaddzb5j0jgpk1hnp5dkckqbn5-slow.nix --argstr tag sw, in 0.09 seconds) client # [ 185.383750] nix[1452]: don't know how to build these paths: client # [ 185.384920] nix[1452]: /nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker2 # [ 185.415859] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:55Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53576 worker2 # [ 185.429545] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:55Z level=debug event=want client=3 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv cached=0 worker2 # [ 185.436587] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:55Z level=debug event=assigned drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 185.476158] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:55Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53576 worker2 # [ 185.493312] nix-grpc-daemon[1796]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 88 ms (attempt 1/5) worker2 # [ 185.588879] nix-grpc-daemon[1796]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 122 ms (attempt 2/5) worker2 # [ 185.718460] nix-grpc-daemon[1796]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 119 ms (attempt 3/5) worker2 # [ 185.845121] nix-grpc-daemon[1796]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 524 ms (attempt 4/5) worker2 # [ 186.376919] nix-grpc-daemon[1796]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 186.382724] nix-grpc-daemon[1796]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker2 # [ 186.401985] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:56Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:53576 client # [ 186.385253] nix[1452]: copying 1 paths to 'lb:50051'... worker2 # [ 186.418619] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:56Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53576 client # [ 186.385567] nix[1452]: copying path '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' to 'lb:50051'... worker2 # [ 186.442299] nix-daemon[866]: accepted connection from pid 1796, user nix-grpc-daemon (trusted) worker2 # [ 186.656974] nix-daemon[866]: accepted connection from pid 1935, user nix-grpc-daemon (trusted) worker2 # [ 186.676734] nix-daemon[866]: reaped child process 1938, status = succeeded worker2 # [ 186.857674] nix-daemon[866]: accepted connection from pid 1940, user nix-grpc-daemon (trusted) worker2 # [ 186.874687] nix-daemon[866]: reaped child process 1943, status = succeeded lb # [ 186.867947] niks3-server[1182]: 2026/09/22 10:59:57 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 186.899199] nix-grpc-daemon[1879]: time=2026-09-22T10:59:57.357Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 186.900777] nix-grpc-daemon[1879]: time=2026-09-22T10:59:57.357Z level=INFO msg="Uploading x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv (416B)" lb # [ 186.898809] niks3-server[1182]: 2026/09/22 10:59:57 INFO Registered completed upload object_key=nar/0jczp45bkr8vp1ymznxf6da1iv8dzh42pnl36wiprbzwhpi2ddia.nar.zst lb # [ 186.915616] niks3-server[1182]: 2026/09/22 10:59:57 INFO Received sign narinfos request method=POST path=/api/pending_closures/54/sign worker2 # [ 186.941073] nix-grpc-daemon[1879]: time=2026-09-22T10:59:57.399Z level=INFO msg="Uploading 1 narinfos" lb # [ 186.927936] niks3-server[1182]: 2026/09/22 10:59:57 INFO Signed narinfos id=54 count=1 lb # [ 186.934793] niks3-server[1182]: 2026/09/22 10:59:57 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.ls lb # [ 186.944286] niks3-server[1182]: 2026/09/22 10:59:57 INFO Received complete upload request method=POST path=/api/pending_closures/54/complete lb # [ 186.950254] niks3-server[1182]: 2026/09/22 10:59:57 INFO Completed upload id=54 lb # [ 186.952832] niks3-server[1182]: 2026/09/22 10:59:57 INFO Registered completed upload object_key=x0m4asg5xx7vgj4dip2crwn6zhxqngvs.narinfo worker2 # [ 186.969216] nix-grpc-daemon[1879]: time=2026-09-22T10:59:57.428Z level=INFO msg="Upload complete. (497ms)" worker2 # [ 186.980741] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:57Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:53576 duration_s=0 paths=1 nar_bytes_in=416 worker2 # [ 186.991640] nix-grpc-daemon[1796]: ts=2026-09-22T10:59:57Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53576 worker2 # [ 186.998889] nix-daemon[866]: accepted connection from pid 1796, user nix-grpc-daemon (trusted) worker2 # [ 187.003692] nix-daemon[866]: reaped child process 1932, status = succeeded client # [ 186.980345] nix[1452]: building '/nix/store/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv' on lb:50051... worker2 # [ 187.016992] nix-daemon[866]: accepted connection from pid 1796, user nix-grpc-daemon (trusted) worker2: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2 worker2 # Checking switch inhibitors... done worker2 # activating the configuration... worker2 # [ 189.834261] nixos[1979]: switching to system configuration /nix/store/3y6vfndpcj9aa9jcwmk2jinamvf8rfcl-nixos-system-worker2-test worker2 # [ 189.840138] systemd[1]: Stopped target Remote File Systems. worker2 # [ 189.840933] systemd[1]: Stopped target Local File Systems. worker2 # setting up /etc... worker2 # [ 190.613272] systemd[1]: Reload requested from client PID 1979 ('.switch-to-conf') (unit backdoor.service)... worker2 # [ 190.615126] systemd[1]: Reloading... worker2 # [ 191.516517] systemd[1]: Reloading finished in 897 ms. worker2 # restarting sysinit-reactivation.target worker2 # [ 191.597915] systemd[1]: Starting Re-setup tmpfiles on a system that is already running.... worker2 # [ 191.749852] systemd[1]: Finished Re-setup tmpfiles on a system that is already running.. worker2 # [ 191.751041] systemd[1]: Reached target Reactivate sysinit units. worker2 # reloading the following units: dbus-broker.service, nix-grpc-daemon.service worker2 # [ 191.760251] dbus-broker-launch[503]: Caught SIGHUP, trigger reload. worker2 # [ 191.768968] systemd[1]: Reloading D-Bus System Message Bus... worker2 # [ 191.771410] systemd[1]: Reloading Nix worker-protocol over gRPC... worker2 # [ 191.786685] dbus-broker-launch[503]: Looking up NSS user entry for 'systemd-timesync'... worker2 # [ 191.792404] dbus-broker-launch[503]: NSS returned no entry for 'systemd-timesync' worker2 # [ 191.802198] dbus-broker-launch[503]: Invalid user-name in /nix/store/602n2r0g7ngz2pj9sxi9qamwpihwymx6-system-path/share/dbus-1/system.d/org.freedesktop.timesync1.conf +16: user="systemd-timesync" worker2 # [ 191.817178] systemd[1]: Reloaded Nix worker-protocol over gRPC. worker2 # [ 191.849809] systemd[1]: Reloaded D-Bus System Message Bus. worker2 # [ 191.855759] systemd[1]: Reached target Remote File Systems. worker2 # [ 191.858942] systemd[1]: Starting linger-users.service... worker2 # [ 191.864398] systemd[1]: Reached target Local File Systems. worker2 # [ 191.868497] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore worker2 # [ 191.880461] systemd[1]: Starting Create SUID/SGID Wrappers... worker2 # [ 191.883419] systemd[1]: Update Boot Loader Random Seed skipped, no trigger condition checks were met. worker2 # [ 191.892180] systemd[1]: Clear Stale Hibernate Storage Info skipped, unmet condition check ConditionPathExists=/sys/firmware/efi/efivars/HibernateLocation-8cf2644b-4b0b-428f-9387-6d876050dc67 worker2 # [ 191.900264] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore worker2 # [ 191.903536] systemd[1]: Early TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 191.908625] systemd[1]: TPM SRK Setup skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 191.911621] systemd[1]: TPM PCR Measurements skipped, unmet condition check ConditionSecurity=measured-os worker2 # [ 191.916681] systemd[1]: Make TPM PCR Policy skipped, unmet condition check ConditionSecurity=measured-uki worker2 # [ 191.950146] systemd[1]: linger-users.service: Deactivated successfully. worker2 # [ 191.951094] systemd[1]: Finished linger-users.service. worker2 # [ 192.406588] systemd[1]: suid-sgid-wrappers.service: Deactivated successfully. worker2 # [ 192.407672] systemd[1]: Finished Create SUID/SGID Wrappers. worker2 # [ 192.580078] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:03Z level=info event=draining worker2 # the following new units were started: sysinit-reactivation.target, systemd-tmpfiles-resetup.service worker2 # [ 192.663335] nixos[1979]: finished switching to system configuration /nix/store/3y6vfndpcj9aa9jcwmk2jinamvf8rfcl-nixos-system-worker2-test worker2: (finished: must succeed: timeout 20 /run/current-system/specialisation/next/bin/switch-to-configuration test >&2, in 3.68 seconds) worker2: must succeed: pgrep -f 'read -t [9]0 x' worker2: (finished: must succeed: pgrep -f 'read -t [9]0 x', in 0.06 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain >&2 worker2 # [ 192.999304] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:03Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:53576 client # don't know how to build these paths: client # /nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker1 # [ 193.025357] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:60204 worker2 # [ 193.032462] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:03Z level=debug event=want client=4 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv cached=0 worker2 # [ 193.034357] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:03Z level=debug event=assigned drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 193.101822] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:03Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 193.116537] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:03Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:60204 client # copying 1 paths to 'lb:50051'... worker1 # [ 193.132405] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:03Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 client # copying path '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' to 'lb:50051'... worker1 # [ 193.157163] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 193.374192] nix-daemon[844]: accepted connection from pid 1622, user nix-grpc-daemon (trusted) worker1 # [ 193.393052] nix-daemon[844]: reaped child process 1625, status = succeeded worker1 # [ 193.571964] nix-daemon[844]: accepted connection from pid 1627, user nix-grpc-daemon (trusted) worker1 # [ 193.589772] nix-daemon[844]: reaped child process 1630, status = succeeded lb # [ 193.566816] niks3-server[1182]: 2026/09/22 11:00:04 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 193.614570] nix-grpc-daemon[1546]: time=2026-09-22T11:00:04.085Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 193.616160] nix-grpc-daemon[1546]: time=2026-09-22T11:00:04.085Z level=INFO msg="Uploading k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv (528B)" lb # [ 193.595095] niks3-server[1182]: 2026/09/22 11:00:04 INFO Registered completed upload object_key=nar/1h3fzlmfdwdiqr732wwrq7jnpw7n6p8f9qx1089c6g6x2i4csv75.nar.zst lb # [ 193.609590] niks3-server[1182]: 2026/09/22 11:00:04 INFO Received sign narinfos request method=POST path=/api/pending_closures/55/sign worker1 # [ 193.653316] nix-grpc-daemon[1546]: time=2026-09-22T11:00:04.124Z level=INFO msg="Uploading 1 narinfos" lb # [ 193.622688] niks3-server[1182]: 2026/09/22 11:00:04 INFO Signed narinfos id=55 count=1 lb # [ 193.630012] niks3-server[1182]: 2026/09/22 11:00:04 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.ls lb # [ 193.641565] niks3-server[1182]: 2026/09/22 11:00:04 INFO Received complete upload request method=POST path=/api/pending_closures/55/complete lb # [ 193.646334] niks3-server[1182]: 2026/09/22 11:00:04 INFO Completed upload id=55 worker1 # [ 193.683047] nix-grpc-daemon[1546]: time=2026-09-22T11:00:04.154Z level=INFO msg="Upload complete. (509ms)" lb # [ 193.649080] niks3-server[1182]: 2026/09/22 11:00:04 INFO Registered completed upload object_key=k3l35rf796jrnjns08398p1l8fcx178a.narinfo worker1 # [ 193.693381] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:04Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=0 paths=1 nar_bytes_in=528 worker1 # [ 193.706766] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:04Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 193.713314] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 193.718543] nix-daemon[844]: reaped child process 1620, status = succeeded client # building '/nix/store/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv' on lb:50051... client # farm-a> node-a: building k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv worker1 # [ 193.732398] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) lb # [ 194.215049] envoy[933]: conn peer=[2001:db8:1::1]:47728 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=3013 tx=851 ms=3503 client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-during-drain worker1 # [ 195.160385] nix-daemon[844]: reaped child process 1634, status = succeeded worker1 # [ 195.322015] nix-daemon[844]: accepted connection from pid 1650, user nix-grpc-daemon (trusted) worker1 # [ 195.340849] nix-daemon[844]: reaped child process 1653, status = succeeded lb # [ 195.317729] niks3-server[1182]: 2026/09/22 11:00:05 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 195.367270] nix-grpc-daemon[1546]: time=2026-09-22T11:00:05.835Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 195.369513] nix-grpc-daemon[1546]: time=2026-09-22T11:00:05.835Z level=INFO msg="Uploading 42c24pndjip9nvdj8j25mkv0650griqf-farm-a (136B)" lb # [ 195.351877] niks3-server[1182]: 2026/09/22 11:00:05 INFO Registered completed upload object_key=log/k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv lb # [ 195.368962] niks3-server[1182]: 2026/09/22 11:00:05 INFO Registered completed upload object_key=nar/0r0px9vw070lid3q4jhnmka11sk2y8kj26ysyvjy9dplavd7mdz4.nar.zst worker1 # [ 195.418621] nix-grpc-daemon[1546]: time=2026-09-22T11:00:05.888Z level=INFO msg="Uploading 1 narinfos" lb # [ 195.383142] niks3-server[1182]: 2026/09/22 11:00:05 INFO Received sign narinfos request method=POST path=/api/pending_closures/56/sign lb # [ 195.391351] niks3-server[1182]: 2026/09/22 11:00:05 INFO Signed narinfos id=56 count=1 worker1 # [ 195.438127] nix-grpc-daemon[1546]: time=2026-09-22T11:00:05.909Z level=INFO msg="Upload complete. (295ms)" lb # [ 195.401719] niks3-server[1182]: 2026/09/22 11:00:05 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.ls lb # [ 195.406218] niks3-server[1182]: 2026/09/22 11:00:05 INFO Received complete upload request method=POST path=/api/pending_closures/56/complete lb # [ 195.411787] niks3-server[1182]: 2026/09/22 11:00:05 INFO Completed upload id=56 worker1 # [ 195.448928] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:05Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=1 drv=k3l35rf796jrnjns08398p1l8fcx178a-farm-a.drv assign_id=4 outputs=1 lb # [ 195.414705] niks3-server[1182]: 2026/09/22 11:00:05 INFO Registered completed upload object_key=42c24pndjip9nvdj8j25mkv0650griqf.narinfo worker2 # [ 195.441122] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:05Z level=debug event=want client=4 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv cached=0 worker2 # [ 195.442939] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:05Z level=debug event=assigned drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 195.462985] nix-daemon[844]: reaped child process 1632, status = succeeded worker1 # [ 195.505693] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:05Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 195.520559] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:05Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 195.535543] nix-grpc-daemon[1546]: time=2026-09-22T11:00:06.006Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... worker1 # [ 195.546802] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:06Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 client # copying path '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' to 'lb:50051'... worker1 # [ 195.569727] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 195.777406] nix-daemon[844]: accepted connection from pid 1657, user nix-grpc-daemon (trusted) worker1 # [ 195.795365] nix-daemon[844]: reaped child process 1660, status = succeeded worker1 # [ 195.969691] nix-daemon[844]: accepted connection from pid 1662, user nix-grpc-daemon (trusted) worker1 # [ 195.987806] nix-daemon[844]: reaped child process 1665, status = succeeded lb # [ 195.963171] niks3-server[1182]: 2026/09/22 11:00:06 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 196.010914] nix-grpc-daemon[1546]: time=2026-09-22T11:00:06.481Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 196.012533] nix-grpc-daemon[1546]: time=2026-09-22T11:00:06.481Z level=INFO msg="Uploading im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv (640B)" lb # [ 196.001413] niks3-server[1182]: 2026/09/22 11:00:06 INFO Registered completed upload object_key=nar/0rxmdnq61ka29n0b2as2hdbz91wi2ap17db890js8r5plhpg436g.nar.zst lb # [ 196.012663] niks3-server[1182]: 2026/09/22 11:00:06 INFO Received sign narinfos request method=POST path=/api/pending_closures/57/sign worker1 # [ 196.055306] nix-grpc-daemon[1546]: time=2026-09-22T11:00:06.526Z level=INFO msg="Uploading 1 narinfos" lb # [ 196.024086] niks3-server[1182]: 2026/09/22 11:00:06 INFO Signed narinfos id=57 count=1 lb # [ 196.026585] niks3-server[1182]: 2026/09/22 11:00:06 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.ls lb # [ 196.035791] niks3-server[1182]: 2026/09/22 11:00:06 INFO Received complete upload request method=POST path=/api/pending_closures/57/complete lb # [ 196.043173] niks3-server[1182]: 2026/09/22 11:00:06 INFO Completed upload id=57 lb # [ 196.046025] niks3-server[1182]: 2026/09/22 11:00:06 INFO Registered completed upload object_key=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l.narinfo worker1 # [ 196.079175] nix-grpc-daemon[1546]: time=2026-09-22T11:00:06.550Z level=INFO msg="Upload complete. (492ms)" worker1 # [ 196.088482] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:06Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=0 paths=1 nar_bytes_in=640 worker1 # [ 196.103065] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:06Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 196.109729] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 196.114721] nix-daemon[844]: reaped child process 1655, status = succeeded client # building '/nix/store/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv' on lb:50051... worker1 # [ 196.126010] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) client # farm-b> node-a: building im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-during-drain worker1 # [ 196.962669] nix-daemon[844]: reaped child process 1669, status = succeeded worker1 # [ 197.132117] nix-daemon[844]: accepted connection from pid 1685, user nix-grpc-daemon (trusted) worker1 # [ 197.150842] nix-daemon[844]: reaped child process 1688, status = succeeded lb # [ 197.126989] niks3-server[1182]: 2026/09/22 11:00:07 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 197.173023] nix-grpc-daemon[1546]: time=2026-09-22T11:00:07.643Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 197.175263] nix-grpc-daemon[1546]: time=2026-09-22T11:00:07.645Z level=INFO msg="Uploading daks7y2c761g04qpzsjx25629wigr8js-farm-b (184B)" lb # [ 197.159274] niks3-server[1182]: 2026/09/22 11:00:07 INFO Registered completed upload object_key=log/im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv lb # [ 197.172978] niks3-server[1182]: 2026/09/22 11:00:07 INFO Registered completed upload object_key=nar/0lnpxl34nrzp147jy5bafczkvxcl00hcf69qjhsljmv2ip9jisfh.nar.zst lb # [ 197.188769] niks3-server[1182]: 2026/09/22 11:00:07 INFO Received sign narinfos request method=POST path=/api/pending_closures/58/sign worker1 # [ 197.231861] nix-grpc-daemon[1546]: time=2026-09-22T11:00:07.702Z level=INFO msg="Uploading 1 narinfos" lb # [ 197.200773] niks3-server[1182]: 2026/09/22 11:00:07 INFO Signed narinfos id=58 count=1 lb # [ 197.207377] niks3-server[1182]: 2026/09/22 11:00:07 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.ls worker1 # [ 197.253737] nix-grpc-daemon[1546]: time=2026-09-22T11:00:07.724Z level=INFO msg="Upload complete. (315ms)" lb # [ 197.217658] niks3-server[1182]: 2026/09/22 11:00:07 INFO Received complete upload request method=POST path=/api/pending_closures/58/complete lb # [ 197.221983] niks3-server[1182]: 2026/09/22 11:00:07 INFO Completed upload id=58 lb # [ 197.226415] niks3-server[1182]: 2026/09/22 11:00:07 INFO Registered completed upload object_key=daks7y2c761g04qpzsjx25629wigr8js.narinfo worker2 # [ 197.257118] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:07Z level=debug event=want client=4 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv cached=0 worker2 # [ 197.259042] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:07Z level=debug event=assigned drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 197.265107] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:07Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=1 drv=im7gh7nqqi41f48wvwjqkgv3nm7w5b4l-farm-b.drv assign_id=5 outputs=1 worker1 # [ 197.285036] nix-daemon[844]: reaped child process 1667, status = succeeded worker1 # [ 197.320631] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:07Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 197.336778] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:07Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 197.351923] nix-grpc-daemon[1546]: time=2026-09-22T11:00:07.823Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' to 'lb:50051'... worker1 # [ 197.362907] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:07Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 197.380830] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 197.583328] nix-daemon[844]: accepted connection from pid 1692, user nix-grpc-daemon (trusted) worker1 # [ 197.600337] nix-daemon[844]: reaped child process 1695, status = succeeded worker1 # [ 197.770351] nix-daemon[844]: accepted connection from pid 1697, user nix-grpc-daemon (trusted) worker1 # [ 197.786273] nix-daemon[844]: reaped child process 1700, status = succeeded lb # [ 197.762468] niks3-server[1182]: 2026/09/22 11:00:08 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 197.808168] nix-grpc-daemon[1546]: time=2026-09-22T11:00:08.279Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 197.810017] nix-grpc-daemon[1546]: time=2026-09-22T11:00:08.280Z level=INFO msg="Uploading lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv (768B)" lb # [ 197.791385] niks3-server[1182]: 2026/09/22 11:00:08 INFO Registered completed upload object_key=nar/066jhl91shf8ar6h3cqqb855j5cswl7170wrg5lzf3fhpka0xywn.nar.zst worker1 # [ 197.840569] nix-grpc-daemon[1546]: time=2026-09-22T11:00:08.311Z level=INFO msg="Uploading 1 narinfos" lb # [ 197.806418] niks3-server[1182]: 2026/09/22 11:00:08 INFO Received sign narinfos request method=POST path=/api/pending_closures/59/sign lb # [ 197.818269] niks3-server[1182]: 2026/09/22 11:00:08 INFO Signed narinfos id=59 count=1 lb # [ 197.827786] niks3-server[1182]: 2026/09/22 11:00:08 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.ls worker1 # [ 197.869665] nix-grpc-daemon[1546]: time=2026-09-22T11:00:08.340Z level=INFO msg="Upload complete. (471ms)" lb # [ 197.833642] niks3-server[1182]: 2026/09/22 11:00:08 INFO Received complete upload request method=POST path=/api/pending_closures/59/complete lb # [ 197.838972] niks3-server[1182]: 2026/09/22 11:00:08 INFO Completed upload id=59 worker1 # [ 197.875213] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:08Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=0 paths=1 nar_bytes_in=768 lb # [ 197.842299] niks3-server[1182]: 2026/09/22 11:00:08 INFO Registered completed upload object_key=lbk8wy49c881163x07gb8z669j2fg1ff.narinfo worker1 # [ 197.887137] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:08Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 197.893228] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 197.897660] nix-daemon[844]: reaped child process 1690, status = succeeded client # building '/nix/store/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv' on lb:50051... worker1 # [ 197.909401] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) client # farm-top> node-a: building lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-during-drain worker1 # [ 199.132583] nix-daemon[844]: reaped child process 1704, status = succeeded worker1 # [ 199.293064] nix-daemon[844]: accepted connection from pid 1720, user nix-grpc-daemon (trusted) worker1 # [ 199.313200] nix-daemon[844]: reaped child process 1723, status = succeeded lb # [ 199.289057] niks3-server[1182]: 2026/09/22 11:00:09 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 199.336385] nix-grpc-daemon[1546]: time=2026-09-22T11:00:09.806Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 199.337835] nix-grpc-daemon[1546]: time=2026-09-22T11:00:09.806Z level=INFO msg="Uploading xdwmx2f4n5n8bggglniz2n35any0xz4w-farm-top (240B)" lb # [ 199.327541] niks3-server[1182]: 2026/09/22 11:00:09 INFO Registered completed upload object_key=log/lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv lb # [ 199.340113] niks3-server[1182]: 2026/09/22 11:00:09 INFO Registered completed upload object_key=nar/1bpy9ga4n2446pary6msidbs8mnil1a34jn8p8yl2scvdg691mz2.nar.zst worker1 # [ 199.391949] nix-grpc-daemon[1546]: time=2026-09-22T11:00:09.862Z level=INFO msg="Uploading 1 narinfos" lb # [ 199.349806] niks3-server[1182]: 2026/09/22 11:00:09 INFO Received sign narinfos request method=POST path=/api/pending_closures/60/sign lb # [ 199.359778] niks3-server[1182]: 2026/09/22 11:00:09 INFO Signed narinfos id=60 count=1 lb # [ 199.366178] niks3-server[1182]: 2026/09/22 11:00:09 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.ls lb # [ 199.375363] niks3-server[1182]: 2026/09/22 11:00:09 INFO Received complete upload request method=POST path=/api/pending_closures/60/complete lb # [ 199.381065] niks3-server[1182]: 2026/09/22 11:00:09 INFO Completed upload id=60 worker1 # [ 199.417821] nix-grpc-daemon[1546]: time=2026-09-22T11:00:09.889Z level=INFO msg="Upload complete. (299ms)" lb # [ 199.384526] niks3-server[1182]: 2026/09/22 11:00:09 INFO Registered completed upload object_key=xdwmx2f4n5n8bggglniz2n35any0xz4w.narinfo worker1 # [ 199.429159] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:09Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=1 drv=lbk8wy49c881163x07gb8z669j2fg1ff-farm-top.drv assign_id=6 outputs=1 worker1 # [ 199.447172] nix-daemon[844]: reaped child process 1702, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain >&2, in 6.73 seconds) client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag during-drain outPath, in 0.23 seconds) worker2: must succeed: pkill -f 'read -t [9]0 x' worker2: (finished: must succeed: pkill -f 'read -t [9]0 x', in 0.06 seconds) client: waiting for success: ! systemctl is-active sw worker2 # [ 199.863081] nix-daemon[866]: reaped child process 1947, status = succeeded worker2 # [ 200.028180] nix-daemon[866]: accepted connection from pid 2126, user nix-grpc-daemon (trusted) worker2 # [ 200.047873] nix-daemon[866]: reaped child process 2129, status = succeeded lb # [ 200.039695] niks3-server[1182]: 2026/09/22 11:00:10 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 200.070166] nix-grpc-daemon[1879]: time=2026-09-22T11:00:10.528Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 200.071680] nix-grpc-daemon[1879]: time=2026-09-22T11:00:10.528Z level=INFO msg="Uploading qr8vyl2zl3nxnin3vw41c6z35vj7087f-slow-sw (120B)" lb # [ 200.076058] niks3-server[1182]: 2026/09/22 11:00:10 INFO Registered completed upload object_key=log/x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv lb # [ 200.087921] niks3-server[1182]: 2026/09/22 11:00:10 INFO Registered completed upload object_key=nar/0vi6cx31wc05wns04pr5vr2bak9iib1idndvy6gik6c2rz9q8c19.nar.zst lb # [ 200.098286] niks3-server[1182]: 2026/09/22 11:00:10 INFO Received sign narinfos request method=POST path=/api/pending_closures/61/sign lb # [ 200.104771] niks3-server[1182]: 2026/09/22 11:00:10 INFO Signed narinfos id=61 count=1 worker2 # [ 200.125290] nix-grpc-daemon[1879]: time=2026-09-22T11:00:10.583Z level=INFO msg="Uploading 1 narinfos" lb # [ 200.111104] niks3-server[1182]: 2026/09/22 11:00:10 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.ls lb # [ 200.121718] niks3-server[1182]: 2026/09/22 11:00:10 INFO Received complete upload request method=POST path=/api/pending_closures/61/complete lb # [ 200.129265] niks3-server[1182]: 2026/09/22 11:00:10 INFO Completed upload id=61 worker2 # [ 200.148570] nix-grpc-daemon[1879]: time=2026-09-22T11:00:10.607Z level=INFO msg="Upload complete. (306ms)" lb # [ 200.132216] niks3-server[1182]: 2026/09/22 11:00:10 INFO Registered completed upload object_key=qr8vyl2zl3nxnin3vw41c6z35vj7087f.narinfo worker2 # [ 200.159675] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:10Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:53576 duration_s=13 drv=x0m4asg5xx7vgj4dip2crwn6zhxqngvs-slow-sw.drv assign_id=3 outputs=1 worker2 # [ 200.176719] nix-daemon[866]: reaped child process 1945, status = succeeded client # [ 200.169884] systemd[1]: sw.service: Deactivated successfully. client # [ 200.175217] systemd[1]: sw.service: Consumed 179ms CPU time over 15.043s wall clock time, 41.2M memory peak, 7.5K incoming IP traffic, 8.5K outgoing IP traffic. worker2 # [ 200.583123] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:11Z level=info event=signal_exit lb # [ 200.796705] niks3-server[1182]: 2026/09/22 11:00:11 INFO lead: released remote=192.168.1.4:47354 client: (finished: waiting for success: ! systemctl is-active sw, in 1.11 seconds) client: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; } client: (finished: must succeed: systemctl show -p Result --value sw | grep -qx success || { journalctl -u sw >&2; false; }, in 0.06 seconds) worker2: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service worker2 # [ 201.025150] systemd[1]: Starting scheduler-token.service... worker2 # [ 201.109845] systemd[1]: scheduler-token.service: Deactivated successfully. worker2 # [ 201.111258] systemd[1]: Finished scheduler-token.service. worker2 # [ 201.112490] systemd[1]: scheduler-token.service: Consumed 30ms CPU time over 84ms wall clock time, 2.1M memory peak, 925B incoming IP traffic, 452B outgoing IP traffic. lb # [ 201.444271] niks3-server[1182]: 2026/09/22 11:00:11 INFO lead: acquired remote=192.168.1.3:45862 worker1 # [ 201.482422] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:11Z level=info event=scheduler_take_over worker2 # [ 201.809589] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:12Z level=info event=scheduler_restarting worker1 # [ 201.826674] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:12Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 201.820212] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:12Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:53560 worker2 # [ 201.825711] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:12Z level=info event=worker_gone addr=192.168.1.3:50051 worker2 # [ 201.830358] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:12Z level=info event=scheduler_restarting addr=lb:50051 worker2 # [ 201.834666] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:12Z level=info event=worker_session_closed peer=ipv4:192.168.1.2:53560 worker2 # [ 201.839618] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:12Z level=info event=worker_gone addr=192.168.1.4:50051 worker2 # [ 201.864128] nix-daemon[866]: reaped child process 1850, status = succeeded worker1 # [ 202.027802] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:12Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 202.033124] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:12Z level=info event=scheduler_disconnected addr=lb:50051 error="no healthy upstream" worker2 # [ 202.037590] nix-grpc-daemon[1796]: ts=2026-09-22T11:00:12Z level=info event=niks3_push_exited status=succeeded why="niks3 push exited" worker2 # [ 202.081195] nix-grpc-daemon[1796]: E0000 00:00:1790074812.540176 1841 external_connection_acceptor_impl.cc:72] NOT handling external connection with fd 19, started true, shutdown true worker1 # [ 202.233980] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:12Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 202.252225] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:12Z level=info event=worker_session_open cn=worker-1 peer=ipv4:192.168.1.2:57690 worker1 # [ 202.258157] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:12Z level=info event=worker_hello addr=192.168.1.3:50051 systems=aarch64-linux max_jobs=1 running=0 worker2 # [ 202.632981] systemd[1]: nix-grpc-daemon.service: Deactivated successfully. worker2 # [ 202.635988] systemd[1]: nix-grpc-daemon.service: Consumed 850ms CPU time over 28.428s wall clock time, 34.7M memory peak, 24.2K incoming IP traffic, 31.8K outgoing IP traffic. worker2 # [ 202.660531] systemd[1]: Starting Nix worker-protocol over gRPC... worker2 # [ 202.855686] nix-grpc-daemon[2146]: ts=2026-09-22T11:00:13Z level=info event=warning msg="--niks3 http:// sends the bearer token in clear" worker2 # [ 202.858306] nix-grpc-daemon[2146]: ts=2026-09-22T11:00:13Z level=info event=oidc_provider name=mock issuer=http://192.168.1.2:8080/oidc worker2 # [ 202.862321] nix-grpc-daemon[2146]: ts=2026-09-22T11:00:13Z level=info event=roles builder=1 scheduler=1 scheduler_addr=lb:50051 advertise=192.168.1.4:50051 niks3=http://lb:5751 max_jobs=1 worker2 # [ 202.869560] nix-grpc-daemon[2146]: WARNING: All log messages before absl::InitializeLog() is called are written to STDERR worker2 # [ 202.871064] nix-grpc-daemon[2146]: E0000 00:00:1790074813.328099 2146 tls_security_connector.cc:771] TlsServerCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2: (finished: waiting for success: systemctl is-active nix-grpc-daemon.service || systemctl start nix-grpc-daemon.service, in 1.95 seconds) lb: waiting until exactly no endpoints of aarch64-linux are unhealthy lb: must succeed: curl -sf localhost:9901/clusters worker2 # [ 202.890277] nix-grpc-daemon[2146]: E0000 00:00:1790074813.334117 2146 tls_security_connector.cc:767] TlsServerCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 202.893535] nix-grpc-daemon[2146]: ts=2026-09-22T11:00:13Z level=info event=startup listen=systemd proxy_socket=/nix/var/nix/daemon-socket/socket worker2 # [ 202.895753] systemd[1]: Started Nix worker-protocol over gRPC. worker2 # [ 202.908262] nix-grpc-daemon[2146]: E0000 00:00:1790074813.356281 2187 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 202.929628] nix-grpc-daemon[2146]: E0000 00:00:1790074813.373829 2187 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 202.932412] nix-grpc-daemon[2146]: E0000 00:00:1790074813.373984 2187 tls_security_connector.cc:493] TlsChannelCertificateWatcher getting identity_cert_error: UNKNOWN:Unable to get latest identity certificates. worker2 # [ 202.934849] nix-grpc-daemon[2146]: E0000 00:00:1790074813.388220 2187 tls_security_connector.cc:489] TlsChannelCertificateWatcher getting root_cert_error: UNKNOWN:Unable to get latest root certificates. worker2 # [ 202.944232] nix-grpc-daemon[2146]: ts=2026-09-22T11:00:13Z level=info event=scheduler_connected addr=lb:50051 worker1 # [ 202.973497] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:13Z level=info event=oidc_jwks provider=mock keys=1 worker1 # [ 202.973764] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:13Z level=info event=worker_session_open cn=oidc:mock:node:worker2 peer=ipv4:192.168.1.2:57690 worker1 # [ 202.973922] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:13Z level=info event=worker_hello addr=192.168.1.4:50051 systems=aarch64-linux max_jobs=1 running=0 lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.10 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.11 seconds) lb: (finished: waiting until exactly no endpoints of aarch64-linux are unhealthy, in 1.20 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.05 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) (finished: subtest: a deploy mid-build drains: the build finishes, the new generation takes over, in 19.14 seconds) subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1 lb # [ 204.216366] envoy[933]: conn peer=[2001:db8:1::1]:37910 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=9638 tx=4413 ms=6427 lb # [ 204.216723] envoy[933]: conn peer=[2001:db8:1::1]:47742 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=2995 tx=1584 ms=14760 lb # [ 204.216845] envoy[933]: conn peer=[2001:db8:1::4]:43438 tls=TLSv1.3 subject="CN=worker-2" sni=- flags=- tls_fail="-" rx=1520 tx=344 ms=27381 worker1 # [ 204.514149] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:14Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:60204 worker2 # [ 204.513395] nix-grpc-daemon[2146]: ts=2026-09-22T11:00:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:38892 worker2 # [ 204.531092] nix-daemon[866]: accepted connection from pid 2146, user nix-grpc-daemon (trusted) worker2 # [ 204.555934] nix-grpc-daemon[2146]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 68 ms (attempt 1/5) worker2 # [ 204.631653] nix-grpc-daemon[2146]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 171 ms (attempt 2/5) worker2 # [ 204.802961] nix-grpc-daemon[2146]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 329 ms (attempt 3/5) worker2 # [ 205.132366] nix-grpc-daemon[2146]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org; retrying in 578 ms (attempt 4/5) worker2 # [ 205.713833] nix-grpc-daemon[2146]: disabling binary cache 'https://cache.nixos.org' for 60 seconds worker2 # [ 205.719686] nix-grpc-daemon[2146]: warning: unable to download 'https://cache.nixos.org/nix-cache-info': Could not resolve hostname (6) Could not resolve host: cache.nixos.org worker1 # [ 205.792575] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:16Z level=debug event=want client=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv cached=0 worker1 # [ 205.797908] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:16Z level=debug event=assigned drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 205.803768] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:16Z level=debug event=want client=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv cached=0 worker1 # [ 205.809142] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:16Z level=debug event=want client=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv cached=0 worker1 # [ 205.836591] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 205.850714] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:16Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 205.865020] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:16Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 205.879773] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 206.090544] nix-daemon[844]: accepted connection from pid 1732, user nix-grpc-daemon (trusted) worker1 # [ 206.108689] nix-daemon[844]: reaped child process 1735, status = succeeded worker1 # [ 206.283524] nix-daemon[844]: accepted connection from pid 1737, user nix-grpc-daemon (trusted) worker1 # [ 206.301366] nix-daemon[844]: reaped child process 1740, status = succeeded lb # [ 206.276825] niks3-server[1182]: 2026/09/22 11:00:16 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 206.323572] nix-grpc-daemon[1546]: time=2026-09-22T11:00:16.794Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 206.325175] nix-grpc-daemon[1546]: time=2026-09-22T11:00:16.794Z level=INFO msg="Uploading 3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv (512B)" lb # [ 206.305696] niks3-server[1182]: 2026/09/22 11:00:16 INFO Registered completed upload object_key=nar/0zxfsmqijfc2dkglls3yp1x3w0w1na7vx39fxc124zddjzpaygd8.nar.zst lb # [ 206.321939] niks3-server[1182]: 2026/09/22 11:00:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/62/sign lb # [ 206.336350] niks3-server[1182]: 2026/09/22 11:00:16 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.ls lb # [ 206.347385] niks3-server[1182]: 2026/09/22 11:00:16 INFO Signed narinfos id=62 count=1 worker1 # [ 206.384399] nix-grpc-daemon[1546]: time=2026-09-22T11:00:16.855Z level=INFO msg="Uploading 1 narinfos" lb # [ 206.356569] niks3-server[1182]: 2026/09/22 11:00:16 INFO Received complete upload request method=POST path=/api/pending_closures/62/complete lb # [ 206.364350] niks3-server[1182]: 2026/09/22 11:00:16 INFO Registered completed upload object_key=3223c4sl4ncpldbvw558vqwddijwzfpd.narinfo lb # [ 206.375517] niks3-server[1182]: 2026/09/22 11:00:16 INFO Completed upload id=62 worker1 # [ 206.414947] nix-grpc-daemon[1546]: time=2026-09-22T11:00:16.886Z level=INFO msg="Upload complete. (518ms)" worker1 # [ 206.424347] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:16Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 206.444355] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:16Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 206.451810] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 206.458023] nix-daemon[844]: reaped child process 1730, status = succeeded worker1 # [ 206.471495] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 207.901010] nix-daemon[844]: reaped child process 1744, status = succeeded worker1 # [ 208.067716] nix-daemon[844]: accepted connection from pid 1760, user nix-grpc-daemon (trusted) worker1 # [ 208.086337] nix-daemon[844]: reaped child process 1763, status = succeeded lb # [ 208.062132] niks3-server[1182]: 2026/09/22 11:00:18 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 208.112374] nix-grpc-daemon[1546]: time=2026-09-22T11:00:18.581Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 208.113851] nix-grpc-daemon[1546]: time=2026-09-22T11:00:18.581Z level=INFO msg="Uploading dla3w0zf02z5jhccwa27i85bbgzrrsvb-farm-a (128B)" lb # [ 208.104549] niks3-server[1182]: 2026/09/22 11:00:18 INFO Registered completed upload object_key=log/3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv lb # [ 208.117282] niks3-server[1182]: 2026/09/22 11:00:18 INFO Registered completed upload object_key=nar/05i7ka1qa7vmhiys96ln3pp3nj3rfdlslnbydrxdqj40cmvmqcwf.nar.zst lb # [ 208.130527] niks3-server[1182]: 2026/09/22 11:00:18 INFO Received sign narinfos request method=POST path=/api/pending_closures/63/sign worker1 # [ 208.171418] nix-grpc-daemon[1546]: time=2026-09-22T11:00:18.642Z level=INFO msg="Uploading 1 narinfos" lb # [ 208.140044] niks3-server[1182]: 2026/09/22 11:00:18 INFO Signed narinfos id=63 count=1 lb # [ 208.142589] niks3-server[1182]: 2026/09/22 11:00:18 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.ls lb # [ 208.154162] niks3-server[1182]: 2026/09/22 11:00:18 INFO Received complete upload request method=POST path=/api/pending_closures/63/complete lb # [ 208.159525] niks3-server[1182]: 2026/09/22 11:00:18 INFO Completed upload id=63 worker1 # [ 208.196255] nix-grpc-daemon[1546]: time=2026-09-22T11:00:18.667Z level=INFO msg="Upload complete. (309ms)" lb # [ 208.162507] niks3-server[1182]: 2026/09/22 11:00:18 INFO Registered completed upload object_key=dla3w0zf02z5jhccwa27i85bbgzrrsvb.narinfo worker1 # [ 208.217162] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:18Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=1 drv=3223c4sl4ncpldbvw558vqwddijwzfpd-farm-a.drv assign_id=1 outputs=1 worker1 # [ 208.231909] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:18Z level=debug event=assigned drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 208.236416] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:18Z level=debug event=want client=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv cached=0 worker1 # [ 208.239650] nix-daemon[844]: reaped child process 1742, status = succeeded worker1 # [ 208.252679] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:18Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 208.266070] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:18Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 208.280913] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:18Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 208.294522] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 208.492353] nix-daemon[844]: accepted connection from pid 1769, user nix-grpc-daemon (trusted) worker1 # [ 208.509121] nix-daemon[844]: reaped child process 1772, status = succeeded worker1 # [ 208.680099] nix-daemon[844]: accepted connection from pid 1774, user nix-grpc-daemon (trusted) worker1 # [ 208.695080] nix-daemon[844]: reaped child process 1777, status = succeeded lb # [ 208.673185] niks3-server[1182]: 2026/09/22 11:00:19 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 208.718621] nix-grpc-daemon[1546]: time=2026-09-22T11:00:19.189Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 208.720215] nix-grpc-daemon[1546]: time=2026-09-22T11:00:19.189Z level=INFO msg="Uploading pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv (512B)" lb # [ 208.699002] niks3-server[1182]: 2026/09/22 11:00:19 INFO Registered completed upload object_key=nar/0g5bxr683biqh358mkd9xrclwllan4cgnv4pzijas586xj68wwx0.nar.zst worker1 # [ 208.749287] nix-grpc-daemon[1546]: time=2026-09-22T11:00:19.220Z level=INFO msg="Uploading 1 narinfos" lb # [ 208.715041] niks3-server[1182]: 2026/09/22 11:00:19 INFO Received sign narinfos request method=POST path=/api/pending_closures/64/sign lb # [ 208.724990] niks3-server[1182]: 2026/09/22 11:00:19 INFO Signed narinfos id=64 count=1 worker1 # [ 208.769255] nix-grpc-daemon[1546]: time=2026-09-22T11:00:19.240Z level=INFO msg="Upload complete. (459ms)" lb # [ 208.731910] niks3-server[1182]: 2026/09/22 11:00:19 INFO Received complete upload request method=POST path=/api/pending_closures/64/complete lb # [ 208.740718] niks3-server[1182]: 2026/09/22 11:00:19 INFO Completed upload id=64 worker1 # [ 208.775649] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:19Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=0 paths=1 nar_bytes_in=512 lb # [ 208.743026] niks3-server[1182]: 2026/09/22 11:00:19 INFO Registered completed upload object_key=pplpzcnjfn295nm16cg0iihpfc9iffry.ls worker1 # [ 208.785902] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:19Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 208.790543] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 208.794236] nix-daemon[844]: reaped child process 1767, status = succeeded worker1 # [ 208.803439] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 210.229812] nix-daemon[844]: reaped child process 1781, status = succeeded worker1 # [ 210.393654] nix-daemon[844]: accepted connection from pid 1797, user nix-grpc-daemon (trusted) worker1 # [ 210.410297] nix-daemon[844]: reaped child process 1800, status = succeeded lb # [ 210.386075] niks3-server[1182]: 2026/09/22 11:00:20 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 210.432361] nix-grpc-daemon[1546]: time=2026-09-22T11:00:20.903Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 210.437060] nix-grpc-daemon[1546]: time=2026-09-22T11:00:20.905Z level=INFO msg="Uploading fhc1zkcygpzppf0bj5fr2k85lf7p87v9-farm-a (128B)" lb # [ 210.426914] niks3-server[1182]: 2026/09/22 11:00:20 INFO Registered completed upload object_key=log/pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv worker1 # [ 210.477161] nix-grpc-daemon[1546]: time=2026-09-22T11:00:20.948Z level=INFO msg="Uploading 1 narinfos" lb # [ 210.438421] niks3-server[1182]: 2026/09/22 11:00:20 INFO Registered completed upload object_key=nar/09na6djn66dpys8h08gqw39p0vw7vxqsdzbiw55rknyna8s239bm.nar.zst lb # [ 210.452933] niks3-server[1182]: 2026/09/22 11:00:20 INFO Received sign narinfos request method=POST path=/api/pending_closures/65/sign lb # [ 210.460715] niks3-server[1182]: 2026/09/22 11:00:20 INFO Signed narinfos id=65 count=1 lb # [ 210.469036] niks3-server[1182]: 2026/09/22 11:00:20 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.ls lb # [ 210.475582] niks3-server[1182]: 2026/09/22 11:00:20 INFO Received complete upload request method=POST path=/api/pending_closures/65/complete lb # [ 210.481994] niks3-server[1182]: 2026/09/22 11:00:20 INFO Completed upload id=65 worker1 # [ 210.511875] nix-grpc-daemon[1546]: time=2026-09-22T11:00:20.983Z level=INFO msg="Upload complete. (298ms)" lb # [ 210.485973] niks3-server[1182]: 2026/09/22 11:00:20 INFO Registered completed upload object_key=fhc1zkcygpzppf0bj5fr2k85lf7p87v9.narinfo worker1 # [ 210.525728] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:20Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=1 drv=pplpzcnjfn295nm16cg0iihpfc9iffry-farm-a.drv assign_id=2 outputs=1 worker1 # [ 210.544252] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:20Z level=debug event=assigned drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 210.552214] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 210.557323] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:21Z level=debug event=want client=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv cached=0 worker1 # [ 210.562007] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:21Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 210.566189] nix-daemon[844]: reaped child process 1779, status = succeeded worker1 # [ 210.569048] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:21Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 210.582432] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 210.762866] nix-daemon[844]: accepted connection from pid 1806, user nix-grpc-daemon (trusted) worker1 # [ 210.780469] nix-daemon[844]: reaped child process 1809, status = succeeded worker1 # [ 210.953930] nix-daemon[844]: accepted connection from pid 1811, user nix-grpc-daemon (trusted) worker1 # [ 210.969746] nix-daemon[844]: reaped child process 1814, status = succeeded lb # [ 210.944196] niks3-server[1182]: 2026/09/22 11:00:21 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 210.989388] nix-grpc-daemon[1546]: time=2026-09-22T11:00:21.460Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 210.990843] nix-grpc-daemon[1546]: time=2026-09-22T11:00:21.460Z level=INFO msg="Uploading sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv (512B)" lb # [ 210.972260] niks3-server[1182]: 2026/09/22 11:00:21 INFO Registered completed upload object_key=nar/1fxz200fmpbckafrn03b2snx9knqpbwf8rwqlbvg68356qj7aj5w.nar.zst lb # [ 210.981893] niks3-server[1182]: 2026/09/22 11:00:21 INFO Received sign narinfos request method=POST path=/api/pending_closures/66/sign worker1 # [ 211.025970] nix-grpc-daemon[1546]: time=2026-09-22T11:00:21.497Z level=INFO msg="Uploading 1 narinfos" lb # [ 210.994675] niks3-server[1182]: 2026/09/22 11:00:21 INFO Signed narinfos id=66 count=1 lb # [ 211.001134] niks3-server[1182]: 2026/09/22 11:00:21 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.ls lb # [ 211.010375] niks3-server[1182]: 2026/09/22 11:00:21 INFO Received complete upload request method=POST path=/api/pending_closures/66/complete worker1 # [ 211.051243] nix-grpc-daemon[1546]: time=2026-09-22T11:00:21.522Z level=INFO msg="Upload complete. (464ms)" lb # [ 211.014938] niks3-server[1182]: 2026/09/22 11:00:21 INFO Completed upload id=66 lb # [ 211.018070] niks3-server[1182]: 2026/09/22 11:00:21 INFO Registered completed upload object_key=sy43wp9v52pni1wms6q0p3qybpc2pwad.narinfo worker1 # [ 211.060226] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:21Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=0 paths=1 nar_bytes_in=512 worker1 # [ 211.071506] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:21Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 211.077550] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 211.082024] nix-daemon[844]: reaped child process 1804, status = succeeded worker1 # [ 211.092664] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 212.735223] nix-daemon[844]: reaped child process 1818, status = succeeded worker1 # [ 212.902563] nix-daemon[844]: accepted connection from pid 1834, user nix-grpc-daemon (trusted) worker1 # [ 212.919034] nix-daemon[844]: reaped child process 1837, status = succeeded lb # [ 212.894857] niks3-server[1182]: 2026/09/22 11:00:23 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 212.942627] nix-grpc-daemon[1546]: time=2026-09-22T11:00:23.412Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker1 # [ 212.944442] nix-grpc-daemon[1546]: time=2026-09-22T11:00:23.412Z level=INFO msg="Uploading g5laypvfk2469p8binw3mksk2hv7zhnh-farm-a (128B)" lb # [ 212.934044] niks3-server[1182]: 2026/09/22 11:00:23 INFO Registered completed upload object_key=log/sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv lb # [ 212.943489] niks3-server[1182]: 2026/09/22 11:00:23 INFO Registered completed upload object_key=nar/1j2671k9mycq8lxyfx5k6a66d5narfqw2wqwlpwxgwzi93vnvbdn.nar.zst lb # [ 212.952564] niks3-server[1182]: 2026/09/22 11:00:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/67/sign worker1 # [ 212.994073] nix-grpc-daemon[1546]: time=2026-09-22T11:00:23.464Z level=INFO msg="Uploading 1 narinfos" lb # [ 212.960558] niks3-server[1182]: 2026/09/22 11:00:23 INFO Signed narinfos id=67 count=1 lb # [ 212.962723] niks3-server[1182]: 2026/09/22 11:00:23 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.ls lb # [ 212.971265] niks3-server[1182]: 2026/09/22 11:00:23 INFO Received complete upload request method=POST path=/api/pending_closures/67/complete lb # [ 212.979587] niks3-server[1182]: 2026/09/22 11:00:23 INFO Completed upload id=67 lb # [ 212.982740] niks3-server[1182]: 2026/09/22 11:00:23 INFO Registered completed upload object_key=g5laypvfk2469p8binw3mksk2hv7zhnh.narinfo worker1 # [ 213.016282] nix-grpc-daemon[1546]: time=2026-09-22T11:00:23.487Z level=INFO msg="Upload complete. (311ms)" worker1 # [ 213.034046] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:23Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=1 drv=sy43wp9v52pni1wms6q0p3qybpc2pwad-farm-a.drv assign_id=3 outputs=1 worker1 # [ 213.052792] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:23Z level=debug event=assigned drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 213.062724] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:23Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 213.068284] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:23Z level=debug event=want client=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv cached=0 worker1 # [ 213.073316] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:23Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 213.077766] nix-daemon[844]: reaped child process 1816, status = succeeded worker1 # [ 213.080974] nix-grpc-daemon[1546]: time=2026-09-22T11:00:23.543Z level=INFO msg="All 1 paths already cached" worker1 # [ 213.090131] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:23Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 213.099687] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 213.276741] nix-daemon[844]: accepted connection from pid 1843, user nix-grpc-daemon (trusted) worker1 # [ 213.293372] nix-daemon[844]: reaped child process 1846, status = succeeded worker1 # [ 213.464191] nix-daemon[844]: accepted connection from pid 1848, user nix-grpc-daemon (trusted) worker1 # [ 213.481967] nix-daemon[844]: reaped child process 1851, status = succeeded lb # [ 213.458047] niks3-server[1182]: 2026/09/22 11:00:23 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 213.503654] nix-grpc-daemon[1546]: time=2026-09-22T11:00:23.974Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 213.505359] nix-grpc-daemon[1546]: time=2026-09-22T11:00:23.974Z level=INFO msg="Uploading cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv (624B)" lb # [ 213.488131] niks3-server[1182]: 2026/09/22 11:00:23 INFO Registered completed upload object_key=nar/1w4sykcjji9zlw0a5xibxycy6x3jyb4drd29x2k9g7x5whfwwpsq.nar.zst worker1 # [ 213.533883] nix-grpc-daemon[1546]: time=2026-09-22T11:00:24.004Z level=INFO msg="Uploading 1 narinfos" lb # [ 213.501188] niks3-server[1182]: 2026/09/22 11:00:23 INFO Received sign narinfos request method=POST path=/api/pending_closures/68/sign lb # [ 213.512718] niks3-server[1182]: 2026/09/22 11:00:23 INFO Signed narinfos id=68 count=1 lb # [ 213.523564] niks3-server[1182]: 2026/09/22 11:00:23 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.ls worker1 # [ 213.564940] nix-grpc-daemon[1546]: time=2026-09-22T11:00:24.036Z level=INFO msg="Upload complete. (461ms)" lb # [ 213.529157] niks3-server[1182]: 2026/09/22 11:00:23 INFO Received complete upload request method=POST path=/api/pending_closures/68/complete lb # [ 213.535957] niks3-server[1182]: 2026/09/22 11:00:23 INFO Completed upload id=68 lb # [ 213.539353] niks3-server[1182]: 2026/09/22 11:00:23 INFO Registered completed upload object_key=cnf7j5vjqyj799isbg26wgjnrlk208gi.narinfo worker1 # [ 213.572191] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:24Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 213.580718] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:24Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 213.590189] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 213.595201] nix-daemon[844]: reaped child process 1841, status = succeeded worker1 # [ 213.607491] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 214.439237] nix-daemon[844]: reaped child process 1855, status = succeeded worker1 # [ 214.611017] nix-daemon[844]: accepted connection from pid 1871, user nix-grpc-daemon (trusted) worker1 # [ 214.629254] nix-daemon[844]: reaped child process 1874, status = succeeded lb # [ 214.606409] niks3-server[1182]: 2026/09/22 11:00:25 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 214.652413] nix-grpc-daemon[1546]: time=2026-09-22T11:00:25.123Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 214.653887] nix-grpc-daemon[1546]: time=2026-09-22T11:00:25.123Z level=INFO msg="Uploading p1334cl91hzsgv8bmiqzkwvhbwq24vn1-farm-b (176B)" lb # [ 214.643940] niks3-server[1182]: 2026/09/22 11:00:25 INFO Registered completed upload object_key=nar/0pa7j07n0n5p9wcfihhx3357xrxs4h8s57kncplymh51lvzvz2kh.nar.zst lb # [ 214.657203] niks3-server[1182]: 2026/09/22 11:00:25 INFO Registered completed upload object_key=log/cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv lb # [ 214.668696] niks3-server[1182]: 2026/09/22 11:00:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/69/sign lb # [ 214.676475] niks3-server[1182]: 2026/09/22 11:00:25 INFO Signed narinfos id=69 count=1 worker1 # [ 214.711993] nix-grpc-daemon[1546]: time=2026-09-22T11:00:25.182Z level=INFO msg="Uploading 1 narinfos" lb # [ 214.679494] niks3-server[1182]: 2026/09/22 11:00:25 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.ls lb # [ 214.691315] niks3-server[1182]: 2026/09/22 11:00:25 INFO Received complete upload request method=POST path=/api/pending_closures/69/complete lb # [ 214.699853] niks3-server[1182]: 2026/09/22 11:00:25 INFO Completed upload id=69 worker1 # [ 214.735922] nix-grpc-daemon[1546]: time=2026-09-22T11:00:25.207Z level=INFO msg="Upload complete. (319ms)" lb # [ 214.703150] niks3-server[1182]: 2026/09/22 11:00:25 INFO Registered completed upload object_key=p1334cl91hzsgv8bmiqzkwvhbwq24vn1.narinfo worker1 # [ 214.749126] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:25Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=1 drv=cnf7j5vjqyj799isbg26wgjnrlk208gi-farm-b.drv assign_id=4 outputs=1 worker1 # [ 214.766952] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:25Z level=debug event=assigned drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 214.771827] nix-daemon[844]: reaped child process 1853, status = succeeded worker1 # [ 214.792144] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 214.799446] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:25Z level=debug event=want client=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv cached=0 worker1 # [ 214.810375] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:25Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 214.823403] nix-grpc-daemon[1546]: time=2026-09-22T11:00:25.294Z level=INFO msg="All 1 paths already cached" worker1 # [ 214.832766] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:25Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 214.845518] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 215.038351] nix-daemon[844]: accepted connection from pid 1880, user nix-grpc-daemon (trusted) worker1 # [ 215.055237] nix-daemon[844]: reaped child process 1883, status = succeeded worker1 # [ 215.226990] nix-daemon[844]: accepted connection from pid 1885, user nix-grpc-daemon (trusted) worker1 # [ 215.243716] nix-daemon[844]: reaped child process 1888, status = succeeded lb # [ 215.219104] niks3-server[1182]: 2026/09/22 11:00:25 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 215.264890] nix-grpc-daemon[1546]: time=2026-09-22T11:00:25.735Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 215.266354] nix-grpc-daemon[1546]: time=2026-09-22T11:00:25.735Z level=INFO msg="Uploading 7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv (624B)" lb # [ 215.246358] niks3-server[1182]: 2026/09/22 11:00:25 INFO Registered completed upload object_key=nar/0l0zss9ls4bhz18slrrmz7l1j93xz7424q7314p64rpgrpqf6b7k.nar.zst worker1 # [ 215.295406] nix-grpc-daemon[1546]: time=2026-09-22T11:00:25.766Z level=INFO msg="Uploading 1 narinfos" lb # [ 215.263310] niks3-server[1182]: 2026/09/22 11:00:25 INFO Received sign narinfos request method=POST path=/api/pending_closures/70/sign lb # [ 215.274529] niks3-server[1182]: 2026/09/22 11:00:25 INFO Signed narinfos id=70 count=1 lb # [ 215.283400] niks3-server[1182]: 2026/09/22 11:00:25 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.ls lb # [ 215.287331] niks3-server[1182]: 2026/09/22 11:00:25 INFO Received complete upload request method=POST path=/api/pending_closures/70/complete worker1 # [ 215.327755] nix-grpc-daemon[1546]: time=2026-09-22T11:00:25.798Z level=INFO msg="Upload complete. (468ms)" lb # [ 215.291744] niks3-server[1182]: 2026/09/22 11:00:25 INFO Completed upload id=70 lb # [ 215.294347] niks3-server[1182]: 2026/09/22 11:00:25 INFO Registered completed upload object_key=7vjzkx8444zm2h7882vsj6bmj73zqjq3.narinfo worker1 # [ 215.336248] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:25Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=0 paths=1 nar_bytes_in=624 worker1 # [ 215.346869] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:25Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 215.352543] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 215.357159] nix-daemon[844]: reaped child process 1878, status = succeeded worker1 # [ 215.369910] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 216.617854] nix-daemon[844]: reaped child process 1892, status = succeeded worker1 # [ 216.791595] nix-daemon[844]: accepted connection from pid 1908, user nix-grpc-daemon (trusted) worker1 # [ 216.811140] nix-daemon[844]: reaped child process 1911, status = succeeded lb # [ 216.787136] niks3-server[1182]: 2026/09/22 11:00:27 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 216.835425] nix-grpc-daemon[1546]: time=2026-09-22T11:00:27.305Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 216.837713] nix-grpc-daemon[1546]: time=2026-09-22T11:00:27.305Z level=INFO msg="Uploading nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8-farm-b (176B)" lb # [ 216.824482] niks3-server[1182]: 2026/09/22 11:00:27 INFO Registered completed upload object_key=nar/0kqa5m59sp6aj1ixqj6q54wwqxkcn98fw1rc9hfkcdk460wjdl0l.nar.zst lb # [ 216.837514] niks3-server[1182]: 2026/09/22 11:00:27 INFO Registered completed upload object_key=log/7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv lb # [ 216.846735] niks3-server[1182]: 2026/09/22 11:00:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/71/sign lb # [ 216.852742] niks3-server[1182]: 2026/09/22 11:00:27 INFO Signed narinfos id=71 count=1 worker1 # [ 216.889632] nix-grpc-daemon[1546]: time=2026-09-22T11:00:27.360Z level=INFO msg="Uploading 1 narinfos" lb # [ 216.858485] niks3-server[1182]: 2026/09/22 11:00:27 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.ls lb # [ 216.867598] niks3-server[1182]: 2026/09/22 11:00:27 INFO Received complete upload request method=POST path=/api/pending_closures/71/complete worker1 # [ 216.910414] nix-grpc-daemon[1546]: time=2026-09-22T11:00:27.381Z level=INFO msg="Upload complete. (329ms)" lb # [ 216.875608] niks3-server[1182]: 2026/09/22 11:00:27 INFO Completed upload id=71 lb # [ 216.878593] niks3-server[1182]: 2026/09/22 11:00:27 INFO Registered completed upload object_key=nrnwwsxw1z9dz81vri9gl1dkhkrc6vh8.narinfo worker1 # [ 216.927655] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:27Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=1 drv=7vjzkx8444zm2h7882vsj6bmj73zqjq3-farm-b.drv assign_id=5 outputs=1 worker1 # [ 216.948909] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:27Z level=debug event=assigned drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 216.959234] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 216.967234] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:27Z level=debug event=want client=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv cached=0 worker1 # [ 216.973825] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:27Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 216.988921] nix-daemon[844]: reaped child process 1890, status = succeeded worker1 # [ 216.992602] nix-grpc-daemon[1546]: time=2026-09-22T11:00:27.443Z level=INFO msg="All 1 paths already cached" worker1 # [ 216.999254] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 217.003227] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:27Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 217.183072] nix-daemon[844]: accepted connection from pid 1917, user nix-grpc-daemon (trusted) worker1 # [ 217.201301] nix-daemon[844]: reaped child process 1920, status = succeeded worker1 # [ 217.372190] nix-daemon[844]: accepted connection from pid 1922, user nix-grpc-daemon (trusted) worker1 # [ 217.387456] nix-daemon[844]: reaped child process 1925, status = succeeded lb # [ 217.365674] niks3-server[1182]: 2026/09/22 11:00:27 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 217.411226] nix-grpc-daemon[1546]: time=2026-09-22T11:00:27.882Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 217.412844] nix-grpc-daemon[1546]: time=2026-09-22T11:00:27.882Z level=INFO msg="Uploading cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv (624B)" lb # [ 217.391041] niks3-server[1182]: 2026/09/22 11:00:27 INFO Registered completed upload object_key=nar/0f6g0vggy0qbqsj64l1nivyqdh6q12kh8620ysnf210ani0727vf.nar.zst worker1 # [ 217.440902] nix-grpc-daemon[1546]: time=2026-09-22T11:00:27.911Z level=INFO msg="Uploading 1 narinfos" lb # [ 217.405285] niks3-server[1182]: 2026/09/22 11:00:27 INFO Received sign narinfos request method=POST path=/api/pending_closures/72/sign lb # [ 217.419329] niks3-server[1182]: 2026/09/22 11:00:27 INFO Signed narinfos id=72 count=1 lb # [ 217.430613] niks3-server[1182]: 2026/09/22 11:00:27 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.ls worker1 # [ 217.471684] nix-grpc-daemon[1546]: time=2026-09-22T11:00:27.942Z level=INFO msg="Upload complete. (463ms)" lb # [ 217.436102] niks3-server[1182]: 2026/09/22 11:00:27 INFO Received complete upload request method=POST path=/api/pending_closures/72/complete lb # [ 217.441526] niks3-server[1182]: 2026/09/22 11:00:27 INFO Completed upload id=72 worker1 # [ 217.477990] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:27Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=0 paths=1 nar_bytes_in=624 lb # [ 217.444954] niks3-server[1182]: 2026/09/22 11:00:27 INFO Registered completed upload object_key=cgwyij19i3fl7sqlyz54vzfr4zajg077.narinfo worker1 # [ 217.487617] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:27Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 217.499066] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 217.503542] nix-daemon[844]: reaped child process 1915, status = succeeded worker1 # [ 217.514376] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 219.162356] nix-daemon[844]: reaped child process 1929, status = succeeded worker1 # [ 219.350049] nix-daemon[844]: accepted connection from pid 1945, user nix-grpc-daemon (trusted) worker1 # [ 219.368394] nix-daemon[844]: reaped child process 1948, status = succeeded lb # [ 219.347146] niks3-server[1182]: 2026/09/22 11:00:29 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 219.397271] nix-grpc-daemon[1546]: time=2026-09-22T11:00:29.866Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker1 # [ 219.398871] nix-grpc-daemon[1546]: time=2026-09-22T11:00:29.866Z level=INFO msg="Uploading hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5-farm-b (176B)" lb # [ 219.385824] niks3-server[1182]: 2026/09/22 11:00:29 INFO Registered completed upload object_key=log/cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv lb # [ 219.397482] niks3-server[1182]: 2026/09/22 11:00:29 INFO Registered completed upload object_key=nar/0x964sxcxsdh3nfjarc1ib52kr50kl8zxl7h34jq5ydx763drras.nar.zst lb # [ 219.408174] niks3-server[1182]: 2026/09/22 11:00:29 INFO Received sign narinfos request method=POST path=/api/pending_closures/73/sign worker1 # [ 219.452288] nix-grpc-daemon[1546]: time=2026-09-22T11:00:29.923Z level=INFO msg="Uploading 1 narinfos" lb # [ 219.415507] niks3-server[1182]: 2026/09/22 11:00:29 INFO Signed narinfos id=73 count=1 lb # [ 219.421268] niks3-server[1182]: 2026/09/22 11:00:29 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.ls lb # [ 219.431855] niks3-server[1182]: 2026/09/22 11:00:29 INFO Received complete upload request method=POST path=/api/pending_closures/73/complete lb # [ 219.440470] niks3-server[1182]: 2026/09/22 11:00:29 INFO Completed upload id=73 worker1 # [ 219.477111] nix-grpc-daemon[1546]: time=2026-09-22T11:00:29.948Z level=INFO msg="Upload complete. (350ms)" lb # [ 219.443536] niks3-server[1182]: 2026/09/22 11:00:29 INFO Registered completed upload object_key=hfsg1k13f2w2fjgkzxk1qi2fh35fdlk5.narinfo worker1 # [ 219.495586] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:29Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=1 drv=cgwyij19i3fl7sqlyz54vzfr4zajg077-farm-b.drv assign_id=6 outputs=1 worker1 # [ 219.513018] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:29Z level=debug event=assigned drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 219.522859] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:29Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 219.527548] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:29Z level=debug event=want client=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv cached=0 worker1 # [ 219.532723] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:29Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 219.536697] nix-daemon[844]: reaped child process 1927, status = succeeded worker1 # [ 219.538418] nix-grpc-daemon[1546]: time=2026-09-22T11:00:30.002Z level=INFO msg="All 2 paths already cached" worker1 # [ 219.540843] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:30Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 219.550204] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 219.733169] nix-daemon[844]: accepted connection from pid 1954, user nix-grpc-daemon (trusted) worker1 # [ 219.751738] nix-daemon[844]: reaped child process 1957, status = succeeded worker1 # [ 219.920819] nix-daemon[844]: accepted connection from pid 1959, user nix-grpc-daemon (trusted) worker1 # [ 219.938759] nix-daemon[844]: reaped child process 1962, status = succeeded lb # [ 219.913579] niks3-server[1182]: 2026/09/22 11:00:30 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 219.958929] nix-grpc-daemon[1546]: time=2026-09-22T11:00:30.429Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 219.960496] nix-grpc-daemon[1546]: time=2026-09-22T11:00:30.429Z level=INFO msg="Uploading dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv (752B)" lb # [ 219.940601] niks3-server[1182]: 2026/09/22 11:00:30 INFO Registered completed upload object_key=nar/0s7ff7rwqx5p9mdzddqgzd9bbd5j8njfc62k07mgnpy19810bs86.nar.zst worker1 # [ 219.988854] nix-grpc-daemon[1546]: time=2026-09-22T11:00:30.459Z level=INFO msg="Uploading 1 narinfos" lb # [ 219.957020] niks3-server[1182]: 2026/09/22 11:00:30 INFO Received sign narinfos request method=POST path=/api/pending_closures/74/sign lb # [ 219.967289] niks3-server[1182]: 2026/09/22 11:00:30 INFO Signed narinfos id=74 count=1 lb # [ 219.974581] niks3-server[1182]: 2026/09/22 11:00:30 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.ls worker1 # [ 220.016290] nix-grpc-daemon[1546]: time=2026-09-22T11:00:30.487Z level=INFO msg="Upload complete. (456ms)" lb # [ 219.980525] niks3-server[1182]: 2026/09/22 11:00:30 INFO Received complete upload request method=POST path=/api/pending_closures/74/complete lb # [ 219.985788] niks3-server[1182]: 2026/09/22 11:00:30 INFO Completed upload id=74 worker1 # [ 220.024219] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:30Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 219.989136] niks3-server[1182]: 2026/09/22 11:00:30 INFO Registered completed upload object_key=dca2k4dvnk52m7gijqfq8q99ih3ni8ny.narinfo worker1 # [ 220.036072] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:30Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 220.044919] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 220.047596] nix-daemon[844]: reaped child process 1952, status = succeeded worker1 # [ 220.058263] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 221.488842] nix-daemon[844]: reaped child process 1966, status = succeeded worker1 # [ 221.658291] nix-daemon[844]: accepted connection from pid 1982, user nix-grpc-daemon (trusted) worker1 # [ 221.675537] nix-daemon[844]: reaped child process 1985, status = succeeded lb # [ 221.651572] niks3-server[1182]: 2026/09/22 11:00:32 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 221.698367] nix-grpc-daemon[1546]: time=2026-09-22T11:00:32.168Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 221.699887] nix-grpc-daemon[1546]: time=2026-09-22T11:00:32.168Z level=INFO msg="Uploading d88xgckxx0s0n5ly2d0jw1qy1641z8pq-farm-top (232B)" lb # [ 221.683124] niks3-server[1182]: 2026/09/22 11:00:32 INFO Registered completed upload object_key=nar/0knxhkhz3x04vk8as5wv5ar2d0h02prv7q5f42qm1xjgr70ancd0.nar.zst lb # [ 221.701637] niks3-server[1182]: 2026/09/22 11:00:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/75/sign worker1 # [ 221.743975] nix-grpc-daemon[1546]: time=2026-09-22T11:00:32.214Z level=INFO msg="Uploading 1 narinfos" lb # [ 221.714986] niks3-server[1182]: 2026/09/22 11:00:32 INFO Signed narinfos id=75 count=1 lb # [ 221.721365] niks3-server[1182]: 2026/09/22 11:00:32 INFO Registered completed upload object_key=log/dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv lb # [ 221.727300] niks3-server[1182]: 2026/09/22 11:00:32 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.ls lb # [ 221.734774] niks3-server[1182]: 2026/09/22 11:00:32 INFO Received complete upload request method=POST path=/api/pending_closures/75/complete lb # [ 221.739508] niks3-server[1182]: 2026/09/22 11:00:32 INFO Completed upload id=75 worker1 # [ 221.775562] nix-grpc-daemon[1546]: time=2026-09-22T11:00:32.246Z level=INFO msg="Upload complete. (306ms)" lb # [ 221.742408] niks3-server[1182]: 2026/09/22 11:00:32 INFO Registered completed upload object_key=d88xgckxx0s0n5ly2d0jw1qy1641z8pq.narinfo worker1 # [ 221.788993] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:32Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=1 drv=dca2k4dvnk52m7gijqfq8q99ih3ni8ny-farm-top.drv assign_id=7 outputs=1 worker1 # [ 221.803551] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:32Z level=debug event=assigned drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 221.808533] nix-daemon[844]: reaped child process 1964, status = succeeded worker1 # [ 221.828831] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 221.842918] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:32Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 221.857365] nix-grpc-daemon[1546]: time=2026-09-22T11:00:32.328Z level=INFO msg="All 2 paths already cached" worker1 # [ 221.866917] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:32Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 221.881105] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 222.074900] nix-daemon[844]: accepted connection from pid 1989, user nix-grpc-daemon (trusted) worker1 # [ 222.093014] nix-daemon[844]: reaped child process 1992, status = succeeded worker1 # [ 222.264110] nix-daemon[844]: accepted connection from pid 1994, user nix-grpc-daemon (trusted) worker1 # [ 222.281300] nix-daemon[844]: reaped child process 1997, status = succeeded lb # [ 222.256114] niks3-server[1182]: 2026/09/22 11:00:32 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 222.301177] nix-grpc-daemon[1546]: time=2026-09-22T11:00:32.772Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 222.302653] nix-grpc-daemon[1546]: time=2026-09-22T11:00:32.772Z level=INFO msg="Uploading 2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv (752B)" lb # [ 222.284598] niks3-server[1182]: 2026/09/22 11:00:32 INFO Received sign narinfos request method=POST path=/api/pending_closures/76/sign worker1 # [ 222.327367] nix-grpc-daemon[1546]: time=2026-09-22T11:00:32.798Z level=INFO msg="Uploading 1 narinfos" lb # [ 222.297743] niks3-server[1182]: 2026/09/22 11:00:32 INFO Signed narinfos id=76 count=1 lb # [ 222.306552] niks3-server[1182]: 2026/09/22 11:00:32 INFO Registered completed upload object_key=nar/0cpiyiv7frlhngqhhc703bglz5q2cpg4n8sgds4sfg7nvw0fcdn5.nar.zst lb # [ 222.319812] niks3-server[1182]: 2026/09/22 11:00:32 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.ls lb # [ 222.324532] niks3-server[1182]: 2026/09/22 11:00:32 INFO Received complete upload request method=POST path=/api/pending_closures/76/complete worker1 # [ 222.366252] nix-grpc-daemon[1546]: time=2026-09-22T11:00:32.837Z level=INFO msg="Upload complete. (471ms)" lb # [ 222.331185] niks3-server[1182]: 2026/09/22 11:00:32 INFO Completed upload id=76 worker1 # [ 222.372264] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:32Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=0 paths=1 nar_bytes_in=752 lb # [ 222.335024] niks3-server[1182]: 2026/09/22 11:00:32 INFO Registered completed upload object_key=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9.narinfo worker1 # [ 222.380734] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:32Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 222.391064] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 222.395620] nix-daemon[844]: reaped child process 1987, status = succeeded worker1 # [ 222.409070] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 224.053716] nix-daemon[844]: reaped child process 2001, status = succeeded worker1 # [ 224.224728] nix-daemon[844]: accepted connection from pid 2017, user nix-grpc-daemon (trusted) worker1 # [ 224.243742] nix-daemon[844]: reaped child process 2020, status = succeeded lb # [ 224.223161] niks3-server[1182]: 2026/09/22 11:00:34 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 224.271467] nix-grpc-daemon[1546]: time=2026-09-22T11:00:34.739Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 224.273149] nix-grpc-daemon[1546]: time=2026-09-22T11:00:34.739Z level=INFO msg="Uploading vz84kyfcw804d0iyx7nj61j47ms9zxmn-farm-top (232B)" lb # [ 224.261737] niks3-server[1182]: 2026/09/22 11:00:34 INFO Registered completed upload object_key=log/2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv lb # [ 224.272395] niks3-server[1182]: 2026/09/22 11:00:34 INFO Registered completed upload object_key=nar/0aaz4iim7fa45zws98gbhp73l23kbm0fvp4anssnxa96v666ljhs.nar.zst worker1 # [ 224.319561] nix-grpc-daemon[1546]: time=2026-09-22T11:00:34.790Z level=INFO msg="Uploading 1 narinfos" lb # [ 224.282762] niks3-server[1182]: 2026/09/22 11:00:34 INFO Received sign narinfos request method=POST path=/api/pending_closures/77/sign lb # [ 224.287947] niks3-server[1182]: 2026/09/22 11:00:34 INFO Signed narinfos id=77 count=1 lb # [ 224.290144] niks3-server[1182]: 2026/09/22 11:00:34 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.ls lb # [ 224.299449] niks3-server[1182]: 2026/09/22 11:00:34 INFO Received complete upload request method=POST path=/api/pending_closures/77/complete lb # [ 224.306510] niks3-server[1182]: 2026/09/22 11:00:34 INFO Completed upload id=77 lb # [ 224.309707] niks3-server[1182]: 2026/09/22 11:00:34 INFO Registered completed upload object_key=vz84kyfcw804d0iyx7nj61j47ms9zxmn.narinfo worker1 # [ 224.341046] nix-grpc-daemon[1546]: time=2026-09-22T11:00:34.812Z level=INFO msg="Upload complete. (321ms)" worker1 # [ 224.353205] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:34Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=1 drv=2bdcl1cxw40gyh0sjgz98vb0jixgnrd9-farm-top.drv assign_id=8 outputs=1 worker1 # [ 224.368237] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:34Z level=debug event=assigned drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv worker=192.168.1.3:50051 clients=1 worker1 # [ 224.373296] nix-daemon[844]: reaped child process 1999, status = succeeded worker1 # [ 224.397353] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:34Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 224.411496] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:34Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 224.427017] nix-grpc-daemon[1546]: time=2026-09-22T11:00:34.897Z level=INFO msg="All 2 paths already cached" worker1 # [ 224.436436] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:34Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 224.449758] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 224.653499] nix-daemon[844]: accepted connection from pid 2024, user nix-grpc-daemon (trusted) worker1 # [ 224.672249] nix-daemon[844]: reaped child process 2027, status = succeeded worker1 # [ 224.843017] nix-daemon[844]: accepted connection from pid 2029, user nix-grpc-daemon (trusted) worker1 # [ 224.860909] nix-daemon[844]: reaped child process 2032, status = succeeded lb # [ 224.838185] niks3-server[1182]: 2026/09/22 11:00:35 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 224.882899] nix-grpc-daemon[1546]: time=2026-09-22T11:00:35.353Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 224.884420] nix-grpc-daemon[1546]: time=2026-09-22T11:00:35.353Z level=INFO msg="Uploading qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv (752B)" lb # [ 224.861383] niks3-server[1182]: 2026/09/22 11:00:35 INFO Registered completed upload object_key=nar/05zmsnqx4aj55yncx917mz6d3hzrvz2rx2x75313vvy1kq1zv9fv.nar.zst lb # [ 224.872216] niks3-server[1182]: 2026/09/22 11:00:35 INFO Received sign narinfos request method=POST path=/api/pending_closures/78/sign worker1 # [ 224.914273] nix-grpc-daemon[1546]: time=2026-09-22T11:00:35.385Z level=INFO msg="Uploading 1 narinfos" lb # [ 224.877540] niks3-server[1182]: 2026/09/22 11:00:35 INFO Signed narinfos id=78 count=1 lb # [ 224.882225] niks3-server[1182]: 2026/09/22 11:00:35 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.ls lb # [ 224.889052] niks3-server[1182]: 2026/09/22 11:00:35 INFO Received complete upload request method=POST path=/api/pending_closures/78/complete lb # [ 224.896572] niks3-server[1182]: 2026/09/22 11:00:35 INFO Completed upload id=78 worker1 # [ 224.933094] nix-grpc-daemon[1546]: time=2026-09-22T11:00:35.404Z level=INFO msg="Upload complete. (466ms)" lb # [ 224.898981] niks3-server[1182]: 2026/09/22 11:00:35 INFO Registered completed upload object_key=qx10pa00pilyp5xdzd48l88d5jz143mr.narinfo worker1 # [ 224.940119] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:35Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=0 paths=1 nar_bytes_in=752 worker1 # [ 224.949848] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:35Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 worker1 # [ 224.954931] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 224.958733] nix-daemon[844]: reaped child process 2022, status = succeeded worker1 # [ 224.968908] nix-daemon[844]: accepted connection from pid 1394, user nix-grpc-daemon (trusted) worker1 # [ 225.809006] nix-daemon[844]: reaped child process 2036, status = succeeded worker1 # [ 225.991448] nix-daemon[844]: accepted connection from pid 2052, user nix-grpc-daemon (trusted) worker1 # [ 226.008187] nix-daemon[844]: reaped child process 2055, status = succeeded lb # [ 225.989428] niks3-server[1182]: 2026/09/22 11:00:36 INFO Received uploads request method=POST path=/api/pending_closures worker1 # [ 226.035609] nix-grpc-daemon[1546]: time=2026-09-22T11:00:36.505Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker1 # [ 226.037336] nix-grpc-daemon[1546]: time=2026-09-22T11:00:36.505Z level=INFO msg="Uploading iagwbb072hnm79qb9wsrjp5907zc7aa2-farm-top (232B)" lb # [ 226.021836] niks3-server[1182]: 2026/09/22 11:00:36 INFO Registered completed upload object_key=nar/053jd22j3kv7v6j0q7cspdpc0h1c1xmywmajl6sd5hzwg50z2yj1.nar.zst lb # [ 226.035557] niks3-server[1182]: 2026/09/22 11:00:36 INFO Registered completed upload object_key=log/qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv lb # [ 226.046790] niks3-server[1182]: 2026/09/22 11:00:36 INFO Received sign narinfos request method=POST path=/api/pending_closures/79/sign worker1 # [ 226.088529] nix-grpc-daemon[1546]: time=2026-09-22T11:00:36.559Z level=INFO msg="Uploading 1 narinfos" lb # [ 226.055284] niks3-server[1182]: 2026/09/22 11:00:36 INFO Signed narinfos id=79 count=1 lb # [ 226.061829] niks3-server[1182]: 2026/09/22 11:00:36 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.ls lb # [ 226.070635] niks3-server[1182]: 2026/09/22 11:00:36 INFO Received complete upload request method=POST path=/api/pending_closures/79/complete lb # [ 226.077461] niks3-server[1182]: 2026/09/22 11:00:36 INFO Completed upload id=79 lb # [ 226.080486] niks3-server[1182]: 2026/09/22 11:00:36 INFO Registered completed upload object_key=iagwbb072hnm79qb9wsrjp5907zc7aa2.narinfo worker1 # [ 226.113296] nix-grpc-daemon[1546]: time=2026-09-22T11:00:36.584Z level=INFO msg="Upload complete. (333ms)" worker1 # [ 226.125311] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:36Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:60204 duration_s=1 drv=qx10pa00pilyp5xdzd48l88d5jz143mr-farm-top.drv assign_id=9 outputs=1 worker1 # [ 226.142165] nix-daemon[844]: reaped child process 2034, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto --expr 'map (tag: import /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix { inherit tag; features = ["vip"]; }) ["f1" "f2" "f3"]' --impure 2>&1, in 22.00 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) client: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features '["gpu"]' client # [ 226.347662] systemd[1]: Started [systemd-run] /run/current-system/sw/bin/nix build -L --store "grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30" --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features "[\"gpu\"]". client # Running as unit: gpu.service; invocation ID: c2857c971fe44315925ab46d34d99964 client: (finished: must succeed: systemd-run --unit gpu nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key&restart-grace=30' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag f5 --arg features '["gpu"]', in 0.09 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.07 seconds) worker2 # [ 226.626981] nix-grpc-daemon[2146]: ts=2026-09-22T11:00:37Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:38892 client # [ 226.602342] nix[1532]: don't know how to build these paths: client # [ 226.603448] nix[1532]: /nix/store/3i1nziwpdqq19lzpqy4rywzxnm8dlps6-farm-top.drv worker1 # [ 226.656722] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:37Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:60204 client # [ 226.632522] nix[1532]: warning: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu}, waiting up to 30s for a worker worker1 # [ 226.681321] nix-grpc-daemon[1394]: ts=2026-09-22T11:00:37Z level=debug event=want client=2 drv=1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv cached=0 lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) client: waiting for success: systemctl show -p Result gpu | grep -q exit-code lb # [ 234.218215] envoy[933]: conn peer=[2001:db8:1::1]:38794 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=27174 tx=12470 ms=21675 client # [ 256.634459] nix[1532]: 1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv: no connected, non-draining worker for system 'aarch64-linux' offers features {gpu} client # [ 256.647713] nix[1532]: error: dependency '1jn0w6x679ijjxwcfb2gxc6lw2fxim2k-farm-a.drv' failed client # [ 256.688264] systemd[1]: gpu.service: Main process exited, code=exited, status=1/FAILURE client # [ 256.693662] systemd[1]: gpu.service: Failed with result 'exit-code'. client # [ 256.699951] systemd[1]: gpu.service: Consumed 180ms CPU time over 30.346s wall clock time, 35.4M memory peak, 5.1K incoming IP traffic, 5K outgoing IP traffic. client: (finished: waiting for success: systemctl show -p Result gpu | grep -q exit-code, in 29.26 seconds) client: must succeed: journalctl -u gpu -o cat client: (finished: must succeed: journalctl -u gpu -o cat, in 0.03 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.08 seconds) lb: must succeed: curl -sf localhost:9901/clusters lb: (finished: must succeed: curl -sf localhost:9901/clusters, in 0.06 seconds) worker1: must succeed: curl -sf http://127.0.0.1:9464/metrics worker1: (finished: must succeed: curl -sf http://127.0.0.1:9464/metrics, in 0.08 seconds) (finished: subtest: requiredSystemFeatures: placed on the worker that has them, refused when none does, in 53.11 seconds) subtest: low disk drains a worker and builds go to the other worker1: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill worker1: (finished: must succeed: fallocate -l $(( $(df --output=avail -B1 /nix/store | tail -1) - 100*1024*1024 )) /nix/.rw-store/fill, in 1.08 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free' worker1 # [ 258.526023] nix-grpc-daemon[1394]: ts=2026-09-22T11:01:08Z level=info event=unhealthy reason=min_free worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=unhealthy reason=min_free', in 1.12 seconds) client: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain >&2 client # don't know how to build these paths: client # /nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker1 # [ 259.830055] nix-grpc-daemon[1394]: ts=2026-09-22T11:01:10Z level=debug event=rpc_start method=QueryMissing cn=ci-1 peer=ipv4:192.168.1.2:60204 worker2 # [ 259.818921] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:38892 worker1 # [ 259.861596] nix-grpc-daemon[1394]: ts=2026-09-22T11:01:10Z level=debug event=want client=3 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv cached=0 worker1 # [ 259.867584] nix-grpc-daemon[1394]: ts=2026-09-22T11:01:10Z level=debug event=assigned drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 259.892475] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:10Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38892 worker2 # [ 259.909040] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:10Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:38892 client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' to 'lb:50051'... worker2 # [ 259.925435] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:10Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38892 worker2 # [ 259.941533] nix-daemon[866]: accepted connection from pid 2146, user nix-grpc-daemon (trusted) worker2 # [ 259.969535] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:10Z level=info event=niks3_push_started pid=2206 worker2 # [ 260.191554] nix-grpc-daemon[2206]: time=2026-09-22T11:01:10.650Z level=INFO msg="Configuring client TLS" cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.pem key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/worker2.key ca="" worker2 # [ 260.370301] nix-daemon[866]: accepted connection from pid 2219, user nix-grpc-daemon (trusted) worker2 # [ 260.388141] nix-daemon[866]: reaped child process 2222, status = succeeded worker2 # [ 260.564756] nix-daemon[866]: accepted connection from pid 2224, user nix-grpc-daemon (trusted) worker2 # [ 260.581665] nix-daemon[866]: reaped child process 2227, status = succeeded lb # [ 260.577226] niks3-server[1182]: 2026/09/22 11:01:11 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 260.607179] nix-grpc-daemon[2206]: time=2026-09-22T11:01:11.065Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 260.610192] nix-grpc-daemon[2206]: time=2026-09-22T11:01:11.065Z level=INFO msg="Uploading kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv (512B)" lb # [ 260.616895] niks3-server[1182]: 2026/09/22 11:01:11 INFO Registered completed upload object_key=nar/1sfy0gcq4znfcj1f68c749r5x2jwmwsfbfp6xdflz5xkndjwjjm3.nar.zst worker2 # [ 260.651741] nix-grpc-daemon[2206]: time=2026-09-22T11:01:11.110Z level=INFO msg="Uploading 1 narinfos" lb # [ 260.631259] niks3-server[1182]: 2026/09/22 11:01:11 INFO Received sign narinfos request method=POST path=/api/pending_closures/80/sign lb # [ 260.642995] niks3-server[1182]: 2026/09/22 11:01:11 INFO Signed narinfos id=80 count=1 lb # [ 260.649432] niks3-server[1182]: 2026/09/22 11:01:11 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.ls lb # [ 260.663184] niks3-server[1182]: 2026/09/22 11:01:11 INFO Received complete upload request method=POST path=/api/pending_closures/80/complete lb # [ 260.667413] niks3-server[1182]: 2026/09/22 11:01:11 INFO Registered completed upload object_key=kksl1jqfg56vabgqvrkz5vki6nq7r59r.narinfo lb # [ 260.671887] niks3-server[1182]: 2026/09/22 11:01:11 INFO Completed upload id=80 worker2 # [ 260.691475] nix-grpc-daemon[2206]: time=2026-09-22T11:01:11.150Z level=INFO msg="Upload complete. (495ms)" worker2 # [ 260.701812] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:11Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38892 duration_s=0 paths=1 nar_bytes_in=512 worker2 # [ 260.712772] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:11Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38892 worker2 # [ 260.725141] nix-daemon[866]: accepted connection from pid 2146, user nix-grpc-daemon (trusted) worker2 # [ 260.733646] nix-daemon[866]: reaped child process 2204, status = succeeded client # building '/nix/store/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv' on lb:50051... client # farm-a> worker2-next: building kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker2 # [ 260.749324] nix-daemon[866]: accepted connection from pid 2146, user nix-grpc-daemon (trusted) client # farm-a> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-a> LOG-farm-a-drain worker2 # [ 261.798705] nix-daemon[866]: reaped child process 2231, status = succeeded worker2 # [ 261.978744] nix-daemon[866]: accepted connection from pid 2247, user nix-grpc-daemon (trusted) worker2 # [ 261.998214] nix-daemon[866]: reaped child process 2250, status = succeeded lb # [ 261.993277] niks3-server[1182]: 2026/09/22 11:01:12 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 262.024435] nix-grpc-daemon[2206]: time=2026-09-22T11:01:12.482Z level=INFO msg="Uploading 1 paths to lb (0 already cached)" worker2 # [ 262.025917] nix-grpc-daemon[2206]: time=2026-09-22T11:01:12.482Z level=INFO msg="Uploading 272b39wn7cbnhkcc2rrz2f6kmhhmcwnr-farm-a (128B)" lb # [ 262.041424] niks3-server[1182]: 2026/09/22 11:01:12 INFO Registered completed upload object_key=log/kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv worker2 # [ 262.074458] nix-grpc-daemon[2206]: time=2026-09-22T11:01:12.532Z level=INFO msg="Uploading 1 narinfos" lb # [ 262.051112] niks3-server[1182]: 2026/09/22 11:01:12 INFO Registered completed upload object_key=nar/1bf9z7y0ikx07b4zjag42zyrv2pq1ycw4461s579kfpmni9kz4dw.nar.zst lb # [ 262.066231] niks3-server[1182]: 2026/09/22 11:01:12 INFO Received sign narinfos request method=POST path=/api/pending_closures/81/sign lb # [ 262.078734] niks3-server[1182]: 2026/09/22 11:01:12 INFO Signed narinfos id=81 count=1 worker2 # [ 262.101752] nix-grpc-daemon[2206]: time=2026-09-22T11:01:12.560Z level=INFO msg="Upload complete. (341ms)" lb # [ 262.082380] niks3-server[1182]: 2026/09/22 11:01:12 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.ls lb # [ 262.087119] niks3-server[1182]: 2026/09/22 11:01:12 INFO Received complete upload request method=POST path=/api/pending_closures/81/complete lb # [ 262.094805] niks3-server[1182]: 2026/09/22 11:01:12 INFO Completed upload id=81 lb # [ 262.098246] niks3-server[1182]: 2026/09/22 11:01:12 INFO Registered completed upload object_key=272b39wn7cbnhkcc2rrz2f6kmhhmcwnr.narinfo worker1 # [ 262.142724] nix-grpc-daemon[1394]: ts=2026-09-22T11:01:12Z level=debug event=want client=3 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv cached=0 worker2 # [ 262.113312] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:12Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38892 duration_s=1 drv=kksl1jqfg56vabgqvrkz5vki6nq7r59r-farm-a.drv assign_id=10 outputs=1 worker2 # [ 262.132562] nix-daemon[866]: reaped child process 2229, status = succeeded worker1 # [ 262.148676] nix-grpc-daemon[1394]: ts=2026-09-22T11:01:12Z level=debug event=assigned drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 262.172082] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:12Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38892 worker2 # [ 262.188404] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:12Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:38892 worker2 # [ 262.201913] nix-grpc-daemon[2206]: time=2026-09-22T11:01:12.661Z level=INFO msg="All 1 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' to 'lb:50051'... worker2 # [ 262.212848] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:12Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38892 worker2 # [ 262.226593] nix-daemon[866]: accepted connection from pid 2146, user nix-grpc-daemon (trusted) worker2 # [ 262.433847] nix-daemon[866]: accepted connection from pid 2254, user nix-grpc-daemon (trusted) worker2 # [ 262.454081] nix-daemon[866]: reaped child process 2257, status = succeeded worker2 # [ 262.631310] nix-daemon[866]: accepted connection from pid 2259, user nix-grpc-daemon (trusted) worker2 # [ 262.648880] nix-daemon[866]: reaped child process 2262, status = succeeded lb # [ 262.642871] niks3-server[1182]: 2026/09/22 11:01:13 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 262.671821] nix-grpc-daemon[2206]: time=2026-09-22T11:01:13.130Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 262.673582] nix-grpc-daemon[2206]: time=2026-09-22T11:01:13.130Z level=INFO msg="Uploading fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv (632B)" lb # [ 262.669632] niks3-server[1182]: 2026/09/22 11:01:13 INFO Registered completed upload object_key=nar/1ax4z88sb8d24gsvydxvgdy2lijkmhnycxhr5war1j3sq97cyzqf.nar.zst lb # [ 262.683472] niks3-server[1182]: 2026/09/22 11:01:13 INFO Received sign narinfos request method=POST path=/api/pending_closures/82/sign worker2 # [ 262.711554] nix-grpc-daemon[2206]: time=2026-09-22T11:01:13.170Z level=INFO msg="Uploading 1 narinfos" lb # [ 262.695818] niks3-server[1182]: 2026/09/22 11:01:13 INFO Signed narinfos id=82 count=1 lb # [ 262.701900] niks3-server[1182]: 2026/09/22 11:01:13 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.ls lb # [ 262.710505] niks3-server[1182]: 2026/09/22 11:01:13 INFO Received complete upload request method=POST path=/api/pending_closures/82/complete lb # [ 262.718546] niks3-server[1182]: 2026/09/22 11:01:13 INFO Registered completed upload object_key=fa7adcm0x8sbgmrsfx912fp8xakk99g6.narinfo lb # [ 262.723118] niks3-server[1182]: 2026/09/22 11:01:13 INFO Completed upload id=82 worker2 # [ 262.743085] nix-grpc-daemon[2206]: time=2026-09-22T11:01:13.202Z level=INFO msg="Upload complete. (499ms)" worker2 # [ 262.752170] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:13Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38892 duration_s=0 paths=1 nar_bytes_in=632 worker2 # [ 262.766214] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:13Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38892 worker2 # [ 262.772604] nix-daemon[866]: accepted connection from pid 2146, user nix-grpc-daemon (trusted) worker2 # [ 262.777382] nix-daemon[866]: reaped child process 2252, status = succeeded client # building '/nix/store/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv' on lb:50051... worker2 # [ 262.789004] nix-daemon[866]: accepted connection from pid 2146, user nix-grpc-daemon (trusted) client # farm-b> worker2-next: building fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv client # farm-b> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-b> LOG-farm-b-drain worker2 # [ 263.635438] nix-daemon[866]: reaped child process 2266, status = succeeded worker2 # [ 263.825560] nix-daemon[866]: accepted connection from pid 2282, user nix-grpc-daemon (trusted) worker2 # [ 263.843898] nix-daemon[866]: reaped child process 2285, status = succeeded lb # [ 263.836657] niks3-server[1182]: 2026/09/22 11:01:14 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 263.867013] nix-grpc-daemon[2206]: time=2026-09-22T11:01:14.325Z level=INFO msg="Uploading 1 paths to lb (1 already cached)" worker2 # [ 263.869008] nix-grpc-daemon[2206]: time=2026-09-22T11:01:14.325Z level=INFO msg="Uploading g774lks418zc45bd9hwhinm61pwcwnr9-farm-b (176B)" lb # [ 263.872477] niks3-server[1182]: 2026/09/22 11:01:14 INFO Registered completed upload object_key=log/fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv lb # [ 263.885247] niks3-server[1182]: 2026/09/22 11:01:14 INFO Registered completed upload object_key=nar/1ammzz4zx9sggy8v73zxk3j08vbqxgpw14n8lx2fh57bx24yzvnj.nar.zst worker2 # [ 263.917809] nix-grpc-daemon[2206]: time=2026-09-22T11:01:14.376Z level=INFO msg="Uploading 1 narinfos" lb # [ 263.898798] niks3-server[1182]: 2026/09/22 11:01:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/83/sign lb # [ 263.907586] niks3-server[1182]: 2026/09/22 11:01:14 INFO Signed narinfos id=83 count=1 lb # [ 263.912883] niks3-server[1182]: 2026/09/22 11:01:14 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.ls lb # [ 263.919443] niks3-server[1182]: 2026/09/22 11:01:14 INFO Received complete upload request method=POST path=/api/pending_closures/83/complete lb # [ 263.926037] niks3-server[1182]: 2026/09/22 11:01:14 INFO Completed upload id=83 worker2 # [ 263.945515] nix-grpc-daemon[2206]: time=2026-09-22T11:01:14.404Z level=INFO msg="Upload complete. (341ms)" lb # [ 263.928277] niks3-server[1182]: 2026/09/22 11:01:14 INFO Registered completed upload object_key=g774lks418zc45bd9hwhinm61pwcwnr9.narinfo worker1 # [ 263.979458] nix-grpc-daemon[1394]: ts=2026-09-22T11:01:14Z level=debug event=want client=3 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv cached=0 worker1 # [ 263.985403] nix-grpc-daemon[1394]: ts=2026-09-22T11:01:14Z level=debug event=assigned drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv worker=192.168.1.4:50051 clients=1 worker2 # [ 263.957185] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:14Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38892 duration_s=1 drv=fa7adcm0x8sbgmrsfx912fp8xakk99g6-farm-b.drv assign_id=11 outputs=1 worker2 # [ 263.979022] nix-daemon[866]: reaped child process 2264, status = succeeded worker2 # [ 264.007663] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:14Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38892 worker2 # [ 264.024126] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:14Z level=debug event=rpc_start method=QueryValidPaths cn=ci-1 peer=ipv4:192.168.1.2:38892 worker2 # [ 264.038503] nix-grpc-daemon[2206]: time=2026-09-22T11:01:14.497Z level=INFO msg="All 2 paths already cached" client # copying 1 paths to 'lb:50051'... client # copying path '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' to 'lb:50051'... worker2 # [ 264.049755] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:14Z level=debug event=rpc_start method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38892 worker2 # [ 264.063862] nix-daemon[866]: accepted connection from pid 2146, user nix-grpc-daemon (trusted) lb # [ 264.219339] envoy[933]: conn peer=[2001:db8:1::1]:35176 tls=TLSv1.3 subject="CN=ci-1" sni=- flags=- tls_fail="-" rx=897 tx=499 ms=30062 worker2 # [ 264.268407] nix-daemon[866]: accepted connection from pid 2289, user nix-grpc-daemon (trusted) worker2 # [ 264.289718] nix-daemon[866]: reaped child process 2292, status = succeeded worker2 # [ 264.464108] nix-daemon[866]: accepted connection from pid 2294, user nix-grpc-daemon (trusted) worker2 # [ 264.484455] nix-daemon[866]: reaped child process 2297, status = succeeded lb # [ 264.477691] niks3-server[1182]: 2026/09/22 11:01:14 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 264.507320] nix-grpc-daemon[2206]: time=2026-09-22T11:01:14.966Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 264.508946] nix-grpc-daemon[2206]: time=2026-09-22T11:01:14.966Z level=INFO msg="Uploading 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv (760B)" lb # [ 264.507731] niks3-server[1182]: 2026/09/22 11:01:14 INFO Registered completed upload object_key=nar/1hrr94bvb37dwmf1giypzl9qnff72xm540ak7q49j34fyzia9ngb.nar.zst lb # [ 264.522005] niks3-server[1182]: 2026/09/22 11:01:14 INFO Received sign narinfos request method=POST path=/api/pending_closures/84/sign worker2 # [ 264.550535] nix-grpc-daemon[2206]: time=2026-09-22T11:01:15.009Z level=INFO msg="Uploading 1 narinfos" lb # [ 264.534127] niks3-server[1182]: 2026/09/22 11:01:14 INFO Signed narinfos id=84 count=1 lb # [ 264.540462] niks3-server[1182]: 2026/09/22 11:01:14 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.ls lb # [ 264.549243] niks3-server[1182]: 2026/09/22 11:01:15 INFO Received complete upload request method=POST path=/api/pending_closures/84/complete lb # [ 264.555799] niks3-server[1182]: 2026/09/22 11:01:15 INFO Registered completed upload object_key=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l.narinfo lb # [ 264.561536] niks3-server[1182]: 2026/09/22 11:01:15 INFO Completed upload id=84 worker2 # [ 264.581405] nix-grpc-daemon[2206]: time=2026-09-22T11:01:15.040Z level=INFO msg="Upload complete. (499ms)" worker2 # [ 264.590652] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:15Z level=info event=rpc method=AddMultipleToStore cn=ci-1 peer=ipv4:192.168.1.2:38892 duration_s=0 paths=1 nar_bytes_in=760 worker2 # [ 264.600707] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:15Z level=debug event=rpc_start method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38892 worker2 # [ 264.612798] nix-daemon[866]: accepted connection from pid 2146, user nix-grpc-daemon (trusted) worker2 # [ 264.617884] nix-daemon[866]: reaped child process 2287, status = succeeded client # building '/nix/store/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv' on lb:50051... worker2 # [ 264.629549] nix-daemon[866]: accepted connection from pid 2146, user nix-grpc-daemon (trusted) client # farm-top> worker2-next: building 77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv client # farm-top> disabling binary cache 'https://cache.nixos.org' for 60 seconds client # farm-top> LOG-farm-top-drain worker2 # [ 265.858795] nix-daemon[866]: reaped child process 2301, status = succeeded worker2 # [ 266.024072] nix-daemon[866]: accepted connection from pid 2317, user nix-grpc-daemon (trusted) worker2 # [ 266.044983] nix-daemon[866]: reaped child process 2320, status = succeeded lb # [ 266.037199] niks3-server[1182]: 2026/09/22 11:01:16 INFO Received uploads request method=POST path=/api/pending_closures worker2 # [ 266.066005] nix-grpc-daemon[2206]: time=2026-09-22T11:01:16.523Z level=INFO msg="Uploading 1 paths to lb (2 already cached)" worker2 # [ 266.067507] nix-grpc-daemon[2206]: time=2026-09-22T11:01:16.523Z level=INFO msg="Uploading bzkv277k4v9r9h4aq5h13ddcavmp36fg-farm-top (232B)" lb # [ 266.067425] niks3-server[1182]: 2026/09/22 11:01:16 INFO Registered completed upload object_key=log/77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv lb # [ 266.082951] niks3-server[1182]: 2026/09/22 11:01:16 INFO Registered completed upload object_key=nar/0l8sy8grn9pmq2n58ag0hx791zdfyxwk9319ynjqmn7yy7wl64s7.nar.zst worker2 # [ 266.119325] nix-grpc-daemon[2206]: time=2026-09-22T11:01:16.578Z level=INFO msg="Uploading 1 narinfos" lb # [ 266.095436] niks3-server[1182]: 2026/09/22 11:01:16 INFO Received sign narinfos request method=POST path=/api/pending_closures/85/sign lb # [ 266.104583] niks3-server[1182]: 2026/09/22 11:01:16 INFO Signed narinfos id=85 count=1 lb # [ 266.106886] niks3-server[1182]: 2026/09/22 11:01:16 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.ls lb # [ 266.116648] niks3-server[1182]: 2026/09/22 11:01:16 INFO Received complete upload request method=POST path=/api/pending_closures/85/complete lb # [ 266.123044] niks3-server[1182]: 2026/09/22 11:01:16 INFO Registered completed upload object_key=bzkv277k4v9r9h4aq5h13ddcavmp36fg.narinfo lb # [ 266.128383] niks3-server[1182]: 2026/09/22 11:01:16 INFO Completed upload id=85 worker2 # [ 266.148385] nix-grpc-daemon[2206]: time=2026-09-22T11:01:16.607Z level=INFO msg="Upload complete. (306ms)" worker2 # [ 266.161117] nix-grpc-daemon[2146]: ts=2026-09-22T11:01:16Z level=info event=rpc method=BuildDerivation cn=ci-1 peer=ipv4:192.168.1.2:38892 duration_s=1 drv=77925dzx1nz9jvb0jf5cxdn4b8gwrv5l-farm-top.drv assign_id=12 outputs=1 worker2 # [ 266.177502] nix-daemon[866]: reaped child process 2299, status = succeeded client: (finished: must succeed: nix build -L --store 'grpc://lb:50051?client-cert=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.pem&client-key=/nix/store/r1p0v55v6igq97giy4hh7w7h1khnwhk6-farm-certs/ci.key' --eval-store auto -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain >&2, in 6.66 seconds) client: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain outPath client: (finished: must succeed: nix eval --raw -f /nix/store/r498lgb40xdi1s9di8nh5bgv2h8vrrp1-job.nix --argstr tag drain outPath, in 0.24 seconds) worker1: must succeed: rm /nix/.rw-store/fill worker1: (finished: must succeed: rm /nix/.rw-store/fill, in 0.15 seconds) worker1: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free' worker1 # [ 267.531309] nix-grpc-daemon[1394]: ts=2026-09-22T11:01:17Z level=info event=healthy reason=min_free worker1: (finished: waiting for success: journalctl -u nix-grpc-daemon -o cat | grep -q 'event=healthy reason=min_free', in 1.12 seconds) (finished: subtest: low disk drains a worker and builds go to the other, in 10.50 seconds) (finished: run the VM test script, in 268.76 seconds) test script finished in 269.02s cleanup kill QemuMachine (pid 50) client # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14) kill QemuMachine (pid 49) lb # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14) kill QemuMachine (pid 51) worker1 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14) kill QemuMachine (pid 52) worker2 # qemu-system-aarch64: terminating on signal 15 from pid 39 (/nix/store/lfh101cfjj8p8npcari9hk0lbqi33yvh-python3-3.14.7/bin/python3.14) (finished: cleanup, in 1.06 seconds) additionally exposed symbols: client, lb, worker1, worker2, vlan1, start_all, test_script, machines, machines_qemu, machines_nspawn, vlans, driver, log, os, create_machine, subtest, run_tests, join_all, retry, serial_stdout_off, serial_stdout_on, polling_condition, BaseMachine, QemuMachine, NspawnMachine, t, debug, dump_machine_ssh